Ship berthing and unberthing environment image sensing method based on multi-view vision

Abstract

The invention discloses a ship berthing and leaving environment image sensing method based on multi-view vision, and relates to the technical field of ships. According to the method, an image collected by a multi-view camera is spherically projected and unfolded in an equator unfolding mode; according to the method, the pixel loss condition can be effectively avoided in the mode that the pixel points of the original image are reversely calculated through the pixel points of the spherical panoramic expanded image; the distortion of the expanded point corresponding to the ship bow and the symmetrical point of the expanded point relative to the sphere center is sacrificed; the parallel characteristic of the images expanded by the two hemispheres divided by the longitude line passing through the two points relative to the longitude line is ensured, so that the integrally spliced sensing panoramic image meets the observation requirement in the ship berthing and leaving scene, and the environment image sensing requirement in the ship berthing and leaving scene can be more efficiently and accurately met in real time.

Description

Technical Field

[0001] The present application relates to the technical field of ships, and in particular to a method for perceiving ship berthing and unberthing environment images based on multi-view vision. Background Art

[0002] When a ship, especially a large ship, berths or unberths, it is necessary to obtain images of the ship's surrounding environment to help the driver timely master the situation around the ship. This can not only assist the driver in judging the distance from the shore base, but also facilitate timely response to dangerous situations, greatly reducing the possibility of collision accidents, and thus being beneficial to improving the safety of berthing and unberthing.

[0003] The current common practice is to install a panoramic camera at a high position on the ship's mast. The panoramic camera can provide a viewing angle of 360° in the horizontal direction and 90° in the vertical direction. However, the panoramic image needs to be stitched and projected using technologies such as image registration and feature point matching. The currently commonly used feature-based stitching method will search for and register features in the entire area of each frame of the image, with the disadvantages of large computational amount, long stitching time, and low frame rate; although the block-based matching stitching method has relatively high computational efficiency, its adaptability to changes in image content is relatively poor, and its robustness to rotation, translation, and scaling transformations of the image is also weak; the deep learning-based stitching method requires a large amount of training data and computational resources, and its effect may be inferior to that of feature- and block-based methods in some scenarios. These existing methods all have certain limitations for the berthing and unberthing scenarios with high real-time requirements. In addition, currently, the spherical fisheye display method and the pole expansion display method are often used, and neither can intuitively display the situation of the ship's side and the shore base, which is not conducive to distance judgment in the berthing and unberthing scenarios and cannot well assist in the perception of the berthing and unberthing environment. Summary of the Invention

[0004] In view of the above problems and technical requirements, the present application proposes a method for perceiving ship berthing and unberthing environment images based on multi-view vision. The technical solution of the present application is as follows:

[0005] A method for perceiving ship berthing and unberthing environment images based on multi-view vision, the method for perceiving ship berthing and unberthing environment images includes:

[0006] Respectively obtain the original images of the ship's surrounding environment at the same moment through each camera included in the multi-view camera installed on the ship. The viewing angles of the cameras in the multi-view camera are inclined downward towards different positions in the circumferential direction of the ship, and the field of view ranges of all cameras cover the 360° circumferential environment of the ship;

[0007] Select the intersection point of the positive direction of the X g axis of the geodetic coordinate system and the projection hemisphere as the unfolding point, and rotate around the X gThe X-axis is expanded by 180° to form a spherical panoramic expansion diagram. The origin of the geodetic coordinate system is located at the center of the circle where each camera is placed in the multi-eye camera. g The positive direction of the axis points to the bow direction, and the projection hemisphere takes the origin of the geodetic coordinate system as the center and the focal length of a single camera in the multi-camera as the radius;

[0008] For any point (α, β) on the spherical panoramic expansion map, determine the intersection coordinates (x g ,y g ,z g ), and convert it to the pixel coordinates (x c ,y c , z c ),in, β∈(0,π);

[0009] According to the correspondence between the points on the spherical panoramic expansion map and the pixel coordinates in the original image, each original image collected by multiple cameras is spherically projected onto the spherical panoramic expansion map to obtain the corresponding projection image, and the image stitching operation is performed on each projection image to obtain the perceived panoramic image of the ship's surrounding environment.

[0010] A further technical solution is that performing an image stitching operation on each projection image includes:

[0011] When performing the stitching operation for the first time, two adjacent projected images with overlapping areas are fused to obtain a perceived panoramic image, so as to complete the image stitching operation and determine a pixel mapping matrix from the projected image to the perceived panoramic image;

[0012] Starting from the second execution of the stitching operation, the pixel mapping matrix corresponding to each projected image is directly used for mapping to obtain a perceived panoramic image, and the pixel mapping matrix is updated once at a predetermined time interval.

[0013] Its further technical solution is that image fusion includes:

[0014] The overlapping areas between the perspective transformed projected images are blended in and out in a gradual manner using the weight of the logistic curve.

[0015] A further technical solution is that image fusion of two adjacent projection images with overlapping areas includes:

[0016] Determine the coincidence line and overlapping area between two adjacent projection images according to the field of view angle and focal length of the camera, perform SIFT registration on the two projection images within the overlapping area to obtain the homography matrix corresponding to each projection image, and perform perspective transformation on the projection images using the homography matrix corresponding to each projection image and then perform image fusion.

[0017] A further technical solution thereof is that image fusion of two adjacent projection images with an overlapping area includes:

[0018] Taking the coincidence line of the imaging planes of the cameras corresponding to the two adjacent projection images as the stitching line, and performing image fusion on the projection images in the area from both sides of the stitching line to the nearer side line of the corresponding imaging plane.

[0019] A further technical solution thereof is that converting to obtain the pixel coordinates (x c , y c , z c ) in the original image includes:

[0020] Determine the installation structure of the multi-camera to determine the equation of the imaging plane of each camera in the multi-camera in the earth coordinate system, and combine the unit vector of the ray corresponding to the point (α, β) to determine the intersection coordinates (x g , y g , z g ) with the imaging plane of the camera; obtain the pixel coordinates (x c , y c , z c ) in the original image corresponding to the point (α, β) according to the coordinate transformation matrix between the camera coordinate system and the earth coordinate system of the camera.

[0021] A further technical solution thereof is that the equation of the imaging plane of each camera in the multi-camera in the earth coordinate system is:

[0022] Ax + By + Cz + f = 0

[0023] where (x, y, z) are the coordinate points in the earth coordinate system, A = sin(β f + π / 2)*sinα f , B = cos(β f + π / 2), C = sin(α f + π / 2)*sinβ f , f is the focal length of the camera, β f is the angle between the normal of the camera and the Y g axis in the earth coordinate system, and α f is the angle of the normal of the camera in the X g Z gProjection on the plane and the angle with the Z g axis; the X of the geodetic coordinate system g Z g plane is parallel to the horizontal plane, and the positive direction of the Y g axis is along the vertical direction upward;

[0024] Determine And determine

[0025] A further technical solution thereof is that the ship berthing and unberthing environment image perception method further includes:

[0026] Performing spherical projection transformation on the original images collected by each camera in the multi-camera after distortion correction processing, and the distortion correction processing includes radial distortion processing and tangential distortion processing.

[0027] A further technical solution thereof is that the ship berthing and unberthing environment image perception method further includes:

[0028] Using a thread to complete distortion correction processing and spherical projection transformation on the original images collected by a single camera in the multi-camera, and performing multi-thread parallel processing on the original images collected by each camera in the multi-camera.

[0029] A further technical solution thereof is that performing distortion correction processing on the original image includes determining that the pixel coordinates (x0, y0) of any pixel point (x, y) in the original image after completing distortion correction processing are:

[0030]

[0031] Wherein, k1, k2, k3 are fitting parameters for correcting radial distortion, and p1, p2 are fitting parameters for correcting tangential distortion.

[0032] The beneficial technical effects of the present application are:

[0033] The present application discloses a ship berthing and unberthing environment image perception method based on multi-view vision. The method performs spherical projection on the images collected by the multi-camera and adopts the equatorial expansion method for expansion. By inversely calculating the original image pixels through the pixels of the spherical panoramic expansion map, the occurrence of pixel loss can be effectively avoided. By sacrificing the distortion of two points, namely the expansion point and its symmetric point about the center of the sphere (which represent the bow and stern of the ship in practical applications), the parallel characteristics of the images expanded from the two hemispheres divided by the meridian passing through these two points with respect to the meridian are ensured, making the overall stitched perception panoramic image meet the observation requirements in the ship berthing and unberthing scenarios, and being able to more efficiently, accurately and real-time meet the environmental image perception requirements in the ship berthing and unberthing scenarios.

[0034] This method greatly reduces the computational load required during the stitching process by regularly updating the transformation matrix for projection stitching and only performing time-consuming feature registration operations on the overlapping areas of the images. Combining with multi-threading technology, it can achieve real-time stitching with low computing power, thereby reducing the configuration requirements for the calculations needed for panoramic display, and realizing a panoramic display method applicable to most ships. Description of the Drawings

[0035] Figure 1 is a schematic flowchart of the ship berthing and unberthing environment image perception method in an embodiment of the present application.

[0036] Figure 2 is a schematic diagram of the coordinate system involved in the ship berthing and unberthing environment image perception method of the present application.

[0037] Figure 3 is a schematic diagram of the perceived panoramic image processed by using the ship berthing and unberthing environment image perception method of the present application in an example scenario. Detailed Embodiments

[0038] The following further describes the detailed embodiments of the present application with reference to the drawings.

[0039] The present application discloses a ship berthing and unberthing environment image perception method based on multi-view vision. Please refer to Figure 1 the flowchart shown, and this ship berthing and unberthing environment image perception method includes:

[0040] Step S1, respectively obtain the original images of the ship's surrounding environment at the same moment through each camera of the multi-view camera installed on the ship. The multi-view camera is installed on the highest mast on the ship's top. The viewing angles of each camera of the multi-view camera are respectively oriented in different directions of the ship's circumference and are all inclined downward. The field of view ranges of all cameras of the multi-view camera cover the 360° circumferential environment of the ship.

[0041] For convenience of description, the present application constructs a geodetic coordinate system OX g Y g Z g , the origin O of the geodetic coordinate system is located at the center of the circumference where each camera of the multi-view camera is placed. The positive direction of the X-axis of the geodetic coordinate system points to the bow direction. The X g Z g plane is parallel to the horizontal plane, and the positive direction of the Y g axis points upward along the vertical direction. Please refer to Figure 2 .

[0042] After installing the multi-view camera, the imaging plane of each camera in the multi-view camera can be determined according to the installation structure of the multi-view camera in the geodetic coordinate system OX g Y g Z gFor the equations in, please combine with Figure 2 the example shown in, where the equation of the imaging plane 210 of one camera in the earth coordinate system is:

[0043] Ax + By + Cz + f = 0

[0044] where (x, y, z) is the coordinate point in the earth coordinate system, A = sin(β f + π / 2)*sinα f , B = cos(β f + π / 2), C = sin(α f + π / 2)*sinβ f , and f is the focal length of each camera in the multi-camera.

[0045] Figure 2 For one of the cameras in the multi-camera, the Z c Y c Z c axis of c is along the normal direction of the camera, and the plane where the X c Y c axis is located in the imaging plane of the camera and the X c axis and the Y c axis are respectively along the two directions of the imaging plane. β f is the angle between the normal direction of the camera and the Y g axis in the earth coordinate system, and α f is the angle between the projection of the normal direction of the camera on the X g Z g plane in the earth coordinate system and the Z g axis.

[0046] Figure 2 Taking the multi-camera including four cameras as an example, the equations of the imaging planes 220, 230, and 240 of the other three cameras in the earth coordinate system can be determined similarly.

[0047] Step S2, select the intersection point of the positive direction of the X g axis of the earth coordinate system and the projection hemisphere as the unfolding point, and unfold it by 180° around the X g axis into a spherical panoramic unfolding diagram.

[0048] where the projection hemisphere has the origin of the earth coordinate system as the center of the sphere and the focal length f of the camera as the radius. Please refer to Figure 2 for the schematic diagram. Different unfolding positions of the projection hemisphere will have different display effects. If unfolded at the pole, it will be displayed in a panoramic way all around, which is not conducive to judging the distance from the shore in the approach and departure scenarios. Therefore, considering the subsequent intuitive display convenience during approach and departure, with the X gThe intersection point of the positive direction of the axis and the projection hemisphere is the unfolding point, which is represented as the bow position in practical applications. Spherical panoramic unfolding Figure 1 Generally, it is an image with an aspect ratio of 2:1. Since only the hemispherical image of the ship needs to be looked down upon when berthing or unberthing, the final spherical panoramic unfolding diagram is 1:1. The coordinates of any point on the spherical panoramic unfolding diagram can be represented by (α, β), and in this application β ∈ (0, π).

[0049] Step S3, perform spherical projection transformation to project the imaging information in the original space onto a unified plane to facilitate subsequent image stitching and fusion. Essentially, it is to find the pixel position of the pixel point on the spherical panoramic unfolding diagram on the corresponding original image. Therefore, for any point (α, β) on the spherical panoramic unfolding diagram, determine the intersection coordinates (x g , y g , z g ) of the ray corresponding to the point (α, β) and the imaging plane of the camera, and convert it to obtain the pixel coordinates (x c , y c , z c ) in the corresponding original image.

[0050] The unit vector (x g1 , y g1 , z g1 ) of the ray corresponding to the point (α, β) can be expressed as:

[0051]

[0052] Then, combined with the equation of the imaging plane of the camera in the geodetic coordinate system, the intersection coordinates (x g , y g , z g ) with the imaging plane of the camera can be determined. As described above, the equation of the imaging plane in the geodetic coordinate system is Ax + By + Cz + f = 0. Therefore, the intersection coordinates (x g , y g , z g ) can be expressed as:

[0053]

[0054] According to the coordinate transformation matrix between the camera coordinate system and the geodetic coordinate system of the camera, the pixel coordinates (x c , y c , z c ) of the point (α, β) corresponding to the original image are obtained and expressed as:

[0055]

[0056] According to the actual pixel size of the corresponding image, x can be determinedc and y c specific ranges of

[0057] Considering the physical properties of the camera lens, light rays at positions farther from the optical center are refracted more, resulting in pillow-shaped or barrel-shaped radial distortion of the formed image centered on the optical center. In addition, there may also be tangential distortion caused by the non-parallelism between the lens and the sensor plane. To improve the quality of subsequent image stitching, first perform distortion correction processing on the original images collected by each camera and then perform spherical projection transformation for this step. The distortion correction processing includes radial distortion processing and tangential distortion processing. Performing distortion correction processing on the original image includes determining the pixel coordinates (x0, y0) of any pixel point (x, y) in the original image after the distortion correction processing as:

[0058]

[0059] wherein, k1, k2, k3 are fitting parameters for correcting radial distortion, and p1, p2 are fitting parameters for correcting tangential distortion.

[0060] Step S4, according to the correspondence between the points on the spherical panoramic unfolded image and the pixel coordinates in the original image, perform spherical projection transformation on the original images collected by each camera to the spherical panoramic unfolded image to obtain corresponding projection images, and perform image stitching operations on the respective projection images to obtain a perceived panoramic image of the ship's surrounding environment.

[0061] When performing the stitching operation for the first time, perform image fusion on two adjacent projection images with an overlapping area to merge multiple results to obtain a perceived panoramic image to complete the image stitching operation.

[0062] One approach is to first determine the coincidence line and overlapping area between two adjacent projection images according to the field of view angle and focal length of the camera, then perform SIFT registration on the two projection images within the overlapping area to obtain the homography matrix corresponding to each projection image, and finally perform perspective transformation on the projection images using the homography matrix corresponding to each projection image and then perform image fusion.

[0063] When it is impossible to capture high-quality registration points due to a single environment, resulting in fewer feature points in the image of environmental variables and unable to complete registration through SIFT, another approach is to use the coincidence line of the imaging planes of the cameras corresponding to two adjacent projection images as the stitching line, and directly perform image fusion on the overlapping area on both sides of the stitching line to the area closer to the adjacent side of the corresponding imaging plane.

[0064] Regardless of which of the above methods is adopted, when performing image fusion, a fade-in and fade-out fusion with logistic curve weights is performed on the overlapping area between the projected images after perspective transformation. Through this method, the brightness difference between the images on both sides of the seam can be eliminated to a certain extent and the seam can be blurred, making the splicing transition effect more natural and forming a good panoramic visual effect.

[0065] In the method of the present application for unfolding at a certain point on the spherical equator, near the unfolding point, the meridians (longitude lines) are almost parallel on the spherical surface and have a uniform spacing. Therefore, the stretching or compression in the meridian direction is small and the distortion is not obvious. However, at the two poles perpendicular to the unfolding point, due to the large spherical curvature and the dense latitude lines, obvious stretching and distortion will occur after unfolding. The approach of the present application sacrifices the distortion at the unfolding point and its symmetric point about the center of the sphere (represented as the bow and stern points in the actual berthing and unberthing scenarios), ensuring the parallel characteristics of the images unfolded from the two hemispheres divided by the meridian passing through these two points with respect to this meridian. Since it is inconvenient to illustrate the actual berthing and unberthing scenarios, in one example, an indoor scenario is used for illustration. The perceived panoramic image obtained by processing and splicing using the method of the present application is as Figure 3 shown Figure 3 Except for a small amount of distortion in the upper and lower parts of the picture, the straight lines on the left and right sides of the image can be ensured to be parallel to the picture border in the middle of the picture. When reflected in the berthing and unberthing scenarios, the perceived panoramic image with such image characteristics can more conveniently and intuitively display the distance between the ship's side and the port when the ship is berthing and unberthing.

[0066] Since it takes a long time to perform one SIFT registration and fade-in and fade-out fusion, in order to meet the real-time requirements in the berthing and unberthing scenarios, in one embodiment, after completing the SIFT registration and fade-in and fade-out fusion during the first execution of the splicing operation, the pixel mapping matrix from the projected image to the perceived panoramic image is determined and recorded. Starting from the second execution of the splicing operation, the perceived panoramic image is directly obtained by mapping using the pixel mapping matrix corresponding to each projected image, and the pixel mapping matrix is updated every predetermined time interval. When it is detected that the time interval from the last update time of the pixel mapping matrix reaches the predetermined time interval, the SIFT registration and fade-in and fade-out fusion are performed again to update the pixel mapping matrix, otherwise, the direct mapping is performed according to the original pixel mapping matrix, thereby improving the real-time performance on the basis of ensuring the image splicing quality.

[0067] In addition, in order to reduce the configuration requirements of the calculations required for panoramic display and further meet the real-time requirements in docking and undocking scenarios, one thread is used to complete the distortion correction and spherical projection transformation of the original image collected by a single camera in the multi-eye camera, and multi-threaded parallel processing is performed on the original images collected by each camera in the multi-eye camera, thereby reducing the configuration requirements of the calculations required for panoramic display, so that the program can complete the stitching of panoramic images in real time using a multi-threaded CPU.

[0068] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A method for ship berthing and unberthing environment image perception based on multi-camera vision, characterized in that, The method for perceiving the ship berthing and unberthing environment image includes: Respectively obtaining the original images of the ship's surrounding environment at the same moment through each camera included in the multi-camera installed on the ship. The viewing angles of the cameras in the multi-camera are inclined downward towards different positions in the circumferential direction of the ship, and the field of view ranges of all cameras cover the 360° circumferential environment of the ship. Select the positive direction of the X-axis of the geodetic coordinate system g The intersection point of the positive direction of the axis and the projection hemisphere is the unfolding point, and it is unfolded by 180° around the X g axis to form a spherical panoramic unfolding diagram. The origin of the geodetic coordinate system is located at the center of the circumference where each camera in the multi-camera is placed. The positive direction of the X g axis of the geodetic coordinate system points to the bow direction, and the projection hemisphere has the origin of the geodetic coordinate system as the center of the sphere and the focal length of a single camera in the multi-camera as the radius; For any point (α, β) on the spherical panoramic unfolding diagram, determine the intersection coordinates (x g , y g , z g ) of the ray corresponding to the point (α, β) and the multi-purpose imaging plane, and convert to obtain the pixel coordinates (x c , y c , z c ) in the corresponding original image, where, β ∈ (0, π); According to the correspondence between the points on the spherical panoramic unfolded map and the pixel coordinates in the original image, respectively perform spherical projection transformation on each original image collected by the multi-camera to the spherical panoramic unfolded map to obtain the corresponding projection image, and perform image stitching operation on each projection image to obtain the perceived panoramic image of the ship's surrounding environment.

2. The method for perceiving the ship berthing and unberthing environment image according to claim 1, wherein Performing the image stitching operation on each projection image includes: When performing the stitching operation for the first time, perform image fusion on two adjacent projection images with an overlapping area to obtain the perceived panoramic image, so as to complete the image stitching operation and determine the pixel mapping matrix from the projection image to the perceived panoramic image. Starting from the second time of performing the stitching operation, directly use the pixel mapping matrix corresponding to each projection image for mapping to obtain the perceived panoramic image, and update the pixel mapping matrix every predetermined time interval.

3. The method for perceiving the ship berthing and unberthing environment image according to claim 2, wherein Performing image fusion includes: Perform fade-in and fade-out fusion with a logistic curve weight on the overlapping area between the perspective-transformed projection images.

4. The method for perceiving the ship berthing and unberthing environment image according to claim 2, wherein Performing image fusion on two adjacent projection images with an overlapping area includes: Determine the coincidence line and overlapping area between two adjacent projection images according to the field of view angle and focal length of the camera, perform SIFT registration on the two projection images within the overlapping area to obtain the homography matrix corresponding to each projection image, and perform perspective transformation on the projection images using the homography matrix corresponding to each projection image and then perform image fusion.

5. The method for perceiving the ship berthing and unberthing environment image according to claim 2, characterized in that, Performing image fusion on two adjacent projection images with an overlapping area includes: Taking the coincidence line of the imaging planes of the cameras corresponding to two adjacent projection images as the stitching line, and performing image fusion on the projection images within the area from both sides of the stitching line to the nearer side edge of the corresponding imaging plane.

6. The method for perceiving the ship berthing and unberthing environment image according to claim 1, wherein The converted corresponding pixel coordinates (x c , y c , z c ) in the original image include: Determine the installation structure of the multi-camera. Determine the equation of the imaging plane of each camera in the multi-camera in the earth coordinate system, and combine the unit vector of the ray corresponding to the point (α, β). Determine the intersection coordinates (x g , y g , z g ) with the imaging plane of the camera; obtain the pixel coordinates (x c , y c , z c ) in the original image corresponding to the point (α, β) according to the coordinate transformation matrix between the camera coordinate system and the earth coordinate system of the camera.

7. The method for perceiving the ship berthing and unberthing environment image according to claim 6, characterized in that The equation of the imaging plane of each camera in the multi-camera in the geodetic coordinate system is: Ax + By + Cz + f = 0 where (x, y, z) are coordinate points in the geodetic coordinate system, A = sin(β f + π / 2) * sinα f , B = cos(β f + π / 2), C = sin(α f + π / 2) * sinβ f , f is the focal length of the camera, β f is the angle between the normal of the camera and the Y g axis in the geodetic coordinate system, α f is the angle between the projection of the normal of the camera on the X g Z g plane in the geodetic coordinate system and the Z g axis; the X g Z g plane of the geodetic coordinate system is parallel to the horizontal plane, and the positive direction of the Y g axis is along the vertical direction upward; Determine And determine 8. The method for perceiving the ship berthing and unberthing environment image according to claim 1, characterized in that The method for perceiving the ship berthing and unberthing environment image further includes: Performing distortion correction processing on the original image collected by each camera in the multi-camera and then performing spherical projection transformation. The distortion correction processing includes radial distortion processing and tangential distortion processing.

9. The method for perceiving the ship berthing and unberthing environment image according to claim 8, characterized in that, The method for perceiving the ship berthing and unberthing environment image further includes: Using one thread to complete the distortion correction processing and spherical projection transformation on the original image collected by a single camera in the multi-camera, and performing multi-thread parallel processing on the original images collected by each camera in the multi-camera.

10. The method for perceiving the ship berthing and unberthing environment image according to claim 8, characterized in that, Performing distortion correction processing on the original image includes determining that the pixel coordinates (x0, y0) of any pixel point (x, y) in the original image after completing the distortion correction processing are: Among them, k1, k2, and k3 are fitting parameters for correcting radial distortion, and p1 and p2 are fitting parameters for correcting tangential distortion.

Images