A ship surrounding environment perception method based on multi-view vision

Abstract

The application discloses a ship surrounding environment sensing method based on multi-vision, which comprises the following steps: image acquisition through a plurality of camera groups arranged and installed on the ship in advance; splicing the images obtained by the single camera group into a 180-degree planar image; extracting the features of the overlapping areas between the images, re-performing image registration and image fusion, and obtaining the spliced 180-degree planar image between the camera groups; performing image splicing on the spliced 180-degree planar images of all the camera groups to generate a 360-degree panoramic image of the surrounding environment of the ship; extracting the video frames of the single camera group, constructing a splicing template based on the image splicing, and performing video splicing to generate a 180-degree panoramic video of the surrounding environment of the ship. The multi-camera image splicing technology, the video splicing to generate panoramic image technology, and real-time ranging and other intelligent auxiliary technologies are applied to large and medium-sized ships, the sensing capability of the surrounding environment of the ship is increased, the operation of the crew on the ship is simplified, and the safety of the berthing of the ship is increased.

Description

Technical Field

[0001] This invention belongs to the field of shipbuilding and relates to machine vision technology, specifically to a method for perceiving the surrounding environment of a ship based on multi-view vision. Background Technology

[0002] The ocean possesses abundant resources and holds a vital transportation position. Maritime freight is the most important mode of transportation in international logistics, accounting for more than two-thirds of total international trade. Currently, the maritime transportation industry is developing rapidly, with the number of ships showing an upward trend. However, despite this trend, significant hidden dangers remain in the berthing and unberthing processes of ships. Insufficient awareness of the surrounding environment is a major reason threatening the safety of ships during berthing and unberthing.

[0003] The berthing and unberthing process is the most complex operation during a ship's entire voyage. During this process, the perception of environmental information is often imprecise, and the restrictive nature of port waters places significant challenges on the ship's maneuverability and the technical skills of the helmsmen. Under the influence of various factors, the ship's controllability is greatly challenged, leading to a year-on-year increase in the accident rate during this operation. In addition, accidents such as collisions with other vessels in the narrow waters of ports, damage to navigation aids such as buoys, contact with breakwaters, and collisions caused by ship waves also occur frequently.

[0004] Currently, large and medium-sized vessels mainly rely on tugboats for berthing, placing excessive emphasis on the operational experience of the captain and pilot. When pilots mishandle the vessels or tugboats enter the port at excessive speed, catastrophic situations such as vessels colliding with the dock can easily occur.

[0005] Therefore, current large and medium-sized vessels have a weak ability to perceive their surrounding environment. When vessels are berthing or sailing, factors such as large waves and insufficient crew technical experience can easily lead to losses of time, manpower, and financial resources. In particular, dangerous situations such as vessels colliding with docks when berthing or hitting obstacles while sailing are prone to occur.

[0006] Therefore, a new technical solution is needed to solve this problem. Summary of the Invention

[0007] Purpose of the invention: In order to overcome the shortcomings of existing technologies in which large and medium-sized ships have weak perception capabilities of their surrounding environment, this invention provides a method for perceiving the ship's surrounding environment based on multi-view vision. This method applies a variety of intelligent auxiliary systems and technologies, such as multi-camera image stitching technology, video stitching to generate panoramic images technology, and real-time ranging, to large and medium-sized ships, thereby increasing the ship's perception capabilities of its surrounding environment, simplifying the crew's operation of the ship, and increasing the safety of the ship's berthing.

[0008] Technical Solution: To achieve the above objectives, this invention provides a method for perceiving the surrounding environment of a ship based on multi-view vision, comprising the following steps:

[0009] S1: Image acquisition is achieved through multiple camera groups that are pre-laid out and installed on the ship;

[0010] S2: For the acquired images, calculate image feature points, register the features of overlapping areas of the images, and stitch the images acquired by the three monocular cameras of a single camera group into a 180° planar image;

[0011] S3: For the 180° planar images of all camera groups in step S2, extract the features of the overlapping areas between the images, re-register the images and fuse the images to obtain the stitched 180° planar images between the camera groups.

[0012] S4: Stitch together the 180° planar images obtained from all camera groups in step S3 to generate a 360° panoramic view of the ship's surrounding environment.

[0013] S5: Extract video frames from a single camera group, construct a stitching template based on image stitching, and perform video stitching to generate a 180° panoramic video of the ship's surrounding environment. Observe the ship's surrounding environment by switching between different camera group perspectives.

[0014] Furthermore, the camera group layout method in step S1 is as follows: determine the number of camera groups based on the ship size and the working distance of the camera groups, so that the camera groups can acquire 360° environmental information around the ship; design the camera groups based on the working distance of the cameras and the field of view of a single camera, and determine the number of monocular cameras in the camera groups so that the field of view of the camera groups reaches 180°.

[0015] Furthermore, the specific method for image stitching in step S2 is as follows:

[0016] A1: Calculate image feature points, perform image registration, and establish the correspondence between the original images;

[0017] A2: After estimating the transformation and registration between images, the registered images are then deformed and projected onto the same plane by selecting an image plane;

[0018] A3: Merge the aligned images onto a large canvas, merge corresponding pixels in overlapping areas between images, and retain pixels in non-overlapping areas to generate a 180° plan view of the ship's surrounding environment.

[0019] Furthermore, the specific method for calculating image feature points in step A1 is as follows: first, feature points are extracted using SURF, and then feature points are extracted again using ORB. This two-step feature extraction method ensures that more matching points are selected.

[0020] Furthermore, the specific method for image feature point matching in step A2 is as follows: FeatureMatcher is used to match the paired features in step A1, and a two-step feature extraction method is adopted to ensure that more matching points are selected and higher transformation accuracy is obtained. Homography is used to map the feature points between images one by one for feature point matching.

[0021] Furthermore, the specific method for image stitching in step S3 is as follows:

[0022] B1: Input the stitched 180° images of the ship's surrounding environment from each camera group, extract the features of each input image, use ORB (OrientedFAST and RotatedBRIEF) for feature extraction, and perform coarse feature point matching.

[0023] B2: Use the RANSAC (Random Sample Consensus) algorithm to remove erroneous matching points, perform fine matching, and recalculate the homography matrix H;

[0024] B3: Use the Levenberg-Marquardt nonlinear iterative minimum approximation method to find the optimal matching point;

[0025] B4: Based on the optimal matching point, obtain a 180° planar image stitched together between camera groups.

[0026] Further, step B2 is specifically implemented as follows: extract several pairs of matching points from the obtained matching point pairs, calculate the transformation matrix, then calculate the mapping error for all matching points, then determine the inliers based on the error threshold, and recalculate the homography matrix H for the set of the largest inliers.

[0027] Furthermore, the specific method for fine matching using the RANSAC method in step B2 is as follows:

[0028] B2-1: Construct a model with a minimum sampling set of cardinality N (N is the minimum number of samples required to initialize the model parameters) and a sample set P, where the number of samples in set P is #(P)>N. Randomly select a subset S of P containing N samples from P to initialize the model M.

[0029] B2-2: The set of samples whose error with model M is less than a certain set threshold t, along with S, constitutes S*. S* is considered to be an interior set, and they constitute the consensus set of S.

[0030] B2-3: If #(S*)≥N, it is considered that the correct model parameters have been obtained, and the new model M* is recalculated using the set S* (inliers) and other methods such as least squares; a new S is randomly selected again, and the above process is repeated.

[0031] B2-4: After completing a certain number of samplings, select the largest consistent set obtained after sampling to determine the inside and outside points.

[0032] Furthermore, the specific method for solving the optimal matching point using the Levenberg-Marquardt nonlinear iterative minimum approximation method in step B3 is as follows:

[0033] Let x(k) represent the vector composed of the weights and thresholds in the k-th iteration. The new vector x(k+1) composed of weights and thresholds can be obtained according to the following rules:

[0034] x(k+1)=x(k)+Δx

[0035]

[0036] In the formula, The Hessian matrix representing the error index E(x); Represents the gradient;

[0037] Let the error index function be:

[0038]

[0039] In the formula, e i (x) represents the error;

[0040] The method for calculating Δx in the Gauss-Newton method is as follows:

[0041] Δx=-[J T (x)J(x)] -1 J T (x)e(x)

[0042] In the formula, J(x) is the Jacobian matrix of E(x);

[0043] The LM algorithm is an improved Gauss-Newton method, and its method for calculating Δx is as follows:

[0044] Δx=-[J T (x)J(x)+μI] -1 JT (x)e(x)

[0045] In the formula, the proportionality coefficient μ > 0 is a constant, and I is the identity matrix.

[0046] Furthermore, the video stitching of the panoramic video in step S5 specifically involves:

[0047] C1: Input video, extract video frames, calculate similarity measures between pixels or feature points to register selected image pairs;

[0048] C2: Deform and align the registered images, and use the image stitching method in step S3 to stitch the selected original video frames. Since multiple calculations of the transformation matrix will accumulate errors when stitching multiple video frames, resulting in a large distortion of the final result, the stitched video frames are used as stitching templates for stitching.

[0049] C3: Determine if the object has moved. If it has not moved, use this template to stitch together the subsequence frames to generate a wide-angle video. If the object has moved, re-register the images.

[0050] C4: Use the three-frame difference method to perform foreground detection on video information to solve potential blurring and ghosting problems in spliced ​​videos. When the target moves in the overlapping area between images, the splicing template should be updated. During the video splicing process, object detection should be performed to update the splicing template.

[0051] C5: Performs image fusion to generate a stitched panoramic video.

[0052] Furthermore, the specific method of the three-frame difference method in step C4 is as follows:

[0053] By taking the difference between the previous frame and the current frame, and then performing an AND operation between the difference and the difference between the next frame and the current frame, the grayscale detection threshold can be set based on the grayscale value of the changing area of ​​the moving target at this time, which can more accurately obtain the position of the moving target in the video image. This can effectively solve the occlusion problem between the previous and next frames.

[0054] Let D k+1 (x, y) represents two adjacent frames in a video frame sequence. k+1 (x,y) and f k The difference between (x,y), D k (x, y) represents two adjacent frames in a video frame sequence. k (x,y) and f k-1 The difference between (x, y) is used to binarize the video frames by selecting an appropriate threshold, transforming the difference video frame sequence into a binary image sequence R. k (x,y),R k+1 (x,y), Rk (x,y) and R k+1 The final moving target can be obtained by ANDing (x,y). The three-frame difference method is described as follows:

[0055] D k (x,y)=|f k (x,y)-f k-1 (x,y)|

[0056] D k+1 (x,y)=|f k+1 (x,y)-f k (x,y)|

[0057]

[0058] R(x,y)=R k (x,y)∩R k+1 (x,y)

[0059] Where T is the threshold and R(x,y) is the final moving target.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0061] 1. This invention, through the layout and installation of multiple camera groups on the ship, combined with multi-camera image stitching technology, video stitching to generate panoramic images technology, real-time ranging and other technologies, can achieve 360° full coverage perception and acquisition of the ship's surrounding environment, ensuring that the ship's surrounding environment can be completely acquired.

[0062] 2. This invention, based on image and video stitching technologies, enhances the perception of the ship's surrounding environment, significantly reducing the probability of dangers during berthing and navigation. Compared to traditional tugboat push-pull berthing, this method makes berthing visible, greatly enhancing safety and providing a new approach to the intelligent development of ship berthing. For maritime navigation, this invention displays the ship's surrounding environment on the central control screen in the bridge, allowing for early detection of obstacles, reducing the difficulty of handling complex situations, and significantly lowering the probability of dangers during berthing and navigation. Attached Figure Description

[0063] Figure 1 This is a design drawing of the camera assembly described in this invention;

[0064] Figure 2 This is a layout diagram of the ship camera group described in this invention;

[0065] Figure 3 This is the image stitching feature point extraction map described in this invention;

[0066] Figure 4 This is the image stitching feature point matching map described in this invention;

[0067] Figure 5 This is a 180° plan view of the camera group described in this invention;

[0068] Figure 6 This is a flowchart of the camera group image stitching process described in this invention;

[0069] Figure 7 This is a 360° panoramic view of the ship's surrounding environment as described in this invention;

[0070] Figure 8 This is a flowchart of the video stitching process for the ship's surrounding environment as described in this invention;

[0071] Figure 9 This is a 180° panoramic video stitched together to depict the ship's surrounding environment as described in this invention.

[0072] Figure 10 This is a video frame extraction image of the ship's surrounding environment as described in this invention. Detailed Implementation

[0073] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0074] This invention provides a method for perceiving the surrounding environment of a ship based on multi-view vision, referring to... Figure 1 It includes the following steps:

[0075] Step 1: Design a sensing scheme. Design a camera group based on the working distance of the camera and the field of view of a single camera, so that the field of view of a single camera group covers 180° of the environment around the ship; arrange the positions of the camera groups according to the size of the ship, so that the field of view of all camera groups covers 360° of the environment around the ship.

[0076] Step 2: Extract the characteristics of the overlapping area of ​​the three images from the camera group, perform feature matching between the images, and stitch the images together to generate a 180° planar view of the environment around the ship.

[0077] Step 3: Take the 180° plan view of the ship's surrounding environment stitched together from each camera group as input again to stitch together a 360° panoramic view of the ship's surrounding environment.

[0078] Step 4: Extract video frames, construct a stitching template based on image stitching, and generate a 180° panoramic video of the ship's surrounding environment by stitching the video together. Observe the ship's surrounding environment by switching between different camera group perspectives.

[0079] The specific method for step 1 is as follows:

[0080] Provided that the working distance is greater than 40 meters and the field of view of a single camera is greater than 60°, the camera focal length can be calculated using the following formula.

[0081]

[0082] Assuming a working distance of 40,000 mm, a target size of 1.05 mm, and a field of view (FOV) of 3500 x 2600 mm.

[0083]

[0084] In this embodiment, three identical cameras are used to form a trinocular vision system, with the middle camera and the two side cameras arranged at a 60° angle. The camera group design is as follows: Figure 2 As shown, for a 200-meter cargo ship, the working distance of a single camera is 40 meters, and the working distance of a camera group is 80 meters. Therefore, in order to obtain 360° environmental information around the ship, six camera groups need to be deployed around the ship: one group each at the bow and stern, and two groups each on the sides. The layout of the ship's camera groups is as follows. Figure 3 As shown, this ensures that each camera group has a 180° field of view. By extracting features from the overlapping areas of each pair of cameras and using image stitching technology, a 180° plane can be formed. Then, by stitching the images from the six camera groups together, a 360° panoramic view of the ship's surroundings can be generated.

[0085] like Figure 4 As shown, the specific method for image stitching in step 2 is as follows:

[0086] (2.1) By pre-calibrating the intrinsic and extrinsic parameters of the camera, pixel-by-pixel correspondence or feature matching is used to estimate the pixel-based motion model and establish the correspondence between the original images;

[0087] (2.2) Calculate the camera image distortion correction parameters, projective transformation parameters, and homography matrix parameters;

[0088] (2.3) Then perform a projective transformation (perspective transformation) on the calibrated image after distortion correction, and calculate and save the projective transformation parameters;

[0089] (2.4) Calculate image feature points and perform feature extraction. First, SURF is used for feature point extraction, and then ORB is used for feature point extraction. ORB is very fast and rotation-invariant. In this embodiment, ORB is used for feature point extraction. The feature point extraction results are as follows: Figure 5 As shown;

[0090] (2.5) Use FeatureMatcher to match the pairwise features in step 2.4. By performing feature extraction twice between images, more matching points can be selected. The more matching points there are, the more accurate the transformation calculated by findHomography will be.

[0091] Homography is used to map feature points between images one-to-one. The homography matrix is ​​written as:

[0092]

[0093] In this embodiment, a homography matrix is ​​used to map a set of corresponding points in two images: points (x1, y1) in the first image and points (x2, y2) in the second image, thus aligning the images. The feature matching result in this embodiment is as follows: Figure 6 As shown.

[0094]

[0095] (2.6) After estimating the transformation and registration between images, a protection plane is selected. Then, the registered images are deformed and aligned with the projection plane. The aligned images are then fused onto a single plane. Pixels or feature points in overlapping areas are fused, while pixels in non-overlapping areas are retained to generate a 180° planar view of the ship's surrounding environment. The image stitching result in this embodiment is as follows: Figure 7 As shown, Figure 7 A 180° plan view of the ship's surroundings, stitched together from three images from a single camera group.

[0096] The specific method for step 3 is as follows:

[0097] (3.1) Input the photos stitched together from each camera group, and prepare to stitch them together again to generate a 360° panoramic image of the ship;

[0098] (3.2) Extract features from each input image, use ORB (Oriented FAST and Rotated BRIEF) feature extraction, and perform coarse feature point matching;

[0099] (3.3) The RANSAC (Random Sample Consensus) algorithm is used to remove erroneous matching points. First, several pairs of matching points are extracted from the obtained matching point pairs, and the transformation matrix is ​​calculated. Then, the mapping error is calculated for all matching points. Next, inliers are determined based on the error threshold. Finally, the homography matrix H is recalculated for the set of the largest inliers. The specific method is as follows:

[0100] (3.3.1) Construct a model with a minimum sampling set of power N (N is the minimum number of samples required to initialize the model parameters) and a sample set P, where the number of samples in set P is #(P)>N. Randomly select a subset S of P containing N samples from P to initialize the model M.

[0101] (3.3.2) The set of samples whose error with model M is less than a certain set threshold t, along with S, constitutes S*. S* is considered an interior set, and they constitute the consensus set of S;

[0102] (3.3.3) If #(S*)≥N, it is considered that the correct model parameters have been obtained, and the new model M* is recalculated using the least squares method with the set S* (inliers); a new S is randomly selected again, and the above process is repeated.

[0103] (3.3.4) After a certain number of samplings, select the largest consistent set obtained after sampling to determine the inside and outside points;

[0104] (3.4) Then use the Levenberg-Marquardt nonlinear iterative minimum approximation method to solve for the best matching point.

[0105] The LM algorithm is a widely used unconditional optimization algorithm that converges quadratically when approaching a certain minimum point. The specific method is as follows:

[0106] Let x(k) represent the vector composed of the weights and thresholds in the k-th iteration. The new vector x(k+1) composed of weights and thresholds can be obtained according to the following rules:

[0107] x(k+1)=x(k)+Δx (5)

[0108]

[0109] In the formula, The Hessian matrix representing the error index E(x); Represents the gradient;

[0110] Let the error index function be:

[0111]

[0112] In the formula, e i (x) represents the error;

[0113] The method for calculating Δx in the Gauss-Newton method is as follows:

[0114] Δx=-[J T (x)J(x)] -1 J T(x)e(x) (8)

[0115] In the formula, J(x) is the Jacobian matrix of E(x);

[0116] The LM algorithm is an improved Gauss-Newton method, and its method for calculating Δx is as follows:

[0117] Δx=-[J T (x)J(x)+μI] -1 J T (x)e(x) (9)

[0118] In the formula, the proportionality coefficient μ>0 is a constant, and I is the identity matrix;

[0119] As can be seen from equation (9), if the proportionality coefficient μ = 0, it is the Gauss-Newton method; if the value of μ is large, the LM algorithm is close to the gradient descent method. Each successful iteration reduces the value by one step, so as it approaches the error target, it gradually becomes similar to the Gauss-Newton method. Since the LM algorithm uses approximate second derivative information, its convergence speed is much faster than that of the gradient descent method.

[0120] (3.5) The images stitched together from multiple camera groups are used as input images again for image stitching to generate a 360° panoramic image of the ship. The six 180° planar images obtained from the six camera groups are then used as input images again for image stitching to generate a 360° panoramic image of the ship's surrounding environment. Specifically, as follows... Figure 8 As shown.

[0121] Step 4 is as follows:

[0122] Video stitching is based on image stitching, combining object detection and stabilization algorithms to create wide-field-of-view videos. First, selected frames of the input video are stitched together to generate a stitching template. Then, subsequence frames are stitched together based on the template, and the template is updated as objects move across overlapping areas between videos.

[0123] Panoramic stitching is a closed-loop stitching process, in which each image is registered with the corresponding image in the sequence, and the registered images are deformed, aligned, and projected onto a cylindrical or spherical surface to seamlessly generate a 180-degree panoramic video.

[0124] Reference Figure 9 The specific splicing method in step 4 is as follows:

[0125] (4.1) Input video, extract video frames, and calculate the similarity measure between pixels or feature points to register selected image pairs;

[0126] (4.2) Deform and align the registered images, and use the image stitching method in step 3 to stitch the selected original video frames. Since multiple calculations of the transformation matrix will accumulate errors when stitching multiple video frames, resulting in a large deformation of the final result, the stitched video frames are used as stitching templates for stitching.

[0127] (4.3) Determine whether the object has moved. If it has not moved, use this template to stitch together the subsequence frames to generate a wide-angle video. If the object has moved, re-register the images.

[0128] (4.4) Use the three-frame difference method to perform foreground detection on video information to solve potential blurring and ghosting problems in spliced ​​video. When the target moves in the overlapping area between images, the splicing template should be updated. During the video splicing process, object detection should be performed to update the splicing template.

[0129] The specific method of the three-frame difference method is as follows:

[0130] By taking the difference between the previous frame and the current frame, and then performing an AND operation between the difference and the difference between the next frame and the current frame, the grayscale detection threshold can be set based on the grayscale value of the changing area of ​​the moving target at this time, which can more accurately obtain the position of the moving target in the video image. This can effectively solve the occlusion problem between the previous and next frames.

[0131] Let D k+1 (x, y) represents two adjacent frames in a video frame sequence. k+1 (x,y) and f k The difference between (x,y), D k (x, y) represents two adjacent frames in a video frame sequence. k (x,y) and f k-1 The difference between (x, y) is used to binarize the video frames by selecting an appropriate threshold, transforming the difference video frame sequence into a binary image sequence R. k (x,y),R k+1 (x,y), R k (x,y) and R k+1 The final moving target can be obtained by ANDing (x,y). The three-frame difference method is described as follows:

[0132] D k (x,y)=|f k (x,y)-f k-1 (x,y)|

[0133] D k+1 (x,y)=|f k+1 (x,y)-f k (x,y)|

[0134]

[0135] R(x,y)=R k (x,y)∩R k+1 (x,y)

[0136] Where T is the threshold and R(x,y) is the final moving target.

[0137] (4.5) Perform image fusion to generate a stitched 180° panoramic video of the ship's surrounding environment. A schematic diagram of the video frames in this embodiment is shown below. Figure 10 As shown, video frames from four time periods are extracted from the stitched video to observe the environment around the ship. By switching between different camera groups, the video environment at different locations around the ship can be observed, improving the ship's environmental awareness.

[0138] This invention provides a multi-view vision-based method for perceiving the ship's surrounding environment, applying image stitching and video stitching technologies to the maritime field. For ship berthing and unberthing operations, this method displays the ship's surrounding environment on the central control screen in the bridge, visualizing the environment and significantly reducing the difficulty of handling complex situations. Compared to existing tugboat push-pull berthing methods, this invention makes berthing and unberthing visible, greatly enhancing safety and providing a new approach to the intelligent development of ship berthing. It increases the perception capability of the ship's surrounding environment, significantly reduces the probability of berthing and navigation hazards, and solves the problem of weak environmental perception capabilities in current large and medium-sized ships.

Claims

1. A method for perceiving the surrounding environment of a ship based on multi-view vision, characterized in that, Includes the following steps: S1: Image acquisition is achieved through multiple camera groups that are pre-laid out and installed on the ship; S2: For the acquired images, calculate image feature points, register the features of overlapping areas of the images, and stitch the images acquired by a single camera group into a 180° planar image; S3: For the 180° planar images of all camera groups in step S2, extract the features of the overlapping areas between the images, re-register the images and fuse the images to obtain the stitched 180° planar images between the camera groups. S4: Stitch together the 180° planar images obtained from all camera groups in step S3 to generate a 360° panoramic view of the ship's surrounding environment. S5: Extract video frames from a single camera group, construct a stitching template based on image stitching, perform video stitching to generate a 180° panoramic video of the ship's surrounding environment, and observe the ship's surrounding environment by switching the perspectives of different camera groups. The specific method for image stitching in step S3 is as follows: B1: Input the stitched 180° images of the ship's surrounding environment from each camera group, extract the features of each input image, use ORB for feature extraction, and perform coarse feature point matching; B2: Use the RANSAC (Random Sample Consensus) algorithm to remove erroneous matching points, perform fine matching, and recalculate the homography matrix H; B3: Use the Levenberg-Marquardt nonlinear iterative minimum approximation method to find the optimal matching point; B4: Based on the optimal matching point, obtain a 180° planar image stitched together between camera groups; The specific steps for video stitching the panoramic video in step S5 are as follows: C1: Input video, extract video frames, calculate similarity measures between pixels or feature points to register selected image pairs; C2: Deform and align the registered images, and use the image stitching method in step S3 to stitch the selected original video frames, using the stitched video frames as stitching templates for stitching. C3: Determine if the object has moved. If it has not moved, use this template to stitch together the subsequence frames to generate a wide-angle video. If the object has moved, re-register the images. C4: Use the three-frame difference method to perform foreground detection on video information to solve potential blurring and ghosting problems in spliced ​​videos. When the target moves in the overlapping area between images, the splicing template should be updated. During the video splicing process, object detection should be performed to update the splicing template. C5: Performs image fusion to generate a stitched panoramic video.

2. The method for perceiving the surrounding environment of a ship based on multi-view vision according to claim 1, characterized in that, The camera group layout method in step S1 is as follows: determine the number of camera groups based on the ship size and the working distance of the camera groups, so that the camera groups can acquire 360° environmental information around the ship; design the camera groups based on the working distance of the cameras and the field of view of a single camera, and determine the number of monocular cameras in the camera groups so that the field of view of the camera groups reaches 180°.

3. The method for perceiving the surrounding environment of a ship based on multi-view vision according to claim 1, characterized in that, The specific method for image stitching in step S2 is as follows: A1: Calculate image feature points, perform image registration, and establish the correspondence between the original images; A2: After estimating the transformation and registration between images, the registered images are then deformed and projected onto the same plane by selecting an image plane; A3: Merge the aligned images onto a large canvas, merge corresponding pixels in overlapping areas between images, and retain pixels in non-overlapping areas to generate a 180° plan view of the ship's surrounding environment.

4. The method for perceiving the surrounding environment of a ship based on multi-view vision according to claim 3, characterized in that, The specific method for calculating image feature points in step A1 is as follows: first, feature points are extracted using SURF, and then feature points are extracted again using ORB.

5. A method for perceiving the surrounding environment of a ship based on multi-view vision according to claim 4, characterized in that, The specific method for image feature point matching in step A2 is as follows: use FeatureMatcher to match the paired features in step A1, and use homography transformation to map the feature points between images one by one for feature point matching.

6. The method for perceiving the surrounding environment of a ship based on multi-view vision according to claim 1, characterized in that, The specific method of step B2 is as follows: extract several pairs of matching points from the obtained matching point pairs, calculate the transformation matrix, then calculate the mapping error for all matching points, then determine the inliers according to the error threshold, and recalculate the homography matrix H for the largest set of inliers.

7. A method for perceiving the surrounding environment of a ship based on multi-view vision according to claim 6, characterized in that, The specific method for fine matching using the RANSAC method in step B2 is as follows: B2-1: The cardinality of constructing a minimum sample set is The model and a sample set ,gather Number of samples ,from Randomly selected from the middle one sample subset of Initialize the model ; B2-2: Remaining Collection With model The sample set whose error is less than a certain set threshold t and constitute , They are considered to be sets of interior points, and they constitute... A consistent set; B2-3: If They believed they had obtained the correct model parameters and utilized the set. The new model was recalculated using the least squares method. ; Re-randomly draw new Repeat the above process; B2-4: After completing a certain number of samplings, select the largest consistent set obtained after sampling to determine the inside and outside points.

8. The method for perceiving the surrounding environment of a ship based on multi-view vision according to claim 1, characterized in that, The specific method for solving the optimal matching point using the Levenberg-Marquardt nonlinear iterative minimum approximation method in step B3 is as follows: set up Indicates the first The vector formed by the weights and thresholds of the previous iteration, and the vector formed by the new weights and thresholds. It can be obtained according to the following rules: ； ； In the formula, Indicator of error The Hessian matrix; Represents the gradient; Let the error index function be: ； In the formula, For error; In the Gauss-Newton method The calculation method is as follows: ； In the formula, for Jacobian matrix; The LM algorithm is an improved Gauss-Newton method, and its calculation... The method is as follows: ； In the formula, the proportionality coefficient It is a constant. It is an identity matrix.

Images