A monocular binocular fusion offshore target depth estimation method

Abstract

The application discloses a monocular-binoocular fusion offshore target depth estimation method, which comprises the following steps: acquiring a stereo image pair by using a binocular camera; performing target frame detection and depth estimation processing on the current frame stereo image pair to obtain a detection frame and a depth map respectively; mapping the detection frame to the depth map to form a region of interest surrounded by the detection frame on the depth map; performing histogram feature extraction and target contour scanning on each region of interest; according to the position and size of the detection frame and the histogram feature, performing one-to-one matching on all regions of interest of the current frame and the previous frame, and creating and assigning a new Kalman filter for the region of interest in the current frame which fails to be matched; and using the matched Kalman filter in the previous frame for the region of interest in the current frame which succeeds in matching. The method can realize stable and accurate detection of depth information under complex maritime scenes such as camera shaking, rare texture and long distance.

Description

Technical Field

[0001] This invention belongs to the field of robot perception technology and relates to a method for estimating the depth of marine targets by combining monocular and binocular vision. Background Technology

[0002] Existing depth estimation algorithms can be categorized into monocular depth estimation and binocular stereo depth estimation based on the number of images used. With the development of deep learning technology, they can be further divided into depth estimation algorithms based on traditional images and depth estimation algorithms based on deep learning.

[0003] Traditional binocular stereo depth estimation algorithms mimic the human eye's imaging and perception process of the real world. They utilize two cameras to acquire two viewpoint images of the same target scene, then calculate the disparity of identical points in the two viewpoint images to obtain the depth information of the target scene. The calculation process mainly includes the following five steps: binocular camera calibration, image distortion correction, stereo image correction, left-right image matching, and triangulation reprojection of the disparity map. Camera calibration primarily aims to obtain camera parameters, thereby determining the transformation relationship from 3D space to the 2D imaging plane, including image distortion correction. The most critical and challenging aspect of traditional binocular stereo vision is image matching. Due to the different spatial positions of the left and right cameras, the images captured will exhibit horizontal and depth disparities, resulting in differences between the two images. By analyzing the differences in identical points in the left and right images, the true distance of the object from the camera can be calculated based on positional relationships. Traditional stereo matching algorithms include SAD, BM, SGBM, and GC algorithms. These algorithms all rely on global or local approaches to search for and match identical points in the left and right images to calculate the disparity. Deep learning-based binocular stereo vision abandons the traditional step-by-step framework, emphasizing end-to-end implementation. This means it takes two images as input and outputs a depth map, where each pixel represents the distance to the camera. The central network is responsible for extracting image features, matching them, and calculating disparity. Deep learning-based binocular stereo vision algorithms include PSMNet and StereoNet.

[0004] Monocular depth estimation has always been a challenging area in computer vision, and recovering depth information from a single image has long been considered an ill-posed problem. Traditional monocular depth estimation algorithms require a large amount of prior knowledge, such as the actual size of the object and camera parameters. They calculate the object's distance from the camera by comparing its size in the image with its actual size. Deep learning-based monocular depth estimation can be broadly categorized into three types based on training methods: supervised algorithms, unsupervised algorithms, and self-supervised algorithms. Supervised algorithms rely on a large number of real depth information labels, using these labels for model training. This method often exhibits poor generalization on datasets, and obtaining real depth information is difficult and expensive. Unsupervised algorithms do not require depth information labels at all; they only need the input image. The model relies on conditions such as near objects appearing larger and distant objects appearing smaller for self-constraint convergence. This method often requires designing large and complex networks to achieve good results. Self-supervised algorithms rely on a small number of depth information labels and a large amount of unlabeled information. They first train the model with a small number of labels and then learn generalization using a large number of unlabeled labels. The basic principle of all these algorithms is to require a continuous sequence of monocular images. The model learns distance changes from the monocular image sequence to estimate the distance. Deep learning-based depth estimation models include monodepth, depthanything, etc.

[0005] Generally, the aforementioned monocular and binocular depth estimation algorithms directly use the depth value of the center of the region of interest (ROI) as the final depth value. However, this approach is unsuitable for maritime scenes. Due to the varying shapes of ships, and the presence of sails or masts, asymmetry can easily occur. Directly using the center point of the ROI may result in the value falling outside the ship's hull, leading to errors. Furthermore, when using unmanned surface vessels (USVs) for filming at sea, the USV's motion—whether stationary, moving, or tossed about by waves and wind—must be continuous and abrupt. Therefore, the camera on the USV will not be stationary but will be in a state of irregular, continuous motion. This camera movement causes changes in the image, posing a significant challenge to depth detection models. Meanwhile, due to the vastness of the sea surface and the large size of ships compared to vehicles, the detection distance on the sea surface is generally over 50 meters. This requires the support of a large baseline binocular camera. However, a problem exists with large baseline binocular cameras: the left part of the image from the left camera and the right part of the image from the right camera cannot be matched because no corresponding point can be found in the other image. Furthermore, this area increases as the baseline increases, which leads to an increase in the blind zone of binocular matching. The effective binocular matching area is only the middle part. When a target appears at the edge, it is easy to obtain inaccurate information. Summary of the Invention

[0006] To address the aforementioned technical problems in the existing technology, this invention proposes a method for estimating the depth of maritime targets using a combination of single and binocular vision, the specific technical solution of which is as follows:

[0007] A method for estimating the depth of marine targets using a binocular camera fusion approach, employing stereo image pairs acquired by binocular camera capture;

[0008] Perform bounding box detection and depth estimation on the current frame stereo image pair to obtain the detection bounding box and depth map, respectively.

[0009] The detection boxes are mapped onto the depth map, forming a region of interest on the depth map surrounded by the detection boxes;

[0010] Histogram feature extraction and target contour scanning are performed for each region of interest;

[0011] Based on the position and size of the detection box and histogram features, all regions of interest in the current frame and the previous frame are matched one by one. For regions of interest in the current frame that cannot be matched, a new Kalman filter is created and assigned to them; for regions of interest in the current frame that are matched, the Kalman filter that was paired in the previous frame is used.

[0012] Furthermore, when the current frame stereo image pair is not the initial frame, the current frame stereo image pair uses its previous frame image pair to perform edge region stitching processing to obtain a new stereo image pair. The new stereo image pair is then input into the depth estimation model for processing to obtain a depth map.

[0013] Furthermore, the stereo image pair includes a left camera image and a right camera image; the regional mono- and binocular fusion operation specifically involves: superimposing the overlapping portions of the left and right camera images of the current frame, stitching the right region image of the previous frame's right camera image onto the right side of the current frame's left camera image, and stitching the left region image of the previous frame's left camera image onto the left side of the current frame's right camera image.

[0014] Furthermore, the target contour scanning is performed using the Sobel edge extraction operator, and the points within the target contour are the depth distance values ​​of the target.

[0015] Furthermore, when the similarity of the detection box position size and histogram features between the current frame and the previous frame is the highest, the two regions of interest are successfully matched, indicating that they belong to the same target object.

[0016] Furthermore, the Kalman filter is used to input the depth value obtained from its corresponding region of interest and perform a smoothing operation to output a final depth estimate, which is the distance between the target and the camera.

[0017] The present invention has the following beneficial effects:

[0018] The method of this invention adjusts the selected output depth point, and does not directly select the center point of the detection box, so as to ensure that the selected output depth point is inside the object. This can reduce the error caused by the selection point being outside the ship due to the asymmetry of the ship. The method also smooths the depth value of the same target, which can effectively reduce the impact of the abrupt change in depth caused by the previous depth estimation model, and ensure the continuity and consistency of the depth information of the same object in the time dimension.

[0019] This invention performs region-based processing on images, while the processing of overlapping areas remains unchanged. It still uses the binocular matching concept, combining the left and right images. For blind spots, it uses information from the previous frame and applies the idea of ​​reflecting depth through time series in monocular depth estimation. It incorporates time information into the input data and then feeds it into the depth estimation model for learning. This effectively reduces the blind spots of binocular stereo vision under large baselines, expands the area of ​​reliable depth, and improves the reliability of depth information for ships appearing at the edges of images. Furthermore, this part is decoupled from the subsequent depth estimation model and can be applied to any depth estimation model. It effectively solves the problem that large baseline binocular cameras are required for ranging of maritime targets, but large baseline binocular cameras cause the overlapping area between the left and right images to become smaller, resulting in a larger blind spot for binocular matching. Attached Figure Description

[0020] Figure 1 This is a flowchart of the single- and binocular fusion method for estimating the depth of maritime targets according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of a large baseline binocular camera imaging according to an embodiment of the present invention;

[0022] Figure 3 This is a flowchart of the monocular and binocular fusion depth detection method according to an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] This invention discloses a method for estimating the depth of maritime targets using a combination of monocular and binocular vision, specifically including the following:

[0025] like Figure 1 and Figure 2 As shown, stereo image pairs are acquired by capturing images with a binocular camera, including the left camera image and the right camera image. Then, one of the stereo image pairs is input into the object detection model to obtain the detection boxes of all objects in the image.

[0026] Before being input into the depth estimation model, stereo image pairs undergo a region-based monocular and binocular fusion operation. For example... Figure 3As shown, based on the binocular detector, the depth information perception brought by the binocular images and video frame sequences is combined. The overlapping parts of the left and right images are superimposed. For the baseline blind area, the images of the previous few frames are combined. The images of the same area in the previous few frames are superimposed on the current image to form a new multi-channel complete image. The normal image is RGB 3-channel, and the newly formed image is a 6-channel image. Then, the multi-channel complete image is fed into the network model for learning. The network model will not only learn the depth relationship between the left and right images, but also the depth relationship of the edge region caused by the motion of a single video sequence.

[0027] Specifically, for the right region of the left camera image and the left region of the right camera image, since the points in these two regions can be found in both images, no processing is performed. For the left region of the left camera image and the right region of the right camera image, since these two regions cannot be found in the other image, the stereo image pair from the previous frame is extracted, and the two parts of the corresponding size are extracted and stitched into the current stereo image pair to generate a new stereo image pair. Compared to the original stereo image pair, the new stereo image pair has a higher resolution and incorporates temporal motion information. The specific stitching method is as follows: for the left camera image, the right part of the previous frame's right camera image is stitched to the right side of the original left camera image, and the left part of the previous frame's left camera image is stitched to the left side of the original right camera image, forming a new stereo image pair.

[0028] Then, the new stereo image pairs are input into the depth estimation model. The model will not only learn the depth relationship between the new stereo image pairs, but also the depth relationship caused by the motion of a single video sequence in the edge region. Finally, a depth map is output, in which each pixel represents the distance between that point and the camera in three-dimensional space.

[0029] After obtaining the outputs of the object detection model and the depth estimation model, the output bounding boxes of the object detection model are mapped onto the output depth map of the depth estimation model to obtain a depth map with bounding box information.

[0030] The region enclosed by the detection box on the depth map is defined as the region of interest (ROI). An edge extraction operator, such as the Sobel operator, is then used to scan the contour of the target within this ROI, and points within that contour are selected as the depth distance values ​​for that target. Simultaneously, histogram feature extraction is performed on the ROI to obtain histogram features corresponding to each ROI; these histogram features also reflect the target characteristics.

[0031] In this invention, an initial Kalman filter is created and assigned to each region of interest (ROI) in the initial frame. In each subsequent frame, the ROIs of the previous frame are matched one-to-one with those of the current frame. Specifically, if the similarity in the position, size, and histogram features of the detection boxes in two frames is highest, the two ROIs are considered a successful match and belong to the same object. The ROI in the current frame will then use the paired Kalman filter from the previous frame. If there are still unmatched ROIs in the current frame, a new Kalman filter is created for the remaining unmatched ROIs. Finally, the depth distance value of the output points selected after contour extraction for each ROI is fed into its respective Kalman filter. After smoothing by the filter, the final result is the depth distance value between the target and the camera.

[0032] The method of this invention can achieve stable and accurate detection of depth information in complex maritime scenarios such as camera shaking, sparse textures, and long distances.

[0033] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for estimating the depth of maritime targets using a combination of monocular and binocular cameras, characterized in that: Stereo image pairs are acquired by using a binocular camera; Perform bounding box detection and depth estimation on the current frame stereo image pair to obtain the detection bounding box and depth map, respectively. The detection boxes are mapped onto the depth map, forming a region of interest on the depth map surrounded by the detection boxes; Histogram feature extraction and target contour scanning are performed for each region of interest; Based on the position and size of the detection box and histogram features, all regions of interest in the current frame and the previous frame are matched one by one. For regions of interest in the current frame that cannot be matched, a new Kalman filter is created and assigned to them; for regions of interest in the current frame that are matched, the Kalman filter that was paired in the previous frame is used. The Kalman filter is used to input the depth value obtained from its corresponding region of interest and perform a smoothing operation to output a final depth estimate, which is the distance between the target and the camera.

2. The method for estimating the depth of a maritime target according to claim 1, characterized in that: When the current frame stereo image pair is not the initial frame, the current frame stereo image pair uses its previous frame image pair to perform a regional monocular and binocular fusion operation to obtain a new stereo image pair. The new stereo image pair is then input into the depth estimation model for processing to obtain a depth map.

3. The method for estimating the depth of a maritime target according to claim 2, characterized in that: The stereo image pair includes a left camera image and a right camera image; The specific method of segmented monocular and binocular fusion operation is as follows: the overlapping parts of the left and right camera images of the current frame are superimposed, the right region image of the right camera image of the previous frame is stitched on the right side of the left camera image of the current frame, and the left region image of the left camera image of the previous frame is stitched on the left side of the right camera image of the current frame.

4. The method for estimating the depth of a maritime target according to claim 1, characterized in that: The target contour scanning is performed using the Sobel edge extraction operator.

Images