Maritime target depth estimation method based on monocular and binocular fusion

Abstract

The invention discloses a monocular and binocular fused sea target depth estimation method. The method comprises the following steps: shooting by using a binocular camera to obtain a stereo image pair; performing target frame detection and depth estimation processing on the current frame stereo image pair to obtain a detection frame and a depth map respectively; mapping the detection frame to the depth map, and forming a region of interest surrounded by the detection frame on the depth map; performing histogram feature extraction and target contour scanning on each region of interest; performing one-to-one matching on all regions of interest of the current frame and the previous frame according to the position size and histogram characteristics of the detection frame, and creating and distributing a new Kalman filter for unsuccessfully matched regions of interest in the current frame; and for the successfully matched region of interest in the current frame, using the paired Kalman filter in the previous frame. According to the method, stable and accurate detection of depth information can be realized in complex maritime scenes such as camera shaking, rare texture and long distance.

Description

Technical Field

[0001] The invention belongs to the technical field of robot perception and relates to a monocular and binocular fusion method for estimating the depth of an offshore target. Background Art

[0002] Existing depth estimation algorithms can be divided into monocular depth estimation and binocular stereo depth estimation according to the number of images used. With the development of deep learning technology, it can be further divided into traditional image-based depth estimation algorithms and deep learning-based depth estimation algorithms.

[0003] The traditional binocular stereo depth estimation algorithm imitates the imaging and perception process of the human eye for the real world. It uses two cameras to obtain two viewpoint images of the same target scene, and then calculates the disparity of the same points in the two viewpoint images to obtain the depth information of the target scene. Its calculation process mainly includes the following five steps: binocular camera calibration, image distortion correction, stereo image correction, left and right image matching, and triangulation reprojection disparity map. Camera calibration is mainly to obtain the parameters of the camera, so as to obtain the conversion relationship of the image from three-dimensional space to two-dimensional imaging plane, including image distortion correction. The most critical and difficult point in traditional binocular stereo vision is image matching. Due to the different spatial positions of the left and right cameras, the images captured will have horizontal and depth disparity in the image, resulting in differences between the two images. Through the difference of the same points in the left and right images, the real distance of the object from the camera can be calculated through the position relationship. Traditional stereo matching algorithms include SAD algorithm, BM algorithm, SGBM algorithm, GC algorithm, etc. These algorithms are based on global or local ideas to search and match the same points in the left and right pictures to calculate the disparity. Binocular stereo vision based on deep learning abandons the traditional binocular stereo vision step framework and emphasizes end-to-end implementation, that is, inputting two left and right pictures and outputting a depth map, where each pixel represents the distance to the camera. The network in the middle is responsible for extracting picture features, matching, and calculating disparity. Binocular stereo vision algorithms based on deep learning include PSMNet, stereoNet, etc.

[0004] Monocular depth estimation has always been a difficult problem in the field of computer vision. Recovering depth information from a single image has always been considered a pathological problem. Traditional monocular depth estimation algorithms require a lot of prior knowledge, such as the actual size of the target, the parameters of the camera, etc., and the depth information of the object from the camera is calculated based on the comparison between the size of the object in the image and the actual size of the object. Monocular depth estimation based on deep learning can be roughly divided into three categories according to the training method: supervised algorithm, unsupervised algorithm and self-supervised algorithm. Supervised algorithms rely on a large number of real depth information labels and use these labels for model training. This method often has poor generalization on the data set, and it is difficult and expensive to obtain real depth information. Unsupervised algorithms do not require depth information labels at all, only the input image is needed. The model relies on conditions such as near big and far small for self-constrained convergence. This method often requires the design of a large and complex network to achieve a better network. Self-supervised algorithms rely on a small number of depth information labels and a large amount of unlabeled information. The model is first trained with a small number of labels, and then generalized with a large number of unlabeled labels. The basic principle is that a continuous sequence of monocular images is required, and the model learns the change of distance from the monocular image sequence to estimate the distance. Depth estimation models based on deep learning include monodepth, depthanything, etc.

[0005] Generally speaking, the above-mentioned monocular and binocular depth estimation algorithms will directly use the depth value in the middle of the region of interest as the final depth value, but this cannot be done for scenes at sea. Since the shapes of ships vary, and some target ships may have sails or masts, this can easily lead to asymmetry. Directly using the center point of the region of interest is likely to cause it to fall outside the hull, resulting in errors. Moreover, when using an unmanned ship to shoot at sea, whether the unmanned ship is stationary, driving, or tossed up and down by waves and wind, its movement should be continuous and cannot change suddenly. Therefore, the camera on the unmanned ship will not be in a stationary state, but in an irregular continuous motion state. Due to the movement of the camera, the image will also change, which is undoubtedly a huge challenge for the depth detection model. At the same time, due to the wide sea environment and the large size of ships compared to vehicles, the detection distance on the sea is generally more than 50m, which requires a large baseline binocular camera to support it. However, there is a problem with the large baseline binocular camera: the left part of the left camera image and the right part of the right camera image cannot be matched because no corresponding points can be found in the other image, and this area will increase with the increase of the baseline, which will lead to an increase in the blind area of ​​binocular matching. The effective binocular matching area is only in the middle part. When a target appears at the edge, it is easy to obtain inaccurate information. Summary of the invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes a method for estimating the depth of marine targets by combining monocular and binocular fusion, and its specific technical solution is as follows: A method for estimating the depth of marine targets by using a single-eye and binocular fusion method, which uses a binocular camera to obtain a stereo image pair; Perform target frame detection and depth estimation on the current frame stereo image pair to obtain a detection frame and a depth map respectively; Mapping the detection frame onto the depth map to form a region of interest surrounded by the detection frame on the depth map; Perform histogram feature extraction and target contour scanning on each region of interest; According to the position and size of the detection frame and the histogram features, all regions of interest in the current frame and its previous frame are matched one by one. For the regions of interest that are not successfully matched in the current frame, a new Kalman filter is created and assigned to them; for the regions of interest that are successfully matched in the current frame, the paired Kalman filter in the previous frame is used.

[0007] Furthermore, when the current frame stereo image pair is not an initial frame, the current frame stereo image pair uses its previous frame image pair to perform edge area stitching processing to obtain a new stereo image pair, and the new stereo image pair is input into the depth estimation model to obtain a depth map.

[0008] Furthermore, the stereo image pair includes a left camera image and a right camera image; the region-specific monocular and binocular fusion operation is specifically: superimposing the overlapping parts of the left camera image and the right camera image of the current frame, splicing the right area image of the right camera image of the previous frame on the right side of the left camera image of the current frame, and splicing the left area image of the left camera image of the previous frame on the left side of the right camera image of the current frame.

[0009] Furthermore, the target contour scanning is performed using a Sobel edge extraction operator, and the points within the target contour are depth distance values ​​of the target.

[0010] Furthermore, when the similarity of the position size and histogram features of the detection box of the current frame and the previous frame is the highest, the two regions of interest are matched successfully, indicating that they belong to the same target object.

[0011] Furthermore, the Kalman filter is used to put the depth value obtained in the corresponding region of interest, perform a smoothing operation, and output a final depth estimation value, that is, the distance value between the target and the camera.

[0012] The present invention has the following beneficial effects: The method of the present invention adjusts the selected output depth point and does not directly select the center point of the detection frame, ensuring that the selected output depth point is inside the object, which can reduce the error caused by the selected point outside the hull due to the asymmetry of the ship. The depth value of the same target is smoothed, which can effectively reduce the influence of the depth mutation caused by the previous depth estimation model, and ensure the continuity and consistency of the depth information of the same object in the time dimension.

[0013] The present invention processes images in different regions, and the processing of overlapping regions remains unchanged. The binocular matching idea is still used, and the left and right images are combined. For the blind area, the information of the previous frame is used, and the idea of ​​time series reflecting depth in monocular depth estimation is applied to integrate the time information into the input data, and then the data is sent to the depth estimation model for learning. The blind area of ​​binocular stereo vision under a large baseline can be effectively reduced, the area of ​​credible depth can be expanded, and the credibility of depth information for some ships appearing at the edge of the picture can be improved. Moreover, this part is decoupled from the subsequent depth estimation model and can be applied to any depth estimation model. The problem that a large baseline binocular camera is needed for ranging of marine targets, but the large baseline binocular camera will cause the overlapping area of ​​the left and right images to become smaller and the blind area of ​​binocular matching to become larger can be effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of a method for estimating the depth of a marine target by using monocular and binocular fusion according to an embodiment of the present invention; Figure 2 is a schematic diagram of large baseline binocular camera imaging according to an embodiment of the present invention; Figure 3 It is a flow chart of the monocular and binocular fusion depth detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0016] The embodiment of the present invention discloses a method for estimating the depth of a marine target by using a monocular and binocular fusion method, which specifically includes the following contents: like Figure 1 and Figure 2 As shown, a stereo image pair is obtained by shooting with a binocular camera, including a left camera image and a right camera image. Then, one of the images of the stereo image pair is input into the target detection model to obtain the detection boxes of all targets in the image.

[0017] Before the stereo image pair is input into the depth estimation model, it will first perform a monocular and binocular fusion operation in different regions. Figure 3As shown, based on the binocular detector, the binocular image and the depth information perception brought by the video frame sequence are combined, and the overlapping parts of the left and right images are superimposed. For the baseline blind area, the images of the previous frames are combined, and the images of the same area of ​​the previous frames are superimposed on the current image to form a new multi-channel complete image. The normal image is RGB3 channels, and the newly formed image is a 6-channel image. The multi-channel complete image is then sent to the network model for learning. The network model will not only learn the depth relationship reflected between the left and right images, but also learn the depth relationship caused by the movement of a single video sequence in the edge area.

[0018] Specifically, for the right area image of the left camera image and the left area image of the right camera image, since the points in these two areas can find corresponding points in the left and right images, they are not processed. For the left area of ​​the left camera image and the right area of ​​the right camera image, since these two areas cannot be found in each other's pictures, the stereo image pair of the previous frame is extracted, and two parts of corresponding sizes are extracted respectively, and they are spliced ​​into the current stereo image pair to generate a new stereo image pair. Compared with the original stereo image pair, the new stereo image pair has a larger resolution, and the new stereo image pair incorporates time motion information. The specific splicing method is as follows: for the left camera image, the right part of the previous frame of the right camera image is spliced ​​on the right side of the original left camera image, and the left part of the previous frame of the left camera image is spliced ​​on the left side of the original right camera image to form a new stereo image pair.

[0019] Afterwards, the new stereo image pair is input into the depth estimation model. The model will not only learn the depth relationship reflected by the new stereo image pair, but also learn the depth relationship caused by the motion of a single video sequence in the edge area. Finally, a depth map is output. Each pixel on the depth map represents the distance between the point and the camera in three-dimensional space.

[0020] After obtaining the outputs of the target detection model and the depth estimation model, the output detection frame of the target detection model is mapped to the output depth map of the depth estimation model to obtain a depth map with detection frame information.

[0021] The area surrounded by the detection box on the depth map is the region of interest. Then, the region of interest is scanned using an edge extraction operator such as the Sobel operator to scan the target contour, and the points within the target contour are selected as the depth distance value of the target. At the same time, the histogram feature extraction is performed on the region of interest to obtain the histogram feature corresponding to each region of interest. The histogram feature also reflects the target feature.

[0022] The present invention creates and allocates an initialized Kalman filter for each region of interest at the initial frame. In each subsequent frame, the region of interest of the previous frame is matched one by one with the region of interest of the current frame, that is: when the position size and similarity of the histogram features of the two-frame detection frame are the highest, it can be considered that the two regions of interest are successfully matched and belong to the same object, then the region of interest of the current frame will continue to use the paired Kalman filter in the previous frame; if there are still unmatched regions of interest in the current frame, a new Kalman filter is created for the remaining unmatched regions of interest in the current frame. Finally, the depth distance value of the output point selected after contour extraction of each region of interest is placed in the respective Kalman filters, and the final result is obtained by smoothing the filter, which is the depth distance value of the target and the camera.

[0023] The method of the present invention can achieve stable and accurate detection of depth information in complex maritime scenes such as camera shaking, sparse texture, and long distance.

[0024] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for estimating the depth of a marine target by using monocular and binocular fusion, characterized in that: A binocular camera is used to obtain a stereo image pair; Perform target frame detection and depth estimation on the current frame stereo image pair to obtain a detection frame and a depth map respectively; Mapping the detection frame onto the depth map to form a region of interest surrounded by the detection frame on the depth map; Perform histogram feature extraction and target contour scanning on each region of interest; According to the position and size of the detection frame and the histogram features, all regions of interest in the current frame and its previous frame are matched one by one. For the regions of interest that are not successfully matched in the current frame, a new Kalman filter is created and assigned to them; for the regions of interest that are successfully matched in the current frame, the paired Kalman filter in the previous frame is used.

2. The method for estimating the depth of a marine target according to claim 1, characterized in that: When the current frame stereo image pair is not the initial frame, the current frame stereo image pair uses its previous frame image pair to perform a region-by-region monocular and binocular fusion operation to obtain a new stereo image pair, and the new stereo image pair is input into a depth estimation model for processing to obtain a depth map.

3. The method for estimating the depth of a marine target according to claim 2, characterized in that: The stereo image pair includes a left camera image and a right camera image; The region-based monocular and binocular fusion operation is specifically as follows: superimposing the overlapping parts of the left camera image and the right camera image of the current frame, splicing the right area image of the right camera image of the previous frame on the right side of the left camera image of the current frame, and splicing the left area image of the left camera image of the previous frame on the left side of the right camera image of the current frame.

4. The method for estimating the depth of a marine target according to claim 1, characterized in that: The target contour scanning is performed using the Sobel edge extraction operator.

5. The method for estimating the depth of a marine target according to claim 1, characterized in that: When the similarity of the position size and histogram features of the detection box of the current frame and the previous frame is the highest, the two regions of interest are matched successfully, indicating that they belong to the same target object.

6. The method for estimating the depth of a marine target according to claim 1, characterized in that: The Kalman filter is used to put the depth value obtained in the corresponding region of interest, perform a smoothing operation, and output a final depth estimation value, that is, the distance value between the target and the camera.

Images