A method for object detection based on disparity segmentation

Abstract

The invention discloses a target detection method based on disparity segmentation, and establishes three detection branches with the same network structure for multi-scale target recognition. Different from other multi-scale networks, each detection branch only receives image information at a corresponding distance scale after disparity segmentation. In addition, before the image information is transmitted to each branch, its resolution is subjected to a specified transformation to enhance the target's feature information expression capability. At the end of the network, the detection results of each scale branch are scaled and merged to obtain a complete detection result. Compared with other multi-scale detection methods, the multi-scale detection network based on disparity segmentation is more accurate and reliable for the detection of small targets at medium and long distances.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and relates to an image segmentation method, and in particular to a target detection method based on disparity segmentation. Background Art

[0002] In recent years, with the rapid development and progress of deep learning, the application of deep convolutional neural networks in the field of target detection has become increasingly widespread and mature. Although there are obvious differences in the structure of target detection networks based on convolutional neural networks, they all follow the basic training process of image sampling, feature extraction, classification of targets under the guidance of loss functions, and position regression. At the same time, the change in target scale caused by the different distances between the target and the viewpoint will also affect the performance of various forms of detectors, especially when dealing with small targets that are within the field of view and far away from the viewpoint. The detection accuracy of the detector will decrease due to reasons such as small target resolution, incomplete feature extraction, and redundant information interference.

[0003] In order to solve the problem that the target scale changes with the viewpoint distance, the existing technology trains the targets of each scale of the image by changing the resolution of the image itself, so as to achieve the effect of multi-scale detection. However, this method has a large amount of calculation and takes a long time. The feature fusion method extracts features at different scale levels and predicts the feature map of each corresponding scale. However, since the network directly integrates the features of each scale, it cannot make good use of the multi-scale features. When predicting a specific scale, the transmission of redundant image information that does not belong to the scale in the image will also lead to reduced detection accuracy. The above solutions to the degradation of model performance caused by the change of target scale have greatly improved the detection performance of the network for small targets. However, in the process of extracting features of each scale, the problem of interference of redundant information that does not belong to the extracted scale on the network to extract semantic information has not been solved. Moreover, for small targets at a long distance and small targets at a close distance with similar sizes in the image, their ability to express features cannot be generalized due to the difference in resolution.

[0004] In the field of image detection, there is no complete and mature technology that can perfectly solve the problem of model performance degradation caused by changes in target scale. In the process of extracting features at each scale, the problem of redundant information that does not belong to the extracted scale interfering with the network's extraction of semantic information also needs to be solved. Summary of the invention

[0005] In response to the shortcomings of the existing technology, this application proposes a target detection method based on disparity segmentation, which calculates the depth distance of each pixel according to the disparity of the binocular image, divides the image based on the depth distance, and then inputs images of different distance scales into networks of different branches for targeted detection. Finally, the output structure of each branch is scaled and merged to obtain a complete detection result, thereby improving the detection accuracy.

[0006] A multi-scale detection method based on disparity segmentation specifically includes the following steps:

[0007] Step 1: Collect the binocular image pair to be detected, and calculate the depth distance Depth of each pixel in the image according to the disparity Disparity of the left and right images in the binocular image pair:

[0008]

[0009] Where f is the focal length of the binocular camera, and baseline is the baseline distance between the two cameras of the binocular camera.

[0010] Step 2: The depth distance Depth of each pixel on the left image calculated in step 1 is used to segment the left image into three images of different distance scales: short distance, medium distance, and long distance. In order to avoid edge feature loss during image segmentation, an overlapping part is set between two adjacent regions of distance scale. For pixels whose depth distance is within the overlapping range, they can be displayed in images of two distance scales at the same time, alleviating the information loss caused by image segmentation. The depth distances of all pixels are sorted from small to large to obtain the maximum value D, and then the images of three different distance scales are divided according to the ranges of [0, 30% D], [28% D, 74% D], and [66% D, D]. In the images corresponding to each distance after segmentation, except for the pixels within the distance range, the other parts are not displayed, which can avoid the interference of redundant information on detection; then the resolution of the medium-distance image and the long-distance image is increased, and the resolution of the close-distance image remains unchanged.

[0011] Preferably, the resolution of the medium-range image and the long-range image are increased by 2 times and 4 times respectively.

[0012] Step 3: collect images at different distance scales to form a data set, and randomly divide them into a close-range image group, a medium-range image group, and a long-range image group, wherein the image resolutions in the medium-range image group and the long-range image group are increased by the same multiple as the images of the corresponding distance scales in step 2.

[0013] Step 4: Establish three neural networks with the same structure as three detection branches. The three detection branches receive the images of the close-range image group, the medium-range image group and the long-range image group in step 3 respectively. Set the following optimization objectives based on the minimization of the mean square error for network training:

[0014]

[0015] Among them, N cls represents the mini-batch size, N reg Indicates the number of Anchor Locations, L cls represents the classification loss, L reg represents the regression loss, j represents the number of detection branches, and μ represents the weight balance parameter.

[0016] Preferably, the detection branch is based on a faster r-cnn or cascade r-cnn network structure.

[0017] Step 5: Use the network trained in step 4 to perform target detection. Input the three images of close distance, medium distance, and long distance segmented in step 2 into the detection branches at the corresponding distance scales respectively. Then scale the output results of the three detection branches to the scale before the resolution is increased and splice them as the final target detection results.

[0018] Preferably, overlapping detection boxes are processed by a non-maximum suppression method.

[0019] The present invention has the following beneficial effects:

[0020] By calculating the disparity to integrate the depth information into the target detection, a method of segmenting the image based on the distance between the target and the viewpoint is proposed. This method is more accurate and reliable than segmenting the scale according to the size of the target in the image. By accurately segmenting the corresponding area on the image at each scale, the segmented image is input into the network branch of the corresponding scale for targeted detection. By setting up multiple branches to perform targeted detection on targets at each scale, the purpose of deleting redundant information and providing a detector with a resolution that matches the resolution of small targets is achieved, further improving the accuracy of multi-scale detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a picture to be detected in the embodiment;

[0022] Figure 2 is a close-range image after segmentation in the embodiment;

[0023] Figure 3 The detection result of the close distance prediction branch in the embodiment;

[0024] Figure 4 is the mid-range image after segmentation in the embodiment;

[0025] Figure 5 The detection result of the distance prediction branch in the embodiment;

[0026] Figure 6 is a long-distance image after segmentation in the embodiment;

[0027] Figure 7 The detection result of the long-distance prediction branch in the embodiment;

[0028] Figure 8 It is the final target detection result in the embodiment. DETAILED DESCRIPTION

[0029] The present invention is further explained below in conjunction with the accompanying drawings.

[0030] A multi-scale detection method based on disparity segmentation specifically includes the following steps:

[0031] Step 1: Build three prediction branches with the same network structure and use the images in the kitti dataset for network training. The resolution of medium-range images and long-range images are increased by 2 times and 4 times respectively.

[0032] Step 2: Figure 1 Taking the image shown as an example, the depth distance Depth of each pixel on the left image is calculated, and then the left image is divided into three images with different distance scales: close distance, medium distance, and long distance according to the size of the depth distance value. The depth distance ranges of the pixels in the close distance, medium distance, and long distance images are [0,77], [71,189], [170,255] respectively. An overlapping part is set between two images with adjacent distance scales. For example, pixels with a depth distance in the range of [71,77] will appear in both the close distance image and the medium distance image. This can avoid the loss of edge features due to image segmentation. Pixels with a depth distance outside the corresponding range are not displayed to avoid interference of redundant information on detection. For example, pixels with a depth distance of 45 will only appear in the close distance image. After segmentation, the close distance, medium distance, and long distance images are as follows Figure 2 , as shown in Figures 4 and 6.

[0033] Step 3: Increase the resolution of the medium-distance image and the long-distance image by 2 and 4 times respectively, and keep the resolution of the close-distance image unchanged, and then input it into the corresponding detection branch. After obtaining the detection results of the three scale areas, scale the results back to the normal size to obtain the final results. When merging the detection results at each scale, we also use the non-maximum suppression method to deal with the detection frames with a high degree of overlap.

[0034] Using the same network structure, the images segmented by this method are compared with the direct detection method. The experimental results are shown in the following table, including the detection method, backbone network, final accuracy, and individual accuracy of the three distances: near, medium, and far:

[0035]

[0036] Among them, those with * represent the present method, and those without * represent the traditional method without image segmentation.

[0037] According to the data in the table, the multi-scale detection network based on disparity segmentation can further improve the accuracy based on the original method after being combined with new methods such as variable kernel convolution and feature pyramid, which shows that the multi-scale detection network based on disparity segmentation can be compatible with these emerging technologies. In addition, in order to further verify the multi-scale detection performance of this method, we applied NAS-FPN to Faster-rcnn as a baseline, compared it with the multi-scale detection network based on disparity segmentation, and showed the evaluation results under the same training settings in the table. The data shows that compared with the baseline, the multi-scale detection network based on disparity segmentation has significantly improved the detection accuracy at all scales, indicating the reliability and accuracy of disparity segmentation for the division of each scale of the image.

[0038] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent transformation made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for object detection based on disparity segmentation, characterized in that: The method specifically comprises the following steps: Step 1: Collect the binocular image pair to be detected, and calculate the depth distance Depth of each pixel in the image according to the disparity Disparity of the left and right images in the binocular image pair; Where, f is the focal length of the binocular camera, and baseline is the baseline distance between the two cameras of the binocular camera; Step 2: Using the depth distance Depth of each pixel point on the left image calculated in step 1, the left image is divided into three images of different distance scales: long distance, medium distance, and close distance. Specifically, the depth distances of the pixels in the long distance, medium distance, and close distance images are: [0, 30% D], [28% D, 74% D], [66% D, D], where D is the maximum depth distance of all pixels on the left image. Then, the resolution of the medium distance image and the long distance image is increased, while the resolution of the close distance image remains unchanged. Step 3: collect images at different distance scales to form a data set, and randomly divide them into a close-range image group, a medium-range image group, and a long-range image group, wherein the image resolutions in the medium-range image group and the long-range image group are increased by the same multiple as the images at the corresponding distance scales in step 2; Step 4: Establish three neural networks with the same structure as three detection branches. The three detection branches receive the images of the close-range image group, the medium-range image group and the long-range image group in step 3 respectively. Set the following optimization objectives based on the minimization of the mean square error for network training: Among them, N cls represents the mini-batch size, N reg Indicates the number of Anchor Locations, L cls represents the classification loss, L reg represents the regression loss, j represents the number of detection branches, and μ represents the weight balance parameter; Step 5: Use the network trained in step 4 to perform target detection. Input the three images of close distance, medium distance, and long distance segmented in step 2 into the detection branches at the corresponding distance scales respectively. Then scale the output results of the three detection branches to the scale before the resolution is increased and splice them as the final target detection results.

2. The object detection method based on disparity segmentation as claimed in claim 1, characterized in that: In step 2, the resolution of the medium-range image and the long-range image are increased by 2 times and 4 times, respectively.

3. The object detection method based on disparity segmentation as claimed in claim 1, characterized in that: In step 4, three detection branches are constructed based on the faster r-cnn or cascade r-cnn network.

4. The object detection method based on disparity segmentation according to claim 1 or 3, characterized in that: Use the kitti dataset for network training.

5. The object detection method based on disparity segmentation as claimed in claim 1, characterized in that: In the output result splicing of step 5, the overlapping detection boxes are processed by the non-maximum suppression method.

Images