Computer vision depth acquisition method

Abstract

The invention relates to the technical field of computer vision, and particularly provides a computer vision depth acquisition method, which comprises the following steps of: acquiring a left view and a right view of a measured object through a binocular camera, and acquiring a dense binocular depth map according to the left view and the right view; projecting a light beam emitted by a laser radar into the dense binocular depth map to obtain sparse radar depth projection points, marking pixel points corresponding to the projection points, performing super-pixel segmentation on the dense binocular depth map according to the marked pixel points to form a plurality of super-pixel areas, and performing super-pixel segmentation on the dense binocular depth map according to a sparse radar depth value corresponding to each projection point to form a plurality of super-pixel regions; and correcting the depth value of each pixel point except the projection point in the superpixel area to obtain the visual depth of the measured object. According to the method, the binocular camera and the laser radar are combined for obtaining the visual depth of the measured object, a dense depth map can be obtained, depth information correction can be carried out through the laser radar, and then more accurate depth information of the measured object is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and specifically provides a method for acquiring depth of computer vision. Background Art

[0002] At present, there are two main ways to obtain depth information in the field of computer vision: one is to actively obtain depth information through hardware devices, such as using laser radar to obtain depth information. The laser radar emits a laser beam and measures the time difference from the emission to the reception of the laser beam to calculate the distance. Since the speed of light is constant, the time difference can be directly converted into distance information. Obtaining depth information through laser radar can obtain high-precision measurement results, which is suitable for scenes that require accurate depth measurement. The laser radar is not affected by ambient light and can work normally at night or in dimly lit environments. However, since the emission and reception of the laser beam are discrete points and cannot fully cover the entire scene, only sparse depth maps can be obtained, and the depth value needs to be supplemented manually, resulting in its limited scope of application. In addition, the cost of high-quality laser radar equipment is high, which further limits its scope of application.

[0003] Another method is to take multiple pictures in the same environment and perform depth estimation based on the image features of the multiple pictures taken, such as the binocular vision method. The binocular vision method uses a binocular camera to take left and right views in the same environment, finds the corresponding matching points in the left and right views through a stereo matching algorithm, obtains the disparity value of the matching points, and then uses triangulation to calculate the depth information of the scene. This method can obtain a dense depth map that can cover the entire scene, and the binocular vision method can collect better edge information, which is helpful for subsequent image processing and recognition. However, the binocular vision method is easily affected by factors such as lighting changes, noise, and occlusion, resulting in fluctuations in the collected depth information. The depth information obtained is less accurate than the depth information obtained using a lidar, and the computational complexity of the stereo matching algorithm is high, requiring more computing resources. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a computer vision depth acquisition method, which combines a laser radar capable of acquiring high-precision depth information and a binocular camera capable of acquiring dense depth information. The laser radar is used to correct the depth information collected by the binocular camera, so as to obtain a depth map with high density, sharp edges and no fluctuation in depth information.

[0005] The computer vision depth acquisition method provided by the present invention comprises: S1: Obtain the left view and right view of the object under test through the binocular camera, and obtain the depth map based on the left view and the right view; S2: Project the light beam emitted by the laser radar into the depth map to obtain the projection point, and mark the pixel points corresponding to the projection point; S3: performing super-pixel segmentation on the depth map according to the marked pixels to form super-pixel regions; S4: According to the depth value corresponding to each projection point, the depth value of each pixel point except the projection point in the superpixel area is corrected to obtain the visual depth of the measured object.

[0006] Preferably, a disparity map is obtained based on the left view and the right view, and a depth map is obtained based on the disparity map.

[0007] Preferably, the method for obtaining the disparity map is: Extract feature points of the object under test in the left view and the right view, and obtain a left feature map corresponding to the left view and a right feature map corresponding to the right view based on the feature points; Preset parallax range , for the left feature map and the right feature map that meet the disparity range Calculate the matching cost of the pixel points; According to the matching cost, obtain the cost body; The cost volume is aggregated to obtain the disparity map.

[0008] Preferably, feature points of the measured object in the left view and the right view are extracted by a convolutional neural network.

[0009] Preferably, after the cost volume is cost aggregated, a disparity regression is performed using a soft argmin method to obtain a disparity map.

[0010] Preferably, the expression of the soft argmin method is: ; in, represents the estimated value of disparity, represents softmax calculation, Represents the estimated matching cost.

[0011] Preferably, a clustering algorithm is used for superpixel segmentation.

[0012] Preferably, the method for correcting the depth value of each pixel point except the projection point in the superpixel area is: Calculate the difference between the depth value corresponding to the projection point in the superpixel area and the depth values ​​of other pixel points excluding the projection point; Sum the differences and take the average , based on the average Correct the depth value of the object being measured.

[0013] Preferably, the expression for depth value correction is: ; in, The coordinates are The corrected depth value corresponding to the pixel point is The coordinates are The pixel point corresponds to the original depth value.

[0014] Compared with the prior art, the present invention can achieve the following beneficial effects: The present invention combines a laser radar with a binocular camera to obtain depth information of the object being measured. By correcting the depth information collected by the binocular camera through the laser radar, the error of the binocular camera in depth measurement can be eliminated, so that the depth map finally obtained is more accurate and reliable. By fusing the data of the laser radar and the binocular camera, the density of the depth information can be significantly enhanced.

[0015] By combining the binocular camera with the lidar, the edge information of the lidar can be used to optimize the depth map of the binocular camera, making the edges of the depth map sharper and clearer, thereby improving the edge sharpness of the depth map.

[0016] Because lidar has the characteristics of high precision and high stability, it can provide more stable depth information. By fusing the data of the two, the fluctuations in the depth measurement of the binocular camera can be eliminated, making the depth map smoother and more consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a method for acquiring depth through computer vision provided according to an embodiment of the present invention; Figure 2 is a schematic diagram of the relative position relationship between a binocular camera and a laser radar according to an embodiment of the present invention; Figure 3 is a schematic diagram of projection points provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the invention clearer, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the invention and do not constitute a limitation to the invention. Similar components in different embodiments use associated similar component numbers. In the following embodiments, many detailed descriptions are to enable the invention to be better understood. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other components, materials, and methods. In some cases, some operations related to the invention are not shown or described in the specification, in order to avoid the core part of the invention being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0019] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to form various implementation methods. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a necessary sequence, unless otherwise specified that a certain sequence must be followed.

[0020] In the description of the invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0021] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0022] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a computer vision depth acquisition method, which uses a binocular camera combined with a laser radar to acquire and correct the visual depth information of the object to be measured, so as to obtain accurate depth information of the object to be measured, as follows: S1: A binocular camera is a stereoscopic vision system composed of two cameras. The two cameras capture images of the same object under test from different perspectives to perceive the geometric structure of the object under test. The binocular camera captures images of the object under test from two different perspectives, which are referred to as the left view and the right view of the object under test in the embodiment of the present invention. Since the left view and the right view are obtained by shooting the same object under test, the feature points of the object under test contained in the left view and the right view are one-to-one corresponding, and the pixel points of each feature point in the left view and the pixel points in the right view should be able to be matched one-to-one. The feature points mentioned here are the corresponding pixel points of a certain point of the object under test itself in the left view or the right view. Usually, points on the object under test with obvious features and easy to distinguish or identify are selected as feature points of the object under test. These feature points are usually points with obvious changes or unique properties in the image, such as edges, corner points, spots, etc. For example, if the object under test is any rectangle, the vertices of each corner can be selected as feature points.

[0024] The pixel points corresponding to the feature points in the left view are extracted, and the pixel points corresponding to these feature points in the right view are extracted at the same time. In the prior art, the method of extracting feature points from an image is relatively mature. The embodiment of the present invention uses a convolutional neural network to extract feature points from the left view and the right view obtained by the binocular camera, and then obtains a left feature map corresponding to the left view and a right feature map corresponding to the right view.

[0025] Since the number of matching pixels in the left feature map and the right feature map is relatively large, a threshold is usually set according to the disparity range obtained by measuring the measured object, and some pixels are selected for matching cost calculation. As the screening threshold, the matching pixel points in the left and right feature maps that meet the disparity range are screened out, and the matching cost is further calculated.

[0026] Matching cost is a quantitative mathematical description used to measure the similarity between each matched pixel in the left and right feature maps. Specifically, matching cost calculation is the process of measuring the correlation or similarity between the pixel to be matched and the candidate pixel. Whether or not two pixels in the left and right feature maps are pixels of the same feature point, a matching cost can be calculated through the matching cost function. The smaller the calculated matching cost value, the greater the correlation between the two pixels and the greater the probability that they are the same feature point. On the contrary, the larger the calculated matching cost value, the greater the difference between the two pixels and the lower the similarity.

[0027] A binocular camera usually uses the left camera as the main camera and the right camera as the auxiliary camera. In the embodiment of the present invention, the left camera is also the main camera. The pixels in the left feature map that meet the disparity range are the pixels to be matched, and the pixels in the right feature map that meet the disparity range are the candidate pixels. The matching cost of each pixel in the left feature map that meets the disparity range is calculated. The matching cost can measure the corresponding matching pixels in the left feature map and the right feature map. Similarity between : ; in, Represents the position coordinates of the pixel point. represents the disparity value and , Represents the calculation function of the matching cost, Indicates that the position coordinates in the left feature map are The characteristics of the pixels, Indicates that in the right feature map, the disparity value is Calculate the similarity obtained by traversing the pixel points. The value of is the matching cost, and the similarity The smaller the value, the higher the similarity between the two corresponding pixels. The larger the value of , the lower the similarity between the two corresponding pixels. It should be noted that the matching cost is calculated based on the left feature map and the right feature map, but the result exists in the corresponding position of the left feature map.

[0028] For any disparity value After the matching cost is calculated, a corresponding matching cost map can be obtained. Multiple matching cost maps can be overlapped to obtain a matching cost volume. Obtaining a matching cost map through matching cost and then obtaining a matching cost volume is an existing technology in the field of computer vision.

[0029] During the matching process, the parallax range It determines the number of times each pixel is matched. Each match obtains the similarity under the disparity. If only the matching cost calculation is performed, the disparity value with the smallest similarity may be randomly selected as the true disparity of the pixel, resulting in more mismatched points. Therefore, cost aggregation is required. Cost aggregation aggregates the cost calculation results of points in an area in a certain way as the cost calculation result of the central pixel, and then obtains the disparity value based on this result, thereby reducing the generation of mismatched points. That is, the similarity (matching cost) mentioned above is subjected to cost aggregation to obtain a new, more stable and accurate matching cost.

[0030] Cost aggregation is performed on the cost volume obtained by the above process, that is, the sum or weighted average of the matching costs is calculated within the specified support domain of each pixel. The specified support domain here refers to the "one region" mentioned in the previous paragraph. The specified support domain is a pixel region centered on one pixel, for example The pixel area of ​​the specified support domain is related to the resolution of the binocular camera, the measurement range, and the distance between the left and right cameras in the binocular camera. The value of the specified support domain is determined by personnel in this field based on the actual situation of the device. It belongs to the existing technology, but the value is different in different situations, so it is impossible to limit its value. Cost aggregation can suppress the noise in the matching cost map, reduce the probability of matching errors, and thus improve the accuracy of the disparity value.

[0031] The cost volume after the cost aggregation process includes the matching cost of each pixel under different disparity values. To obtain the final disparity information, the embodiment of the present invention adopts the softargmin method to perform disparity regression to output a dense disparity map.

[0032] The expression of the Softargmin method is as follows: ; in, represents the estimated value of disparity, represents softmax calculation, It represents the estimated matching cost, that is, the matching cost calculated by the above process. It is called the estimated matching cost because the final matching point result is solved by the Softargmin method for the cost aggregation, and the accurate matching cost is obtained. Therefore, the matching cost calculated above is called the estimated matching cost.

[0033] The disparity map obtained through the above process can then be used to obtain a corresponding dense binocular depth map. Obtaining a depth map from a disparity map is a mature existing technology.

[0034] S2: Figure 2 As shown, in the embodiment of the present invention, the binocular camera and the laser radar are two independent individuals. Before implementing the method of the present invention, the binocular camera and the laser radar need to be fixed according to the predetermined position and angle. The predetermined position and angle mentioned here need to meet the following conditions: the field of view of the left camera and the right camera in the binocular camera can completely cover the object to be measured, and the object to be measured is completely within the radiation range of the laser radar. After the position and angle of the binocular camera and the laser radar are selected, the relative position of the two is fixed, that is, the binocular camera and the laser radar are relatively still, and the two form a fixed whole. The two can be moved as a whole, but it is necessary to ensure that there is no relative movement between the binocular camera and the laser radar.

[0035] In the embodiment of the present invention, the left camera is used as the main camera. Therefore, the left camera is used as the coordinate system of the binocular camera. The binocular camera and the laser radar are regarded as a whole. It is necessary to convert the laser radar coordinate system to the binocular camera coordinate system and unify the binocular camera and the laser radar into one coordinate system. Convert to the stereo camera coordinate system The transformation relationship is as follows: ; in, and Represents the relative position relationship between the binocular camera coordinate system and the laser radar coordinate system, including rotation and translation. express The rotation matrix of express The translation vector of .

[0036] like Figure 3 As shown, in the embodiment of the present invention, when the depth information is corrected by the laser radar, the laser beam of the laser radar needs to be projected onto the dense binocular depth map. Therefore, the three-dimensional point in the binocular camera coordinate system needs to be projected onto the binocular camera imaging plane, and then the three-dimensional point coordinates in the binocular camera coordinate system need to be converted into pixel coordinates in the image coordinate system. In the prior art, this process needs to be completed based on a binocular camera imaging model, which includes a binocular camera coordinate system, a binocular camera imaging plane, and a pixel coordinate system of the binocular camera. Figure 3 middle, represents the binocular camera coordinate system, Represents the coordinate point on the imaging plane of the binocular camera.

[0037] There are multiple projection points of the laser beam of the lidar on the dense binocular depth map, and the multiple projection points form a point cloud, that is, the point cloud contains multiple projection points. The projection points in the point cloud are marked at the corresponding pixel positions on the dense binocular depth map, and multiple marked pixel points are obtained accordingly.

[0038] S3: After obtaining the marked pixel points corresponding to all the projection points on the depth map, the depth map is segmented into superpixels. Image superpixel segmentation belongs to the mature existing technology. The embodiment of the present invention uses a clustering algorithm to segment the depth map into superpixels, which is as follows: The pixel point marked with the projection point on the depth map is used as the initial superpixel center. Represents the number of pixels marking the projection points on the depth map, that is, the number of initial superpixel centers. , The initial superpixel center is represented as , calculate the distance from each pixel in the depth map to each initial superpixel center, and assign each pixel to the area where the initial superpixel center is closest to it, forming An initial superpixel region.

[0039] For each initial superpixel area, the cluster center is calculated: ; in, represents the cluster center, The serial number of the cluster center. The calculated cluster center is used as the new superpixel center. Repeat the above pixel allocation method based on distance to allocate all pixels to the new superpixel center to form a new superpixel area. The cluster center is recalculated based on the new superpixel area. The above process is iterated until all superpixel centers no longer change. At this time, the final superpixel area is obtained. Each area contains multiple pixels. Based on these final superpixel areas, the depth information of the object under test is corrected.

[0040] S4: For each final superpixel region, calculate the difference between the depth value corresponding to the projection point in the superpixel region and the depth values ​​of other pixels excluding the projection point. ,in, Indicates the number of other pixels in the superpixel area, that is, the sequence number.

[0041] Sum all the above differences and take the average : ; in, Indicates the difference Any difference in the sequence number, .

[0042] Then the depth value of each pixel except the projection point in the final superpixel area is corrected: ; in, The coordinates are The corrected depth value corresponding to the pixel point is The coordinates are The pixel point corresponds to the original depth value, and the original depth value mentioned here is the depth value of the pixel point without correction.

[0043] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0044] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A computer vision depth acquisition method, characterized in that: include: S1: Obtain a left view and a right view of the object to be measured through a binocular camera, and obtain a depth map according to the left view and the right view; S2: Projecting the light beam emitted by the laser radar into the depth map to obtain a projection point, and marking the pixel point corresponding to the projection point; S3: performing super-pixel segmentation on the depth map according to the marked pixel points to form a super-pixel region; S4: According to the depth value corresponding to each projection point, the depth value of each pixel point in the superpixel area except the projection point is corrected to obtain the visual depth of the measured object.

2. The computer vision depth acquisition method according to claim 1, characterized in that: A disparity map is obtained according to the left view and the right view, and a depth map is obtained according to the disparity map.

3. The computer vision depth acquisition method according to claim 2, characterized in that: The method for obtaining the disparity map is: Extracting feature points of the object under test in the left view and the right view, and obtaining a left feature map corresponding to the left view and a right feature map corresponding to the right view according to the feature points; Preset parallax range , for the left feature map and the right feature map that meet the disparity range Calculate the matching cost of the pixel points; Obtaining a cost body according to the matching cost; Cost aggregation is performed on the cost volume to obtain the disparity map.

4. The computer vision depth acquisition method according to claim 3, characterized in that: The feature points of the object under test in the left view and the right view are extracted by a convolutional neural network.

5. The computer vision depth acquisition method according to claim 3, characterized in that: After cost aggregation is performed on the cost volume, disparity regression is performed using a soft argmin method to obtain the disparity map.

6. The computer vision depth acquisition method according to claim 5, characterized in that: The expression of the soft argmin method is: ； in, represents the estimated value of disparity, represents softmax calculation, Represents the estimated matching cost.

7. The computer vision depth acquisition method according to claim 1, characterized in that: Clustering algorithm is used for superpixel segmentation.

8. The computer vision depth acquisition method according to claim 1, characterized in that: The method for correcting the depth value of each pixel point except the projection point in the superpixel area is: Calculate the difference between the depth value corresponding to the projection point in the superpixel area and the depth values ​​of other pixel points excluding the projection point; The difference is summed and the average is taken , according to the average value Correct the depth value of the object being measured.

9. The computer vision depth acquisition method according to claim 8, characterized in that: The expression of the depth value correction is: ； in, The coordinates are The corrected depth value corresponding to the pixel point is The coordinates are The pixel point corresponds to the original depth value.

Images