Depth estimation method, device and equipment integrating laser radar and binocular camera

Abstract

The present invention provides a depth estimation method, device and equipment integrating laser radar and binocular camera, the method comprising: collecting scene depth information and image information through a radar camera, the radar camera comprising a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera; projecting the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map; performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; fusing the upsampled radar disparity map with the image information to obtain a target disparity map, which can effectively adapt to scenes where conditions such as outdoor lighting and object texture are difficult to control, and can effectively improve the accuracy of matching.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a depth estimation method, device and equipment integrating laser radar and binocular camera. Background Art

[0002] Many tasks require obtaining depth information of the environment, especially mobile robots and computer vision applications, such as 3D reconstruction, SLAM, autonomous driving, and robot path planning. Depth information measurement is a fundamental problem in 3D vision. In recent years, more and more scholars have begun to pay attention to this field and proposed several new methods. These methods are mainly divided into two categories: traditional methods and deep learning methods. Deep learning methods can also be divided into supervised methods and unsupervised methods. Supervised methods require a large amount of real depth data as ground truth. These ground truths require high-precision structured light sensors or lidars to obtain, which is very expensive. However, unsupervised methods do not require these ground truths. The basic idea is to use photometric errors to guide the training of left and right image neural networks.

[0003] But unsupervised methods also require a large number of high-quality stereo images to train parameters. On the other hand, there are some traditional techniques for achieving depth estimation, but they usually have some limitations. For example, RGB-D cameras can quickly obtain disparity maps, while the quality of disparity depends on the outdoor environment. Because sunlight affects the structured light sensor of the RGB-D camera. At the same time, light also has an impact on the stereo algorithm. For example, due to some reasons, part of the image may be overexposed. In addition, the stereo algorithm also has certain requirements for the scene in the picture. For example, the scene needs to have rich textures and suitable lighting conditions.

[0004] Therefore, for challenging outdoor scenes where conditions such as lighting and object texture are difficult to control, the matching accuracy of the binocular stereo matching method is relatively low. Summary of the invention

[0005] The present invention provides a depth estimation method, device and equipment integrating laser radar and binocular camera, so as to solve the defects of the binocular stereo matching method in the prior art in terms of dependence on target texture and illumination change, and effectively improve the matching accuracy.

[0006] The present invention provides a depth estimation method integrating laser radar and binocular camera, comprising:

[0007] Collect scene depth information and image information through a radar camera, wherein the radar camera includes a laser radar and a binocular camera;

[0008] Calibrate the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera;

[0009] According to the posture transformation matrix, projecting the scene depth information onto the image plane of the image information to obtain a radar disparity map;

[0010] Performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map;

[0011] The upsampled radar disparity map is fused with the image information to obtain a target disparity map.

[0012] According to a depth estimation method that integrates a laser radar and a binocular camera provided by the present invention, before performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map, the method further includes:

[0013] Abnormal projection points in the radar disparity map are identified and removed.

[0014] According to a depth estimation method for integrating a laser radar and a binocular camera provided by the present invention, the identifying of abnormal projection points in the radar disparity map includes:

[0015] Completing the scan line of the laser radar;

[0016] When the radar depths of the target scan line are greater than the first radar depth of the first scan line and the second radar depth of the second scan line adjacent to the scan line, it is determined that the target scan line is composed of abnormal projection points.

[0017] According to a depth estimation method for fusing a laser radar and a binocular camera provided by the present invention, the scene depth information and the image information are calibrated to obtain a pose transformation matrix of the laser radar and the binocular camera, including:

[0018] Calibrate a first spatial relative position relationship between a first camera and a second camera of the binocular camera according to the image information;

[0019] Calibrate a second spatial relative position relationship between the laser radar and the binocular camera according to the scene depth information and the image information;

[0020] According to the first spatial relative position relationship and the second spatial relative position relationship, a posture transformation matrix of the laser radar and the binocular camera is constructed.

[0021] According to a depth estimation method for fusing a laser radar and a binocular camera provided by the present invention, the upsampled radar disparity map is fused with the image information to obtain a target disparity map, including:

[0022] Determining tilt window model parameters according to the upsampled radar disparity map and the image information;

[0023] Determine an inclined plane according to the inclined window model parameters, and determine a matching cost determination rule for any pixel in the inclined plane;

[0024] According to the matching cost determination rule, a disparity map that minimizes the cost is determined as a target disparity map.

[0025] According to a depth estimation method for fusing a laser radar and a binocular camera provided by the present invention, the method of determining a disparity map that minimizes the cost as a target disparity map according to the matching cost determination rule includes:

[0026] Randomly initializing the inclined plane to obtain an initialized plane model;

[0027] Based on the initialized plane model, disparity propagation is performed to determine the disparity map corresponding to the plane with the minimum matching cost when the disparity propagation reaches a preset number of times as the target disparity map.

[0028] According to a depth estimation method that integrates a laser radar and a binocular camera provided by the present invention, after performing the parallax propagation, the method further includes:

[0029] Based on the predetermined maximum disparity and normal vector variation range, the plane with the minimum matching cost when the disparity propagation reaches a preset number of times is optimized.

[0030] The present invention also provides a depth estimation device integrating a laser radar and a binocular camera, comprising:

[0031] An acquisition module, used to acquire scene depth information and image information through a radar camera, wherein the radar camera includes a laser radar and a binocular camera;

[0032] A calibration module, used to calibrate the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera;

[0033] A projection module, used for projecting the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map;

[0034] An upsampling module, used for performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map;

[0035] A fusion module is used to fuse the upsampled radar disparity map with the image information to obtain a target disparity map.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the depth estimation method for fusing a laser radar and a binocular camera as described in any one of the above are implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the depth estimation method for fusing a laser radar and a binocular camera as described in any one of the above.

[0038] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned depth estimation methods for fusing a laser radar and a binocular camera.

[0039] The present invention provides a depth estimation method, device and equipment for fusing a laser radar and a binocular camera. The method comprises: collecting scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera; projecting the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map; performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; fusing the upsampled radar disparity map with the image information to obtain a target disparity map, which can effectively adapt to scenes where conditions such as outdoor lighting and object texture are difficult to control, can effectively improve the matching accuracy, and can quickly and robustly estimate depth values ​​in complex environments, and has strong practicality and engineering value. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a flowchart of a depth estimation method integrating a laser radar and a binocular camera provided in an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of the installation position relationship of the radar camera provided in an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of defining a radar scan line and points between the scan lines provided by an embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of a tilted window and a parallel window provided in an embodiment of the present invention;

[0045] Figure 5 It is a schematic diagram of the structure of a depth estimation device integrating a laser radar and a binocular camera provided in an embodiment of the present invention;

[0046] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Combine the following Figure 1-Figure 6 The present invention describes a depth estimation method, device and equipment that integrates laser radar and binocular camera.

[0049] Figure 1 It is a flowchart of a depth estimation method integrating a laser radar and a binocular camera provided in an embodiment of the present invention; Figure 2 is a schematic diagram of the installation position relationship of the radar camera provided in an embodiment of the present invention; Figure 3 is a schematic diagram of defining a radar scan line and points between the scan lines provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of a tilted window and a parallel window provided in an embodiment of the present invention.

[0050] like Figure 1 As shown, an embodiment of the present invention provides a depth estimation method integrating a laser radar and a binocular camera, comprising the following steps:

[0051] 101. The scene depth information and image information are collected through a radar camera, which includes a laser radar and a binocular camera.

[0052] Specifically, the radar camera refers to a laser radar and a binocular camera, wherein the laser radar can be a mechanical rotating multi-line laser radar, a solid-state multi-line laser radar, or a single-line laser radar that generates multiple scan lines through registration. Figure 2The figure shows the installation position relationship diagram of the laser radar and the binocular camera provided in this embodiment, LiDAR represents the laser radar, and Zed represents the binocular camera, wherein the relative position of the binocular camera and the laser radar can be changed, and the position change range is different according to the different viewing angles of the binocular camera, as long as the radar data can be projected to the required position. The specific method of collecting scene depth information can be that the laser radar emits a laser beam to the object to be measured, receives the reflected wave of the laser beam, and records the time difference to determine the distance between the object to be measured and the test point. The binocular camera is composed of two ordinary cameras, and uses optical principles and triangulation principles to obtain scene depth.

[0053] 102. Calibrate the scene depth information and image information to obtain the posture transformation matrix of the laser radar and the binocular camera.

[0054] Specifically, the scene depth information and image information are calibrated to obtain the pose transformation matrix of the laser radar and the binocular camera, which can be divided into two parts, one is the calibration of the binocular camera, and the other is the calibration of the radar camera. The calibration of the binocular camera can be based on the image information, calibrating the first spatial relative position relationship between the first camera and the second camera of the binocular camera, that is, obtaining the image information of the two cameras of the binocular camera, and then calculating the internal parameters of the binocular camera, the lens distortion parameters and the spatial relative position relationship of the two cameras through the calibration algorithm. The process of calibrating the radar camera can be based on the scene depth information and image information, calibrating the second spatial relative position relationship between the laser radar and the binocular camera, that is, according to the scene depth information and image information, obtaining the spatial relative position relationship between the laser radar and the left (right) camera, that is, the first camera and the second camera through the algorithm. Then, based on the first spatial relative position relationship and the second spatial relative position relationship, the pose transformation matrix of the laser radar and the binocular camera is constructed.

[0055] 103. According to the pose transformation matrix, the scene depth information is projected onto the image plane of the image information to obtain a radar disparity map.

[0056] Specifically, based on the posture transformation matrix, the radar point is projected onto the image plane of the camera's image information through the relative position relationship between the radar and the camera, the internal parameters of the camera, and the distortion parameters to obtain the radar disparity map.

[0057] 104. Perform bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map.

[0058] After obtaining the radar disparity map, bilinear interpolation is performed on the radar disparity map to fill the gaps between the scan lines to obtain an upsampled radar disparity map. The formula for bilinear interpolation is shown in formula (1):

[0059]

[0060] Where π(x,y) is the disparity value at the coordinate (x,y). down and up is the point on the radar scan line (x,y) in the y-axis direction. Figure 3 As shown, y down is the y-coordinate of the scan line below the point (x,y). up is the y-coordinate on the scan line above the point (x,y). If there is no radar scan line above or below a pixel, the pixel is assigned a global maximum.

[0061] 105. The upsampled radar disparity map is fused with the image information to obtain a target disparity map.

[0062] After upsampling to obtain the upsampled radar disparity map, the upsampled radar disparity map and the RGB information of the image are fused to obtain the optimal disparity map, that is, the target disparity map. The specific method is to determine the tilted window model parameters according to the upsampled radar disparity map and the image information; determine the tilted plane according to the tilted window model parameters, and the tilted window is composed of the pixel coordinates (x, y), the disparity d and the normal vector n = (n x ,n y ,n z ) Determine the unique inclined plane If the normal vector n is 0, the tilted window degenerates into a parallel window; the plane can be expressed as formula (2)

[0063]

[0064] Among them, (x, y, d) is a point on the parallax plane, and are the three parameters of the inclined plane; these three parameters are determined by the following formulas (3)(4)(5):

[0065]

[0066]

[0067]

[0068] like Figure 4 As shown, it is a comparison between the tilted window and the parallel window. Figure 4 It can be clearly seen that the slanted window has a better adaptability to the inclined plane. Among them, slanted support window refers to the slanted window, and front-parallel window refers to the parallel window.

[0069] After all the parameters of the inclined planes are determined, the matching cost determination rule of any pixel in the inclined plane is determined. The matching cost of pixel p in the plane can be defined as shown in formula (6):

[0070]

[0071] in, is the plane where pixel p is located, w p is a square window centered at pixel p, and (p′, d′) is the pixel and disparity within the window of pixel p.

[0072] The formula ω(p,p′) can be expressed as formula (7):

[0073]

[0074] Among them, ‖I p -I p′ ‖ is the L1 distance between pixel p and pixel p′ in RGB space, γ is a custom parameter. In this algorithm, the value of γ is set to 10;

[0075] formula It is expressed as formula (8):

[0076] ρ(p′,q)=α·min(‖I p′ -I q ‖,τ col )+β·min(|π p′ -d′|,τ disp ) (8)

[0077] Among them, q represents Indicates the same point in another view; min() means to get the minimum value between two values; α and β are the proportions of color and disparity, and their sum is 1; τ col and τ disp are the truncation costs of color and disparity, respectively. When there is occlusion, the truncation cost prevents the matching cost from being too large. p′ -I q ‖ represents the L1 distance between pixel p′ and pixel q in RGB space; |π p′ -d′| represents the disparity of pixel p′ in the upsampled disparity map and the disparity of pixel p′ in the plane The absolute difference in parallax.

[0078] Then, the matching cost determination rule can be used to determine the disparity map that minimizes the cost as the target disparity map.

[0079] According to the matching cost determination rule, the disparity map with the minimum cost is determined as the target disparity map, including: randomly initializing the inclined plane to obtain an initialized plane model, that is, after the inclined plane model is determined, the model parameters x, y, d, n x ,n y ,n z needs to be determined; if the pixel p′(x, y) has a disparity d′ in the upsampled disparity map, assign the value of d′ to d; if there is no such value, assign the value between the maximum and minimum disparity to d; and select a random unit normal vector and assign it to n x ,n y ,n z .

[0080] Then, based on the initialized plane model, parallax propagation is performed. Each pixel is assigned a plane, and then the plane with the lowest matching cost on each pixel is found. Adjacent pixels may have similar planes. By expanding the planes of adjacent pixels, a plane with a lower matching cost can be found. There are many ways to propagate, such as from left to right, from top to bottom, from top left to bottom right, etc. The propagation method used in the present invention can be expressed as follows: in the odd-numbered propagation process, the propagation direction propagates from the upper left to the lower right, and in the even-numbered propagation process, the propagation direction propagates from the lower right to the upper left. And in the odd-numbered process, if but In the even number of processes, pixel p needs to be up and p left Compare; if the specified number of times has been completed, the whole process ends, that is, the disparity map corresponding to the plane with the minimum matching cost when the disparity propagation reaches the preset number of times is determined as the target disparity map.

[0081] In order to ensure the optimization of the disparity map, after the disparity propagation, it also includes: based on the predetermined maximum disparity and normal vector change range, optimizing the plane with the lowest matching cost when the disparity propagation reaches the preset number of times. Specifically, after the disparity propagation process, in order to further reduce the matching cost, the plane needs to be optimized; the optimized plane is located between the two propagation processes, and a new plane with lower aggregation cost is provided for the propagation process as much as possible; in this process, the plane needs to use a point (x, y, d) and a normal vector (n x ,n y ,n z ) means; at the same time and is defined as the maximum range of parallax and normal vector variation; if a pixel is between two radar scan lines, then If not, then is the maximum disparity of the image; similarly, is also the maximum allowed range of a vector; from Randomly select Δ d , calculate d′=d+Δ d ; Also in A random value in the range, and It can be expressed as The function of unit() is unitization; now a new plane All parameters of are determined if Then the plane Used to replace This is an iterative process. In the first calculation, is set to and In each iteration, Updated to and if Optimize plane It means it has been completed; then continue the disparity propagation process to obtain the final target disparity map.

[0082] The present invention provides a depth estimation method for fusing a laser radar and a binocular camera, comprising: collecting scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera; projecting the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map; performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; fusing the upsampled radar disparity map with the image information to obtain a target disparity map, which can effectively adapt to scenes where conditions such as outdoor lighting and object texture are difficult to control, can effectively improve the accuracy of matching, and can quickly and robustly estimate depth values ​​in complex environments, has strong practicality and engineering value, can also overcome the disadvantages of traditional methods being sensitive to noise and deep learning methods being relatively dependent on data sets, and can be applicable to large scenes such as outdoor scenes.

[0083] Furthermore, in this embodiment, before bilinear interpolation is performed on the radar disparity map to obtain the upsampled radar disparity map, it can also include: identifying abnormal projection points in the radar disparity map and clearing the abnormal projection points. In the radar projection process, due to the relative position relationship between the camera and the radar, erroneous projection points may be caused, so an outlier clearing module is required to clear them. Among them, the identification of abnormal projection points in the radar disparity map can be specifically that since the radar points on the radar scan line are discontinuous, the scan line of the laser radar is first completed; when the radar depth of the target scan line is greater than the first radar depth of the first scan line adjacent to the scan line and the second radar depth of the second scan line, it is determined that the target scan line is composed of abnormal projection points, that is, when a radar scan line with a relatively large depth value appears between two scan lines with shallow radar depths, it is considered that this scan line is composed of abnormal projection points, and this scan line is cleared, so that the abnormal values ​​in the radar projected disparity map can be effectively removed.

[0084] Based on the same general inventive concept, the present application also protects a depth estimation device that integrates a laser radar and a binocular camera. The depth estimation device that integrates a laser radar and a binocular camera provided by the present invention is described below. The depth estimation device that integrates a laser radar and a binocular camera described below and the depth estimation method that integrates a laser radar and a binocular camera described above can be referenced to each other.

[0085] Figure 5 It is a schematic diagram of the structure of a depth estimation device integrating a laser radar and a binocular camera provided in an embodiment of the present invention.

[0086] like Figure 5 As shown, an embodiment of the present invention provides a depth estimation device integrating a laser radar and a binocular camera, comprising:

[0087] An acquisition module 51 is used to acquire scene depth information and image information through a radar camera, wherein the radar camera includes a laser radar and a binocular camera;

[0088] A calibration module 52, used to calibrate the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera;

[0089] A projection module 53, configured to project the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map;

[0090] An upsampling module 54, configured to perform bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map;

[0091] The fusion module 55 is used to fuse the upsampled radar disparity map with the image information to obtain a target disparity map.

[0092] An embodiment of the present invention provides a depth estimation device that integrates a laser radar and a binocular camera, comprising: collecting scene depth information and image information through a radar camera, the radar camera comprising a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera; projecting the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map; performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; and fusing the upsampled radar disparity map with the image information to obtain a target disparity map. The device can effectively adapt to scenes where conditions such as outdoor lighting and object texture are difficult to control, can effectively improve the matching accuracy, and can quickly and robustly estimate depth values ​​in complex environments, and has strong practicality and engineering value.

[0093] Furthermore, this embodiment also includes an exception handling module, which is used to:

[0094] Abnormal projection points in the radar disparity map are identified and removed.

[0095] Furthermore, this embodiment also includes an exception handling module, which is specifically used to:

[0096] Completing the scan line of the laser radar;

[0097] When the radar depths of the target scan line are greater than the first radar depth of the first scan line and the second radar depth of the second scan line adjacent to the scan line, it is determined that the target scan line is composed of abnormal projection points.

[0098] Furthermore, the calibration module 52 in this embodiment is specifically used for:

[0099] Calibrate a first spatial relative position relationship between a first camera and a second camera of the binocular camera according to the image information;

[0100] Calibrate a second spatial relative position relationship between the laser radar and the binocular camera according to the scene depth information and the image information;

[0101] According to the first spatial relative position relationship and the second spatial relative position relationship, a posture transformation matrix of the laser radar and the binocular camera is constructed.

[0102] Furthermore, the fusion module 55 in this embodiment is specifically used for:

[0103] Determining tilt window model parameters according to the upsampled radar disparity map and the image information;

[0104] Determine an inclined plane according to the inclined window model parameters, and determine a matching cost determination rule for any pixel in the inclined plane;

[0105] According to the matching cost determination rule, a disparity map that minimizes the cost is determined as a target disparity map.

[0106] Furthermore, the fusion module 55 in this embodiment is also specifically used for:

[0107] Randomly initializing the inclined plane to obtain an initialized plane model;

[0108] Based on the initialized plane model, disparity propagation is performed to determine the disparity map corresponding to the plane with the minimum matching cost when the disparity propagation reaches a preset number of times as the target disparity map.

[0109] Furthermore, the fusion module 55 in this embodiment is also specifically used for:

[0110] Based on the predetermined maximum disparity and normal vector variation range, the plane with the minimum matching cost when the disparity propagation reaches a preset number of times is optimized.

[0111] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0112] like Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the depth estimation method of fusing the laser radar and the binocular camera, the method comprising: collecting scene depth information and image information through a radar camera, the radar camera comprising a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain the posture transformation matrix of the laser radar and the binocular camera; projecting the scene depth information onto the image plane of the image information according to the posture transformation matrix to obtain a radar disparity map; performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; fusing the upsampled radar disparity map with the image information to obtain a target disparity map.

[0113] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the depth estimation method of fusing a laser radar and a binocular camera provided by the above methods, and the method includes: collecting scene depth information and image information through a radar camera, and the radar camera includes a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera; according to the posture transformation matrix, projecting the scene depth information onto an image plane of the image information to obtain a radar disparity map; performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; fusing the upsampled radar disparity map with the image information to obtain a target disparity map.

[0115] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the depth estimation method for fusing a laser radar and a binocular camera provided by the above-mentioned methods, the method comprising: collecting scene depth information and image information through a radar camera, the radar camera comprising a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera; projecting the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map; performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; fusing the upsampled radar disparity map with the image information to obtain a target disparity map.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A depth estimation method integrating laser radar and binocular camera, characterized in that: include: Collect scene depth information and image information through a radar camera, wherein the radar camera includes a laser radar and a binocular camera; Calibrate the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; According to the posture transformation matrix, projecting the scene depth information onto the image plane of the image information to obtain a radar disparity map; Performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; Fusing the upsampled radar disparity map with the image information to obtain a target disparity map; The step of fusing the upsampled radar disparity map with the image information to obtain a target disparity map includes: Determining tilt window model parameters according to the upsampled radar disparity map and the image information; Determine an inclined plane according to the inclined window model parameters, and determine a matching cost determination rule for any pixel in the inclined plane; According to the matching cost determination rule, the disparity map that minimizes the cost is determined as the target disparity map; the expression of the inclined plane is: in, represents the tilted plane; (x, y, d) is a point on the parallax plane; and are three tilted window model parameters of the tilted plane; the tilted window model parameters are respectively expressed as: Among them, (n x ,n y ,n z ) represents the normal vector of a point (x, y, d) on the parallax plane; The expression of the matching cost determination rule is: in, represents the matching cost determination rule of pixel p; is the plane where pixel p is located, w p is a square window centered at pixel p, (p ′ ,d ′ ) is the pixel sum disparity within the pixel p window; ω(p,p ′ ) is expressed as: Among them, ‖I p -I p′ ‖ is the L1 distance between pixel p and pixel p′ in RGB space, and γ is a custom parameter; It is expressed as: p(p ′ ,q)=α·min(‖I p′ -I q ‖,t col )+β·min(|π p′ -d ′ |,t disp ) Among them, q represents Indicates the same point in another view; min() means to get the minimum value between two values; α and β are the proportions of color and disparity, and the sum of α and β is 1; τ col and τ disp are the truncation costs of color and disparity, respectively; ‖I p′ -I q ‖ represents the L1 distance between pixel p′ and pixel q in RGB space; |π p′ -d ′ | represents the disparity of pixel p′ in the upsampled radar disparity map and the disparity of pixel p′ in the plane The absolute difference in median parallax; The step of determining the disparity map that minimizes the cost as the target disparity map according to the matching cost determination rule includes: Randomly initializing the inclined plane to obtain an initialized plane model; Based on the initialized plane model, disparity propagation is performed to determine the disparity map corresponding to the plane with the minimum matching cost when the disparity propagation reaches a preset number of times as the target disparity map; the disparity propagation includes: unfolding the plane of adjacent pixels, and determining the plane with the minimum matching cost among the planes of multiple adjacent pixels.

2. The depth estimation method of fusing laser radar and binocular camera according to claim 1, characterized in that: Before performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map, the method further includes: Abnormal projection points in the radar disparity map are identified and removed.

3. The depth estimation method of fusing laser radar and binocular camera according to claim 2, characterized in that: The identifying of abnormal projection points in the radar disparity map comprises: Completing the scan line of the laser radar; When the radar depths of the target scan line are greater than the first radar depth of the first scan line and the second radar depth of the second scan line adjacent to the scan line, it is determined that the target scan line is composed of abnormal projection points.

4. The depth estimation method of fusing laser radar and binocular camera according to claim 1, characterized in that: The step of calibrating the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera includes: Calibrate a first spatial relative position relationship between a first camera and a second camera of the binocular camera according to the image information; Calibrate a second spatial relative position relationship between the laser radar and the binocular camera according to the scene depth information and the image information; According to the first spatial relative position relationship and the second spatial relative position relationship, a posture transformation matrix of the laser radar and the binocular camera is constructed.

5. The depth estimation method of fusing laser radar and binocular camera according to claim 1, characterized in that: After the parallax propagation is performed, the method further includes: Based on the predetermined maximum disparity and normal vector variation range, the plane with the minimum matching cost when the disparity propagation reaches a preset number of times is optimized.

6. A depth estimation device integrating laser radar and binocular camera, characterized in that: include: An acquisition module, used to acquire scene depth information and image information through a radar camera, wherein the radar camera includes a laser radar and a binocular camera; A calibration module, used to calibrate the scene depth information and the image information to obtain a posture transformation matrix of the laser radar and the binocular camera; A projection module, used for projecting the scene depth information onto an image plane of the image information according to the posture transformation matrix to obtain a radar disparity map; An upsampling module, used for performing bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map; A fusion module, used for fusing the upsampled radar disparity map with the image information to obtain a target disparity map; The step of fusing the upsampled radar disparity map with the image information to obtain a target disparity map includes: Determining tilt window model parameters according to the upsampled radar disparity map and the image information; Determine an inclined plane according to the inclined window model parameters, and determine a matching cost determination rule for any pixel in the inclined plane; According to the matching cost determination rule, the disparity map that minimizes the cost is determined as the target disparity map; the expression of the inclined plane is: in, represents the tilted plane; (x, y, d) is a point on the parallax plane; and are three tilted window model parameters of the tilted plane; the tilted window model parameters are respectively expressed as: Among them, (n x ,n y ,n z ) represents the normal vector of a point (x, y, d) on the parallax plane; The expression of the matching cost determination rule is: in, represents the matching cost determination rule of pixel p; is the plane where pixel p is located, w p is a square window centered at pixel p, (p ′ ,d ′ ) is the pixel sum disparity within the pixel p window; ω(p,p ′ ) is expressed as: Among them, ‖I p -I p′ ‖ is the L1 distance between pixel p and pixel p′ in RGB space, and γ is a custom parameter; It is expressed as: p(p ′ ,q)=α·min(‖I p′ -I q ‖,t col )+β·min(|π p′ -d ′ |,t disp ) Among them, q represents Indicates the same point in another view; min() means to get the minimum value between two values; α and β are the proportions of color and disparity, and the sum of α and β is 1; τ col and τ disp are the truncation costs of color and disparity, respectively; ‖I p′ -I q ‖ represents the L1 distance between pixel p′ and pixel q in RGB space; |π p′ -d ′ | represents the disparity of pixel p′ in the upsampled radar disparity map and the disparity of pixel p′ in the plane The absolute difference in median parallax; The step of determining the disparity map that minimizes the cost as the target disparity map according to the matching cost determination rule includes: Randomly initializing the inclined plane to obtain an initialized plane model; Based on the initialized plane model, disparity propagation is performed to determine the disparity map corresponding to the plane with the minimum matching cost when the disparity propagation reaches a preset number of times as the target disparity map; the disparity propagation includes: unfolding the plane of adjacent pixels, and determining the plane with the minimum matching cost among the planes of multiple adjacent pixels.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the depth estimation method of fusing the laser radar and the binocular camera as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the depth estimation method for fusing a laser radar and a binocular camera as described in any one of claims 1 to 5 are implemented.

Images