Sparse radar depth completion method based on Boolean mask constraint

Abstract

The invention relates to the technical field of sparse radar depth completion, in particular to a sparse radar depth completion method based on Boolean mask constraint. The method comprises the following steps: generating a plurality of Boolean masks, processing the Boolean masks, and superposing the processed areas into a depth map; performing depth reversal operation to obtain a reversed depth value; performing multi-scale convolution kernel expansion operation to obtain an expanded depth map; performing mask-constrained hole closing operation to obtain a depth map after hole closing; performing depth value upward extension operation to obtain an extended depth map; performing mask-constrained hole filling operation to obtain a depth map after hole filling; performing median filtering and Gaussian filtering operation on the depth map after hole filling to obtain a filtered depth map; and performing depth map reversal operation on the filtered depth map to obtain a final depth map. The method can effectively protect the deep fracture area, and improves the performance of the depth completion algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of sparse radar depth completion, and in particular to a sparse radar depth completion method based on Boolean mask constraints. Background Art

[0002] At present, deep completion algorithms include traditional algorithms and algorithms based on deep learning. Traditional methods are easy to transplant and more universal, but they require careful design of the methods and steps of algorithm processing data; learning-based methods are data-driven and require less human design, but they require high computing power of the computing platform and have worse portability and divergence.

[0003] In actual scenes, since the positions of objects in the scene are discontinuous, the depth values ​​at the edges of objects will jump, but most end-to-end depth expansion algorithms will process the actual depth jump area smoothly, which will affect the effect of downstream algorithms. How to protect the depth jump area in the depth completion task is inspiring. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a sparse radar depth completion method based on Boolean mask constraints, and the dense depth map finally outputted can effectively protect the depth jump area.

[0005] To achieve the above-mentioned object and other related objects, the present invention provides a sparse radar depth completion method based on Boolean mask constraints, comprising:

[0006] S1. Perform panorama segmentation on the RGB image to generate multiple Boolean masks, process the area defined by each Boolean mask respectively, and superimpose the processed areas into a depth map;

[0007] S2. performing a depth inversion operation on the effective depth value of the depth map to obtain an inverted depth value;

[0008] S3, performing a multi-scale convolution kernel dilation operation on the effective depth value and the reversed depth value to obtain an expanded depth map;

[0009] S4, performing a mask-constrained hole closing operation on the expanded depth map to obtain a hole-closed depth map;

[0010] S5, performing an operation of extending the depth value upward on the depth map after the hole is closed to obtain an extended depth map;

[0011] S6, performing a mask-constrained hole filling operation on the extended depth map to obtain a hole-filled depth map;

[0012] S7, performing median filtering and Gaussian filtering operations on the hole-filled depth map to obtain a filtered depth map;

[0013] S8. Perform a depth map inversion operation on the filtered depth map to obtain a final depth map.

[0014] In one embodiment of the present invention, performing panoptic segmentation on the RGB image to generate a plurality of Boolean masks in step S1 includes:

[0015] The image panoptic segmentation algorithm includes Panoptic SegFormer and Point2Mask to generate multiple Boolean masks, which satisfy the following formula:

[0016]

[0017] Where n is the total number of objects in the RGB image, Mask i is the Boolean mask corresponding to object i, J W×L It is a W×L all-1 matrix, W is the width of the RGB image, and L is the length of the RGB image.

[0018] In one embodiment of the present invention, performing a depth reversal operation on the effective depth value of the depth map in step S2 to obtain a reversed depth value includes:

[0019] The formula for performing a depth inversion operation on the effective depth value of the depth map is:

[0020] I inverse =100-I input ,

[0021] Among them, I inverse is the depth value after inversion, I input is the effective depth value of the depth map.

[0022] In one embodiment of the present invention, performing a multi-scale convolution kernel dilation operation on the effective depth value and the inverted depth value in step S3 to obtain an expanded depth map includes:

[0023] S31, the effective depth value I input According to 0~30m, 30~60m, and 60~80m, it is divided into short-distance depth value, medium-distance depth value, and long-distance depth value, corresponding to three convolution kernels of 7×7, 5×5, and 3×3 respectively;

[0024] S32, under the Boolean mask constraint, the inverted depth value I inverse A multi-scale convolution kernel dilation operation is performed to obtain a dilated depth map.

[0025] In one embodiment of the present invention, in step S32, under the Boolean mask constraint, the inverted depth value I inverse Perform a multi-scale convolution kernel dilation operation to obtain the expanded depth map including:

[0026] Under the Boolean mask constraint, the inverted depth value I inverse The formula for multi-scale convolution kernel expansion operation is:

[0027]

[0028] Among them, I inverse,j ,j=near,med,far are depth maps that only retain the near-distance depth value, the medium-distance depth value, and the far-distance depth value, respectively. Mask i is the Boolean mask corresponding to object i, This is a morphological dilation operation, done with OpenCV, Cross j is the convolution kernel corresponding to the j-th depth interval, I dilated is the depth map after dilation.

[0029] In one embodiment of the present invention, performing a mask-constrained hole closing operation on the expanded depth map in step S4 to obtain a hole-closed depth map includes:

[0030] Using a 5×5 full convolution kernel to perform a hole closing operation on the expanded depth map;

[0031] Under the Boolean mask constraint, the formula for hole closing operation is:

[0032]

[0033] Among them, Mask i is the Boolean mask corresponding to object i, J 5×5 is a 5×5 matrix of all 1s, I closed It is the depth map after the hole is closed.

[0034] In one embodiment of the present invention, the step S5 of performing an operation of extending the depth value upward on the depth map after the hole is closed to obtain the extended depth map includes:

[0035] The depth map after closing the hole I closed The area before the first valid value in each column from top to bottom is filled, and the filled value is the first valid value in the column;

[0036] The formula for extending the depth value upward is:

[0037] I extend =F extend (Iclosed ),

[0038] Among them, F extend The depth value extends upward, I extend is the extended depth map.

[0039] In one embodiment of the present invention, performing a mask-constrained hole filling operation on the extended depth map in step S6 to obtain a hole-filled depth map includes:

[0040] Use 9×9 full convolution kernel to extend the depth map I extend Perform morphological dilation operation on the extended depth map I extend Fill the holes whose size is smaller than the set threshold;

[0041] Under the Boolean mask constraint, the hole filling operation formula is:

[0042]

[0043] Among them, Mask i is the Boolean mask corresponding to object i, J 9×9 is a 9×9 matrix of all 1s, I fill This is the depth map after hole filling.

[0044] In one embodiment of the present invention, performing median filtering and Gaussian filtering operations on the hole-filled depth map in step S7 to obtain a filtered depth map includes:

[0045] Use 5×5 median filtering to remove the hole-filled depth map I fill The outer points in the image are then filtered using a 5×5 Gaussian filter to smooth the plane area and make the edges of the object clear.

[0046] The median filter and Gaussian filter operation formula is:

[0047] I blur =F MedGauss (I output ),

[0048] Among them, F MedGauss is the median Gaussian combination filter, I blur is the depth map after filtering.

[0049] In one embodiment of the present invention, performing a depth map inversion operation on the filtered depth map in step S8 to obtain a final depth map includes:

[0050] The depth map inversion operation formula is:

[0051] I output =100-Iblur ,

[0052] Among them, I output is the final depth map.

[0053] As described above, the sparse radar depth completion method based on Boolean mask constraint of the present invention has the following beneficial effects:

[0054] (1) The sparse LIDAR depth completion method based on Boolean mask constraints of the present invention enables RGB image information to provide guidance for the depth completion task, and relies on basic image processing operations to execute sparse LIDAR depth data to complete the depth completion task. The present invention can effectively protect the depth fracture area and improve the performance of the depth completion algorithm.

[0055] (2) The sparse radar depth completion method based on Boolean mask constraint of the present invention labels each pixel in the RGB image based on panoptic segmentation, generates a Boolean mask according to the labeling result, and performs depth completion based on morphological filtering on the sparse depth map under the constraint of the Boolean mask. The present invention ensures the versatility of the algorithm and the structural unification of the RGB image and the depth map.

[0056] (3) The sparse radar depth completion method based on Boolean mask constraints of the present invention can effectively protect the depth break area in the depth map, and can effectively improve the performance of the depth completion algorithm, providing a new idea for improving the depth completion algorithm, and can be integrated with other existing depth completion algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a sparse radar depth completion method based on Boolean mask constraints provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0059] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner, and thus the illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0060] Terms such as first or second can be used to describe various components, but these components are not limited by the above terms. The above terms are used to distinguish one component from another component, for example, without departing from the scope of the concept according to the present disclosure, a first component can be referred to as a second component, and similarly, a second component can be referred to as a first component.

[0061] In addition, "connected / coupled" means that one component is directly electrically coupled to another component or is indirectly electrically coupled through another component. The singular form may include the plural form unless explicitly stated in the sentence. In addition, "include / comprise" or "includes / comprises" used in this specification indicates that one or more components, steps, operations, and elements exist or have been added. The specific structural or functional descriptions of the examples of the implementation of the concepts disclosed in this specification are only illustrated to describe the examples of the implementation of the concepts, and the examples of the implementation of the concepts can be implemented in various forms, but these descriptions are not limited to the examples of the implementation described in this specification.

[0062] According to the concept, various modifications and changes can be applied to the examples of the embodiments, so that the examples of the embodiments will be illustrated in the drawings and described in the specification. However, the examples of the embodiments according to the concept are not limited to the specific embodiments, but include all changes, equivalents or replacements included in the spirit and technical scope of the present disclosure.

[0063] It should be understood that when an element is described as being "coupled" or "connected" to another element, the element may be directly coupled or directly connected to the other element, or may be coupled or connected to the other element through a third element. Conversely, it should be understood that when an element is referred to as being "directly coupled to" or "directly coupled to" another element, no other element is placed between them. Other expressions describing the relationship between components (i.e., "between" and "directly between" or "adjacent to" and "directly adjacent to") need to be interpreted in the same manner.

[0064] The terms used in this specification are only used to describe specific examples of the embodiments and are not intended to limit the present disclosure. If there is no clear opposite meaning in the context, the singular form may include the plural form. In this specification, it should be understood that the term "including" or "having" indicates the presence of the features, quantities, steps, operations, components, parts or combinations thereof described in the specification, but the possibility of the presence or addition of one or more other features, quantities, steps, operations, components, parts or combinations thereof cannot be precluded.

[0065] If there is no contrary definition, all terms (including technical terms or scientific terms) used herein have the same meaning as those generally understood by ordinary technicians in the field. If the terms defined in the commonly used dictionary are not clearly defined in this specification, they should be interpreted as having the same meaning as in the context of the relevant technology, and not interpreted as ideal or overly formal meanings.

[0066] Descriptions of well-known components and processing techniques may be omitted so as not to unnecessarily obscure the embodiments of the disclosure.

[0067] Throughout the specification, the same reference numerals refer to the same elements. Thus, even if a reference numeral is not mentioned or described with reference to one figure, it may be mentioned or described with reference to another figure. Furthermore, even if a reference numeral is not shown in one figure, it may be mentioned or described with reference to another figure.

[0068] In addition, the logic level of the signal may be different or opposite to the described logic level. For example, a signal described as having a logic "high" level may alternatively have a logic "low" level, and a signal described as having a logic "low" level may alternatively have a logic "high" level.

[0069] The following will describe the various embodiments of the present disclosure in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present disclosure, many technical details are provided in order to enable the reader to better understand the present disclosure. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present disclosure can be implemented.

[0070] See also Figure 1 , Figure 1 A flow chart of a sparse radar depth completion method based on Boolean mask constraints provided in an embodiment of the present application. The present invention provides a sparse radar depth completion method based on Boolean mask constraints. The sparse radar depth completion method based on Boolean mask constraints of the present invention includes two core parts. The first is to use a panoramic segmentation algorithm to assign each pixel in an RGB image to a specific category or object to achieve pixel-level image segmentation, and generate a Boolean mask based on the segmentation result; the second is to use OpenCV's morphological filtering to process the sparse depth map, and use convolution operations to fill in invalid depth values. Based on these two core parts, the present invention implements depth completion through the following 8 steps, specifically including:

[0071] Step S1, performing panoramic segmentation on the RGB image to generate multiple Boolean masks, processing each area defined by the Boolean mask respectively, and superimposing the processed areas into a depth map.

[0072] Specifically, performing panorama segmentation on the RGB image in step S1 to generate multiple Boolean masks includes:

[0073] The image panoptic segmentation algorithm includes Panoptic SegFormer and Point2Mask to generate multiple Boolean masks, which satisfy the following formula:

[0074]

[0075] Where n is the total number of objects in the RGB image, Mask i is the Boolean mask corresponding to object i, J W×L It is a W×L all-1 matrix, W is the width of the RGB image, and L is the length of the RGB image.

[0076] In one embodiment of the present invention, in step S1, a panoramic segmentation is performed on the input RGB image, and each pixel of the image is assigned to a specific object. The image panoramic segmentation algorithm used in the present invention includes but is not limited to PanopticSegFormer, Point2Mask, etc. The image panoramic segmentation algorithm outputs a number of Boolean masks, and the number of masks is related to the number of objects recognized by the algorithm.

[0077] Specifically, the mask constraint, i.e., the generated Boolean mask, can limit the morphological filtering operation to only a specific pixel range when operating the depth value, thereby preventing the mutual coverage of depth values ​​between different individuals, thereby achieving the effect of protecting the depth fracture area.

[0078] The mask constraints in subsequent operations are mainly reflected in performing a specific processing (such as multi-scale convolution expansion, hole closure, etc.) on each area defined by the Boolean mask, and then superimposing these areas into a depth map and outputting it.

[0079] Step S2: performing a depth inversion operation on the effective depth value of the depth map to obtain an inverted depth value.

[0080] Specifically, performing a depth reversal operation on the effective depth value of the depth map in step S2 to obtain a reversed depth value includes:

[0081] The formula for performing a depth inversion operation on the effective depth value of the depth map is:

[0082] I inverse =100-I input ,

[0083] Among them, I inverse is the depth value after inversion, I input is the effective depth value of the depth map.

[0084] In one embodiment of the present invention, step S2 is to perform a depth inversion operation on all valid depth values ​​of the input depth map. The reasons for performing the inversion operation are as follows:

[0085] 1) The core idea of ​​depth expansion is to use larger pixel values ​​to replace smaller pixel values. For the depth map, if the depth expansion is performed directly, the farther pixel values ​​will replace the closer pixel values, resulting in the loss of the depth value of the edge of the nearby object;

[0086] 2) Usually, in the spatial point cloud data collected by LiDAR, the smaller depth value is close to 0m, and the invalid pixel value is also 0m. If we do not try to distinguish them, the two will be confused during depth completion, resulting in inaccurate output results.

[0087] Step S3: performing a multi-scale convolution kernel dilation operation on the effective depth value and the inverted depth value to obtain a dilated depth map.

[0088] Specifically, in step S3, performing a multi-scale convolution kernel dilation operation on the effective depth value and the inverted depth value to obtain an expanded depth map includes:

[0089] Step S31: input The depth values ​​are divided into short-distance depth value, medium-distance depth value and long-distance depth value according to 0-30m, 30-60m and 60-80m, corresponding to the three convolution kernels of 7×7, 5×5 and 3×3 respectively.

[0090] In one embodiment of the present invention, the multi-scale convolution kernel dilation converts the effective depth I in the input depth map into input The depth values ​​are divided into short-distance depth value, medium-distance depth value and long-distance depth value according to 0-30m, 30-60m and 60-80m, corresponding to three convolution kernels of 7×7, 5×5 and 3×3 respectively, as shown in Table 1. Table 1 is a multi-scale convolution kernel expansion table. The convolution kernel corresponding to the near distance is larger to prevent the near depth value from being covered.

[0091] Table 1:

[0092]

[0093] Step S32: Under the Boolean mask constraint, the inverted depth value I inverse A multi-scale convolution kernel dilation operation is performed to obtain a dilated depth map.

[0094] Specifically, in step S32, under the Boolean mask constraint, the inverted depth value I inverse Perform a multi-scale convolution kernel dilation operation to obtain the expanded depth map including:

[0095] Under the Boolean mask constraint, the inverted depth value I inverse The formula for multi-scale convolution kernel expansion operation is:

[0096]

[0097] Among them, I inverse,j ,j=near,med,far are depth maps that only retain the near-distance depth value, the medium-distance depth value, and the far-distance depth value, respectively. Mask i is the Boolean mask corresponding to object i, This is a morphological dilation operation, done with OpenCV, Cross j is the convolution kernel corresponding to the j-th depth interval, I dilated is the depth map after dilation.

[0098] Step S4: performing a mask-constrained hole closing operation on the expanded depth map to obtain a hole-closed depth map.

[0099] In one embodiment of the present invention, this step uses a 5×5 full convolution kernel to perform a hole closing operation on the input depth map, which can link the narrow cracks in the depth map. After the multi-scale convolution kernel is expanded, there are still many empty depth values ​​in the output depth map. First, these areas are linked into several closed areas to prepare for the subsequent filling operation.

[0100] Specifically, performing a mask-constrained hole closing operation on the expanded depth map in step S4 to obtain a hole-closed depth map includes:

[0101] Using a 5×5 full convolution kernel to perform a hole closing operation on the expanded depth map;

[0102] Under the Boolean mask constraint, the formula for hole closing operation is:

[0103]

[0104] Among them, Mask i is the Boolean mask corresponding to object i, J 5×5 is a 5×5 matrix of all 1s, I closed It is the depth map after the hole is closed.

[0105] Step S5: performing an operation of extending the depth value upward on the depth map after the hole is closed to obtain an extended depth map.

[0106] In one embodiment of the present invention, the depth map I closedEach column of is filled from top to bottom with the area before the first valid value, and the filled values ​​are all the first valid value in the column. Generally, due to the difference in the installation position of the lidar and the camera, the LIDAR point cloud may be distributed in the lower part of the RGB image, resulting in no depth value in the higher area. Considering that there are usually targets such as high-rise buildings, trees, and pillars in the environment, it is necessary to extend the depth upward. In the actual operation process, there will be some objects that exceed the actual height after extension. Since the top area of ​​GroundTruth is generally empty, this will not affect the final test results.

[0107] Specifically, in step S5, performing an operation of extending the depth value upward on the depth map after the hole is closed to obtain an extended depth map includes:

[0108] The depth map after closing the hole I closed The area before the first valid value in each column from top to bottom is filled, and the filled value is the first valid value in the column;

[0109] The formula for extending the depth value upward is:

[0110] I extend =F extend (I closed ),

[0111] Among them, F extend The depth value extends upward, I extend is the extended depth map.

[0112] Step S6: performing a mask-constrained hole filling operation on the extended depth map to obtain a hole-filled depth map.

[0113] In one embodiment of the present invention, this step uses a 9×9 full convolution kernel to calibrate the input depth map I extend Perform a morphological dilation operation to fill the small holes in the depth map.

[0114] Specifically, performing a mask-constrained hole filling operation on the extended depth map in step S6 to obtain a hole-filled depth map includes:

[0115] Use 9×9 full convolution kernel to extend the depth map I extend Perform morphological dilation operation on the extended depth map I extend Fill the holes whose size is smaller than the set threshold;

[0116] Under the Boolean mask constraint, the hole filling operation formula is:

[0117]

[0118] Among them, Mask i is the Boolean mask corresponding to object i, J 9×9 is a 9×9 matrix of all 1s, I fill This is the depth map after hole filling.

[0119] Step S7: performing median filtering and Gaussian filtering operations on the hole-filled depth map to obtain a filtered depth map.

[0120] In one embodiment of the present invention, the sparse depth map has been completed into a dense depth map. On this basis, a 5×5 median filter is first used to remove the depth map I fill Then, a 5×5 Gaussian filter is used to smooth the plane area and make the edge of the object clear.

[0121] According to the results of the different filters and the effects of the two-by-two combinations on the algorithm performance, the combined filtering of median filtering + Gaussian filtering can provide the lowest RMSE and MAE for the deep expansion task, and the running time is also shorter. Therefore, the present invention also uses this combined filtering in the last filtering operation. At the same time, compared with Gaussian filtering, bilateral filtering can retain the local structure of the object and is more suitable for practical applications, so the present invention also retains this filtering option.

[0122] Specifically, performing median filtering and Gaussian filtering operations on the hole-filled depth map in step S7 to obtain a filtered depth map includes:

[0123] Use 5×5 median filtering to remove the hole-filled depth map I fill The outer points in the image are then filtered using a 5×5 Gaussian filter to smooth the plane area and make the edges of the object clear.

[0124] The median filter and Gaussian filter operation formula is:

[0125] I blur =F MedGauss (I output ),

[0126] Among them, F MedGauss is the median Gaussian combination filter, I blur is the depth map after filtering.

[0127] Step S8: performing a depth map inversion operation on the filtered depth map to obtain a final depth map.

[0128] Specifically, performing a depth map inversion operation on the filtered depth map in step S8 to obtain a final depth map includes:

[0129] The depth map inversion operation formula is:

[0130] I output =100-I blur ,

[0131] Among them, I output is the final depth map.

[0132] The sparse radar depth completion method based on Boolean mask constraints of the present invention is compiled and run using Python. In addition, a series of Torch, OpenCV, and Detectron2 functions are also used. The final output result of the present invention is a dense depth map that can be directly used for tasks such as 3D target detection and simultaneous localization and mapping (SLAM).

[0133] The sparse radar depth completion method based on Boolean mask constraint of the present invention first uses a panoramic segmentation algorithm to mark each pixel in a color image, and the pixel labels belonging to the same object are consistent. A Boolean mask is generated according to the marking result. Under the constraint of the mask, the sparse depth map is subjected to depth completion based on morphological filtering to ensure the algorithm versatility and the structural unification of the color map and the depth map. The dense depth map finally outputted can effectively protect the depth jump area.

[0134] The present invention also proposes an electronic device, which includes a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to implement the above-mentioned sparse radar depth completion method based on Boolean mask constraints. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components; the memory may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The memory can also be an internal memory of the random access memory (RAM) type, and the processor and memory can be integrated into one or more independent circuits or hardware, such as an application-specific integrated circuit (ASIC). It should be noted that the computer program in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0135] The present invention also proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned sparse radar depth completion method based on Boolean mask constraints. The computer-readable storage medium can be an electronic medium, a magnetic medium, an optical medium, an electromagnetic medium, an infrared medium, or a semiconductor system or a propagation medium. The computer-readable storage medium can also include a semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a hard disk, and an optical disk. The optical disk can include a compact disk-read only memory (CD-ROM), a compact disk-read / write (CD-RW), and a DVD.

[0136] In summary, the sparse radar depth completion method based on Boolean mask constraints of the present invention enables RGB image information to provide guidance for the depth completion task, and relies on basic image processing operations to execute sparse LIDAR depth data to complete the depth completion task. The present invention can effectively protect the depth fracture area and improve the performance of the depth completion algorithm.

[0137] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A sparse radar depth completion method based on Boolean mask constraints, characterized in that: include: S1. Perform panorama segmentation on the RGB image to generate multiple Boolean masks, process the area defined by each Boolean mask respectively, and superimpose the processed areas into a depth map; S2. performing a depth inversion operation on the effective depth value of the depth map to obtain an inverted depth value; S3, performing a multi-scale convolution kernel dilation operation on the effective depth value and the reversed depth value to obtain an expanded depth map; S4, performing a mask-constrained hole closing operation on the expanded depth map to obtain a hole-closed depth map; S5, performing an operation of extending the depth value upward on the depth map after the hole is closed to obtain an extended depth map; S6, performing a mask-constrained hole filling operation on the extended depth map to obtain a hole-filled depth map; S7, performing median filtering and Gaussian filtering operations on the hole-filled depth map to obtain a filtered depth map; S8. Perform a depth map inversion operation on the filtered depth map to obtain a final depth map.

2. The sparse radar depth completion method based on Boolean mask constraint according to claim 1, characterized in that: The panoptic segmentation of the RGB image in step S1 to generate a plurality of Boolean masks includes: The image panoptic segmentation algorithm includes PanopticSegFormer and Point2Mask to generate multiple Boolean masks, which satisfy the following formula: Where n is the total number of objects in the RGB image, Mask i is the Boolean mask corresponding to object i, J W×L It is a W×L all-1 matrix, W is the width of the RGB image, and L is the length of the RGB image.

3. The sparse radar depth completion method based on Boolean mask constraint according to claim 2 is characterized in that: The step S2 of performing a depth reversal operation on the effective depth value of the depth map to obtain a reversed depth value includes: The formula for performing a depth inversion operation on the effective depth value of the depth map is: I inverse =100-I input , Among them, I inverse is the depth value after inversion, I input is the effective depth value of the depth map.

4. The sparse radar depth completion method based on Boolean mask constraint according to claim 3 is characterized in that: The step S3 of performing a multi-scale convolution kernel dilation operation on the effective depth value and the inverted depth value to obtain an expanded depth map includes: S31, the effective depth value I input According to 0~30m, 30~60m, and 60~80m, it is divided into short-distance depth value, medium-distance depth value, and long-distance depth value, corresponding to three convolution kernels of 7×7, 5×5, and 3×3 respectively; S32, under the Boolean mask constraint, the inverted depth value I inverse A multi-scale convolution kernel dilation operation is performed to obtain a dilated depth map.

5. The sparse radar depth completion method based on Boolean mask constraint according to claim 4 is characterized in that: In step S32, under the Boolean mask constraint, the inverted depth value I inverse Perform a multi-scale convolution kernel dilation operation to obtain the expanded depth map including: Under the Boolean mask constraint, the inverted depth value I inverse The formula for multi-scale convolution kernel expansion operation is: Among them, I inverse,j ,j=near,med,far are depth maps that only retain the near-distance depth value, the medium-distance depth value, and the far-distance depth value, respectively. Mask i is the Boolean mask corresponding to object i, This is a morphological dilation operation, done with OpenCV, Cross j is the convolution kernel corresponding to the j-th depth interval, I dilated is the depth map after dilation.

6. The sparse radar depth completion method based on Boolean mask constraint according to claim 5, characterized in that: The step S4 of performing a mask-constrained hole closing operation on the expanded depth map to obtain a hole-closed depth map includes: Using a 5×5 full convolution kernel to perform a hole closing operation on the expanded depth map; Under the Boolean mask constraint, the formula for hole closing operation is: Among them, Mask i is the Boolean mask corresponding to object i, J 5×5 is a 5×5 matrix of all 1s, I closed It is the depth map after the hole is closed.

7. The sparse radar depth completion method based on Boolean mask constraint according to claim 6, characterized in that: In step S5, the depth map after the hole is closed is extended upward to obtain an extended depth map, which includes: The depth map after closing the hole I closed The area before the first valid value in each column from top to bottom is filled, and the filled value is the first valid value in the column; The formula for extending the depth value upward is: I extend =F extend (I closed ), Among them, F extend The depth value extends upward, I extend is the extended depth map.

8. The sparse radar depth completion method based on Boolean mask constraint according to claim 7, characterized in that: The step S6 of performing a mask-constrained hole filling operation on the extended depth map to obtain a hole-filled depth map includes: Use 9×9 full convolution kernel to extend the depth map I extend Perform morphological dilation operation on the extended depth map I extend Fill the holes whose size is smaller than the set threshold; Under the Boolean mask constraint, the hole filling operation formula is: Among them, Mask i is the Boolean mask corresponding to object i, J 9×9 is a 9×9 matrix of all 1s, I fill This is the depth map after hole filling.

9. The sparse radar depth completion method based on Boolean mask constraint according to claim 8, characterized in that: The step S7 of performing median filtering and Gaussian filtering operations on the hole-filled depth map to obtain a filtered depth map includes: Use 5×5 median filtering to remove the hole-filled depth map I fill The outer points in the image are then filtered using a 5×5 Gaussian filter to smooth the plane area and make the edges of the object clear. The median filter and Gaussian filter operation formula is: I blur =F MedGauss (I output ), Among them, F MedGauss is the median Gaussian combination filter, I blur is the depth map after filtering.

10. The sparse radar depth completion method based on Boolean mask constraint according to claim 9, characterized in that: The step S8 of performing a depth map inversion operation on the filtered depth map to obtain a final depth map includes: The depth map inversion operation formula is: I output =100-I blur , Among them, I output is the final depth map.

Images