Distance information completion method, device, storage medium and computer program product

Abstract

The present invention relates to the field of data processing and provides a distance information completion method, device, storage medium, and computer program product. The method comprises: determining a reference area based on a to-be-completed area of ​​a depth image; determining candidate distance values ​​based on the distance values ​​of valid distance points within the reference area; and completing the distance values ​​of distance-missing points in the to-be-completed area based on the candidate distance values. The distance information completion method provided by the present invention improves the efficiency of distance information completion by completing information on distance-missing points in a specific area of ​​an image. Furthermore, by comprehensively considering valid distance points within the reference area corresponding to the to-be-completed area to obtain candidate distance information, the accuracy of distance information completion for distance-missing points in the to-be-completed area is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distance information completion method, device, storage medium and computer program product. Background Art

[0002] With the rapid development of computer technology, artificial intelligence has ushered in wave after wave of enthusiasm, and vision-based intelligent recognition technology has become part of people's daily lives. In the conventional object detection and recognition process, the plane information of the object is first determined through the planar image obtained by the RGB camera. Then, a 3D camera (a camera capable of ranging imaging or radar) introduces additional distance points for overall spatial positioning. However, due to defects in the 3D camera hardware or interference caused by strongly reflective objects on the internal photosensitive elements, the imaging effect is often unsatisfactory. Some image areas cannot obtain accurate and effective distance point information, resulting in errors in robot grasping, navigation and ranging tasks.

[0003] Currently, the main methods for completing missing distance points are to use the closest points around the missing point or to complete the global point using a completion network model. However, the former method suffers from low accuracy due to visual differences caused by objects at different distances or angles from the camera. The latter method, however, requires extensive training data and requires global point completion, resulting in low efficiency. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention proposes a distance information completion method that can complete distance points in a specific area and improve the accuracy and efficiency of distance information completion.

[0005] The present invention also provides an electronic device.

[0006] The present invention also provides a storage medium.

[0007] The present invention also provides a computer program product.

[0008] According to an embodiment of the first aspect of the present invention, a distance information completion method includes:

[0009] Determine a reference area based on the area to be completed in the depth image;

[0010] determining a candidate distance value based on the distance values ​​of the valid distance points within the reference area;

[0011] Completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values;

[0012] The reference area is within the depth image; and the valid distance point is a point where no distance value is missing.

[0013] According to the distance information completion method of an embodiment of the present invention, the efficiency of distance information completion is improved by completing the information of distance missing points in a specific area of ​​the image. At the same time, by comprehensively considering the valid distance points in the reference area corresponding to the area to be completed to obtain candidate distance information, the accuracy of distance information completion for the distance missing points in the area to be completed is improved.

[0014] According to one embodiment of the present invention, the reference area includes a contraction area and an expansion area; wherein the contraction area and the expansion area are both within the depth image, and the contraction area is located inside the area to be completed, and the expansion area is located outside the area to be completed;

[0015] The determining of candidate distance values ​​according to the distance values ​​of the valid distance points in the reference area includes:

[0016] determining a first candidate distance value based on the distance values ​​of the effective distance points in the contraction region, and determining a second candidate distance value based on the distance values ​​of the effective distance points in the expansion region;

[0017] The step of completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values ​​includes:

[0018] Distance values ​​of distance-missing points in the to-be-completed area are completed based on the first candidate distance value and the second candidate distance value.

[0019] According to the distance information completion method of an embodiment of the present invention, candidate distance information is obtained by comprehensively considering the effective distance points in the contraction area and the expansion area corresponding to the area to be completed, thereby further improving the accuracy of completing the distance information of the distance missing points in the area to be completed.

[0020] According to one embodiment of the present invention, the step of completing the distance values ​​of the distance-missing points in the to-be-completed area based on the first candidate distance value and the second candidate distance value includes:

[0021] Determining that a difference between the first candidate distance value and the second candidate distance value is greater than a preset difference threshold, and using the first candidate distance value to complete the distance value of the distance-missing point in the to-be-completed area;

[0022] Determine that the difference between the first candidate distance value and the second candidate distance value is not greater than the difference threshold, and use the average of the first candidate distance value and the second candidate distance value to complete the distance value of the distance missing point in the to-be-completed area.

[0023] According to the distance information completion method of the embodiment of the present invention, the distance values ​​of the distance missing points in the area to be completed are completed according to different situations based on the difference between the first candidate distance value and the second candidate distance value, thereby further improving the accuracy of distance information completion.

[0024] According to one embodiment of the present invention, the area to be completed is a polygon;

[0025] The determining of the reference area based on the area to be completed in the depth image includes:

[0026] Determining an extension point on at least one edge line of the area to be completed, determining a plurality of vertices of the contraction area based on each of the extension points and a preset inward distance, and obtaining the contraction area according to the plurality of vertices;

[0027] Based on each of the extension points and a preset outward expansion distance, a center point of the expansion region corresponding to the extension point is determined, and the expansion region is obtained according to the center point of the expansion region.

[0028] According to the distance information completion method of an embodiment of the present invention, a polygon is used to mark the area to be completed, and one or more contraction areas and one or more expansion areas are determined based on the edge of the polygonal area to be completed and a preset inward or outward distance. This allows for uniform acquisition of multiple related areas near the area to be completed, and a first candidate distance value and a second candidate distance value are comprehensively calculated based on the effective distance values ​​of these related areas, further improving the accuracy of distance information completion.

[0029] According to one embodiment of the present invention, after determining the reference area based on the to-be-completed area in the depth image, and before determining the first candidate distance value according to the distance values ​​of the effective distance points in the contracted area, and before determining the second candidate distance value according to the distance values ​​of the effective distance points in the expanded area, the method further includes:

[0030] Determining that the number of valid distance points within the contraction area is less than a preset first threshold, and re-determining the contraction area by adjusting the retraction distance;

[0031] It is determined that the number of valid distance points in the expansion area is less than a preset second threshold, and the expansion area is redefined by adjusting the outward expansion distance.

[0032] According to the distance information completion method of an embodiment of the present invention, when it is determined that there are few valid distance points in a contraction area or an expansion area, it is considered that the distance information of the relevant area is insufficient to calculate the first candidate distance value or the second candidate distance value. Therefore, the contraction area or the expansion area is redefined by adjusting the degree of inward or outward expansion, ensuring that sufficient distance information is obtained to calculate the first candidate distance value and the second candidate distance value of the area to be completed, thereby further improving the accuracy of distance information completion.

[0033] According to one embodiment of the present invention, determining the first candidate distance value based on the distance values ​​of the effective distance points in the contraction area, and determining the second candidate distance value based on the distance values ​​of the effective distance points in the expansion area, includes:

[0034] determining a first distance average value of the contraction area, and determining the first candidate distance value based on the first distance average value;

[0035] determining a second distance average value of the expansion area, and determining the second candidate distance value based on the second distance average value;

[0036] The first distance average value is obtained by averaging the distance values ​​of all effective distance points in the contraction area; and the second distance average value is obtained by averaging the distance values ​​of all effective distance points in the expansion area.

[0037] According to an embodiment of the present invention, the distance information completion method uses the average of the effective distance values ​​of the relevant areas as the basis for calculating candidate distance values. Specifically, for multiple expanded or contracted areas, the average of the effective distance values ​​of each expanded or contracted area is first obtained. Then, the corresponding candidate distance value is calculated based on the average distance value of the multiple expanded or contracted areas. This allows for uniform acquisition of distance point information for the relevant areas surrounding the area to be completed, further improving the universality and accuracy of distance information completion.

[0038] According to one embodiment of the present invention, before determining the reference area based on the to-be-completed area in the depth image, the method further includes:

[0039] Determining the area to be completed from the depth image based on the two-dimensional coordinate information of the target recognition area in the color imaging image;

[0040] The coordinate system of the color imaging image and the coordinate system of the depth image are pre-registered with each other; and the target recognition area is the area where the object to be detected is located.

[0041] According to the distance information completion method of an embodiment of the present invention, there is no need to directly obtain the area to be completed through the depth image. Instead, the area to be completed in the depth image is determined by the target recognition area of ​​the corresponding color imaging image. Since the color imaging image can identify the area of ​​the target object more quickly and accurately than the depth image, the efficiency and accuracy of the distance information completion are further improved as a whole.

[0042] According to one embodiment of the present invention, before determining the area to be completed from the depth image based on the two-dimensional coordinate information of the target recognition area of ​​the color imaging image, the method further includes:

[0043] Inputting the color imaging image into a target recognition model to obtain a color imaging image marked with a target recognition area output by the target recognition model;

[0044] The target recognition model is obtained by training based on the sample color imaging image and the target recognition area results corresponding to the sample color imaging image.

[0045] According to the distance information completion method of an embodiment of the present invention, a trained target recognition model is used to identify the target object area in the color imaging image, which can further improve the accuracy of selecting the area to be completed, thereby further improving the accuracy of distance information completion.

[0046] According to an embodiment of the second aspect of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, any of the distance information completion methods described above is implemented.

[0047] According to a third aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the above-described distance information completion methods.

[0048] A computer program product according to an embodiment of the fourth aspect of the present invention includes a computer program, which, when executed by a processor, implements the distance information completion method as described in any one of the above items.

[0049] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0050] By completing the information of missing distance points in a specific area of ​​the image, the efficiency of distance information completion is improved. At the same time, by comprehensively considering the effective distance points in the reference area corresponding to the area to be completed to obtain candidate distance information, the accuracy of distance information completion for missing distance points in the area to be completed is improved.

[0051] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 1 is a flow chart of a distance information completion method provided by an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the overall process of the distance information completion method provided by an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of regional solution provided by an embodiment of the present invention;

[0056] Figure 4 is a structural diagram of a distance information completion device provided by an embodiment of the present invention;

[0057] Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0059] Figure 1 This is one of the flow charts of the distance information completion method provided by the embodiment of the present invention. Figure 1 , an embodiment of the present invention provides a distance information completion method, comprising the steps of:

[0060] S1. Determine a reference area based on a to-be-completed area of ​​a depth image; wherein the reference area is within the depth image.

[0061] It is understandable that due to defects in the 3D camera hardware itself or interference caused by strongly reflective objects on the internal photosensitive elements, the depth image (Depth map, which represents the distance imaging map of the object from the camera) obtained by the 3D camera has some distance points missing, resulting in the inability to obtain the real spatial position information of certain objects in the image, affecting subsequent robot grasping, navigation and ranging tasks. In the prior art, for the completion of missing distance points, the closest distance points in the surrounding area are directly used to complete the completion, ignoring the different distance points measured due to the distance of the object from the camera in 3D space. There is a certain visual difference, and the supplemented distance points do not conform to the current effective distance. In addition, if a 3D data completion network model is used for completion, it needs to be trained on a large amount of data on the GPU, and it is a global point completion method, and cannot be completed for specific points, which consumes too many additional computing resources, and the model prediction is time-consuming and unstable.

[0062] In an embodiment of the present invention, the area to be completed is the area for which distance information needs to be completed. It can be a region selected by boxing an object, a component, or a part of an object. For the selected area to be completed, the embodiment of the present invention uses the distance point information of reference areas inside or outside the area to be completed to perform a comprehensive calculation to complete the distance information of the missing distance points in the area to be completed. The area to be completed can be a circle, polygon, or irregular shape, etc.

[0063] In an embodiment of the present invention, there may be one or more reference areas. The reference area may be determined by taking the center point of the area to be completed as the center point of the reference area to be obtained, and based on this center point, taking an area that is a plurality of times smaller than the area to be completed as the reference area; or taking a plurality of small areas around the center point of the area to be completed as the plurality of shrinkage areas. Furthermore, the area enclosed between two contours formed by magnifying the area to be completed by different magnifications may be used as the reference area; or taking a plurality of evenly distributed small areas around the area to be completed as the plurality of reference areas. The specific method for obtaining the reference area may be set according to actual needs.

[0064] S2. Determine candidate distance values ​​based on the distance values ​​of valid distance points in the reference area, wherein the valid distance points are points with no missing distance values.

[0065] In an embodiment of the present invention, for a determined reference area, a candidate distance value is obtained by calculating the distance values ​​of the valid distance points within the reference area (an average calculation can be performed, or a weighted average can be performed by determining a weight based on the distance of each distance point from the center point of the reference area).

[0066] S3: completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values.

[0067] It should be noted that after obtaining the candidate distance values ​​corresponding to the reference area, the distance values ​​of the missing points in the area to be completed can be completed based on the candidate distance values ​​according to different situations. For example, if there are multiple reference areas, the average of the candidate distance values ​​corresponding to the multiple reference areas can be used as the distance value for the missing point in the area to be completed. Alternatively, corresponding weights can be set for the candidate distance values ​​of the multiple reference areas to perform a weighted average, and the value obtained by the weighted average is used as the distance value for the missing point in the area to be completed.

[0068] According to the distance information completion method of an embodiment of the present invention, by completing the information of the distance missing points in a specific area of ​​the image, there is no need to perform model training and global completion on the entire depth image, which saves a lot of computing resources and operation time, thereby improving the efficiency of distance information completion; at the same time, by comprehensively calculating the candidate distance values ​​based on multiple valid distance points in the reference area corresponding to the area to be completed, the problem of low accuracy caused by the existing technology of using a single distance point near the target point to supplement the distance value is overcome, thereby improving the accuracy of distance information completion for the distance missing points in the area to be completed.

[0069] In one embodiment, the reference area includes a contraction area and an expansion area; wherein the contraction area and the expansion area are both within the depth image, and the contraction area is located inside the to-be-completed area, and the expansion area is located outside the to-be-completed area;

[0070] Step S2 may specifically include:

[0071] S21 . Determine a first candidate distance value according to the distance values ​​of the effective distance points in the contraction area, and determine a second candidate distance value according to the distance values ​​of the effective distance points in the expansion area.

[0072] Step S3 may specifically include:

[0073] S31: Complete the distance values ​​of the distance-missing points in the area to be completed based on the first candidate distance value and the second candidate distance value.

[0074] In an embodiment of the present invention, a first candidate distance value and a second candidate distance value are respectively calculated by simultaneously obtaining a contraction region and an expansion region of the region to be completed. The candidate distance values ​​are calculated by comprehensively considering the valid distance points of the reference region inside and outside the region to be completed. Specifically, the contraction region can be determined by using the center point of the region to be completed as the center point of the contraction region, and using an area that is a plurality of times smaller than the region to be completed as the contraction region; or by selecting a plurality of small areas around the center point of the region to be completed as the contraction region. For the expansion region, the area enclosed by two contours formed by magnifying the region to be completed by different magnifications can be used as the expansion region; or, a plurality of evenly distributed small areas outside the region to be completed can be selected as the expansion region. The number of both the contraction region and the expansion region can be one or more, and the specific method for obtaining the contraction region and the expansion region can be defined according to actual needs.

[0075] In an embodiment of the present invention, for a determined contraction region and expansion region, a first candidate distance value and a second candidate distance value are calculated based on the distance values ​​of valid distance points within each region (this may be averaged or weighted averaged based on the distance of each distance point from the center point of the region). If there are multiple contraction regions, an average or weighted average is further calculated based on the candidate distance values ​​corresponding to the multiple contraction regions to obtain the first candidate distance value. Similarly, if there are multiple expansion regions, an average or weighted average is further calculated based on the candidate distance values ​​corresponding to the multiple expansion regions to obtain the first candidate distance value.

[0076] It should be noted that after obtaining the first candidate distance value corresponding to the contraction region and the second candidate distance value corresponding to the expansion region, the distance value of the missing distance point in the region to be completed can be completed based on the first candidate distance value and the second candidate distance value, depending on different situations. For example, the average of the first candidate distance value and the second candidate distance value can be used as the distance value for the missing distance point in the region to be completed, or corresponding weights can be assigned to the first candidate distance value and the second candidate distance value to perform a weighted average, and the value obtained by the weighted average is used as the distance value for the missing distance point in the region to be completed.

[0077] According to the distance information completion method of an embodiment of the present invention, candidate distance values ​​are comprehensively calculated based on multiple valid distance points in the contracted area and the expanded area corresponding to the area to be completed, and distance information is completed by comprehensively considering both the internal and external reference areas of the area to be completed, thereby further improving the accuracy of distance information completion for distance missing points in the area to be completed.

[0078] In one embodiment, step S31 may include the steps of:

[0079] S311: Determine whether the difference between the first candidate distance value and the second candidate distance value is greater than a preset difference threshold, and use the first candidate distance value to complete the distance value of the distance-missing points in the area to be completed;

[0080] S312: Determine that the difference between the first candidate distance value and the second candidate distance value is not greater than the difference threshold, and use the average of the first candidate distance value and the second candidate distance value to complete the distance value of the distance missing points in the area to be completed.

[0081] It should be noted that, in the embodiment of the present invention, distance values ​​of distance-missing points in the area to be completed may be completed according to different situations based on the difference between the first candidate distance value and the second candidate distance value.

[0082] Specifically, when the difference between the first candidate distance value and the second candidate distance value is greater than the preset difference threshold, it is considered that the distance point information of the expanded area outside the area to be completed has little reference value for distance completion of the area to be completed. Therefore, the first candidate distance value corresponding to the contracted area is directly used to complete the distance value of the missing distance points in the area to be completed.

[0083] Correspondingly, when the difference between the first candidate distance value and the second candidate distance value is not greater than the gap threshold, it is considered that the distance point information of the expanded area outside the area to be completed has a certain reference value for distance completion of the area to be completed. Therefore, it is necessary to perform an average operation on the first candidate distance value and the second candidate distance value to complete the distance value of the missing points in the area to be completed.

[0084] According to the distance information completion method of an embodiment of the present invention, the distance values ​​of the distance missing points in the area to be completed are completed according to different situations based on the difference between the first candidate distance value and the second candidate distance value, so that the distance information completed in the area to be completed is closer to the actual scene, thereby further improving the accuracy of the distance information completion.

[0085] In other embodiments, the reference value weight of the expanded region outside the region to be completed for distance completion of the region to be completed may be determined based on the difference between the first candidate distance value and the second candidate distance value, and the calculation weights of the first candidate distance value and the second candidate distance value may be determined based on the reference value weights. For example, different threshold intervals may be pre-set, each corresponding to a different inner and outer reference area weight ratio. A target threshold interval corresponding to the difference between the first candidate distance value and the second candidate distance value may be determined, and then a weighted average calculation may be performed on the first candidate distance value and the second candidate distance value based on the corresponding inner and outer reference area weight ratio to complete the distance information of the missing distance point.

[0086] In one embodiment, the area to be completed is a polygon;

[0087] Step S1 may include the steps of:

[0088] S11, determining an extension point on at least one edge line of the to-be-completed region, determining a plurality of vertices of the contraction region based on each of the extension points and a preset inward distance, and obtaining the contraction region based on the plurality of vertices;

[0089] S12: Determine a center point of an expansion region corresponding to each extension point based on the extension point and a preset outward expansion distance, and obtain the expansion region according to the center point of the expansion region.

[0090] It should be noted that for the selection of the area to be completed, a polygon can be used to mark it, and further, a rectangle can be used to mark it. The following takes the rectangular area to be completed as an example to illustrate that for the contraction area, the center points of the four sides of the rectangle can be used as extension points, and the preset inward distance can be extended from these extension points (preferably in the direction perpendicular to the side lines) to the inside of the area to be completed to obtain four extended end points. Connecting these four end points can obtain a diamond area, which is the contraction area. For the expansion area, similarly, according to the extension points on the four sides of the rectangle mentioned above, the preset outward distance can be extended from these extension points (preferably in the direction perpendicular to the side lines) to the outside of the area to be completed to obtain four extended end points. With these end points as the center points, the corresponding four expansion areas can be formed according to the preset side length. It should be noted that the inward distance (outward expansion distance) corresponding to different extension points can be different. When the inward distances corresponding to different extension points are different, the contraction area formed can be an irregular shape. In other embodiments, one or more contraction areas and one or more expansion areas may be determined based on different numbers of extension points and different inward or outward distances.

[0091] According to the distance information completion method of an embodiment of the present invention, a polygon is used to mark the area to be completed, and a contraction area and several expansion areas are determined based on the edge of the polygonal area to be completed and a preset inward or outward distance. This allows for uniform acquisition of multiple related areas near the area to be completed, and a first candidate distance value and a second candidate distance value are comprehensively calculated based on the effective distance values ​​of these related areas, further improving the accuracy of distance information completion.

[0092] In one embodiment, after step S1 and before step S21, the following steps may be further included:

[0093] S13, determining that the number of valid distance points in the contraction area is less than a preset first threshold, and re-determining the contraction area by adjusting the retraction distance;

[0094] S14: Determine that the number of valid distance points in the expansion area is less than a preset second threshold, and redefine the expansion area by adjusting the outward expansion distance.

[0095] It is understandable that when the range of the contraction region or expansion region determined in step S1 is relatively small, it is very likely that the number of valid distance points in the contraction region or expansion region is small, or even zero. Therefore, to ensure that a sufficient number of valid distance points can be obtained for calculating the candidate distance value, the embodiment of the present invention sets a first threshold and a second threshold. When the number of valid distance points in the contraction region is less than the first threshold, the contraction region is redefined by adjusting the inward contraction distance (reducing the inward contraction distance can obtain a larger contraction region); when the number of valid distance points in at least one of the expansion regions is less than the preset second threshold, the outward expansion distance is adjusted to redefined multiple expansion regions (increasing the outward expansion distance can obtain an expansion region farther from the region to be completed). Alternatively, the range of the expansion region can be expanded by adjusting the side length.

[0096] According to the distance information completion method of an embodiment of the present invention, when it is determined that there are few valid distance points in a contraction area or an expansion area, it is considered that the distance information of the relevant area is insufficient to calculate the first candidate distance value or the second candidate distance value. Therefore, the contraction area or the expansion area is redefined by adjusting the degree of inward or outward expansion, ensuring that sufficient distance information is obtained to calculate the first candidate distance value and the second candidate distance value of the area to be completed, thereby further improving the accuracy of distance information completion.

[0097] In one embodiment, step S21 may include the steps of:

[0098] S21, determining a first distance average value of the contraction area, and determining the first candidate distance value according to the first distance average value;

[0099] S22, determining a second distance average value of the expansion area, and determining the second candidate distance value according to the second distance average value;

[0100] The first distance average value is obtained by averaging the distance values ​​of all effective distance points in the contraction area; and the second distance average value is obtained by averaging the distance values ​​of all effective distance points in the expansion area.

[0101] It should be noted that, in the embodiment of the present invention, an average value calculation method is used to obtain the first candidate distance value and the second candidate distance value. Specifically, if there is only one contraction area and one expansion area, the distance values ​​of all valid distance points within the contraction area or the expansion area are used to calculate the average value as the first candidate distance value or the second candidate distance value. If there are multiple contraction areas or expansion areas, the average distance value of all valid distance points within each contraction area (expansion area) is first calculated, and then the second candidate distance value is calculated based on the average distance values ​​of the multiple contraction areas (expansion areas) (which can be an average value or a weighted average determined by different weights based on areas in different directions).

[0102] According to the distance information completion method of an embodiment of the present invention, the average value of the effective distance values ​​of the relevant areas is used as the basis for calculating the candidate distance values. Specifically, for multiple shrinking or expanding areas, the average value of the effective distance values ​​of each area is first obtained, and then the candidate distance value is calculated based on the distance average value of the multiple areas. This allows for uniform acquisition of distance point information of the relevant areas surrounding the area to be completed, further improving the universality and accuracy of distance information completion.

[0103] In one embodiment, before step S1, the method may further include the following steps:

[0104] S15, determining the area to be completed from the depth image based on the two-dimensional coordinate information of the target recognition area in the color imaging image;

[0105] The coordinate system of the color imaging image and the coordinate system of the depth image are pre-registered with each other; and the target recognition area is the area where the object to be detected is located.

[0106] It should be noted that in practical applications, the distance information of the target scene is generally obtained by combining color imaging and depth images. Since color imaging can identify the area of ​​the target object more quickly and accurately than depth images, the coordinate systems of the color imaging and depth images can be aligned with each other first, and then the color imaging is used to perform target recognition to obtain the two-dimensional coordinate information of the target recognition area (such as the coordinates of the four vertices of the rectangular box), and then the corresponding area to be completed can be selected in the depth image according to the same coordinates.

[0107] According to the distance information completion method of an embodiment of the present invention, there is no need to directly obtain the area to be completed through the depth image. Instead, the area to be completed in the depth image is determined by the target recognition area of ​​the corresponding color imaging image. Since the color imaging image can identify the area of ​​the target object more quickly and accurately than the depth image, the efficiency and accuracy of the distance information completion are further improved as a whole.

[0108] In one embodiment, before step S15, the following steps may be further included:

[0109] S16, inputting the color imaging image into a target recognition model to obtain a color imaging image marked with a target recognition area output by the target recognition model;

[0110] The target recognition model is obtained by training based on the sample color imaging image and the target recognition area results corresponding to the sample color imaging image.

[0111] It should be noted that for object recognition in color images, pre-trained object recognition models (such as the Faster-RCNN series and the YOLO series) can be used for recognition. For object recognition model training, sample color images with target objects of different shapes (such as cups and boxes) can be used, and then combined with the marked target object boxes for model training.

[0112] According to the distance information completion method of an embodiment of the present invention, a trained target recognition model is used to identify the target object area in the color imaging image, which can further improve the accuracy of selecting the area to be completed, thereby further improving the accuracy of distance information completion.

[0113] See Figure 2-3 Based on the above solution, in order to better understand the distance information completion method provided by the embodiment of the present invention, the following specific examples are given for detailed description:

[0114] In order to solve the problems of low efficiency and low accuracy in the existing technology for completing missing distance points, the distance information completion method provided in the embodiment of the present invention estimates the target distance points by using some distance points in the contracted area of ​​the area to be completed. This does not rely on a single distance and has a certain redundancy. At the same time, some distance points in the expanded area of ​​the area to be completed are used to estimate the target distance points. The estimated distance points in the area can be corrected by using the spatial structural features to obtain more accurate target distance points for completion. It should be noted that the completed distance points obtained by multi-directional calculation on the spatial structure do not rely on a powerful GPU and do not require the consumption of additional computing resources. The estimated target distance points are more stable and can be easily transplanted into various recognition algorithm frameworks. The specific steps of the embodiment of the present invention are as follows:

[0115] 1) Capture the current scene data through a 3D camera or other device to obtain the registered RGB image (color image) and Depth image (depth image), which are represented by S1 and S2 respectively.

[0116] 2) Use the target recognition model to train and predict the currently collected RGB image, where the target recognition model can be any standard 2D target detection network (such as the faster-rcnn series, yolo series, etc.), which can predict a rectangular box around the target (it can also be other graphics, this example uses a rectangle as an example), and use the rectangular box to mark the area where the target object is located. The embodiment of the present invention does not limit the type of 2D target detection network. Let the upper left corner of the RGB image be the coordinate origin, the positive direction of the x-axis from left to right, and the positive direction of the y-axis from top to bottom. The predicted corner coordinates of the rectangular box containing the target object area are in and Respectively represent the coordinates of the upper left corner of the rectangular box, and Represents the coordinates of the lower right corner of the rectangle. Similarly, set the same coordinate system on the Depth map as in the RGB map, so that the corner point B of the rectangle can be directly transferred to the Depth map to obtain the area to be completed in the Depth map. Each coordinate value in the Depth map corresponds to the actual distance point P, so the target distance point P can be selected in the area to be completed in the Depth map. * .

[0117] 3) According to the depth map and the coordinates of the area to be completed, the candidate points for completing the distance information of the missing points in the area to be completed are calculated based on the contraction area and expansion area corresponding to the area to be completed. The specific area solution is shown in the figure Figure 3 shown.

[0118] a) Calculate the candidate distance value of the shrinking area. First calculate the midpoints of the four sides of the rectangular box of the area to be completed and By setting four shrinkage factors (inward distances) [cg0>0,cg1>0,cg2>0,cg3>0], a new shrunken diamond area can be obtained. The four corner points of the diamond area are and in and Using the shrunk diamond area to calculate the candidate distance value is more effective in filtering out irrelevant areas than directly shrinking the rectangle. Suppose the distance value set of the shrunk diamond area points is The following formula is used to calculate the candidate distance value corresponding to the shrinkage area

[0119]

[0120] in, k is The number of all valid distance points (distance values ​​not missing) in the set. When k = 0 (can also be set to less than the preset value), that is, there are no valid distance points in the region, the shrinkage factor [cg0, cg1, cg2, cg3] can be adjusted to obtain a new diamond region for calculation.

[0121] Using the same calculation method, the vertices of the contracted diamond area can be connected with the extended points on the edge of the area to be completed to form a valid point set of the four directions between the diamond area and the area to be completed. and The candidate distance is calculated for each set, and then the average is taken to obtain the final candidate distance value.

[0122] b) Calculate the candidate distance value of the expansion area. According to the midpoints of the four sides of the rectangular box in step a) above and According to the four expansion factors (extension distance) set in the current scene [ep0>0,ep1>0,ep2>0,ep3>0], the coordinates of the distance point after expansion can be obtained. and in and To prevent the distance points selected after expansion from appearing invalid missing values ​​or mutation values ​​of 0 due to objective factors such as equipment and angle, set The distance value set of the four expansion area points with a center and a side length of d is and The candidate distance value corresponding to the expansion area is calculated using the following formula:

[0123]

[0124] in, m is The number of all valid distance points in the set; n is The number of all valid distance points in the set; w is The number of all valid distance points in the set; u is The number of all valid distance points in the set. When m, n, w, u are 0 (can also be set to the default value), the expansion factor [ep0, ep1, ep2, ep3] can be adjusted (or the side length can be adjusted) to obtain a new expansion area for calculation. The candidate distance value calculated by the four-direction comprehensive calculation It can effectively utilize the spatial information around the target to obtain a more accurate distance.

[0125] 4) Fill in the distance value of the missing points in the area to be filled. First, determine the target distance point P directly collected by the device. * Whether the distance value exists, when P * When the distance value does not exist (distance missing point), the candidate distance value is used for calculation. Set the threshold φ>0, when The target distance point when Target distance point The target distance point P for all missing distance values ​​in the completion area * Complete the completion.

[0126] It should be noted that the above embodiment describes a solution for completing distance information by simultaneously referring to the effective distance values ​​of the contraction area and the expansion area. In other embodiments, distance information completion can also be achieved by referring to the effective distance value of the contraction area alone, wherein the acquisition of the contraction area can refer to the contraction area determination method described in the above embodiment; in addition, distance information completion can also be achieved by referring to the effective distance value of the expansion area alone, wherein the acquisition of the expansion area can refer to the contraction area determination method described in the above embodiment.

[0127] It should be noted that the embodiments of the present invention are applicable to the deployment of a variety of platforms with visual perception: smartphones, robots (such as home service robots), surveillance cameras, smart appliances, etc. Data collected by a variety of sensors can be used as input: RGB-D infrared cameras, RGB cameras + lidar, RGB cameras + millimeter-wave radar, RGB cameras + infrared cameras, and other components capable of RGB imaging and ranging imaging.

[0128] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0129] 1) The calculation of the embodiment of the present invention is stable and fast, but does not rely on high-performance computing resources, and is easy to transplant into devices.

[0130] 2) The embodiments of the present invention can be plug-and-play and can be flexibly adapted to existing recognition algorithms.

[0131] 3) The embodiment of the present invention uses distance points in a specific area for completion, and the specific area can be freely adjusted according to different scene tasks, which has a certain universality.

[0132] 4) The embodiment of the present invention does not rely too much on the imaging performance of the 3D camera, saving hardware cost.

[0133] refer to Figure 4 , Figure 4: is a schematic diagram of a module of a distance information completion device provided in an embodiment of the present invention. The distance information completion device provided in an embodiment of the present invention includes:

[0134] The region determination module 1 is configured to determine a reference region based on the region to be completed in the depth image;

[0135] An information candidate module 2 is configured to determine candidate distance values ​​according to the distance values ​​of the valid distance points in the reference area;

[0136] An information completion module 3 is configured to complete the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values;

[0137] The reference area is within the depth image; and the valid distance point is a point where no distance value is missing.

[0138] In one embodiment, the reference area includes a contraction area and an expansion area; wherein the contraction area and the expansion area are both within the depth image, and the contraction area is located inside the to-be-completed area, and the expansion area is located outside the to-be-completed area;

[0139] The information candidate module 2 is specifically used for:

[0140] determining a first candidate distance value based on the distance values ​​of the effective distance points in the contraction region, and determining a second candidate distance value based on the distance values ​​of the effective distance points in the expansion region;

[0141] The information completion module 3 is specifically used to:

[0142] Distance values ​​of distance-missing points in the to-be-completed area are completed based on the first candidate distance value and the second candidate distance value.

[0143] In one embodiment, the information completion module 3 is specifically configured to:

[0144] Determining that a difference between the first candidate distance value and the second candidate distance value is greater than a preset difference threshold, and using the first candidate distance value to complete the distance value of the distance-missing point in the to-be-completed area;

[0145] Determine that the difference between the first candidate distance value and the second candidate distance value is not greater than the difference threshold, and use the average of the first candidate distance value and the second candidate distance value to complete the distance value of the distance missing point in the to-be-completed area.

[0146] In one embodiment, the area to be completed is a polygon;

[0147] The area determination module 1 is specifically configured to:

[0148] Determining an extension point on at least one edge line of the area to be completed, determining a plurality of vertices of the contraction area based on each of the extension points and a preset inward distance, and obtaining the contraction area according to the plurality of vertices;

[0149] Based on each of the extension points and a preset outward expansion distance, a center point of the expansion region corresponding to the extension point is determined, and the expansion region is obtained according to the center point of the expansion region.

[0150] In one embodiment, the apparatus further includes a region adjustment module, which is configured to:

[0151] Determining that the number of valid distance points within the contraction area is less than a preset first threshold, and re-determining the contraction area by adjusting the retraction distance;

[0152] It is determined that the number of valid distance points in the expansion area is less than a preset second threshold, and a plurality of expansion areas are redefined by adjusting the outward expansion distance.

[0153] In one embodiment, the information candidate module 2 is specifically configured to:

[0154] determining a first distance average value of the contraction area, and determining the first candidate distance value based on the first distance average value;

[0155] determining a second distance average value of the expansion area, and determining the second candidate distance value based on the second distance average value;

[0156] The first distance average value is obtained by averaging the distance values ​​of all effective distance points in the contraction area; and the second distance average value is obtained by averaging the distance values ​​of all effective distance points in the expansion area.

[0157] In one embodiment, the apparatus further includes a region matching module, which is configured to:

[0158] Determining the area to be completed from the depth image based on the two-dimensional coordinate information of the target recognition area in the color imaging image;

[0159] The coordinate system of the color imaging image and the coordinate system of the depth image are pre-registered with each other; and the target recognition area is the area where the object to be detected is located.

[0160] In one embodiment, the apparatus further comprises a target recognition module, which is configured to:

[0161] Inputting the color imaging image into a target recognition model to obtain a color imaging image marked with a target recognition area output by the target recognition model;

[0162] The target recognition model is obtained by training based on the sample color imaging image and the target recognition area results corresponding to the sample color imaging image.

[0163] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the following method:

[0164] S1. Determine a reference area based on the area to be completed in the depth image;

[0165] S2. determining candidate distance values ​​according to the distance values ​​of the valid distance points within the reference area;

[0166] S3, completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values;

[0167] The reference area is within the depth image; and the valid distance point is a point where no distance value is missing.

[0168] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology, or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0169] On the other hand, an embodiment of the present invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the methods provided in the above-mentioned method embodiments, for example, including:

[0170] S1. Determine a reference area based on the area to be completed in the depth image;

[0171] S2. determining candidate distance values ​​according to the distance values ​​of the valid distance points within the reference area;

[0172] S3, completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values;

[0173] The reference area is within the depth image; and the valid distance point is a point where no distance value is missing.

[0174] In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the transmission method provided in each of the above embodiments is implemented, for example, including:

[0175] S1. Determine a reference area based on the area to be completed in the depth image;

[0176] S2. determining candidate distance values ​​according to the distance values ​​of the valid distance points within the reference area;

[0177] S3, completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values;

[0178] The reference area is within the depth image; and the valid distance point is a point where no distance value is missing.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0180] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are intended to illustrate the present invention only and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and are intended to be encompassed by the claims of the present invention.

Claims

1. A distance information completion method, characterized in that: include: Determine a reference area based on the area to be completed in the depth image; determining a candidate distance value based on the distance values ​​of the valid distance points within the reference area; Completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values; The reference area is within the depth image; the valid distance point is a point where no distance value is missing; the reference area includes a contraction area and an expansion area; the contraction area and the expansion area are both within the depth image, and the contraction area is located inside the area to be completed, and the expansion area is located outside the area to be completed; the area to be completed is a polygon; The determining of the reference area based on the area to be completed in the depth image includes: Determining an extension point on at least one edge line of the area to be completed, determining a plurality of vertices of the contraction area based on each of the extension points and a preset inward distance, and obtaining the contraction area according to the plurality of vertices; Based on each of the extension points and a preset outward expansion distance, a center point of the expansion region corresponding to the extension point is determined, and the expansion region is obtained according to the center point of the expansion region.

2. The distance information completion method according to claim 1, characterized in that: The determining of candidate distance values ​​according to the distance values ​​of the valid distance points in the reference area includes: determining a first candidate distance value based on the distance values ​​of the effective distance points in the contraction region, and determining a second candidate distance value based on the distance values ​​of the effective distance points in the expansion region; The step of completing the distance values ​​of the distance-missing points in the area to be completed based on the candidate distance values ​​includes: Distance values ​​of distance-missing points in the to-be-completed area are completed based on the first candidate distance value and the second candidate distance value.

3. The distance information completion method according to claim 2, characterized in that The method of completing the distance value of the missing points in the to-be-completed area based on the first candidate distance value and the second candidate distance value includes: Determining that a difference between the first candidate distance value and the second candidate distance value is greater than a preset difference threshold, and using the first candidate distance value to complete the distance value of the distance-missing point in the area to be completed; Determine that the difference between the first candidate distance value and the second candidate distance value is not greater than the difference threshold, and use the average of the first candidate distance value and the second candidate distance value to complete the distance value of the distance missing point in the to-be-completed area.

4. The distance information completion method according to claim 2, characterized in that: After determining the reference area based on the to-be-completed area in the depth image, and before determining the first candidate distance value according to the distance values ​​of the effective distance points in the contracted area, and before determining the second candidate distance value according to the distance values ​​of the effective distance points in the expanded area, the method further includes: Determining that the number of valid distance points within the contraction area is less than a preset first threshold, and re-determining the contraction area by adjusting the retraction distance; It is determined that the number of valid distance points in the expansion area is less than a preset second threshold, and the expansion area is redefined by adjusting the outward expansion distance.

5. The distance information completion method according to claim 2, characterized in that The determining of the first candidate distance value according to the distance value of the effective distance point in the contraction area, and the determining of the second candidate distance value according to the distance value of the effective distance point in the expansion area, comprises: determining a first distance average value of the contraction area, and determining the first candidate distance value based on the first distance average value; determining a second distance average value of the expansion area, and determining the second candidate distance value based on the second distance average value; The first distance average value is obtained by averaging the distance values ​​of all effective distance points in the contraction area; and the second distance average value is obtained by averaging the distance values ​​of all effective distance points in the expansion area.

6. The distance information completion method according to claim 1, characterized in that: Before determining the reference area based on the area to be completed in the depth image, the method further includes: Determining the area to be completed from the depth image based on the two-dimensional coordinate information of the target recognition area in the color imaging image; The coordinate system of the color imaging image and the coordinate system of the depth image are pre-registered with each other; and the target recognition area is the area where the object to be detected is located.

7. The distance information completion method according to claim 6, characterized in that: Before determining the area to be completed from the depth image based on the two-dimensional coordinate information of the target recognition area of ​​the color imaging image, the method further includes: Inputting the color imaging image into a target recognition model to obtain a color imaging image marked with a target recognition area output by the target recognition model; The target recognition model is obtained by training based on the sample color imaging image and the target recognition area results corresponding to the sample color imaging image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the distance information completion method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distance information completion method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the distance information completion method according to any one of claims 1 to 7 is implemented.

Images