Depth map processing method and system, and three-dimensional image reconstruction method

Abstract

The invention provides a depth map processing method and system and a three-dimensional image reconstruction method. The depth map processing method comprises the following steps: traversing each pixel point of a depth map, and initializing a mask; respectively setting filtering window radiuses in the row direction and the column direction based on the resolution of the depth map; adopting expansion convolution to determine sampling points of a filtering window; judging the effectiveness of each sampling point in the filtering window, gradually reducing the calculation range of the adjacent points, determining pixel points corresponding to the pseudo edge point set according to the judgment of the adjacent points, and resetting the mask; and in combination with morphological operation, the depth map is processed, and a final image with accurate three-dimensional data is obtained. According to the error information of the depth map data output by the structured light camera and the line spectrum confocal sensor, the depth map processing method is provided, so that the three-dimensional surface reconstructed by the output final image can output accurate three-dimensional data.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a depth map processing method and system, and a three-dimensional image reconstruction method. Background Technology

[0002] With the development of portable and affordable depth cameras, the fundamental research and application of depth images in the field of image processing have been greatly enhanced. It is becoming increasingly important. By applying information from depth images, the performance of related research and applications in the field of machine vision can be improved, such as image segmentation, object tracking, image recognition, and image reconstruction.

[0003] Due to factors such as varying reflectivities on the surface of the object under test or stray light in the application scenario, abnormal pseudo-edge point sets may appear in the measurement imaging. For 3D surfaces reconstructed using the brightest peak reconstruction strategy, obvious abnormal pseudo-edges appear, forming upward or downward spike-like surfaces far from the original surface. This interferes with the application of depth information from the depth image. Therefore, to obtain more accurate information, depth enhancement processing is needed to remove abnormal pseudo-edge point sets from the depth image.

[0004] Chinese patent CN 109242782 B proposes a noise processing method and apparatus. This patent targets a TOF camera, selecting pixels from an array and calculating the relationship between the pixel and the remaining pixels in the array based on the pixel's corresponding depth information. The discreteness of pixels is determined, and when at least one discreteness of a pixel exceeds a preset threshold, the pixel is identified as noise and removed from the area array. This achieves the identification and removal of noise in the area array, thus solving the technical problem that the generated noise will affect the accuracy of the interference depth information.

[0005] Chinese patent CN 111242861 B proposes a method, apparatus, electronic device, and storage medium for removing stray light from a TOF camera. This patent calculates a confidence map from the grayscale image acquired by the TOF camera; instead of performing stray light suppression processing on the original grayscale image, the confidence map is processed to suppress stray light. In principle, this assimilates the influence of stray light into the influence of background noise, facilitating seamless integration with subsequent depth calculations.

[0006] The aforementioned patent is based on the imaging principle of TOF cameras. It solves the problem of edge distortion of depth maps caused by the contamination of real signals by indirect path signals by separating or suppressing indirect path signals.

[0007] Different depth maps are acquired using different imaging principles. The false edge point set problem caused by the reconstruction of depth maps acquired by line-spectrum confocal sensors and structured light cameras is due to light or signals "running around" in space, generating new geometry at incorrect locations, or affecting weaker nearby signals, leading to imaging errors. Therefore, the solution to the false edge problem proposed for TOF cameras is not applicable to depth maps acquired by line-spectrum confocal sensors and structured light cameras.

[0008] Chinese patent CN 116523768 A proposes a denoising method for depth images. This patent treats both noise and edges as abrupt changes, using the number of abrupt changes in adjacent pixel directions as confidence weights. This achieves good noise filtering while preserving more texture details and other information. This patent's technical approach differs from ours.

[0009] Chinese patent CN 118096809 A discloses a star map preprocessing method combining local thresholding and morphological processing. This patent divides an image captured by a visible light camera into regions, calculates the mean and variance of grayscale values ​​for each region, and sets a threshold for each region based on the mean and variance. The image is then segmented according to the threshold. Morphological processing is then performed on the thresholded image to eliminate isolated noise points. Finally, the processed image is fused with the original image to complete the star map preprocessing, effectively eliminating the influence of stray light and noise points on the image. The patent addresses stray visible light, focusing on identifying and removing or reducing unwanted brightness and color information in the image. However, the depth map obtained by structured light and line spectral confocal sensors is a grayscale image, where pixel values ​​represent depth information, not brightness information. Therefore, it is impossible to use mean and variance calculations to filter out "isolated noise points" (which are not the same as isolated noise in visible light images) in the depth map, and to identify and correct erroneous depth measurements caused by unwanted light. In other words, the technical solution proposed in this patent cannot eliminate the set of false edge points in the depth map. Summary of the Invention

[0010] This invention proposes a depth map processing method and system, and a three-dimensional image reconstruction method, which solves at least one of the above-mentioned technical problems.

[0011] To achieve the above objectives, the present invention proposes the following technical solution: A depth map processing method, comprising: The depth map is output by any one of a line structured light camera, a surface structured light camera, or a line spectral confocal sensor, and includes: Iterate through each pixel in the depth map. If the pixel value of the pixel is less than the height threshold, then the mask of the pixel is set to 1; otherwise, the mask of the pixel is set to 0. Based on the resolution of the depth map, the radii of the filtering windows in the row and column directions are set respectively; based on the radii of the filtering windows in the row and column directions, the dilation rate of the dilated convolution is set, and the number of sampling points in the filtering window is determined. Iterate through the pixel values ​​of each sampling point in the filtering window. If the pixel value of the sampling point falls into the invalid pixel range, it indicates that the sampling point is invalid. Then, remove the invalid points and use the sampling points that do not fall into the invalid pixel range as the neighboring point calculation range. In the neighboring point calculation range, calculate the neighboring points and the number of neighboring points for each sampling point. If the number of neighboring points of the sampling point is less than the number threshold, the mask of the sampling point is set to 1; otherwise, the mask of the sampling point is set to 0. Based on a mask, morphological closing operations are used to process the depth map and obtain the processed image. Based on the processed image, pixels with a mask value of 1 are set as invalid points, and the final image is output.

[0012] Furthermore, before determining the range of the filtering window, If a corresponding confidence map exists for the depth map, the mask of the pixel with a confidence level less than the confidence threshold in the confidence map is set to 1; otherwise, the mask of the pixel is set to 0.

[0013] Furthermore, using the brightness map as the confidence map, the mask of pixels with gray values ​​less than the gray value threshold is set to 1; otherwise, the mask of the pixel is set to 0.

[0014] Furthermore, before obtaining the calculation range of neighboring points, the method also includes: if the difference between the maximum and minimum values ​​of the sampled pixel values ​​in any filtering window is greater than the height threshold, then the relationship between each sampled point in the filtering window and the preset threshold is determined; otherwise, the filtering window is skipped.

[0015] Furthermore, before calculating the nearest neighbor points, the following steps are also included: within the nearest neighbor point calculation range of the current filtering window, if the pixel value of the current sampling point is greater than the direction threshold, it indicates that the sampling point is a pixel with the peak direction upward; if the pixel value of the current sampling point is less than the direction threshold, it indicates that the sampling point is a pixel with the peak direction downward; if only peaks in a single direction are removed, the sampling point in the corresponding direction is selected for nearest neighbor point calculation.

[0016] Furthermore, it also includes: Within the neighbor point calculation range of the current filtering window, the number of sampling points with pixel values ​​greater than the direction threshold and the number of sampling points with pixel values ​​less than the direction threshold are counted respectively. If the number of sampling points in any direction is greater than the direction number threshold, it indicates that there is a peak in that direction, so the neighbor points of the sampling points in that direction are calculated. Otherwise, there is no peak in that direction, and the neighbor point calculation of the sampling points in that direction is skipped. If only peaks in a single direction are removed, the sampling points in the corresponding direction are selected for neighbor point calculation.

[0017] Furthermore, neighboring points include: Within the neighboring point calculation range, if the pixel value difference between the current sampling point and each sampling point is not greater than the height threshold, then all sampling points within the neighboring point calculation range are determined to be neighboring points of the current sampling point.

[0018] Based on the same inventive concept, a depth map processing system, wherein the depth map is output by any one of a line structured light camera, a surface structured light camera, and a line spectral confocal sensor, includes: The initialization module iterates through each pixel in the depth map. If the pixel value of a pixel is less than the height threshold, the mask of that pixel is set to 1; otherwise, the mask of that pixel is set to 0. The sampling module sets the filter window radii in the row and column directions based on the resolution of the depth map; based on the filter window radii in the row and column directions, it sets the dilation rate of the dilated convolution and determines the number of sampling points in the filter window. The window calculation module iterates through the pixel values ​​of each sampling point in the filtering window. If the pixel value of a sampling point falls into the invalid pixel range, it indicates that the sampling point is invalid. Invalid points are then removed, and sampling points that do not fall into the invalid pixel range are used as the neighboring point calculation range. Within the neighboring point calculation range, the neighboring points and the number of neighboring points for each sampling point are calculated. If the number of neighboring points for a sampling point is less than the number threshold, the mask for that sampling point is set to 1; otherwise, the mask for that sampling point is set to 0. Invalid points are pixels whose pixel values ​​in the depth map fall within the invalid pixel range; The morphology module, based on a mask, uses morphological closing operations to process the depth map and obtain a processed image; based on the processed image, pixels with a mask value of 1 are set as invalid points, and the final image is output.

[0019] On the other hand, the present invention also proposes a three-dimensional image reconstruction method, comprising: Based on the depth map processing method described above, the final image with abnormal spikes removed is obtained; The spatial coordinates of each pixel in the final image are calculated based on back projection to obtain a 3D image.

[0020] Based on the same inventive concept, this application also proposes an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the method described above.

[0021] On the other hand, the present invention also proposes a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the method described above.

[0022] The beneficial effects of the embodiments of the present invention are as follows: This embodiment proposes a depth map processing method based on erroneous information in the depth map data output by a structured light camera and a line-spectral confocal sensor. The method initializes a mask by traversing each pixel of the depth map; sets the filtering window radii in the row and column directions based on the depth map resolution; uses dilated convolution to determine the sampling points of the filtering window; determines the validity of each sampling point in the filtering window, gradually narrowing the calculation range of neighboring points; determines the pixels corresponding to the pseudo-edge point set based on the neighboring point determination; and resets the mask; combines morphological operations to process the depth map and obtain an accurate final image of the 3D data. This allows the reconstructed 3D surface based on the output final image to output accurate 3D data for subsequent applications.

[0023] The depth map processing method proposed in this embodiment combines confidence images and local maximum height difference information to avoid deleting effective information in high-confidence areas and flat areas, while avoiding subsequent complex calculations, thus accelerating the execution time of the entire algorithm and effectively solving the problem that existing technologies cannot quickly process large-area pseudo-edge regions.

[0024] The depth map processing method proposed in this embodiment adopts the idea of ​​dilated convolution to perform interval sampling processing on the data of large windows. At the same time, it combines masking and morphological processing to eliminate the mesh effect. This maintains a large receptive field to process large areas of pseudo-edge regions, while reducing the amount of computation, accelerating the algorithm's running time, and improving computational efficiency.

[0025] In the depth map processing method proposed in this embodiment, the current region is determined to be an upper peak or a lower peak by comparing the height of the current point with the average height of the neighborhood. The method can flexibly remove pseudo-edge regions by specifying the filtering type, which is flexible. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the depth map processing method in this invention; Figure 2 This is the depth map output by the linear spectral confocal sensor in this invention; Figure 3 yes Figure 2 3D reconstruction image; Figure 4 yes Figure 2 The calculated 3D reconstruction image. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0028] like Figure 1 As shown, this application proposes a depth map processing method, wherein the depth map is output by any one of a line structured light camera, a surface structured light camera, and a line spectral confocal sensor, including: Iterate through each pixel in the depth map. If the pixel value of the pixel is less than the height threshold, then the mask of the pixel is set to 1; otherwise, the mask of the pixel is set to 0. Based on the resolution of the depth map, the radii of the filtering windows in the row and column directions are set respectively; based on the radii of the filtering windows in the row and column directions, the dilation rate of the dilated convolution is set, and the number of sampling points in the filtering window is determined. Iterate through the pixel values ​​of each sampling point in the filtering window. If the pixel value of the sampling point falls into the invalid pixel range, it indicates that the sampling point is invalid. Then, remove the invalid points and use the sampling points that do not fall into the invalid pixel range as the neighboring point calculation range. In the neighboring point calculation range, calculate the neighboring points and the number of neighboring points for each sampling point. If the number of neighboring points of the sampling point is less than the number threshold, the mask of the sampling point is set to 1; otherwise, the mask of the sampling point is set to 0. Based on a mask, morphological closing operations are used to process the depth map and obtain the processed image. Based on the processed image, pixels with a mask value of 1 are set as invalid points, and the final image is output. Invalid points are pixels whose pixel values ​​in the depth map fall within the invalid pixel range.

[0029] Stray light: refers to all non-imaging rays in an optical system that do not follow the predetermined optical path to reach the imaging device. It may come from reflections or scattering within the system, or from interference from external ambient light.

[0030] False edge point sets refer to edges, contours, or floating "thin-skin" structures in 3D data that do not exist in the real physical world. These point sets typically appear as isolated lines or surfaces, or as 3D data attached to the edges of real objects but in the wrong position, severely interfering with subsequent data processing for identification, measurement, and reconstruction.

[0031] This application addresses the issue that in line spectral confocal sensors, the signal on the actual sample surface is weak due to factors such as the tilt angle of the test sample surface or the presence of various reflectivities on the sample surface. The stray light signal is relatively strong due to the influence of the reflected light spots from adjacent flat areas, causing the line center extraction position to be located at the stray light position. Therefore, in the three-dimensional surface reconstructed based on the brightest peak reconstruction strategy, a significantly abnormal pseudo-edge appears, forming a "pseudo-edge" with an incorrect position and a highly abrupt change; it looks like a small step or burr superimposed on the real edge (i.e., forming an upward or downward peak-shaped surface away from the original surface).

[0032] This application addresses the problem that (line / surface) structured light cameras, due to factors such as the tilt angle of the test sample surface or the measurement of areas with sharp object edges, result in a weak signal on the actual sample surface. This is further compounded by the influence of strong reflected light spots from adjacent flat areas, leading to relatively strong stray light signals. Consequently, the extraction position at the center of the line is located at the stray light position, resulting in a reconstructed 3D surface with obvious abnormal pseudo-edges, forming upward or downward peak-shaped surfaces far from the original surface.

[0033] Given that line spectral confocal sensors and (line / surface) structured light cameras are affected by the test sample surface or the test environment, the 3D surface reconstructed based on their output depth map may exhibit obvious abnormal pseudo edges, forming upward or downward spike-shaped surfaces far from the original surface.

[0034] False edges may have one or more of the following characteristics: they are always attached to real high-contrast physical edges; they manifest as "jitter," "burrs," or "false steps" at the edge position; the magnitude (height deviation) of the false edge is positively correlated with the difference in reflectivity of the surface material of the test sample, and the greater the difference, the more obvious the false edge.

[0035] To address the aforementioned technical problems, this embodiment proposes a depth map processing method, wherein the depth map is output by any one of a line structured light camera, a surface structured light camera, and a line spectral confocal sensor, and includes: Iterate through each pixel in the depth map. If the pixel value of the pixel is less than the height threshold, then the mask of the pixel is set to 1; otherwise, the mask of the pixel is set to 0.

[0036] Preferably, if the depth map has a corresponding confidence map, the mask of the pixel with a confidence level less than the confidence threshold in the confidence map is set to 1; otherwise, the mask of the pixel is set to 0.

[0037] The confidence map is an image generated by the (line / area) structured light camera along with the depth map. The higher the confidence value of a pixel, the higher the confidence of the depth test of the current pixel.

[0038] Preferably, the brightness map is used as the confidence map, and the mask of the pixel whose gray value is less than the gray value threshold is set to 1; otherwise, the mask of the pixel is set to 0.

[0039] If the depth map is output by a line spectral confocal sensor, then the brightness map is used as the confidence map. The larger the gray value of a pixel, the stronger the signal, and the higher the probability that it is a real signal rather than stray light, which indicates the reliability of the current test depth.

[0040] In this embodiment, it is proposed to use one image as the basis for filtering the target region. In practical applications, it is sufficient to select one image as the basis for filtering, and try to avoid using multiple images to avoid increasing the error in subsequent calculations due to non-unique filtering criteria.

[0041] This embodiment uses a spatial selector (mask) to mark the target area with pixel values. It traverses each pixel in the depth map or confidence map (or brightness map) to initialize the mask, thereby reducing the amount of subsequent calculations and improving computational efficiency.

[0042] A mask is typically a digital image or matrix of the same size as the target region. In this embodiment, the target region is a full depth map.

[0043] This embodiment uses a binary mask, which marks each pixel in the depth map with two pixel values ​​to form a digital image of the same size as the depth map. In this embodiment, 0 and 1 are used to mark the pixels respectively, with 1 indicating the target pixel (to be processed later) and 0 indicating the background / excluded pixel (no processing, retaining the original data).

[0044] Alternatively, 0 and 255 can be used as markers, or other pixel values ​​can be set for markers.

[0045] Based on the resolution of the depth map, the radii of the filtering windows in the row and column directions are set respectively; based on the radii of the filtering windows in the row and column directions, the dilation rate of the dilated convolution is set, and the number of sampling points in the filtering window is determined.

[0046] Since it is necessary to filter out sharp peak-like regions of a certain area, a relatively large filtering window is required. However, calculating a large filtering window for each pixel in the entire image is very time-consuming. Therefore, this embodiment introduces the idea of ​​dilated convolution to maintain a large receptive field while reducing the amount of computation, thereby improving computational efficiency.

[0047] Dilated Convolution, also known as Atrous Convolution or Expanded Convolution, is a deep learning technique that improves traditional convolution operations. By introducing the dilation rate, holes are inserted between the elements of the convolution kernel, significantly expanding the receptive field without increasing the number of parameters and computational complexity, allowing each convolution output to contain information in a larger range; at the same time, it can ensure that the size of the output feature map remains unchanged.

[0048] For example, a 3×3 convolution kernel with a dilation rate of 2 has the same receptive field as a 5×5 convolution kernel, but the number of parameters is only 9, which is 36% of the number of parameters of a 5×5 convolution.

[0049] The receptive field is the size that the convolution kernel sees on the image. For example, the receptive field size of a 3x3 convolution kernel is 9.

[0050] In addition, in this embodiment, a filtering window radius based on the row and column directions is proposed to adaptively adjust the dilation rate of the dilated convolution to select a filtering window of an appropriate size.

[0051] It is known that there are H pixel points in the row direction and W pixel points in the column direction of the depth map; the radius r of the filtering window includes that the radius of the filtering window in the row direction is r m (r m <H), and the radius of the filtering window in the column direction is r n (r n <W). If the radii of the filtering windows in the row and column directions are r, the diameter of the filtering window is 2r + 1. If the radii of the filtering windows in the row and column directions are different, the diameter of the filtering window in the row direction is 2r m + 1; the diameter of the filtering window in the column direction is 2r n + 1.

[0052] If the radius r of the filtering window is not greater than the first radius a, the dilation rate d is set to 1; if the radius of the filtering window is less than the second radius b and greater than the first radius, the dilation rate is set to 2; if the radius of the filtering window is not less than the second radius, the dilation rate is set to 3. Here, the second radius is greater than the first radius (b > a), and b < H, b < W.

[0053] Preferably, the specific values of the first radius and the second radius are related to the resolution of the depth map and also depend on the current computing precision requirements and hardware conditions. For example, the first radius a = 10 and the second radius b = 20 can be set; or a = 15 and b = 25 can be set, etc.

[0054] Preferably, if the radii of the filtering windows in the row and column directions are different, the determination conditions for the dilation rate are further set according to the resolution of the depth map.

[0055] For example, if r m <a, r n <a, then d = 1; if a ≤ r m ≤ b, a ≤ r n ≤ b, then d = 2; if r m > b, r n > b, then d = 3.

[0056] In addition, the judgment conditions in the row and column directions can be set respectively according to the different numbers of pixels in the rows and columns of the depth map to determine the sampling range of the filtering window.

[0057] For example, if r m <a1, r n <a2, then d = 1; if a1 ≤ r m ≤ b1, a2 ≤ r n ≤ b2, then d = 2; if r m > b1, r n > b2, then d = 3. Here, a1 < b1, a2 < b2; a1 and b1 are the judgment parameters for the radius of the filtering window in the row direction, and a2 and b2 are the judgment parameters for the radius of the filtering window in the column direction. Similarly, both a1 and b1 are less than H, and both a2 and b2 are less than W.

[0058] In this embodiment, the pixel coordinates (i, j) of the depth map are known. Dilated convolution is used to sample the pixel points (i, j) within the filtering window, and the sampling point coordinates are (i - r m + m × d, j - r n + n × d); where, (m, n) are the coordinates of the pixel point in the filtering window, and the value ranges of m and n are [0, floor(2r / d)+1], where the function floor(·) represents rounding down.

[0059] The pixel value of the known depth image pixel point (i, j) is z(i, j); The pixel value of any sampling point (m, n) within the filtering window is w(m, n) = z((i - r m + m × d, j - r n + n × d).

[0060] Based on the above process, to determine the size of the filtering window and the sampling range of the dilated convolution, several filtering windows and several sampling points in each filtering window can be obtained.

[0061] Preferably, in order to accelerate the calculation speed of obtaining the filtering window, the filtering window can also be screened to narrow the calculation range.

[0062] Before obtaining the calculation range of neighboring points, the method also includes: if the difference between the maximum and minimum values ​​of the sampled pixel values ​​in any filtering window is greater than the height threshold, then the relationship between each sampled point in the filtering window and the preset threshold is determined; otherwise, the filtering window is skipped.

[0063] Based on practical experience, the height threshold for the difference in pixel values ​​of the sampling points in the filtering window is generally the same as the height threshold for the mask judgment of each pixel in the previous depth map.

[0064] Preferably, other values ​​can be set as the criteria for whether the filter window can be skipped.

[0065] Since the abnormal pseudo-edge point set in the 3D image does not have a real edge, the pixel value of the peak region corresponding to the pseudo-edge point set is extremely different from the normal pixel value in the depth map.

[0066] In this embodiment, a decision to skip the filtering window is added. The difference between the maximum and minimum pixel values ​​in the filtering window (the difference in pixel values ​​of the sampled points in the current filtering window) is used to further determine whether there are spikes in the pixel region corresponding to the current filtering window. If the difference between the maximum and minimum pixel values ​​of the sampled points in the current filtering window is less than the height threshold, the current filtering window can be considered a flat region, and the current filtering window is skipped without further processing of the sampled points in the current filtering window.

[0067] Preferably, the difference between the pixel value of each sampling point in the current filtering window and the specified sampling point can be calculated as a judgment condition for the filtering window; or the difference between the pixel value of each sampling point and the empirical value can be calculated directly using empirical values ​​as a judgment condition.

[0068] In the above process of judging the filtering window, directly judging whether there is a spike in the current filtering window based on two pixels may result in randomness. Therefore, the mean value of the sampling points of the current filtering window can be used as the comparison object for the pixel value of each sampling point to improve the robustness of the judgment result.

[0069] Preferably, in addition to simply judging based on the difference between the maximum and minimum pixel values ​​of the sampling points in the filtering window, the average pixel value of the sampling points in the current filtering window can also be used as the judgment condition. That is, it is judged whether the difference between any sampling point in the current filtering window and the average pixel value of the sampling points in the current filtering window is greater than the height threshold. If it is greater than the height threshold, the current filtering window is the window to be processed; otherwise, the current filtering window is skipped.

[0070] Based on practical experience, the height threshold for the difference in pixel values ​​at the sampling points of the filtering window is generally the same as the height threshold for the mask judgment of each pixel in the previous depth map. Preferably, other values ​​can also be set as the judgment condition for whether the filtering window can be skipped.

[0071] Preferably, the average pixel value of the sampling points in the current filtering window can be replaced with the average pixel value of all pixels in the current filtering window to avoid calculation errors caused by sampling.

[0072] In the above-mentioned filtering window judgment process, if the difference between the pixel value of any sampling point in the filtering window and the judgment condition (including the difference between the maximum and minimum values, the difference between any sampling point and the average pixel value of the filtering window (the average pixel value of the sampling points in the filtering window or the average of all pixels in the filtering window), and the difference between any sampling point and a specified pixel value (or empirical value)) is greater than the height threshold, it is judged that the current filtering window has a spike. The judgment result may be random. In order to further increase the robustness of the judgment, the number of sampling points in the current filtering window that are greater than the height threshold can be counted based on the judged pixel value.

[0073] Preferably, in any filtering window, if the difference between the pixel value of multiple sampling points and the judgment condition is greater than the height threshold, and the number of sampling points in the current filtering window that are greater than the height threshold is greater than the number threshold, then the current filtering window is a window to be processed; otherwise, the current filtering window is skipped.

[0074] The quantity threshold is set based on practical experience and depends on factors such as hardware computing conditions, usage environment, and the material of the object under test.

[0075] Due to the influence of the application environment and factors such as the surface reflectivity and material of the object under test, existing image acquisition equipment (the limitations of image acquisition equipment are also one of the influencing factors) will have invalid points in the depth map, which are usually manifested as black pixels, specific nano values ​​or abnormal values.

[0076] Generally, the pixel value of an invalid point differs significantly from that of a normal point (i.e., a non-invalid point), and the pixel value of an invalid point is much smaller than that of a normal point. However, there may be a very small number of invalid points whose pixel values ​​are significantly larger than those of normal points.

[0077] Preferably, invalid points can be filtered out by setting a preset threshold to remove their influence. If invalid points are not filtered out, normal points near invalid points are easily identified as false edges in the subsequent calculation of neighboring points, affecting the final processing effect.

[0078] The preset threshold is set based on the current device parameters, the application environment, and the material of the object being measured. Alternatively, it can be set based on practical experience, or according to the measurement height or the depth of the object being measured.

[0079] To further determine the validity of the sampling points and improve the accuracy of the processing, this embodiment further filters the sampling points to remove invalid points. Invalid points are pixels whose pixel values ​​fall within invalid ranges (including invalid pixel ranges, invalid grayscale ranges, and invalid confidence ranges).

[0080] In this embodiment, invalid points are pixels whose pixel values ​​in the depth map fall within the invalid pixel range.

[0081] Preferably, a confidence map or a brightness map can also be used as the screening criterion for invalid points. That is, an invalid confidence range is set in the confidence map, and any pixel whose confidence falls within the invalid confidence range is considered an invalid point.

[0082] In the brightness map, any pixel whose grayscale image falls within the invalid grayscale range is considered an invalid pixel.

[0083] Preferably, only one of the depth map, confidence map, and brightness map is used as the screening criterion to avoid calculation errors caused by cross-use.

[0084] Iterate through the pixel values ​​of each sampling point in the filtering window. If the pixel value of the sampling point falls into the invalid pixel range, it indicates that the sampling point is invalid. Then, remove the invalid point and use the sampling points that do not fall into the invalid pixel range as the neighboring points to calculate the range.

[0085] In this embodiment, invalid points are removed by resetting their pixel values.

[0086]

[0087] In the formula, w(m,n) represents the sampling point in the filtering window; when the pixel value w(m,n) of the sampling point is determined to fall into the invalid range (nan), that is, when calculating the neighboring points of a certain sampling point, the pixel value of the invalid point is made to be consistent with the pixel value z(i,j) of the sampling point, so as to avoid the invalid point being identified as the neighboring point of the sampling point.

[0088] Preferably, another way to remove invalid points is to directly set the pixel (i.e., invalid point) to 1 in the mask.

[0089] After the above calculations, we can obtain a number of sampling points in the current filtering window after excluding invalid points, which can be used as the calculation range for neighboring points.

[0090] In addition, there may be a need to filter out spikes in specific directions (upward or downward). Before calculating neighboring points, we can determine the sampling points with different spike directions to reduce the amount of neighboring point calculation.

[0091] Specifically, this includes: within the neighbor point calculation range of the current filtering window, if the pixel value of the current sampling point is greater than the direction threshold, it indicates that the sampling point is a pixel with the peak direction upward; if the pixel value of the current sampling point is less than the direction threshold, it indicates that the sampling point is a pixel with the peak direction downward; if only peaks in a single direction are removed, the sampling point in the corresponding direction is selected for neighbor point calculation.

[0092] Preferably, the direction threshold can be the average pixel value of the current filtering window sampling points, or any specified value (or empirical value), or the average pixel value of the current filtering window sampling points after removing invalid points. This embodiment does not specifically limit the setting method and value of the direction threshold; it can be determined according to the actual situation.

[0093] Preferably, if only pixel value is used as the judgment condition, false judgments may occur. To further improve the accuracy of peak detection, the number of pixels can also be considered, as follows: Within the neighbor point calculation range of the current filtering window, the number of sampling points with pixel values ​​greater than the direction threshold and the number of sampling points with pixel values ​​less than the direction threshold are counted respectively. If the number of sampling points in any direction is greater than the direction number threshold, it indicates that there is a peak in that direction, so the neighbor points of the sampling points in that direction are calculated. Otherwise, there is no peak in that direction, and the neighbor point calculation of the sampling points in that direction is skipped. If only peaks in a single direction are removed, the sampling points in the corresponding direction are selected for neighbor point calculation.

[0094] If there is no need to filter out spikes in a specific direction, the above spike direction determination process can be skipped.

[0095] After filtering out or not filtering out directions in the peak direction, the neighboring points of each sampling point are calculated based on the range of neighboring points determined by the screening, and the number of neighboring points is counted to determine the valid points.

[0096] Preferably, the method for calculating neighboring points includes: within the neighboring point calculation range, if the difference (|w(m,n)-z(i,j)|) between the pixel value w(m,n) of the current sampling point and the pixel value z(i,j) of each sampling point is not greater than the height threshold thr, then all sampling points within the neighboring point calculation range are determined to be neighboring points of the current sampling point.

[0097]

[0098] In the formula, count(·) represents the counting function for neighboring points, used to count the number of neighboring points for each sampling point; isNear(·) represents the determination function for neighboring points.

[0099] According to the neighbor point calculation method, if the difference between the pixel value w(m,n) of the current sampling point and the pixel value z(i,j) of each sampling point (|w(m,n)-z(i,j)|) is not greater than the height threshold thr, then the sampling point is identified as a neighbor point of the current sampling point and counted as 1; otherwise, the sampling point is not a neighbor point of the current sampling point and counted as 0.

[0100] According to the determination function proposed in this embodiment, based on the above-mentioned invalid point determination conditions, if the sampling point is invalid, the pixel value of the invalid point is reset to the pixel value of the current sampling point, and the difference between the two is zero, that is, the invalid point is not a neighboring point of the current sampling point, so as to avoid misjudging the invalid point as a neighboring point and reduce calculation error.

[0101] Preferably, if other methods for calculating neighboring points are set, invalid points should be effectively excluded from the neighboring point filtering criteria to reduce calculation errors.

[0102] Preferably, other nearest neighbor calculation methods can also be used. For example, Chinese Patent CN111275810A discloses a K-nearest neighbor point cloud filtering method, apparatus, and storage medium based on image processing. This patent proposes to select several points in a two-dimensional image using a K-nearest neighbor filtering template (which can be regarded as several sampling points within the nearest neighbor calculation range in this embodiment), calculate the Euclidean distance between the center point and the K-nearest neighbor points, and determine whether the center point is a noise point based on the Euclidean distance and a pre-set distance threshold and proportion threshold. For center points that meet the distance threshold and proportion threshold, the number of K-nearest neighbor points in the template is counted based on a quantity threshold. Center points of the template with a quantity less than the quantity threshold are set as outlier noise points. By setting the quantity threshold, spatial restrictions are achieved, effectively removing outliers, scattered, and elongated noise point clouds, and improving the accuracy and applicability of filtering.

[0103] Alternatively, a point cloud data denoising method and system, as proposed in Chinese Patent CN 106548520 A, can be used. This method employs the Euclidean distance calculation formula from that patent to calculate the neighboring points of each sampling point and count the number of neighboring points.

[0104] Within the range of neighbor point calculation, calculate the neighbor points and the number of neighbor points for each sampling point; if the number of neighbor points count(i,j) of the sampling point is less than the number threshold nums, then the mask mask(i,j) of the sampling point is set to 1; otherwise, the mask of the sampling point is set to 0.

[0105]

[0106] Based on the above calculation process, the above calculation is repeated for each filtering window or filtering window that meets the conditions in the depth map to complete the determination of all sampling points and obtain a mask of the same size as the entire depth map. In this embodiment, a mask value of 1 is determined to be a set of pseudo edge points, and a mask value of 0 is determined to be a valid point (normal point).

[0107] Based on a mask, morphological closing operations are used to process the depth map and obtain the processed image.

[0108] The mask obtained by the above process may have valid point determination in the center of the peak region, and the mask will have a mesh effect due to dilation convolution. Therefore, further morphological processing is required: mask=close(mask).

[0109] Morphological closing operations can remove holes inside the mask and in the mesh, thus smoothing the processing results (image processing).

[0110] Based on the processed image, pixels with a mask value of 1 are set as invalid points, and the final image is output.

[0111]

[0112] In the formula, z res (i,j) represents the pixel value of each pixel in the final image.

[0113] like Figure 2 As shown in the figure, this embodiment takes the depth map output by the line spectrum confocal sensor as an example. It can be seen that there are pixel data in the depth map that are far away from the actual physical location.

[0114] Three-dimensional reconstruction is performed based on the depth map output by the line-spectral confocal sensor. Figure 3 As shown), the final image is reconstructed in three dimensions. Figure 4 As shown in the figure, the depth map processing method proposed in this application can effectively remove the false edge problem in the three-dimensional image by using a three-dimensional reconstruction map.

[0115] Based on the same inventive concept, this application also proposes a depth map processing system, wherein the depth map is output by any one of a line structured light camera, a surface structured light camera, and a line spectral confocal sensor, comprising: The initialization module iterates through each pixel in the depth map. If the pixel value of a pixel is less than the height threshold, the mask of that pixel is set to 1; otherwise, the mask of that pixel is set to 0. The sampling module sets the filter window radii in the row and column directions based on the resolution of the depth map; based on the filter window radii in the row and column directions, it sets the dilation rate of the dilated convolution and determines the number of sampling points in the filter window. The window calculation module iterates through the pixel values ​​of each sampling point in the filtering window. If the pixel value of a sampling point falls into the invalid pixel range, it indicates that the sampling point is invalid. Invalid points are then removed, and sampling points that do not fall into the invalid pixel range are used as the neighboring point calculation range. Within the neighboring point calculation range, the neighboring points and the number of neighboring points for each sampling point are calculated. If the number of neighboring points for a sampling point is less than the number threshold, the mask for that sampling point is set to 1; otherwise, the mask for that sampling point is set to 0. Invalid points are pixels whose pixel values ​​in the depth map fall within the invalid pixel range; The morphology module, based on a mask, uses morphological closing operations to process the depth map and obtain the processed image. Based on the processed image, pixels with a mask value of 1 are set as invalid points, and the final image is output.

[0116] Based on the same inventive concept, this application also proposes a three-dimensional image reconstruction method, which obtains a final image with abnormal spikes removed according to the above-mentioned depth map processing method. The spatial coordinates of each pixel in the final image are calculated based on back projection to obtain a 3D image.

[0117] Based on the same inventive concept, this application also proposes an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the method described above.

[0118] On the other hand, the present invention also proposes a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the method described above.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A depth map processing method, characterized in that, The depth map is output by any one of a line structured light camera, a surface structured light camera, or a line spectral confocal sensor, and includes: Iterate through each pixel in the depth map. If the pixel value of the pixel is less than the height threshold, then the mask of the pixel is set to 1; otherwise, the mask of the pixel is set to 0. Based on the resolution of the depth map, the radii of the filtering windows in the row and column directions are set respectively; based on the radii of the filtering windows in the row and column directions, the dilation rate of the dilated convolution is set, and the number of sampling points in the filtering window is determined. Iterate through the pixel values ​​of each sampling point in the filtering window. If the pixel value of the sampling point falls into the invalid pixel range, it indicates that the sampling point is invalid. Then, remove the invalid points and use the sampling points that do not fall into the invalid pixel range as the neighboring point calculation range. In the neighboring point calculation range, calculate the neighboring points and the number of neighboring points for each sampling point. If the number of neighboring points of the sampling point is less than the number threshold, the mask of the sampling point is set to 1; otherwise, the mask of the sampling point is set to 0. Based on a mask, morphological closing operations are used to process the depth map and obtain the processed image. Based on the processed image, pixels with a mask value of 1 are set as invalid points, and the final image is output.

2. The depth map processing method according to claim 1, characterized in that, Before determining the range of the filtering window If a corresponding confidence map exists for the depth map, the mask of the pixel with a confidence level less than the confidence threshold in the confidence map is set to 1; otherwise, the mask of the pixel is set to 0.

3. The depth map processing method according to claim 2, characterized in that, Using the brightness map as the confidence map, the mask of pixels with gray values ​​less than the gray threshold is set to 1. Conversely, the mask for that pixel is set to 0.

4. The depth map processing method according to any one of claims 1, characterized in that, Before obtaining the calculation range of neighboring points, the method also includes: if the difference between the maximum and minimum values ​​of the sampled pixel values ​​in any filtering window is greater than the height threshold, then the relationship between each sampled point in the filtering window and the preset threshold is determined; otherwise, the filtering window is skipped.

5. The depth map processing method according to any one of claims 1, characterized in that, Before calculating neighboring points, the following steps are also included: within the neighboring point calculation range of the current filtering window, if the pixel value of the current sampling point is greater than the direction threshold, it indicates that the sampling point is a pixel with the peak direction upward; if the pixel value of the current sampling point is less than the direction threshold, it indicates that the sampling point is a pixel with the peak direction downward; if only peaks in a single direction are removed, sampling points in the corresponding direction are selected for neighboring point calculation.

6. The depth map processing method according to claim 5, characterized in that, Also includes: Within the neighboring point calculation range of the current filtering window, count the number of sampling points with pixel values ​​greater than the direction threshold and the number of sampling points with pixel values ​​less than the direction threshold, respectively. If the number of sampling points in any direction is greater than the direction number threshold, it indicates that there is a peak in that direction. Then, the neighboring points of the sampling points in that direction are calculated. Conversely, if there is no spike in that direction, skip the calculation of neighboring points of the sampling point in that direction; if only spikes in a single direction are removed, select the sampling point in the corresponding direction for neighboring point calculation.

7. The depth map processing method according to any one of claims 1, characterized in that, Neighboring points, including: Within the neighboring point calculation range, if the pixel value difference between the current sampling point and each sampling point is not greater than the height threshold, then all sampling points within the neighboring point calculation range are determined to be neighboring points of the current sampling point.

8. A depth map processing system, characterized in that, The depth map is output by any one of a line structured light camera, a surface structured light camera, or a line spectral confocal sensor, and includes: The initialization module iterates through each pixel in the depth map. If the pixel value of a pixel is less than the height threshold, the mask of that pixel is set to 1; otherwise, the mask of that pixel is set to 0. The sampling module sets the filter window radii in the row and column directions based on the resolution of the depth map; based on the filter window radii in the row and column directions, it sets the dilation rate of the dilated convolution and determines the number of sampling points in the filter window. The window calculation module iterates through the pixel values ​​of each sampling point in the filtering window. If the pixel value of a sampling point falls into the invalid pixel range, it indicates that the sampling point is invalid. Invalid points are then removed, and sampling points that do not fall into the invalid pixel range are used as the neighboring point calculation range. Within the neighboring point calculation range, the neighboring points and the number of neighboring points for each sampling point are calculated. If the number of neighboring points for a sampling point is less than the number threshold, the mask for that sampling point is set to 1; otherwise, the mask for that sampling point is set to 0. Invalid points are pixels whose pixel values ​​in the depth map fall within the invalid pixel range; The morphology module, based on a mask, uses morphological closing operations to process the depth map and obtain a processed image; based on the processed image, pixels with a mask value of 1 are set as invalid points, and the final image is output.

9. A three-dimensional image reconstruction method, characterized in that, include; According to any one of the depth map processing methods in claims 1-7, a final image with abnormal spikes removed is obtained; The spatial coordinates of each pixel in the final image are calculated based on back projection to obtain a 3D image.

10. A computer-readable storage medium storing at least one instruction or at least one program, said at least one instruction or at least one program being loaded and executed by a processor to implement the depth map processing method as claimed in any one of claims 1-7 or the three-dimensional image reconstruction method as claimed in claim 9.

Images