Depth map processing method and device, equipment and medium

Abstract

The invention discloses a depth map processing method and device, equipment and a medium, and the method comprises the steps: obtaining continuous n frames of depth maps, each pixel point in each depth map comprising a depth value, and n being a natural number greater than 1; obtaining a depth difference value of the same pixel point between the two frames of depth maps selected randomly; and performing corresponding smoothing processing according to the depth difference value. According to the method, the depth maps at multiple moments are collected, the depth differences between different frames of the same pixel point are compared, noise points with large depth changes are filtered, recognized and removed according to the depth difference values, the overall quality of the depth maps is improved, and the influence of the noise of the depth sensor on the precision is effectively reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a depth map processing method, device, equipment and medium. Background Art

[0002] Depth maps are usually obtained by depth sensors. However, in practical applications, depth sensors are easily affected by various noises, such as measurement errors, holes caused by occlusion, and inaccuracies in long-distance measurements. These situations significantly affect the quality of depth maps, and further affect tasks that rely on high-precision depth information, such as 3D reconstruction, SLAM (Simultaneous Localization and Mapping), object recognition and grasping, obstacle detection in autonomous driving, etc. Smoothing processing of depth maps is a key preprocessing step in these tasks.

[0003] Common depth map smoothing algorithms include mean filtering, Gaussian filtering, median filtering, bilateral filtering, weighted moving average, and guided filtering, etc. Each algorithm provides an optimized effect according to the characteristics of different scenarios, but it also has certain limitations. For example, these algorithms usually do not fully consider the noise changes when the depth sensor collects data at different times, which may lead to large fluctuations in depth values at the same position, thus affecting the accuracy of subsequent processing.

[0004] The 3D face reconstruction scenario has relatively high requirements for the refinement of depth information. The noise of the depth sensor will significantly reduce the reconstruction accuracy and affect the reconstruction effect, especially when dealing with complex facial geometries. Summary of the Invention

[0005] To solve the above one or more technical problems, the present invention provides a depth map processing method and device, which filter noise through the depth difference of the same pixel point in multiple frames of depth maps, improve the overall quality of the depth map, and effectively reduce the influence of depth sensor noise on accuracy.

[0006] To solve the above problems, the first aspect of the present invention provides a depth map processing method, including:

[0007] Obtain n consecutive frames of depth maps, where each pixel point in each depth map includes a depth value, and n is a natural number greater than 1;

[0008] Obtain the depth difference between the same pixel point in two randomly selected frames of depth maps;

[0009] Perform corresponding smoothing processing according to the depth difference.

[0010] Further, the obtaining the depth difference between the same pixel point in two randomly selected frames of depth maps includes:

[0011] Obtain the depth values of the same pixel in consecutive n depth maps;

[0012] Sort the depth values in ascending or descending order;

[0013] Randomly select two depth values for comparison to obtain the depth difference.

[0014] Further, the corresponding smoothing process according to the depth difference includes:

[0015] If the depth difference is greater than the preset threshold, set the depth value of the pixel to zero;

[0016] If the depth difference is less than the preset threshold, take the average value of the depth values between the two compared depth values as the smoothing result of the pixel.

[0017] Further, it also includes:

[0018] Obtain the mask of the smoothed depth map;

[0019] Perform an erosion operation on the mask with a kernel of preset coordinates for edge filtering.

[0020] The second aspect of the present invention provides a depth map processing device for implementing the above depth map processing method, including:

[0021] An acquisition module for acquiring consecutive n depth maps, where each pixel in each depth map includes a depth value, and n is a natural number greater than 1;

[0022] A data processing module for obtaining the depth difference between two randomly selected depth maps of the same pixel;

[0023] A smoothing module for performing corresponding smoothing processing according to the depth difference.

[0024] The third aspect of the present invention provides a three-dimensional face reconstruction method based on depth map temporal smoothing, including:

[0025] Collect face image data, where the face image data includes the color pattern of the face and the face depth image;

[0026] Use the above depth map processing method to perform depth map smoothing processing on the face depth image;

[0027] Input the processed depth map data into a three-dimensional reconstruction model to obtain three-dimensional face image data.

[0028] The fourth aspect of the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0029] The fifth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention discloses a depth map processing method and device. The method includes obtaining n consecutive depth maps, where each pixel point in each depth map includes a depth value; obtaining the depth difference between the same pixel point between two randomly selected depth maps; and performing corresponding smoothing processing according to the depth difference. Different from the prior art, the present invention collects depth maps at multiple moments, compares the depth differences of the same pixel point between different frames, filters according to the depth differences, identifies and removes noise points with large depth changes, improves the overall quality of the depth map, and effectively reduces the influence of depth sensor noise on the accuracy. Description of the Drawings

[0032] The following further describes in detail the specific embodiments of the present invention with reference to the drawings, where:

[0033] Figure 1 is the flowchart of the depth map processing method described in Embodiment 1; Figure 1 ;

[0034] Figure 2 is the flowchart of the depth map processing method described in Embodiment 1; Figure 2 ;

[0035] Figure 3 is the structural schematic diagram of the depth map processing device described in Embodiment 2;

[0036] Figure 4 is the flowchart of the three-dimensional face reconstruction method based on depth map temporal smoothing described in Embodiment 3;

[0037] Figure 5 is the structural schematic diagram of the computer device described in Embodiment 4;

[0038] Marking description: 110, acquisition module; 120, data processing module; 130, smoothing module. Specific Embodiments

[0039] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0040] Embodiment 1

[0041] This embodiment discloses a depth map processing method, as Figure 1 and2 , including:

[0042] S1. Obtain consecutive n depth maps depth_image_1, depth_image_2, …, depth_image_n. Each pixel point <col, row> in each depth map contains a depth value depth_image_i(col, row), where n is a natural number greater than 1.

[0043] S2. Obtain the depth difference between two randomly selected depth maps for the same pixel point.

[0044] Specifically, step S2 includes:

[0045] Obtain the depth values depth_image_1(col, row), depth_image_2(col, row),..., depth_image_n(col, row) of the same pixel point in consecutive n depth maps.

[0046] Sort the depth values in ascending or descending order. In this embodiment, sort the depth values of the same pixel point obtained in consecutive n depth maps in ascending order, that is, sort them in ascending order from smallest to largest, to obtain the sorted depth values sorted_value_1, sorted_value_2,..., sorted_value_n.

[0047] Randomly select two depth values for comparison to obtain the depth difference. Specifically, select the k-th largest and the k-th smallest depth values sorted_value_k and sorted_value_(n-k-1) for comparison to obtain the depth difference Δsorted_value between them, where k is a natural number greater than or equal to 1.

[0048] S3. Perform corresponding smoothing processing according to the depth difference.

[0049] Specifically, step S3 includes:

[0050] If the depth difference Δsorted_value is greater than a preset threshold thres, it is considered that the pixel point <col, row> has fluctuations, and set the depth value depth_image(col, row) of the pixel point to zero.

[0051] If the depth difference Δsorted_value is less than the preset threshold thres, it is considered that the pixel point <col, row> has no fluctuations, take the average of the depth values between sorted_value_k and sorted_value_(n-k-1), and this average value is used as the smoothing result of the pixel point.

[0052] To solve the problem of inaccurate depth values at the edges of the depth map, in this embodiment, it further includes:

[0053] Obtain the mask of the smoothed depth map. Specifically, binarize the smoothed depth map to obtain the mask.

[0054] Perform an erosion operation on the mask to reduce the range of the edges, multiply the mask by the depth map, and filter out the depth values where the mask is 0.

[0055] There are always some specific positions in space that are difficult to obtain accurate depth values, and these positions may amplify the deviation of the depth sensor. Therefore, by filtering the positions that are prone to fluctuations during multi-frame acquisition and smoothing the remaining positions, where a certain position is filtered by judging whether there are large fluctuations on the depth map.

[0056] In addition, since multiple sets of depth values are obtained at the same position, it can be considered that the regions with small fluctuation ranges have higher confidence levels. Therefore, the multiple sets of depth values in these regions are averaged for smoothing to achieve better results.

[0057] The present invention provides a depth map processing method, which collects depth maps at multiple adjacent moments, compares the depth differences between different frames, filters out noise through the depth differences at the same positions in multiple depth maps, identifies and removes the noise points with large depth changes, improves the overall quality of the depth map, and effectively reduces the influence of depth sensor noise on the accuracy.

[0058] Different from the existing depth map smoothing methods, the method proposed by the present invention combines the spatial information in the depth map with the relative position information of the depth maps obtained at different times. This way of combining spatial position and time information can obtain a depth map with obvious noise filtering effect.

[0059] Embodiment 2

[0060] This embodiment discloses a depth map processing device for implementing the depth map processing method described in Embodiment 1, as Figure 3 , including an acquisition module 110, a data processing module 120, and a smoothing module 130. Specifically, the acquisition module 110 is used to acquire continuous n frames of depth maps, and each pixel point in each depth map contains a depth value, where n is a natural number greater than 1; the data processing module 120 is used to obtain the depth difference between two randomly selected frames of depth maps at the same pixel point; the smoothing module 130 is used to perform corresponding smoothing processing according to the depth difference.

[0061] For other specific implementation details, please refer to Embodiment 1 and will not be elaborated here.

[0062] Each module in the above-mentioned depth map processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in the processor of a computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0063] Embodiment 3

[0064] This embodiment discloses a method for 3D face reconstruction based on depth map temporal smoothing, as Figure 4 , including:

[0065] Collect face image data, where the face image data includes the color pattern of the face and the face depth image.

[0066] Use the depth map processing method described in Embodiment 1 to perform depth map smoothing processing on the face depth image.

[0067] Input the processed depth map data into the 3D reconstruction model to obtain the 3D face image data.

[0068] For other specific implementation details, please refer to Embodiment 1 and will not be elaborated here.

[0069] Combined with the depth map processing method provided in Embodiment 1, it can identify and remove the noise points with large depth changes during preprocessing in the 3D face reconstruction process, improve the overall quality of the depth map, effectively reduce the influence of depth sensor noise on the accuracy, and provide more stable and reliable depth information support for high-precision 3D reconstruction.

[0070] Embodiment 4

[0071] This embodiment discloses a computer device, which can be a server or a terminal integrated with a scheduler, and its internal structure diagram can be as Figure 5 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a depth map processing method.

[0072] Those skilled in the art can understand that Figure 5The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. Specifically, the computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0073] In this embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0074] Obtain n consecutive depth maps, and each pixel point in each depth map contains a depth value, where n is a natural number greater than 1;

[0075] Obtain the depth difference between the same pixel point between two randomly selected depth maps;

[0076] Perform corresponding smoothing processing according to the depth difference.

[0077] In this embodiment, when the processor executes the computer program, the following steps are also implemented:

[0078] Obtain the depth values of the same pixel point in n consecutive depth maps;

[0079] Sort the depth values in ascending or descending order;

[0080] Randomly select two depth values for comparison to obtain the depth difference.

[0081] In this embodiment, when the processor executes the computer program, the following steps are also implemented:

[0082] If the depth difference is greater than a preset threshold, set the depth value of the pixel point to zero;

[0083] If the depth difference is less than the preset threshold, take the average value of the depth values between the two compared depth values as the smoothing result of the pixel point.

[0084] In this embodiment, when the processor executes the computer program, the following steps are also implemented:

[0085] Obtain the mask of the smoothed depth map;

[0086] Perform erosion operation on the mask with a kernel of preset coordinates for edge filtering.

[0087] Embodiment 5

[0088] This embodiment discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0089] Obtain n consecutive depth maps, where each pixel point in each depth map contains a depth value, and n is a natural number greater than 1;

[0090] Obtain the depth difference between the same pixel point between two randomly selected depth maps;

[0091] Perform corresponding smoothing processing according to the depth difference.

[0092] In this embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0093] Obtain the depth values of the same pixel point in n consecutive depth maps;

[0094] Sort the depth values in ascending or descending order;

[0095] Randomly select two depth values for comparison to obtain the depth difference.

[0096] In this embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0097] If the depth difference is greater than a preset threshold, set the depth value of the pixel point to zero;

[0098] If the depth difference is less than a preset threshold, take the average value of the depth values between the two compared depth values as the smoothing result of the pixel point.

[0099] In this embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0100] Obtain the mask of the smoothed depth map;

[0101] Perform an erosion operation on the mask with a kernel at a preset coordinate for edge filtering.

[0102] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0103] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0104] In the description of this specification, the descriptions referring to terms such as "in this embodiment" or "specifically" mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0105] As described above, it is only a preferred embodiment of the present invention, and there is no any form of limitation to the present invention. Therefore, any modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A depth map processing method, characterized in that: include: Obtain n consecutive depth maps, each of which contains a depth value for each pixel, where n is a natural number greater than 1; Get the depth difference between two randomly selected depth images of the same pixel; According to the depth difference, corresponding smoothing processing is performed.

2. The depth map processing method according to claim 1, characterized in that: The obtaining of the depth difference between two randomly selected depth images of the same pixel point includes: Get the depth value of the same pixel in the depth map of consecutive n frames; Sort each depth value in ascending or descending order; Randomly select two depth values ​​for comparison to obtain the depth difference.

3. The depth map processing method according to claim 1, characterized in that: The performing corresponding smoothing processing according to the depth difference includes: If the depth difference is greater than a preset threshold, the depth value of the pixel is set to zero; If the depth difference is less than a preset threshold, an average value of the depth values ​​between the two compared depth values ​​is taken as the smoothing result of the pixel point.

4. The depth map processing method according to claim 1, characterized in that: Also includes: Get the mask of the smoothed depth map; Perform an erosion operation on the mask to perform edge filtering.

5. A depth map processing device, used to implement the depth map processing method according to any one of claims 1 to 4, characterized in that: include: An acquisition module is used to acquire n consecutive depth maps, each of which contains a depth value for each pixel, where n is a natural number greater than 1; A data processing module is used to obtain the depth difference between two randomly selected depth images of the same pixel; The smoothing module is used to perform corresponding smoothing processing according to the depth difference.

6. A method for 3D reconstruction of human face based on temporal smoothing of depth map, characterized in that: include: Collecting facial image data, the facial image data including a color pattern of the face and a depth image of the face; Performing depth map smoothing processing on the face depth image using the depth map processing method described in any one of claims 1 to 4; The processed depth map data is input into the 3D reconstruction model to obtain the 3D image data of the face.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images