Method, apparatus and device for improving TOF data by using RGB data, and medium

Abstract

The application discloses a method, device, equipment and medium for improving TOF data by using RGB data, the method comprising: acquiring RGB data and TOF data synchronously shot; estimating a pseudo-depth map according to the RGB data, and correcting the pseudo-depth map by using the TOF data to obtain a corrected depth map; extracting a gradient map of the RGB data, and calculating a weight coefficient of the TOF data according to the gradient map and the RGB data; and fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data. The application combines the corrected depth map determined based on the RGB data with the TOF data, and simultaneously uses the details and structural information provided by the corrected depth map and the real scale and stability provided by the TOF data, so that the jitter problem of the TOF data is solved, and the accuracy of the TOF depth data is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for improving TOF data by using RGB data. Background Art

[0002] With the rapid advancement of science and technology, depth camera technology has ushered in a wide range of applications and development opportunities. Currently, many smart products on the market, such as sweeping robots, lidar, and facial detection and recognition devices, have adopted TOF depth cameras as core components. In practical applications, the accuracy of TOF depth cameras is particularly critical. The higher the accuracy, the more stable the detection data it collects, which in turn can ensure significantly improved accuracy of the detection results. However, TOF depth cameras are susceptible to interference from strong light and reflective materials during use, resulting in significant jitter in the collected depth data, affecting the accuracy of the depth data.

[0003] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide a method, device, equipment and medium for improving TOF data by using RGB data in response to the shortcomings of the existing technology.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a method for improving TOF data by using RGB data, wherein the method for improving TOF data by using RGB data specifically includes:

[0006] Get the synchronously captured RGB data and TOF data;

[0007] estimating a pseudo depth map based on the RGB data, and correcting the pseudo depth map using the TOF data to obtain a corrected depth map;

[0008] Extracting a gradient map of the RGB data, and calculating a weight coefficient of the TOF data based on the gradient map and the RGB data;

[0009] The corrected depth map and the TOF data are fused based on the weight coefficient to obtain target TOF data.

[0010] The method for improving TOF data using RGB data, wherein the correcting the pseudo depth map using the TOF data to obtain a corrected depth map specifically includes:

[0011] Calculating pixel differences between the TOF data and the pseudo depth map to obtain a depth residual map;

[0012] Determine a correction image corresponding to the pseudo depth image according to the depth residual image;

[0013] The corrected depth map is added to the pseudo depth map to obtain a corrected depth map.

[0014] The method for improving TOF data using RGB data, wherein calculating the weight coefficient of the TOF data based on the gradient map and the RGB data specifically includes:

[0015] Based on the pixel position, the gradient value corresponding to each true depth pixel in the TOF data is selected in the gradient map, and the RGB pixel value corresponding to each true depth pixel in the TOF data is selected in the RGB data;

[0016] The weight coefficient of each real depth pixel is calculated according to the gradient value and RGB pixel value corresponding to each real depth pixel to obtain the weight coefficient of the TOF data.

[0017] In the method for improving TOF data using RGB data, the weight coefficient of the real depth pixel is:

[0018] ,

[0019] in, Represents the true depth pixel in TOF data The weight coefficient of represents the gradient value, represents the RGB pixel value, are adjustment parameters, and , Represents TOF data.

[0020] The method for improving TOF data using RGB data, wherein the fusing of the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data specifically includes:

[0021] Calculating a correction weight coefficient corresponding to the corrected depth map according to the weight coefficient;

[0022] The corrected depth map and the TOF data are weighted according to the weight coefficient and the corrected weight coefficient to obtain target TOF data.

[0023] The method for improving TOF data using RGB data, wherein, after acquiring the synchronously captured RGB data and TOF data, the method further comprises:

[0024] The RGB data is mapped to the coordinate system of the TOF data, and the mapped RGB data is downsampled so that the resolution of the RGB data is the same as the resolution of the TOF data.

[0025] The method for improving TOF data using RGB data, wherein, after fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data, the method further includes:

[0026] The target TOF data is repaired using gradient direction guidance to remove invalid holes in the target TOF data.

[0027] A second aspect of the present application provides a device for improving TOF data using RGB data, wherein the device for improving TOF data using RGB data specifically includes:

[0028] Acquisition module, used to obtain synchronously captured RGB data and TOF data;

[0029] an estimation module, configured to estimate a pseudo depth map based on the RGB data, and correct the pseudo depth map using the TOF data to obtain a corrected depth map;

[0030] An extraction module, configured to extract a gradient map of the RGB data and calculate a weight coefficient of the TOF data based on the gradient map and the RGB data;

[0031] A fusion module is used to fuse the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data.

[0032] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in any of the above methods for improving TOF data using RGB data.

[0033] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0034] The memory stores a computer-readable program executable by the processor;

[0035] When the processor executes the computer-readable program, the steps in any of the above methods for improving TOF data using RGB data are implemented.

[0036] Beneficial effects: Compared with the prior art, the present application provides a method, apparatus, device and medium for improving TOF data using RGB data, the method comprising acquiring synchronously captured RGB data and TOF data; estimating a pseudo depth map based on the RGB data, and correcting the pseudo depth map using the TOF data to obtain a corrected depth map; extracting a gradient map of the RGB data, and calculating a weight coefficient of the TOF data based on the gradient map and the RGB data; and fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data. The present application combines the corrected depth map determined based on the RGB data with the TOF data, and at the same time utilizes the details and structural information provided by the corrected depth map and the true scale and stability provided by the TOF data to solve the jitter problem of the TOF data and improve the accuracy of the TOF depth data. At the same time, when fusing the corrected depth map with the TOF data, the present application utilizes the gradient information of the RGB data to control the fusion weight, effectively retaining the edge information of the object and further improving the accuracy of the TOF depth data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 Flowchart of a method for improving TOF data using RGB data provided in an embodiment of the present application.

[0039] Figure 2 This is an example image of RGB data.

[0040] Figure 3 This is an example graph of TOF data.

[0041] Figure 4 The schematic diagram of the electronic device used to obtain RGB data and TOF data.

[0042] Figure 5 This is an example of a pseudo depth map.

[0043] Figure 6 The following is an example flow chart of the process of obtaining a corrected depth map.

[0044] Figure 7 This is an example flow chart of the process of obtaining weight coefficients.

[0045] Figure 8This is a block diagram of the principle of an apparatus for enhancing TOF data using RGB data provided in an embodiment of the present application.

[0046] Figure 9 This is a block diagram of the principles of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application provide a method, apparatus, device, and medium for enhancing TOF data using RGB data. To make the objectives, technical solutions, and effects of the present application more clear and explicit, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to explain the present application and are not intended to limit the present application.

[0048] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0050] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0051] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0052] This embodiment provides a method for improving TOF data by using RGB data, such as Figure 1 As shown, the method of using RGB data to improve TOF data specifically includes:

[0053] S10: Acquire synchronously captured RGB data and TOF data.

[0054] Specifically, RGB data and TOF data are obtained by shooting an electronic device, and the electronic device is provided with a TOF module and an RGB module in the same plane and in the same direction, wherein the number of TOF modules can be one, and the number of RGB modules can be one or two; the TOF module and the RGB module are calibrated with module internal parameters and external parameters. That is to say, before obtaining the synchronously shot RGB data and TOF data, the internal parameters of the RGB module and the internal parameters of the ToF module are calibrated respectively, and the TOF module and the RGB module are calibrated with joint heterogeneous external parameters. For example, the electronic device may include a parameter calibration module, which is used to calibrate the internal parameters of the RGB module and the internal parameters of the TOF module before the synchronously shot RGB data and TOF data, and to calibrate the TOF module and the RGB module with joint heterogeneous external parameters, wherein the parameter calibration module can use the Zhang Zhengyou calibration method to realize parameter calibration, which is common knowledge to those skilled in the art and will not be described in detail here.

[0055] The TOF module and the RGB module will receive the synchronous shooting signal respectively, and shoot based on the synchronous signal to obtain a frame of RGB data and a frame of TOF data. For example, Figure 2 The RGB data shown and Figure 3 The TOF data shown. Among them, the synchronization signal can be sent to the TOF module and the RGB module by the data processing module deployed in the electronic device, that is, the electronic device can be deployed with a data processing module, and the data processing module is respectively connected to the TOF module and the RGB module, for sending synchronization signals to the TOF module and the RGB module, receiving the TOF data shot by the TOF module based on the synchronization signal and the RGB data shot by the RGB module based on the synchronization signal, so that the electronic device can synchronously shoot RGB data and TOF data. In addition, in order to facilitate the transmission of RGB data and TOF data, the electronic device can also be deployed with a data transmission module, and the data transmission module is connected to the data processing module for transmitting the TOF data and RGB data received by the data processing module. Among them, the data transmission module can be USB, NFC, WIFI, wired, etc., and the data processing module can be a processing chip, a processing circuit, a data processor (such as a CPU, etc.), etc. For example, as Figure 4 As shown, the data transmission module adopts USB and the data processing module adopts CPU.

[0056] Furthermore, in actual applications, the resolution of the RGB data captured by the RGB module is generally different from the resolution of the TOF data captured by the TOF module. For example, the resolution of the RGB data is 1920*1200, and the resolution of the TOF data is 240*180. Therefore, after acquiring the RGB data and TOF data, the resolution of the RGB data can be adjusted so that the resolution of the adjusted RGB data is the same as the resolution of the TOF data. Based on this, after acquiring the synchronously captured RGB data and TOF data, the method further includes:

[0057] The RGB data is mapped to the coordinate system of the TOF data, and the mapped RGB data is downsampled so that the resolution of the RGB data is the same as the resolution of the TOF data.

[0058] Specifically, when mapping the RGB data to the coordinate system of the TOF data, the internal parameters of the RGB module, the internal parameters of the TOF module, and the external parameters between the RGB module and the TOF module can be used to convert the RGB data to the coordinate system set by the TOF module (i.e., the coordinate system of the TOF data) to obtain the mapped RGB data. After obtaining the mapped RGB data, the mapped RGB data is downsampled so that the resolution of the RGB data is the same as the resolution of the TOF data. For example, if the resolution of the TOF data is 240*180, the resolution of the downsampled RGB data will also be 240*180.

[0059] It should be noted that after acquiring the RGB data and TOF data, the RGB data and TOF data may be preprocessed, wherein the preprocessing may include de-distortion processing, denoising processing, etc. In the embodiment of the present application, after acquiring the RGB data and TOF data, de-distortion processing is performed on the RGB data and TOF data respectively to improve the data quality of the RGB data and TOF data.

[0060] S20: Estimate a pseudo depth map according to the RGB data, and correct the pseudo depth map using the TOF data to obtain a corrected depth map.

[0061] Specifically, the pseudo depth map is obtained by performing depth estimation on the RGB data. For example, the depth estimation can be performed on the RGB data by using a monocular depth estimation algorithm or a trained depth estimation model to obtain the pseudo depth map. In other words, the pixel value of each pseudo depth pixel in the pseudo depth map is an estimated value obtained by performing depth estimation based on the RGB data. For example, Figure 3 The pseudo depth map estimated from the RGB data shown is as follows Figure 5 shown.

[0062] The pixel value of each true depth pixel in the TOF data is the true depth value, which is calculated by the frequency difference or time difference between the pixel pulse waves. For example, the true depth value is calculated by calculating the flight time of infrared light. Therefore, TOF data is more reliable than pseudo depth maps, so TOF data can be used to correct pseudo depth maps to improve their reliability.

[0063] For example, Figure 6 As shown, the use of the TOF data to correct the pseudo depth map to obtain a corrected depth map specifically includes:

[0064] S21, performing pixel difference calculation on the TOF data and the pseudo depth map to obtain a depth residual map;

[0065] S22, determining a correction image corresponding to the pseudo depth image according to the depth residual image;

[0066] S23: Add the corrected depth map to the pseudo depth map to obtain a corrected depth map.

[0067] Specifically, the resolution of the depth residual map is the same as that of the pseudo-depth map, and is used to reflect the difference between the TOF data and the pseudo-depth map. The pixel value of each depth residual pixel in the depth residual map is determined based on the pixel value of the real depth pixel in the TOF data and the pixel value of the pseudo-depth pixel in the pseudo-depth map, and is used to reflect the difference between the real depth pixel and the pseudo-depth pixel. That is, for each pseudo-depth pixel in the pseudo-depth map, a real depth pixel is selected in the TOF data according to the pixel position of the pseudo-depth pixel. The pixel position of the real depth pixel in the TOF data is the same as the pixel position of the pseudo-depth pixel in the pseudo-depth map. The difference between the pixel value of the pseudo-depth pixel and the pixel value of the selected real depth pixel is then calculated, and the difference is used as the depth residual value corresponding to the pseudo-depth pixel. The depth residual values ​​corresponding to all pseudo-depth pixels constitute the depth residual map corresponding to the pseudo-depth map.

[0068] In one embodiment, the calculation formula for the depth residual pixel in the depth residual map can be:

[0069] ,

[0070] in, represents the pixel value of the depth residual pixel, represents the pixel value of the true depth pixel, represents the pixel value of the pseudo depth pixel, Represents RGB data.

[0071] It should be noted that in actual applications, the depth residual map can also be obtained by other methods, for example, inputting the pseudo depth map and TOF data into the prediction residual network model, and outputting the depth residual map between the TOF data and the pseudo depth map through the residual network model.

[0072] After obtaining the depth residual map, a correction term for each pseudo-depth pixel is determined based on the depth residual map to obtain a correction map. For example, the depth residual map can be directly used as the correction map corresponding to the pseudo-depth map, or the depth residual map can be first edge-preserving filtered or locally curve-fitted, and then the depth residual map after edge-preserving filtering or local curve fitting is used as the correction map. In a specific embodiment, after obtaining the depth residual map, the depth residual map is first edge-preserving filtered (i.e., the value is kept unchanged in high variance areas and the average value of neighboring pixels is used in smooth areas), and then the depth residual map after edge-preserving filtering is used as the correction map. This not only well preserves the edge information and detail information of the depth residual map, but also reduces image noise.

[0073] Furthermore, after obtaining the corrected image, the pseudo depth map is corrected using the corrected image to resolve the scale deviation problem of the pseudo depth map and improve the accuracy of the pseudo depth map. Specifically, after obtaining the corrected image, the corrected image and the pseudo depth map can be added pixel by pixel to obtain a corrected depth map, wherein the pixel value calculation formula of the corrected depth pixel of the corrected depth map can be:

[0074] ,

[0075] in, represents the pixel value of the corrected depth pixel in the corrected depth map, represents the pixel value of the pseudo depth pixel in the pseudo depth map, Indicates the pixel value of the correction pixel in the correction image.

[0076] S30 , extracting a gradient map of the RGB data, and calculating a weight coefficient of the TOF data based on the gradient map and the RGB data.

[0077] Specifically, the gradient map is used to reflect the edge strength of the RGB data. The gradient map includes the gradient of each RGB pixel in the RGB data. That is, after acquiring the RGB data, the gradient of each RGB pixel in the RGB data is extracted to obtain the gradient map of the RGB data. The gradient can be extracted using methods such as Sobel and Laplacian.

[0078] Furthermore, since the gradient map can reflect the edge strength of RGB data, the gradient map can be used to construct the weight coefficient, that is, the gradient information of the RGB data is used to control the weight coefficient of the TOF data, and the weight coefficient of each real depth pixel in the TOF data is adaptively adjusted, so that the fused target TOF data can effectively retain the boundaries and detail contours, improve the visual consistency and authenticity of the target TOF data, and thus improve the accuracy of the target TOF data.

[0079] For example, Figure 7 As shown, the weight coefficient of the TOF data calculated according to the gradient map and the RGB data specifically includes:

[0080] S31. Based on the pixel position, select, in the gradient map, a gradient value corresponding to each true depth pixel in the TOF data, and select, in the RGB data, an RGB pixel value corresponding to each true depth pixel in the TOF data;

[0081] S32. Calculate a weight coefficient of each real depth pixel according to the gradient value and the RGB pixel value corresponding to each real depth pixel to obtain a weight coefficient of the TOF data.

[0082] Specifically, the RGB data is RGB data mapped to the size of the TOF data. That is, when selecting from the TOF data and RGB data based on pixel position, the resolution of the RGB data is the same as the resolution of the TOF data. That is, the RGB pixels in the RGB data correspond one-to-one to the true depth pixels in the TOF data. Therefore, for each true depth pixel in the TOF data, a corresponding RGB pixel can be selected from the RGB data (that is, the RGB pixel corresponding to the pixel position is selected). Similarly, since the gradient map is obtained by performing gradient extraction on each RGB pixel in the RGB data, the resolution of the gradient map is the same as the resolution of the RGB data. Therefore, the resolution of the gradient map is also the same as the resolution of the TOF data. That is, for each true depth pixel in the TOF data, a corresponding gradient pixel can be selected from the gradient map (that is, the gradient pixel corresponding to the pixel position is selected). To this end, when constructing the weight coefficient for TOF data, the pixel position can be used as the basis to select the corresponding gradient pixel in the gradient map for each true depth pixel in the TOF data, select the RGB pixel in the RGB data, and then read the gradient value of the selected gradient pixel and the RGB pixel value of the RGB pixel to obtain the gradient value and RGB pixel value corresponding to each true depth pixel.

[0083] After obtaining the gradient value and RGB pixel value corresponding to each true depth pixel, a weight coefficient is constructed for the true depth pixel according to the gradient value and RGB pixel value, and the TOF data is fused with the corrected pseudo depth map based on the weight coefficient. This makes comprehensive use of the knowledge information carried by the RGB data and the gradient information of the RGB data, which can effectively retain the object boundaries and detailed contours in the shooting scene, and improve the accuracy of the fused target TOF data.

[0084] In one embodiment, the weight coefficient of the real depth pixel is:

[0085] ,

[0086] in, Represents the true depth pixel in TOF data The weight coefficient of represents the gradient value, represents the RGB pixel value, are adjustment parameters, and , Represents TOF data.

[0087] The embodiment of the present application uses an exponential function of the gradient value to dynamically adjust the weight coefficient, which can better utilize the gradient information to adaptively adjust the weight of the real depth pixel, thereby better improving the reliability of the fused target TOF data. Of course, in actual applications, other methods can also be used to construct weight coefficients for TOF data, for example, directly weighting the gradient value with the RGB pixel value.

[0088] S40 , fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data.

[0089] Specifically, the weight coefficient is used as the weight coefficient of the TOF data, which includes the weight coefficient of each real depth pixel in the TOF data. That is, after obtaining the weight coefficient of the TOF data, the weight coefficient corresponding to each real depth pixel can be read from the weight coefficient of the TOF data. Then, based on the weight coefficient corresponding to each real depth pixel, each real depth pixel is compared with the corrected depth map and the corrected weight coefficient of its corresponding corrected depth pixel. Finally, based on the weight coefficient and the corrected weight coefficient, the TOF data and the corrected depth map are weighted to obtain the target TOF data.

[0090] Exemplarily, fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data specifically includes:

[0091] Calculating a correction weight coefficient corresponding to the corrected depth map according to the weight coefficient;

[0092] The corrected depth map and the TOF data are weighted according to the weight coefficient and the corrected weight coefficient to obtain target TOF data.

[0093] Specifically, the weight coefficient of the TOF data corresponds to the correction weight coefficient of the corrected depth map in a one-to-one correspondence, that is, for each weight coefficient item in the weight coefficient of the TOF data, there is a correction weight coefficient item in the correction weight coefficient, and the pixel position of the real depth pixel corresponding to the weight coefficient item corresponds to the pixel position of the corrected depth pixel corresponding to the correction weight coefficient item, and the values ​​of the weight coefficient item and the correction weight coefficient item are both located at In the equation, the sum of the weight coefficient term and the modified weight coefficient term is equal to 1. That is, each modified weight coefficient term in the modified weight coefficient can be expressed as , which is the corrected depth pixel in the corrected depth map The modified weight coefficient of is the real depth pixel in TOF data The weight coefficient of .

[0094] After obtaining the corrected weight coefficient corresponding to the corrected depth map, the corrected depth map and the TOF data are weighted based on the weight coefficient and the corrected weight coefficient, wherein the weighting process can be expressed as:

[0095] ,

[0096] in, Represents the pixel value of the target depth pixel in the target TOF data, Represents the weight coefficient of the real depth pixel in TOF data, Represents the pixel value of the true depth pixel in the OF data, Represents the pixel value of the corrected depth pixel in the corrected depth map.

[0097] Furthermore, after acquiring the target TOF data, the target TOF data can be post-processed to further improve the depth accuracy of the target TOF data. Post-processing can include filtering and / or hole repair. Filtering can use guided filtering (such as a guided filter) or edge-preserving filtering (such as a bilateral filter) to enhance edge clarity; hole repair can be performed using upsampling.

[0098] In a specific embodiment, after fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data, the method further includes:

[0099] The target TOF data is repaired using gradient direction guidance to remove invalid holes in the target TOF data.

[0100] Specifically, patching the target TOF data using gradient direction guidance refers to interpolating the TOF data using gradient guidance to patch invalid holes in the target TOF data. For example, a gradient map corresponding to the target TOF data is first obtained, and then a neighborhood estimation method guided by the gradient map is used to interpolate the target TOF data to patch invalid holes in the target TOF data. The embodiments of the present application use a gradient direction-guided interpolation method to patch the target TOF data, which can better ensure the edges of the target TOF data and improve the accuracy of the patched target TOF data.

[0101] Of course, in practical applications, other methods may also be used to patch the target TOF data, for example, a bilinear interpolation method or a nearest neighbor interpolation method, which are not described here one by one.

[0102] In summary, this embodiment provides a method, apparatus, device, and medium for enhancing time-of-flight (TOF) data using RGB data. The method includes acquiring simultaneously captured RGB data and time-of-flight (TOF) data; estimating a pseudo-depth map based on the RGB data, and correcting the pseudo-depth map using the TOF data to obtain a corrected depth map; extracting a gradient map from the RGB data, and calculating a weight coefficient for the TOF data based on the gradient map and the RGB data; and fusing the corrected depth map with the TOF data based on the weight coefficient to obtain target TOF data. This application combines the corrected depth map determined based on the RGB data with the TOF data, simultaneously utilizing the detail and structure information provided by the corrected depth map and the true scale and stability provided by the TOF data to address the jitter problem associated with TOF data. Furthermore, when fusing the corrected depth map with the TOF data, this application utilizes the gradient information of the RGB data to control the fusion weight, effectively preserving object edge information. This can improve the accuracy of TOF depth data by 80%-90%, thereby improving the accuracy of downstream tasks based on depth data, such as object detection and face recognition.

[0103] Based on the above method of using RGB data to improve TOF data, this embodiment provides a device for using RGB data to improve TOF data, such as Figure 8 As shown, the device for improving TOF data by using RGB data specifically includes:

[0104] An acquisition module 100 is used to acquire synchronously captured RGB data and TOF data;

[0105] an estimating module 200 for estimating a pseudo depth map based on the RGB data, and correcting the pseudo depth map using the TOF data to obtain a corrected depth map;

[0106] An extraction module 300 is configured to extract a gradient map of the RGB data and calculate a weight coefficient of the TOF data based on the gradient map and the RGB data;

[0107] The fusion module 400 is configured to fuse the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data.

[0108] Based on the above-mentioned method of using RGB data to enhance TOF data, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the method of using RGB data to enhance TOF data as described in the above embodiment.

[0109] Based on the above method of using RGB data to improve TOF data, the present application also provides a terminal device, such as Figure 9 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logic instructions in the memory 22 to execute the method described in the above embodiment.

[0110] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0111] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0112] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0113] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for improving TOF data using RGB data, characterized in that: The method of using RGB data to improve TOF data specifically includes: Get the synchronously captured RGB data and TOF data; estimating a pseudo depth map based on the RGB data, and correcting the pseudo depth map using the TOF data to obtain a corrected depth map; Extracting a gradient map of the RGB data, and calculating a weight coefficient of the TOF data based on the gradient map and the RGB data; fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data; The method of correcting the pseudo depth map using the TOF data to obtain a corrected depth map specifically includes: Calculating pixel differences between the TOF data and the pseudo depth map to obtain a depth residual map; Determine a correction image corresponding to the pseudo depth image according to the depth residual image; Adding the corrected depth map to the pseudo depth map to obtain a corrected depth map; Calculating the weight coefficient of the TOF data according to the gradient map and the RGB data specifically includes: Based on the pixel position, the gradient value corresponding to each true depth pixel in the TOF data is selected in the gradient map, and the RGB pixel value corresponding to each true depth pixel in the TOF data is selected in the RGB data; The weight coefficient of each real depth pixel is calculated according to the gradient value and RGB pixel value corresponding to each real depth pixel to obtain the weight coefficient of the TOF data, wherein the weight coefficient of the real depth pixel is: ， in, Represents the true depth pixel in TOF data The weight coefficient of represents the gradient value, represents the RGB pixel value, are adjustment parameters, and , Represents TOF data.

2. The method for improving TOF data using RGB data according to claim 1, characterized in that: The fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data specifically includes: Calculating a correction weight coefficient corresponding to the corrected depth map according to the weight coefficient; The corrected depth map and the TOF data are weighted according to the weight coefficient and the corrected weight coefficient to obtain target TOF data.

3. The method for improving TOF data using RGB data according to claim 1, wherein: After acquiring the synchronously captured RGB data and TOF data, the method further includes: The RGB data is mapped to the coordinate system of the TOF data, and the mapped RGB data is downsampled so that the resolution of the RGB data is the same as the resolution of the TOF data.

4. The method for improving TOF data using RGB data according to claim 1, wherein: After fusing the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data, the method further includes: The target TOF data is repaired using gradient direction guidance to remove invalid holes in the target TOF data.

5. A device for improving TOF data using RGB data, characterized in that: The device for improving TOF data by using RGB data specifically includes: Acquisition module, used to obtain synchronously captured RGB data and TOF data; an estimation module, configured to estimate a pseudo depth map based on the RGB data, and correct the pseudo depth map using the TOF data to obtain a corrected depth map; An extraction module, configured to extract a gradient map of the RGB data and calculate a weight coefficient of the TOF data based on the gradient map and the RGB data; a fusion module, configured to fuse the corrected depth map and the TOF data based on the weight coefficient to obtain target TOF data; The method of correcting the pseudo depth map using the TOF data to obtain a corrected depth map specifically includes: Calculating pixel differences between the TOF data and the pseudo depth map to obtain a depth residual map; Determine a correction image corresponding to the pseudo depth image according to the depth residual image; Adding the corrected depth map to the pseudo depth map to obtain a corrected depth map; Calculating the weight coefficient of the TOF data according to the gradient map and the RGB data specifically includes: Based on the pixel position, the gradient value corresponding to each true depth pixel in the TOF data is selected in the gradient map, and the RGB pixel value corresponding to each true depth pixel in the TOF data is selected in the RGB data; The weight coefficient of each real depth pixel is calculated according to the gradient value and RGB pixel value corresponding to each real depth pixel to obtain the weight coefficient of the TOF data, wherein the weight coefficient of the real depth pixel is: ， in, Represents the true depth pixel in TOF data The weight coefficient of represents the gradient value, represents the RGB pixel value, are adjustment parameters, and , Represents TOF data.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the method for enhancing TOF data using RGB data as described in any one of claims 1 to 4.

7. A terminal device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the method for improving TOF data using RGB data are implemented as described in any one of claims 1 to 4.

Images