Depth image denoising method, device, electronic device and storage medium

Abstract

The present application relates to a method, device, electronic device and storage medium for denoising a depth image. The method comprises: obtaining original depth information corresponding to a target object; performing a first denoising process on the original depth information after preprocessing to obtain first intermediate depth information; performing a second denoising process on the original depth information and then performing a preprocessing to obtain second intermediate depth information; fusing the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generating a first depth image based on the fused depth information; generating a second depth image based on the original depth information, performing a third denoising process on the second depth image to obtain a third depth image; fusing the first depth image and the third depth image to obtain a target depth image corresponding to the target object. The use of this method can improve the accuracy of the depth image.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a method, device, electronic device and storage medium for deep image denoising. Background Art

[0002] With the development of image processing technology, depth image denoising technology has emerged. Due to the influence of factors such as device characteristics and external light, the depth image captured by the terminal has large noise in the output depth image, resulting in low accuracy of the depth image. Depth image denoising refers to removing noise from the depth image to improve the accuracy of the final output depth image. In traditional technology, denoising is usually only performed on the depth image that has been output by the terminal, and the denoising effect is very limited, resulting in low accuracy of the depth image finally output by the terminal. Summary of the invention

[0003] The embodiments of the present application provide a depth image denoising method, device, electronic device and computer-readable storage medium, which can improve the accuracy of depth images.

[0004] A depth image denoising method, the method comprising:

[0005] Get the original depth information corresponding to the target object;

[0006] Preprocessing the original depth information and then performing a first denoising process to obtain first intermediate depth information;

[0007] Performing a second denoising process and then preprocessing on the original depth information to obtain second intermediate depth information;

[0008] Fusing the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generating a first depth image based on the fused depth information;

[0009] Generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image;

[0010] The first depth image and the third depth image are fused to obtain a target depth image corresponding to the target object.

[0011] A depth image denoising device, the device comprising:

[0012] An acquisition module is used to obtain original depth information corresponding to the target object;

[0013] A first processing module, configured to pre-process the original depth information and then perform a first denoising process to obtain first intermediate depth information;

[0014] A second processing module, configured to perform a second denoising process and then preprocess the original depth information to obtain second intermediate depth information;

[0015] an information fusion module, configured to fuse the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generate a first depth image based on the fused depth information;

[0016] A generating module, configured to generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image;

[0017] An image fusion module is used to fuse the first depth image and the third depth image to obtain a target depth image corresponding to the target object.

[0018] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0019] Get the original depth information corresponding to the target object;

[0020] Preprocessing the original depth information and then performing a first denoising process to obtain first intermediate depth information;

[0021] Performing a second denoising process and then preprocessing on the original depth information to obtain second intermediate depth information;

[0022] Fusing the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generating a first depth image based on the fused depth information;

[0023] Generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image;

[0024] The first depth image and the third depth image are fused to obtain a target depth image corresponding to the target object.

[0025] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0026] Get the original depth information corresponding to the target object;

[0027] Preprocessing the original depth information and then performing a first denoising process to obtain first intermediate depth information;

[0028] Performing a second denoising process and then preprocessing on the original depth information to obtain second intermediate depth information;

[0029] Fusing the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generating a first depth image based on the fused depth information;

[0030] Generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image;

[0031] The first depth image and the third depth image are fused to obtain a target depth image corresponding to the target object.

[0032] The above-mentioned depth image denoising method, device, electronic device and storage medium obtain the original depth information corresponding to the target object; perform a first denoising process on the original depth information after preprocessing to obtain the first intermediate depth information; perform a second denoising process on the original depth information and then perform a preprocessing to obtain the second intermediate depth information; fuse the first intermediate depth information and the second intermediate depth information to obtain the fused depth information, and generate a first depth image based on the fused depth information; generate a second depth image according to the original depth information, perform a third denoising process on the second depth image to obtain the third depth image; fuse the first depth image and the third depth image to obtain the target depth image corresponding to the target object. In this way, through three effective denoising paths, while denoising the output depth image, the intermediate data of the generated depth image is also jointly denoised, which ensures the denoising effect of the depth image and improves the accuracy of the final output depth image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A diagram showing an application scenario of a depth image denoising method in an embodiment;

[0035] Figure 2 is a schematic diagram of a flow chart of a depth image denoising method in one embodiment;

[0036] Figure 3 It is a schematic diagram of a process of spatiotemporal joint denoising of multiple phase images in one embodiment;

[0037] Figure 4A schematic diagram of a two-round denoising principle for multiple phase images in one embodiment;

[0038] Figure 5 A schematic diagram of a first round of denoising framework for multiple phase images in one embodiment;

[0039] Figure 6 A schematic diagram of a second round of denoising framework for multiple phase images in one embodiment;

[0040] Figure 7 is a schematic diagram of a generation process of a third depth image in one embodiment;

[0041] Figure 8 is a schematic diagram of a denoising framework for a second depth image in one embodiment;

[0042] Fig. 9 is a schematic diagram of a process for generating a target depth image in one embodiment;

[0043] Fig.10 This is a schematic diagram of the traditional deep image denoising principle;

[0044] Fig.11 A schematic diagram of the depth image denoising principle of the present application in one embodiment;

[0045] Fig.12 is a structural block diagram of a depth image denoising device in one embodiment;

[0046] Fig.13 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] Figure 1 FIG. 1 is a schematic diagram of an application environment of a depth image denoising method in an embodiment. Figure 1 As shown, the application environment includes a target object 102 and a terminal 104. The terminal 104 may specifically include a desktop terminal or a mobile terminal. The mobile terminal may specifically include at least one of a mobile phone, a tablet computer, and a laptop computer. Those skilled in the art will appreciate that Figure 1 The application environment shown in is only a part of the scenarios related to the present application solution and does not constitute a limitation on the application environment of the present application solution.

[0049] The terminal 104 can obtain the original depth information corresponding to the target object 102; perform a first denoising process on the original depth information after preprocessing to obtain first intermediate depth information; perform a second denoising process on the original depth information and then perform a preprocessing to obtain second intermediate depth information; fuse the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generate a first depth image based on the fused depth information; generate a second depth image according to the original depth information, perform a third denoising process on the second depth image to obtain a third depth image; fuse the first depth image and the third depth image to obtain a target depth image corresponding to the target object.

[0050] Figure 2 FIG. 1 is a flowchart of a method for denoising a depth image in one embodiment. The method for denoising a depth image in this embodiment is executed on Figure 1 The description is made by taking the terminal 104 in FIG. Figure 2 As shown, the deep image denoising method includes the following steps:

[0051] Step 202: Acquire original depth information corresponding to the target object.

[0052] The target object is the object for which the depth image is to be captured, for example, the target object can be a person, an animal, an object, a scene, etc. The original depth information is the original depth information required to generate the depth image. Optionally, the original depth information can be a plurality of original phase images, or image data after certain preprocessing of the light signal. Each pixel value in the depth image represents the distance between each point of the target object and the camera.

[0053] In one embodiment, the terminal may transmit a modulated signal to the target object, and the target object may reflect an optical signal to the terminal based on the modulated signal. The terminal may receive the optical signal reflected by the target object and process the optical signal to obtain original depth information corresponding to the target object.

[0054] In one embodiment, the original depth information corresponding to the target object may be stored in a third-party storage device, and the terminal may communicate with the third-party storage device and directly obtain the original depth information corresponding to the target object from the third-party storage device.

[0055] Step 204 , pre-processing the original depth information and then performing a first denoising process to obtain first intermediate depth information.

[0056] The preprocessing may include at least one of filtering, encoding and data format conversion. The first intermediate depth information is intermediate image data necessary for generating a depth image.

[0057] Specifically, the terminal may pre-process the original depth information and then perform a first denoising process to obtain the first intermediate depth information.

[0058] Step 206 , performing a second denoising process and then a preprocessing on the original depth information to obtain second intermediate depth information.

[0059] The second intermediate depth information is intermediate image data necessary for generating a depth image.

[0060] Specifically, the terminal may perform a second denoising process on the original depth information and then perform preprocessing to obtain the second intermediate depth information.

[0061] Step 208: Fuse the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generate a first depth image based on the fused depth information.

[0062] The fused depth information is depth information having data features of both the first intermediate depth information and the second intermediate depth information. In other words, the fused depth information is obtained by fusing the denoised intermediate depth information. The first depth image is generated based on the fused depth information, that is, the first depth image is a depth image generated based on the denoised intermediate depth information.

[0063] Specifically, the terminal may fuse the first intermediate depth information and the second intermediate depth information to obtain fused depth information. Furthermore, the terminal may input the fused depth information into a depth image generator. The depth image generator may generate a first depth image based on the fused depth information.

[0064] Step 210: Generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image.

[0065] The second depth image is a depth image generated based on the original depth information before denoising. The third depth image is a depth image obtained after performing a third denoising on the second depth image.

[0066] Specifically, the terminal may generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image.

[0067] In one embodiment, the terminal may directly generate the second depth image based on the original depth information, or may pre-process the original depth information before generating the second depth image.

[0068] Step 212: Fusing the first depth image and the third depth image to obtain a target depth image corresponding to the target object.

[0069] Among them, the target depth image is the depth image finally output by the terminal. It can be understood that the target depth image contains less noise and has higher accuracy.

[0070] Specifically, the terminal may fuse the first depth image and the third depth image to obtain a target depth image corresponding to the target object.

[0071] In the above-mentioned depth image denoising method, the original depth information corresponding to the target object is obtained; the original depth information is preprocessed and then subjected to the first denoising process to obtain the first intermediate depth information; the original depth information is subjected to the second denoising process and then subjected to the preprocessing process to obtain the second intermediate depth information; the first intermediate depth information and the second intermediate depth information are fused to obtain the fused depth information, and a first depth image is generated based on the fused depth information; a second depth image is generated according to the original depth information, and the second depth image is subjected to the third denoising process to obtain the third depth image; the first depth image and the third depth image are fused to obtain the target depth image corresponding to the target object. In this way, through three effective denoising paths, while the output depth image is denoised, the intermediate data for generating the depth image is also jointly denoised, thereby ensuring the denoising effect of the depth image and improving the accuracy of the depth image finally output.

[0072] In one embodiment, step 202, i.e., the step of obtaining original depth information corresponding to the target object, specifically includes: obtaining more than one light signal reflected by the target object; generating a light signal sequence according to the more than one light signal; and determining the original depth information corresponding to the target object according to the light signal sequence.

[0073] Specifically, the terminal may transmit more than one group of modulation signals to the target object, and the target object may reflect more than one optical signal to the terminal based on the more than one group of modulation signals. The terminal may obtain the more than one optical signals reflected by the target object, and generate an optical signal sequence based on the more than one optical signals, and determine the original depth information corresponding to the target object based on the optical signal sequence.

[0074] Optionally, the terminal may sort more than one optical signal according to the reception time to generate an optical signal sequence. It can be understood that each optical signal in the optical signal sequence is sorted one by one according to the order of their corresponding reception time. Optionally, the terminal may also sort more than one optical signal according to other attributes of the optical signal instead of the reception time to generate an optical signal sequence.

[0075] In one embodiment, the first denoising process and the second denoising process of the depth image denoising method are the same, that is, the denoising processes of the first denoising process and the second denoising process are the same.

[0076] In one embodiment, the first denoising process, the second denoising process and the third denoising process of the above-mentioned depth image denoising method can adopt the same denoising process. It is understandable that the first denoising process, the second denoising process and the third denoising process can also adopt different denoising processes.

[0077] In one embodiment, step 206, that is, the step of performing a second denoising process on the original depth information and then preprocessing it to obtain the second intermediate depth information, specifically includes: performing a second denoising process on the original depth information to obtain denoised depth information, and preprocessing the denoised depth information to obtain the second intermediate depth information; the original depth information is a first image sequence including multiple original phase images corresponding to the optical signal sequence; and the denoised depth information is a denoised phase image corresponding to the original phase image.

[0078] The denoised depth information is the original depth information after denoising. It can be understood that the denoised depth information has less noise than the original depth information. The original phase image is an image obtained by converting the light signal and has different phases. The denoised phase image is a phase image after denoising the original phase image.

[0079] Specifically, the terminal may perform denoising on the multiple original phase images to obtain corresponding denoised phase images. Furthermore, the terminal may perform preprocessing on the denoised phase images to obtain the second intermediate depth information.

[0080] Alternatively, if Figure 3 As shown, the original depth information is subjected to a second denoising process to obtain denoised depth information, that is, a plurality of original phase images are subjected to a second denoising process to obtain corresponding denoised phase images. The following steps are included:

[0081] Step 302: Perform spatial denoising processing on the current original phase image to obtain a spatial denoising image corresponding to the current original phase image.

[0082] Among them, the spatial domain denoising process is to denoise the original phase image in the spatial domain / pixel domain. It can be understood that the denoising process in the spatial domain is denoising at the pixel level. The spatial domain denoising image is the phase image after the spatial domain denoising of the original phase image.

[0083] Specifically, there are multiple original phase images, for example, 2 frames of original phase images, 4 frames of original phase images, or 8 frames of original phase images. For each original phase image, the terminal can perform spatial denoising on the current original phase image to obtain a spatial denoising image corresponding to the current original phase image.

[0084] Step 304, based on the reference denoised image, perform a first time domain denoising process on the spatial domain denoised image corresponding to the current original phase image to obtain a time domain denoised image corresponding to the current original phase image; the reference denoised image is a time domain denoised image corresponding to the previous original phase image of the current original phase image in the first image sequence.

[0085] For example, if the terminal obtains 4 frames of original phase images and is currently processing the second frame of original phase image, the time domain denoised image corresponding to the first frame of original phase image can be used as a reference denoised image for processing the second frame of original phase image.

[0086] Specifically, the terminal can input the reference denoised image and the spatial denoised image corresponding to the current original phase image into the corresponding denoising module. The denoising module can use the reference denoised image as a reference to perform a first time domain denoising process on the spatial denoised image corresponding to the current original phase image to obtain the time domain denoised image corresponding to the current original phase image.

[0087] Step 306: Generate a denoised phase image corresponding to the current original phase image based on the time domain denoised image corresponding to the current original phase image.

[0088] In one embodiment, step 306, that is, the step of generating a denoised phase image corresponding to the current original phase image based on the time domain denoised image corresponding to the current original phase image, specifically includes: performing a second time domain denoising process on the time domain denoised image corresponding to the current original phase image based on the time domain denoised images corresponding to other original phase images among the multiple original phase images, to obtain the denoised phase image corresponding to the current original phase image.

[0089] Specifically, the multiple original phase images include the current original phase image and other original phase images except the current original phase image. The terminal can input the current original phase image and other original phase images in the multiple original phase images into the corresponding time domain denoising module. The time domain denoising module can use the time domain denoising images corresponding to other original phase images in the multiple original phase images as a reference, perform a second time domain denoising process on the time domain denoising image corresponding to the current original phase image, and obtain a denoised phase image corresponding to the current original phase image.

[0090] like Figure 4 As shown, the number of original phase images is n. The terminal can sequentially use each original phase image as the input of the corresponding denoising module, and the corresponding denoising module performs spatial denoising and first time domain denoising on the input original phase image, and outputs a time domain denoised image corresponding to the original phase image. Then, the time domain denoised image is input to the corresponding denoising module, and the corresponding denoising module performs a second time domain denoising on the time domain denoised image, and outputs a denoised phase image corresponding to the original phase image.

[0091] like Figure 5 As shown, during the spatial denoising process and the first temporal denoising process, the number of original phase images is n (Curfrm_1, Curfrm_2, ..., Curfrm_n). The terminal can sequentially use the original phase image as the input of the corresponding spatial denoising module (SNR, Spatial Noise Reduce, i.e., spatial denoising module). The corresponding SNR module can perform spatial denoising on the original phase image and output a spatial denoising image corresponding to the original phase image. The terminal can use the temporal denoising image and the corresponding reference denoising image obtained from the reference denoising image list as the input of the corresponding temporal denoising module (TNR, Temporal noise reduce, i.e., temporal denoising module). The TNR module can refer to the corresponding reference denoising image, perform the first temporal denoising on the spatial denoising image, and output a temporal denoising image corresponding to the original phase image. At the same time, the terminal can also write the output temporal denoising image into the reference denoising image list.

[0092] like Figure 6 As shown, the number of time-domain denoised images is n (Outfrm_1, Outfrm_2, ..., Outfrm_n). During the second time-domain denoising process, the terminal may use the n time-domain denoised images as inputs of the corresponding TNR module. The corresponding TNR module may perform multi-phase denoising on the n time-domain denoised images to obtain corresponding denoised phase images.

[0093] In the above embodiment, two rounds of denoising are performed on the intermediate data necessary for generating the depth image, so that the intermediate data necessary for generating the depth image has higher accuracy, thereby making the generated depth image contain less noise.

[0094] In one embodiment, step 304, that is, referring to the denoised image, performs a first time-domain denoising process on the spatial denoised image corresponding to the current original phase image to obtain the time-domain denoised image corresponding to the current original phase image, specifically comprising: generating a first boundary mapping image based on the difference between the spatial denoised image corresponding to the current original phase image and the reference denoised image; the first boundary mapping image is used to reflect the difference between the spatial denoised image and the reference denoised image; generating a first boundary weight image according to the first boundary mapping image and a pre-set first boundary threshold; generating a time-domain denoised image corresponding to the current original phase image based on the first boundary weight image, the spatial denoised image corresponding to the current original phase image, and the reference denoised image.

[0095] Among them, the first boundary mapping image is a differential image used to reflect the difference between the spatial denoised image and the reference denoised image. The larger the pixel value of the pixel point in the first boundary mapping image, the greater the difference between the corresponding pixel point in the spatial denoised image and the corresponding pixel point in the reference denoised image, and vice versa. The first boundary weight image is an image that represents the weight of the spatial denoised image at each pixel position. The pixel value of each pixel in the first boundary weight image can represent the weight value of the spatial denoised image at each pixel position. The larger the pixel value of the pixel point in the first boundary mapping image, the smaller the pixel value of the corresponding pixel point in the first boundary weight image, and vice versa.

[0096] Specifically, the terminal may compare each pixel in the spatial denoised image corresponding to the current original phase image with each pixel in the reference denoised image to determine the difference between the spatial denoised image corresponding to the current original phase image and the reference denoised image. Furthermore, the terminal may generate a first boundary mapping image based on the difference between the spatial denoised image corresponding to the current original phase image and the reference denoised image. The terminal may generate a first boundary weight image based on the first boundary mapping image and a pre-set first boundary threshold, and generate a time domain denoised image corresponding to the current original phase image based on the first boundary weight image, the spatial denoised image corresponding to the current original phase image, and the reference denoised image.

[0097] Optionally, the terminal may generate a first boundary weight image according to the first boundary mapping image and a preset first boundary threshold, which may be specifically implemented by any one of the following three generating methods:

[0098] The first generation method generates a first boundary weight image through hard judgment of multiple first boundary thresholds. The specific generation process is as follows:

[0099] If map>thr_0, wgt=d_0;

[0100] If map>thr_1 and map<=thr_0, wgt=d_1;

[0101] If map>thr_2 and map<=thr_1, wgt=d_2;

[0102] If map>thr_3 and map<=thr_2, wgt=d_3;

[0103] If map>thr_k+1 and map<=thr_k, wgt=d_k+1;

[0104] If map <thr_k+1,wgt=d_k+2。

[0105] The second generation method is to generate the first boundary weight image by linear fitting of multiple first boundary thresholds. The specific generation process is as follows:

[0106] If map>thr_0, wgt=d_0;

[0107] If map>thr_1 and map<=thr_0, wgt=d_1+(map-thr_1)*(d_1-d_0) / (thr_0-thr_1);

[0108] If map>thr_2 and map<=thr_1, wgt=d_2+(map-thr_2)*(d_2-d_1) / (thr_1-thr_2);

[0109] If map>thr_3 and map<=thr_2, wgt=d_3+(map-thr_3)*(d_3-d_2) / (thr_2-thr_3);

[0110] If map>thr_k+1 and map<=thr_k, wgt=d_k+1+(map-thr_k+1)*(d_k+1-d_k) / (thr_k-thr_k+1);

[0111] If map <thr_k+1,wgt=d_k+2+(map-thr_k+2)*(d_k+2-d_k+1) / (thr_k+1)。

[0112] In the third generation method, the first boundary mapping image and the fitting function of the first boundary threshold generate the first boundary weight image. The specific generation process is as follows:

[0113] wgt=f(thr,map).

[0114] Among them, map is the pixel value of each pixel in the first boundary mapping image; k is a natural number; thr_0 to thr_k+1 are multiple pre-set first boundary thresholds; wgt is the weight value of the spatial denoising image at each pixel position, that is, the pixel value of each pixel in the first boundary weight image to be generated; d_0 to d_k+2 are the specific values ​​of wgt.

[0115] In the above embodiment, a first boundary mapping image is generated that can characterize the difference between the spatial denoised image and the reference denoised image, and then the weights corresponding to each spatial denoised image at each pixel position are determined based on the size of the difference to generate a time domain denoised image after the first time domain denoising processing, thereby improving the denoising effect of the depth image.

[0116] In one embodiment, the time domain denoised image is calculated by the following formula:

[0117]

[0118] Among them, Tnrfrm i,j Snrfrm represents the pixel value of the time-domain denoised image at pixel position (i, j); i,j Represents the pixel value of the spatial denoised image at pixel position (i, j); reffrm i,j represents the pixel value of the reference denoised image at pixel position (i, j); bdr - wgt i,j Represents the pixel value of the first boundary weight image at pixel position (i, j).

[0119] In one embodiment, step 306, that is, the step of performing a second time-domain denoising process on the time-domain denoised image corresponding to the current original phase image based on the time-domain denoised images corresponding to other original phase images among the multiple original phase images to obtain the denoised phase image corresponding to the current original phase image, specifically includes: determining a denoising coefficient according to the time-domain denoised image corresponding to the current original phase image, the time-domain denoised images corresponding to other original phase images among the multiple original phase images, and a preset difference threshold; generating a denoised phase image corresponding to the current original phase image according to the denoising coefficient, the time-domain denoised image corresponding to the current original phase image, and the time-domain denoised images corresponding to other original phase images among the multiple original phase images.

[0120] Among them, the denoising coefficient is a necessary coefficient for denoising the time domain denoised image.

[0121] Specifically, the terminal can determine the denoising coefficient based on the time domain denoised image corresponding to the current original phase image, the time domain denoised images corresponding to other original phase images among multiple original phase images, and a pre-set difference threshold, and generate a denoised phase image corresponding to the current original phase image based on the denoising coefficient, the time domain denoised image corresponding to the current original phase image, and the time domain denoised images corresponding to other original phase images among multiple original phase images.

[0122] In the above embodiment, multi-phase denoising is performed on the time-domain denoised images corresponding to each original phase image through the denoising coefficient corresponding to each pixel position to generate a corresponding denoised phase image, so that the intermediate data required to generate the depth image has higher accuracy, thereby making the generated depth image contain less noise.

[0123] In one embodiment, the denoising coefficient is calculated by the following formula:

[0124] coef_k i,j = abs(outfrm_t i,j-outfrm_k i,j ) <thr?1:0

[0125] Among them, coef_k i,j Indicates the denoising coefficient corresponding to the pixel position (i, j); outfrm_t i,j Indicates the time domain denoised image corresponding to the current original phase image; outfrm_k i,j represents the time domain denoised image corresponding to other original phase images in the multiple original phase images; thr represents a preset difference threshold;

[0126] The denoised phase image is calculated using the following formula:

[0127]

[0128] Among them, phase_t i,j represents the pixel value of the denoised phase image at pixel position (i, j); n represents the number of original phase images.

[0129] In one embodiment, the reference denoised image is obtained from a reference denoised image list, and the above-mentioned depth image denoising method further includes: writing the time domain denoised image corresponding to the current original phase image into the reference denoised image list.

[0130] The reference denoised image list is a list recording reference denoised images.

[0131] Specifically, the terminal may write the time domain denoised image corresponding to the current original phase image into the reference denoised image list to prepare the corresponding reference denoised image for denoising the next original phase image of the current original phase image, thereby improving the depth image denoising rate.

[0132] In one embodiment, the preprocessing includes at least one of filtering, encoding and data format conversion; the number of phase images in the second intermediate depth information after preprocessing is the same as the number of original phase images in the denoised depth information before preprocessing.

[0133] For example, if the number of phase images in the denoised depth information before preprocessing is 4, that is, 4 phase images, the number of phase images in the second intermediate depth information after preprocessing is also 4, that is, 4 phase images.

[0134] In one embodiment, the step of generating the second depth image according to the original depth information in step 210 specifically includes: preprocessing the original depth information to obtain preprocessed depth information; and generating the second depth image based on the preprocessed depth information.

[0135] Specifically, the terminal may preprocess the original depth information to obtain third intermediate depth information, and generate the second depth image based on the third intermediate depth information.

[0136] In the above embodiment, the original depth information is preprocessed first, and then the second depth image is generated based on the third intermediate depth information obtained by the preprocessing. In this way, the generation rate of the second depth image can be increased, thereby further improving the denoising efficiency of the depth image.

[0137] In one embodiment, the second depth image is a second image sequence including a plurality of original depth images corresponding to the optical signal sequence, and the third depth image is a denoised depth image corresponding to the original depth image. Figure 7 As shown, in step 210, performing a third denoising process on the second depth image to obtain a third depth image includes the following steps:

[0138] Step 702, based on the difference between the current original depth image and the reference depth image, generate a second boundary mapping image; the second boundary mapping image is used to reflect the difference between the current original depth image and the reference depth image; the reference depth image is a denoised depth image corresponding to the previous original depth image of the current original depth image in the second image sequence.

[0139] Among them, the original depth image is a depth image directly generated based on the original depth. The denoised depth image is a depth image after the third denoising process is performed on the original depth image. The second boundary mapping image is a differential image used to reflect the difference between the current original depth image and the reference depth image. The larger the pixel value of the pixel point in the second boundary mapping image, the greater the difference between the corresponding pixel point in the current original depth image and the corresponding pixel point in the reference depth image, and vice versa.

[0140] Specifically, the terminal may compare each pixel in the current original depth image with each pixel in the reference depth image to determine the difference between the current original depth image and the reference depth image. Furthermore, the terminal may generate a second boundary mapping image based on the difference between the current original depth image and the reference depth image.

[0141] Step 704: Generate a second boundary weight image according to the second boundary mapping image and a preset second boundary threshold.

[0142] The second boundary weight image is an image representing the weight of each pixel position of the current original depth image, and the pixel value of each pixel in the second boundary weight image can represent the weight value of the current original depth image at each pixel position. The larger the pixel value of the pixel point in the second boundary mapping image, the smaller the pixel value of the corresponding pixel point in the second boundary weight image, and vice versa.

[0143] Specifically, the terminal may generate a second boundary weight image according to the second boundary mapping image and a preset second boundary threshold.

[0144] Optionally, the terminal may generate a second boundary weight image based on the second boundary mapping image and a preset second boundary threshold, which may be achieved specifically by any of the following three generation methods, namely, generating the second boundary weight image by hard judgment of multiple second boundary thresholds, generating the second boundary weight image by linear fitting of multiple second boundary thresholds, and generating the second boundary weight image by fitting function of the second boundary mapping image and the second boundary threshold. The specific generation process of these three generation methods is similar to that of generating the first boundary weight image described above, and will not be described in detail here.

[0145] Step 706: Generate a third depth image based on the second boundary weight image, the second depth image, and the reference depth image.

[0146] Specifically, the terminal may generate a denoised depth image corresponding to the current original depth image based on the second boundary weight image, the current original depth image and the reference depth image.

[0147] In the above embodiment, by generating a second boundary mapping image that can characterize the difference between the current original depth image and the reference depth image, and then determining the weights corresponding to each pixel position of each original depth image based on the size of the difference, a denoised depth image corresponding to the current original depth image after the second time domain denoising process is generated, thereby improving the denoising effect of the depth image.

[0148] In one embodiment, the reference depth image is obtained from a reference depth image list, and the depth image denoising method further includes: writing the denoised depth image corresponding to the current original depth image into the reference depth image list.

[0149] The reference depth image list is a list recording reference depth images.

[0150] Specifically, the terminal may write the denoised depth image corresponding to the current original depth image into the reference depth image list to prepare a corresponding reference depth image for denoising the next original depth image of the current original depth image, thereby improving the depth image denoising rate.

[0151] In one embodiment, Figure 8As shown, the terminal can use the current original depth image (Curfrm) and the reference depth image (reffrm) corresponding to the current original depth image as the input of the corresponding time domain denoising module (TNR). The corresponding time domain denoising module can refer to the reference depth image corresponding to the current original depth image, perform multi-frame denoising on the current original depth image, and output the denoised depth image (Tnrfrm) corresponding to the current original depth image. At the same time, the terminal can also write the denoised depth image corresponding to the current original depth image into the reference depth image list.

[0152] In one embodiment, Fig. 9 As shown, step 212, that is, fusing the first depth image and the third depth image to obtain a target depth image corresponding to the target object, specifically includes the following steps:

[0153] Step 902 : determining a first fusion weight of the first depth image at each pixel position, and determining a second fusion weight of the third depth image at the corresponding pixel position.

[0154] The first fusion weight is the weight value corresponding to each pixel position of the first depth image, and the second fusion weight is the weight value corresponding to each pixel position of the third depth image.

[0155] Specifically, the terminal may determine a first fusion weight of the first depth image at each pixel position, and determine a second fusion weight of the third depth image at a corresponding pixel position.

[0156] Optionally, the first fusion weight and the second fusion weight may be manually set or calculated by a corresponding adaptive algorithm.

[0157] Step 904: Generate a fused image according to the first depth image, the third depth image, the first fusion weight, and the second fusion weight.

[0158] The fused image is an image corresponding to each pixel obtained by fusing each pixel in the first depth image with the corresponding pixel in the third depth image.

[0159] Specifically, the terminal may generate a fused image according to the first depth image, the third depth image, the first fusion weight, and the second fusion weight.

[0160] Step 906: Generate a target depth image corresponding to the target object based on the fused image.

[0161] Specifically, the terminal may generate a target depth image corresponding to the target object based on the fused image.

[0162] In the above embodiment, each pixel of the first depth image and the third depth image is fused and then input into the depth image generator to generate the final target depth image, thereby further improving the accuracy of the depth image.

[0163] In one embodiment, the fused image is calculated using the following formula:

[0164]

[0165] Wherein, i represents the i-th pixel in the first depth image and the third depth image; Represents the pixel value of the i-th pixel in the fused image; The pixel representing the i-th pixel in the first depth image; The pixel representing the i-th pixel in the third depth image; express The weight corresponding to the i-th pixel position; express The weight corresponding to the i-th pixel position.

[0166] like Fig.10 As shown, the traditional depth image denoising system architecture usually includes only one path, that is, the signal transmitter of the terminal transmits a modulated signal to the target object, and the target object reflects the optical signal to the terminal. The terminal receives the optical signal through the signal receiver and generates original depth information based on the optical signal. Then, the terminal can pre-process the original depth information through the processing module, and input it into the depth image generator to generate a depth image, and then denoise the output depth image. The traditional depth image denoising method only denoises the depth image that has been output by the terminal, and its denoising effect is very limited, resulting in low accuracy of the depth image finally output by the terminal.

[0167] However, the deep image denoising system architecture of this application, such as Fig.11 As shown, three denoising channels are included. Among them, channel A and channel B are responsible for denoising the intermediate data required for generating the depth image, and channel C is responsible for denoising the depth image that has been generated. The depth image denoising method of the present application, through three effective denoising channels, denoises the output depth image while also jointly denoising the intermediate data for generating the depth image, thereby ensuring the denoising effect of the depth image and improving the accuracy of the final output depth image.

[0168] In one embodiment, the signal transmitter transmits 8 groups of modulated signals to the target object by controlling the integration time, and the signal receiver can receive 8 groups of optical signals reflected by the target object, perform photoelectric conversion on the 8 groups of optical signals, and then perform 10-bit quantization to obtain 8 original phase images. The terminal can pre-process the 8 original phase images through channel A and then perform a first denoising process to obtain the first intermediate depth information, and perform a second denoising process on the 8 original phase images through channel B and then perform a pre-processing process to obtain the second intermediate depth information. Furthermore, the terminal can fuse the first intermediate depth information and the second intermediate depth information to generate a first depth image. At the same time, the terminal can pre-process the 8 original phase images through channel C to generate a second depth image, and perform a third denoising process on the second depth image to obtain a third depth image. The terminal can fuse the first depth image and the third depth image to obtain a target depth image corresponding to the target object.

[0169] In one embodiment, the signal transmitter transmits 4 groups of modulated signals to the target object by controlling the integration time, and the signal receiver can receive 4 groups of optical signals reflected by the target object, perform photoelectric conversion on the 4 groups of optical signals and then perform 10-bit quantization to obtain 4 original phase images. The terminal bypasses channel A and channel C, and only performs the second denoising process and then pre-processing on the 4 original phase images through channel B to generate a target depth image corresponding to the target object.

[0170] In a specific embodiment, a depth image denoising method is provided, which specifically includes the following process:

[0171] (1) Obtain more than one light signal reflected by the target object.

[0172] (2) Generate an optical signal sequence according to the reception time of more than one optical signal.

[0173] (3) Determine original depth information corresponding to the target object according to the optical signal sequence; the original depth information is a first image sequence including a plurality of original phase images corresponding to the optical signal sequence.

[0174] (4) Preprocessing the original depth information and then performing a first denoising process to obtain first intermediate depth information.

[0175] (5) Perform spatial denoising on the current original phase image to obtain a spatial denoising image corresponding to the current original phase image.

[0176] (6) Based on the difference between the spatial denoised image corresponding to the current original phase image and the reference denoised image, a first boundary mapping image is generated; the first boundary mapping image is used to reflect the difference between the spatial denoised image and the reference denoised image; the reference denoised image is a time domain denoised image corresponding to the previous original phase image of the current original phase image in the first image sequence; the reference denoised image is obtained from the reference denoised image list.

[0177] (7) Generate a first boundary weight image based on the first boundary mapping image and a preset first boundary threshold.

[0178] (8) Based on the first boundary weight image, the spatial domain denoised image corresponding to the current original phase image, and the reference denoised image, a temporal domain denoised image corresponding to the current original phase image is generated.

[0179] Optionally, the time domain denoised image is calculated by the following formula:

[0180]

[0181] Among them, Tnrfrm i,j Snrfrm represents the pixel value of the time-domain denoised image at pixel position (i, j); i,j Represents the pixel value of the spatial denoised image at pixel position (i, j); reffrm i,j represents the pixel value of the reference denoised image at pixel position (i, j); bdr-wgt i,j Represents the pixel value of the first boundary weight image at pixel position (i, j).

[0182] (9) Determine a denoising coefficient according to a time-domain denoised image corresponding to the current original phase image, time-domain denoised images corresponding to other original phase images among the multiple original phase images, and a preset difference threshold.

[0183] (10) Generate a denoised phase image corresponding to the current original phase image according to the denoising coefficient, the time domain denoised image corresponding to the current original phase image, and the time domain denoised images corresponding to other original phase images among the multiple original phase images.

[0184] Optionally, the denoising coefficient is calculated by the following formula:

[0185] coef_k i,j = abs(outfrm_t i,j -outfrm_k i,j ) <thr?1:0

[0186] Among them, coef_k i,jIndicates the denoising coefficient corresponding to the pixel position (i, j); outfrm_t i,j Indicates the time domain denoised image corresponding to the current original phase image; outfrm_k i,j represents the time domain denoised image corresponding to other original phase images in the multiple original phase images; thr represents a preset difference threshold;

[0187] The denoised phase image is calculated using the following formula:

[0188]

[0189] Among them, phase_t i,j represents the pixel value of the denoised phase image at pixel position (i, j); n represents the number of original phase images.

[0190] (11) Write the time domain denoised image corresponding to the current original phase image into the reference denoised image list.

[0191] Optionally, the preprocessing includes at least one of filtering, encoding and data format conversion; the number of phase images in the second intermediate depth information after the preprocessing is the same as the number of original phase images in the denoised depth information before the preprocessing.

[0192] (12) Preprocess the denoised phase image to obtain second intermediate depth information.

[0193] (13) Fusing the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generating a first depth image based on the fused depth information.

[0194] (14) Preprocess the original depth information to obtain third intermediate depth information.

[0195] (15) Generate a second depth image based on the third intermediate depth information; the second depth image is a second image sequence including multiple original depth images corresponding to the optical signal sequence, and the third depth image is a denoised depth image corresponding to the original depth image.

[0196] (16) Based on the difference between the current original depth image and the reference depth image, a second boundary mapping image is generated; the second boundary mapping image is used to reflect the difference between the current original depth image and the reference depth image; the reference depth image is obtained from the reference depth image list.

[0197] (17) Generate a second boundary weight image based on the second boundary mapping image and a preset second boundary threshold.

[0198] (18) Based on the second boundary weight image, the current original depth image and the reference depth image, a denoised depth image corresponding to the current original depth image is generated.

[0199] (19) Write the denoised depth image corresponding to the current original depth image into the reference depth image list.

[0200] (20) Determine a first fusion weight of the first depth image at each pixel position, and determine a second fusion weight of the denoised depth image at the corresponding pixel position.

[0201] (21) Generate a fused image based on the first depth image, the denoised depth image, the first fusion weight and the second fusion weight.

[0202] (22) Based on the fused image, a target depth image corresponding to the target object is generated.

[0203] Optionally, the fused image is calculated using the following formula:

[0204]

[0205] Wherein, i represents the i-th pixel in the first depth image and the denoised depth image; Represents the pixel value of the i-th pixel in the fused image; represents the pixel value of the i-th pixel in the first depth image; represents the pixel value of the i-th pixel in the denoised depth image; express The weight corresponding to the i-th pixel position; express The weight corresponding to the i-th pixel position.

[0206] It should be understood that although Figure 2 , 3 The steps in the flowcharts of , 7, and 9 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 3 At least part of the steps in , 7, and 9 may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequentially, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0207] In one embodiment, Fig.12As shown, a depth image denoising device 1200 is provided, comprising: an acquisition module 1201, a first processing module 1202, a second processing module 1203, an information fusion module 1204, a generation module 1205 and an image fusion module 1206, wherein:

[0208] The acquisition module 1201 is used to acquire original depth information corresponding to the target object.

[0209] The first processing module 1202 is used to perform a first denoising process on the original depth information after preprocessing to obtain first intermediate depth information.

[0210] The second processing module 1203 is used to perform a second denoising process and then preprocessing on the original depth information to obtain second intermediate depth information.

[0211] An information fusion module 1204 is used to fuse the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generate a first depth image based on the fused depth information;

[0212] The generating module 1205 is configured to generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image.

[0213] The image fusion module 1206 is configured to fuse the first depth image and the third depth image to obtain a target depth image corresponding to the target object.

[0214] In one embodiment, the acquisition module 1201 is further used to acquire more than one optical signal reflected by the target object; generate an optical signal sequence according to the reception time of the more than one optical signal; and determine the original depth information corresponding to the target object according to the optical signal sequence.

[0215] In one embodiment, the first denoising process and the second denoising process are the same.

[0216] In one embodiment, the second processing module 1203 is also used to perform a second denoising process on the original depth information to obtain denoised depth information, and pre-process the denoised depth information to obtain second intermediate depth information; the original depth information is a first image sequence containing multiple original phase images corresponding to the optical signal sequence; the denoised depth information is a denoised phase image corresponding to the original phase image. The second processing module 1203 is also used to perform a spatial denoising process on the current original phase image to obtain a spatial denoised image corresponding to the current original phase image; based on the reference denoised image, perform a first time domain denoising process on the spatial denoised image corresponding to the current original phase image to obtain a time domain denoised image corresponding to the current original phase image; the reference denoised image is a time domain denoised image corresponding to the previous original phase image of the current original phase image in the first image sequence; based on the time domain denoised image corresponding to the current original phase image, generate a denoised phase image corresponding to the current original phase image.

[0217] In one embodiment, based on the time domain denoising images corresponding to other original phase images among the multiple original phase images, a second time domain denoising process is performed on the time domain denoising image corresponding to the current original phase image to obtain a denoised phase image corresponding to the current original phase image.

[0218] In one embodiment, the second processing module 1203 is also used to generate a first boundary mapping image based on the difference between the spatial denoised image corresponding to the current original phase image and the reference denoised image; the first boundary mapping image is used to reflect the difference between the spatial denoised image and the reference denoised image; a first boundary weight image is generated according to the first boundary mapping image and a pre-set first boundary threshold; based on the first boundary weight image, the spatial denoised image corresponding to the current original phase image and the reference denoised image, a time domain denoised image corresponding to the current original phase image is generated.

[0219] In one embodiment, the time domain denoised image is calculated by the following formula:

[0220]

[0221] Among them, Tnrfrm i,j Snrfrm represents the pixel value of the time-domain denoised image at pixel position (i, j); i,j Represents the pixel value of the spatial denoised image at pixel position (i, j); reffrm i,j represents the pixel value of the reference denoised image at pixel position (i, j); bdr - wgt i,j Represents the pixel value of the first boundary weight image at pixel position (i, j).

[0222] In one embodiment, the second processing module 1203 is also used to determine a denoising coefficient based on a time-domain denoised image corresponding to the current original phase image, time-domain denoised images corresponding to other original phase images among multiple original phase images, and a pre-set difference threshold; and generate a denoised phase image corresponding to the current original phase image based on the denoising coefficient, the time-domain denoised image corresponding to the current original phase image, and the time-domain denoised images corresponding to other original phase images among multiple original phase images.

[0223] In one embodiment, the denoising coefficient is calculated by the following formula:

[0224] coef_k i,j = abs(outfrm_t i,j -outfrm_k i,j ) <thr?1:0

[0225] Among them, coef_k i,j Indicates the denoising coefficient corresponding to the pixel position (i, j); outfrm_t i,j Indicates the time domain denoised image corresponding to the current original phase image; outfrm_k i,j represents the time domain denoised image corresponding to other original phase images in the multiple original phase images; thr represents a preset difference threshold;

[0226] The denoised phase image is calculated using the following formula:

[0227]

[0228] Among them, phase_t i,j represents the pixel value of the denoised phase image at pixel position (i, j); n represents the number of original phase images.

[0229] In one embodiment, the reference denoised image is obtained from a reference denoised image list, and the depth image denoising apparatus 1201 further includes a first writing module for writing the time domain denoised image into the reference denoised image list.

[0230] In one embodiment, the preprocessing includes at least one of filtering, encoding and data format conversion; the number of phase images in the second intermediate depth information after preprocessing is the same as the number of original phase images in the denoised depth information before preprocessing.

[0231] In one embodiment, the generating module 1205 is further configured to pre-process the original depth information to obtain third intermediate depth information; and generate the second depth image based on the third intermediate depth information.

[0232] In one embodiment, the generation module 1205 is also used to generate a second boundary mapping image based on the difference between the current original depth image and the reference depth image; the second boundary mapping image is used to reflect the difference between the current original depth image and the reference depth image; the reference depth image is a denoised depth image corresponding to the previous original depth image of the current original depth image in the second image sequence; a second boundary weight image is generated based on the second boundary mapping image and a pre-set second boundary threshold; based on the second boundary weight image, the current original depth image and the reference depth image, a denoised depth image corresponding to the current original depth image is generated.

[0233] In one embodiment, the reference depth image is obtained from a reference depth image list, and the depth image denoising device 1201 further includes a second writing module for writing the denoised depth image corresponding to the current original depth image into the reference depth image list.

[0234] In one embodiment, the image fusion module 1206 is also used to determine a first fusion weight of the first depth image at each pixel position, and determine a second fusion weight of the third depth image at a corresponding pixel position; generate a fused image according to the first depth image, the third depth image, the first fusion weight, and the second fusion weight; and generate a target depth image corresponding to the target object based on the fused image.

[0235] In one embodiment, the fused image is calculated using the following formula:

[0236]

[0237] Wherein, i represents the i-th pixel in the first depth image and the third depth image; Represents the pixel value of the i-th pixel in the fused image; represents the pixel value of the i-th pixel in the first depth image; represents the pixel value of the i-th pixel in the third depth image; express The weight corresponding to the i-th pixel position; express The weight corresponding to the i-th pixel position.

[0238] The above-mentioned depth image denoising device obtains the original depth information corresponding to the target object; performs a first preprocessing and then a denoising process on the original depth information to obtain the first intermediate depth information; performs a second denoising process and then a preprocessing on the original depth information to obtain the second intermediate depth information; fuses the first intermediate depth information and the second intermediate depth information to obtain the fused depth information, and generates a first depth image based on the fused depth information; generates a second depth image according to the original depth information, performs a third denoising process on the second depth image to obtain the third depth image; fuses the first depth image and the third depth image to obtain the target depth image corresponding to the target object. In this way, through three effective denoising paths, while denoising the output depth image, the intermediate data for generating the depth image is also jointly denoised, thereby ensuring the denoising effect of the depth image and improving the accuracy of the depth image finally output.

[0239] The division of the various modules in the above-mentioned depth image denoising device is only for illustration. In other embodiments, the depth image denoising device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned depth image denoising device.

[0240] For the specific definition of the depth image denoising device, please refer to the definition of the depth image denoising method above, which will not be repeated here. Each module in the above-mentioned depth image denoising device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0241] Fig.13 FIG. 1 is a schematic diagram of the internal structure of an electronic device in one embodiment. Fig.13 As shown, the electronic device includes a processor and a memory connected via a system bus. Among them, the processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement a depth image denoising method provided in each of the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc.

[0242] Those skilled in the art will understand that Fig.13The structure shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied. The specific server may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0243] The implementation of each module in the depth image denoising device provided in the embodiment of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. The program modules constituted by the computer program may be stored in the memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.

[0244] The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the depth image denoising method.

[0245] A computer program product comprising instructions, when executed on a computer, causes the computer to perform a deep image denoising method.

[0246] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0247] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A deep image denoising method, It is characterized in that The method comprises: Get the original depth information corresponding to the target object; Preprocessing the original depth information and then performing a first denoising process to obtain first intermediate depth information; the preprocessing includes at least one of filtering, encoding and data format conversion; Performing a second denoising process on the original depth information to obtain denoised depth information, and preprocessing the denoised depth information to obtain second intermediate depth information; the original depth information is a first image sequence including a plurality of original phase images corresponding to the optical signal sequence; and the denoised depth information is a denoised phase image corresponding to the original phase image; Fusing the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generating a first depth image based on the fused depth information; Generate a second depth image according to the original depth information, perform a third denoising process on the second depth image to obtain a third depth image; the second depth image is a second image sequence including a plurality of original depth images corresponding to the optical signal sequence, and the third depth image is a denoised depth image corresponding to the original depth image; the third denoising process on the second depth image to obtain the third depth image includes: generating a second boundary mapping image based on the difference between the current original depth image and the reference depth image; the second boundary mapping image is used to reflect the difference between the current original depth image and the reference depth image; the reference depth image is a denoised depth image corresponding to the previous original depth image of the current original depth image in the second image sequence; generate a second boundary weight image according to the second boundary mapping image and a preset second boundary threshold; generate a denoised depth image corresponding to the current original depth image based on the second boundary weight image, the current original depth image and the reference depth image; The first depth image and the third depth image are fused to obtain a target depth image corresponding to the target object.

2. The method according to claim 1, It is characterized in that The obtaining of original depth information corresponding to the target object includes: Acquire more than one optical signal reflected by the target object; generating an optical signal sequence according to the more than one optical signal; Original depth information corresponding to the target object is determined according to the optical signal sequence.

3. The method according to claim 2, It is characterized in that The first denoising process and the second denoising process are the same.

4. The method according to claim 3, It is characterized in that The performing a second denoising process on the original depth information to obtain denoised depth information includes: Performing spatial domain denoising processing on the current original phase image to obtain a spatial domain denoised image corresponding to the current original phase image; Using the reference denoised image as a reference, performing a first time domain denoising process on the spatial domain denoised image corresponding to the current original phase image to obtain a time domain denoised image corresponding to the current original phase image; the reference denoised image is a time domain denoised image corresponding to an original phase image preceding the current original phase image in the first image sequence; Based on the time-domain denoised image corresponding to the current original phase image, a denoised phase image corresponding to the current original phase image is generated.

5. The method according to claim 4, It is characterized in that The step of generating a denoised phase image corresponding to the current original phase image based on the time domain denoised image corresponding to the current original phase image comprises: Based on the time domain denoising images corresponding to other original phase images among the multiple original phase images, a second time domain denoising process is performed on the time domain denoising image corresponding to the current original phase image to obtain a denoised phase image corresponding to the current original phase image.

6. The method according to claim 4, It is characterized in that The step of using the reference denoised image as a reference and performing a first time domain denoising process on the spatial domain denoised image corresponding to the current original phase image to obtain the time domain denoised image corresponding to the current original phase image includes: Generate a first boundary mapping image based on the difference between the spatial domain denoised image corresponding to the current original phase image and the reference denoised image; the first boundary mapping image is used to reflect the difference between the spatial domain denoised image and the reference denoised image; generating a first boundary weight image according to the first boundary mapping image and a preset first boundary threshold; A time domain denoised image corresponding to the current original phase image is generated based on the first boundary weight image, the spatial domain denoised image corresponding to the current original phase image, and the reference denoised image.

7. The method according to claim 6, It is characterized in that The time domain denoised image is calculated by the following formula: Among them, Tnrfrm i,j Snrfrm represents the pixel value of the time-domain denoised image at pixel position (i, j); i,j Represents the pixel value of the spatial denoised image at pixel position (i, j); reffrm i,j bdr_wgt represents the pixel value of the reference denoised image at pixel position (i, j); i,j Represents the pixel value of the first boundary weight image at pixel position (i, j).

8. The method according to claim 5, It is characterized in that The step of performing a second time domain denoising process on the time domain denoised image corresponding to the current original phase image based on the time domain denoised images corresponding to other original phase images among the multiple original phase images to obtain the denoised phase image corresponding to the current original phase image includes: Determine a denoising coefficient according to a time-domain denoised image corresponding to the current original phase image, time-domain denoised images corresponding to other original phase images among the multiple original phase images, and a preset difference threshold; A denoised phase image corresponding to the current original phase image is generated according to the denoising coefficient, the time domain denoised image corresponding to the current original phase image, and the time domain denoised images corresponding to other original phase images among the multiple original phase images.

9. The method according to claim 8, It is characterized in that The denoising coefficient is calculated by the following formula: coef_k i,j =abs(outfrm_t i,j -outfrm_k i,j )<thr?1:0 Among them, coef_k i,j Indicates the denoising coefficient corresponding to the pixel position (i, j); outfrm_t i,j Indicates the time domain denoised image corresponding to the current original phase image; outfrm_k i,j represents the time domain denoised image corresponding to other original phase images in the multiple original phase images; thr represents a preset difference threshold; The denoised phase image is calculated by the following formula: Among them, phase_t i,j represents the pixel value of the denoised phase image at pixel position (i, j); n represents the number of original phase images.

10. The method according to claim 4, It is characterized in that The reference denoised image is obtained from a reference denoised image list, and the method further includes: The time domain denoised image corresponding to the current original phase image is written into the reference denoised image list.

11. The method according to claim 4, It is characterized in that The number of phase images in the second intermediate depth information after preprocessing is the same as the number of phase images in the denoised depth information before preprocessing.

12. The method according to claim 1, It is characterized in that The generating a second depth image according to the original depth information comprises: Preprocessing the original depth information to obtain third intermediate depth information; A second depth image is generated based on the third intermediate depth information.

13. The method according to claim 1, It is characterized in that The reference depth image is obtained from a reference depth image list, and the method further includes: The denoised depth image corresponding to the current original depth image is written into the reference depth image list.

14. The method according to claim 1, It is characterized in that The fusing the first depth image and the third depth image to obtain a target depth image corresponding to the target object includes: Determine a first fusion weight of the first depth image at each pixel position, and determine a second fusion weight of the third depth image at a corresponding pixel position; generating a fused image according to the first depth image, the third depth image, the first fusion weight, and the second fusion weight; Based on the fused image, a target depth image corresponding to the target object is generated.

15. The method according to claim 14, It is characterized in that The fused image is calculated by the following formula: Wherein, i represents the i-th pixel in the first depth image and the third depth image; Represents the pixel value of the i-th pixel in the fused image; represents the pixel value of the i-th pixel in the first depth image; represents the pixel value of the i-th pixel in the third depth image; express The weight corresponding to the i-th pixel position; express The weight corresponding to the i-th pixel position.

16. A deep image denoising device, It is characterized in that The device comprises: An acquisition module is used to obtain original depth information corresponding to the target object; A first processing module, configured to perform a first denoising process on the original depth information after preprocessing, so as to obtain first intermediate depth information; the preprocessing includes at least one of filtering, encoding and data format conversion; A second processing module is used to perform a second denoising process on the original depth information to obtain denoised depth information, and pre-process the denoised depth information to obtain second intermediate depth information; the original depth information is a first image sequence including a plurality of original phase images corresponding to the optical signal sequence; and the denoised depth information is a denoised phase image corresponding to the original phase image; an information fusion module, configured to fuse the first intermediate depth information and the second intermediate depth information to obtain fused depth information, and generate a first depth image based on the fused depth information; A generating module, configured to generate a second depth image according to the original depth information, and perform a third denoising process on the second depth image to obtain a third depth image; the second depth image is a second image sequence including a plurality of original depth images corresponding to the optical signal sequence, and the third depth image is a denoised depth image corresponding to the original depth image; the performing the third denoising process on the second depth image to obtain the third depth image comprises: generating a second boundary mapping image based on a difference between a current original depth image and a reference depth image; the second boundary mapping image is used to reflect the difference between the current original depth image and the reference depth image; the reference depth image is a denoised depth image corresponding to an original depth image preceding the current original depth image in the second image sequence; generating a second boundary weight image according to the second boundary mapping image and a preset second boundary threshold; generating a denoised depth image corresponding to the current original depth image based on the second boundary weight image, the current original depth image and the reference depth image; An image fusion module is used to fuse the first depth image and the third depth image to obtain a target depth image corresponding to the target object.

17. An electronic device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images