Indirect time flight camera depth image denoising method, device and equipment and storage medium

Abstract

The invention discloses an indirect time flight camera depth image denoising method, device and equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining an original color image and an original depth image collected by an indirect time flight camera; performing instance segmentation on the original color image to obtain an instance segmentation image; performing abnormal depth value detection on the instance segmentation region to obtain an abnormal depth value binary image; performing diffusion model denoising processing on the original depth image based on the instance segmentation image and the abnormal depth value binary image to obtain a denoised depth image; and according to the texture complexity of the original depth image, adaptive weighted fusion is carried out on the de-noised depth image and the original depth image to obtain a target depth image, and the de-noising effect and the depth estimation precision of the depth image of the indirect time flight camera are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for denoising depth images from an indirect time-of-flight camera. Background Technology

[0002] With the rapid development of fields such as robot vision, 3D reconstruction, and augmented reality, indirect time-of-flight cameras have become core sensors for acquiring scene depth information due to their advantages of small size, low cost, and high frame rate. These cameras calculate depth by modulating the phase difference of light signals and undertake key tasks of environmental perception and spatial modeling in applications such as industrial inspection, smart homes, and autonomous driving. The market demand for high-quality, high-precision depth images is growing daily.

[0003] Existing depth image denoising techniques primarily rely on traditional filtering or single-depth-domain processing, which struggles to effectively address the unique noise characteristics of indirect time-of-flight cameras. Differences in light reflectivity on object surfaces, multipath interference, and object occlusion generate numerous flying spots at the edges of depth images, deviating significantly from the true values, and causing inconsistencies between the depth and color images. Traditional methods cannot distinguish noise from true structural details, easily losing edge sharpness or producing over-smoothing during denoising. Furthermore, existing solutions are mostly software-based, failing to meet the millisecond-level processing latency requirements of real-time scenarios such as robot vision, and lacking high-efficiency denoising solutions that combine algorithmic and hardware co-design.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for denoising depth images from an indirect time-of-flight camera, aiming to solve the technical problem of how to improve the denoising effect and depth estimation accuracy of depth images from an indirect time-of-flight camera.

[0006] To achieve the above objectives, the present invention provides a method for denoising depth images from an indirect time-of-flight camera, the method comprising the following steps: Acquire raw color and raw depth images captured by the indirect time-of-flight camera; The original color image is segmented into instances to obtain an instance segmentation map; Anomaly depth value detection is performed on the segmented region of the instance to obtain a binary map of the anomaly depth value; Based on the instance segmentation map and the abnormal depth value binary map, the original depth image is subjected to diffusion model denoising to obtain a denoised depth image. Based on the texture complexity of the original depth image, the denoised depth image and the original depth image are adaptively weighted and fused to obtain the target depth image.

[0007] In one embodiment, the step of performing instance segmentation on the original color image to obtain an instance segmentation map includes: The original color image is input into a preset instance segmentation model for inference to obtain an initial segmentation mask; The initial segmentation mask is subjected to an opening operation to remove isolated noise regions, resulting in a first mask. Perform a closing operation on the first optimized mask to fill the internal holes of the mask and obtain the second mask; Perform connectivity analysis on the second optimized mask, remove regions with an area smaller than a preset area threshold, and obtain the third mask; The third mask is subjected to Gaussian filtering to smooth the mask edges and obtain the instance segmentation map.

[0008] In one embodiment, the step of detecting abnormal depth values ​​in the instance segmentation map to obtain an abnormal depth value binary map includes: The pixel depth values ​​of each instance segmentation map are statistically analyzed to obtain the mean depth value and the standard deviation of the depth value; Pixels whose mean depth value deviates from twice the standard deviation of the depth value are marked as preliminary outliers. Determine the rate of change of the nearest neighbor depth value of the preliminary anomaly point, and when the rate of change of the nearest neighbor depth value is greater than a preset rate of change threshold, mark the pixel point as the final anomaly point; Generate a binary map of anomaly depth values ​​based on each of the final anomaly points.

[0009] In one embodiment, the step of performing diffusion model denoising on the original depth image based on the instance segmentation map and the abnormal depth value binary map to obtain a denoised depth image includes: The original depth image, the instance segmentation map, and the binary map of abnormal depth values ​​are input into the conditional diffusion model to obtain the first noise prediction value. The original depth image is input into the unconditional diffusion model to obtain the second noise prediction value; Based on a preset guiding coefficient, the first noise prediction value and the second noise prediction value are weighted and combined to obtain a joint noise prediction value. The original depth image is denoised by performing a preset number of iterations based on the joint noise prediction value to obtain a denoised depth image.

[0010] In one embodiment, the step of adaptively weighting and fusing the denoised depth image and the original depth image based on the texture complexity of the original depth image to obtain the target depth image includes: The horizontal and vertical gradients of each pixel in the original depth image are calculated using the Sobel operator. Calculate the gradient magnitude based on the horizontal gradient and the vertical gradient; The gradient magnitude is averaged within a preset local window to obtain the texture complexity of each pixel. The texture complexity is input into a preset activation function to obtain the fusion weight value; At the anomaly point locations marked in the binary map of the anomaly depth values, the denoised depth image and the original depth image are weighted and fused according to the fusion weight value to obtain the target depth image.

[0011] In one embodiment, the method further includes: The instance segmentation logic, abnormal depth value detection logic, diffusion model denoising logic, and adaptive weighted fusion logic are converted into a hardware description language to obtain the algorithm hardware description code. The algorithm hardware description code is subjected to logical synthesis processing and converted into a gate-level netlist; The gate-level netlist is subjected to layout and routing processing to obtain the physical layout; Functional simulation and timing analysis were performed on the physical layout to verify the results. After the verification result indicates that the verification is successful, an integrated circuit is designed to obtain a chip, which is then embedded into an indirect time-of-flight camera.

[0012] In one embodiment, the step of converting the instance segmentation logic, the abnormal depth value detection logic, the diffusion model denoising logic, and the adaptive weighted fusion logic into a hardware description language to obtain the algorithm hardware description code includes: The instance splitting processing logic is converted into the first functional unit code; The anomaly detection and handling logic is converted into the code of the second functional unit. The diffusion model denoising logic is converted into third functional unit code; The adaptive fusion processing logic is converted into the fourth functional unit code; The first functional unit code, the second functional unit code, the third functional unit code, and the fourth functional unit code are interconnected through a bus interface to obtain the algorithm hardware description code.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes an indirect time-flying camera depth image denoising device, the device comprising: The image reading module is used to acquire the raw color image and raw depth image captured by the indirect time-of-flight camera; The instance segmentation module is used to perform instance segmentation on the original color image to obtain an instance segmentation map; The depth value detection module is used to detect abnormal depth values ​​in the segmented region of the instance and obtain a binary map of abnormal depth values. The diffusion denoising module is used to perform diffusion model denoising on the original depth image based on the instance segmentation map and the abnormal depth value binary map to obtain a denoised depth image. An adaptive fusion module is used to adaptively weight and fuse the denoised depth image and the original depth image according to the texture complexity of the original depth image to obtain a target depth image.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes an indirect time-of-flight camera depth image denoising device, the device comprising: a memory, a processor, and an indirect time-of-flight camera depth image denoising program stored in the memory and executable on the processor, the indirect time-of-flight camera depth image denoising program being configured to implement the steps of the indirect time-of-flight camera depth image denoising method described above.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing an indirect time-of-flight camera depth image denoising program, which, when executed by a processor, implements the steps of the indirect time-of-flight camera depth image denoising method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the indirect time-of-flight camera depth image denoising method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: By using instance segmentation results and abnormal depth value detection information as dual conditions to guide the diffusion model for denoising, flying point noise in indirect time-flying camera depth images is effectively removed while preserving edge details. The denoising results are combined with texture complexity adaptive fusion with the original data to avoid over-smoothing and improve depth estimation accuracy. The algorithm hardening is achieved through dedicated integrated circuits to meet the millisecond-level processing latency requirements of real-time application scenarios such as robot vision and 3D reconstruction, enabling the acquisition of high-quality depth images for complex and variable scenes. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the indirect time-flying camera depth image denoising method of this application. Figure 2 This is a chip workflow diagram provided for Embodiment 1 of the indirect time-of-flight camera depth image denoising method of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the indirect time-flying camera depth image denoising method of this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the indirect time-flying camera depth image denoising method of this application; Figure 5 This is a schematic diagram of the module structure of the indirect time-of-flight camera depth image denoising device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the indirect time-flying camera depth image denoising method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as an indirect time-of-flight camera depth image denoising device. The following description uses an indirect time-of-flight camera depth image denoising device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, embodiments of this application provide a method for denoising depth images from an indirect time-of-flight camera, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the indirect time-of-flight camera depth image denoising method of this application.

[0026] In this embodiment, the indirect time-flying camera depth image denoising method includes steps S10~S50: Step S10: Acquire the raw color image and raw depth image captured by the indirect time-of-flight camera; It's important to note that an indirect time-of-flight camera, also known as an iTOF camera, is an imaging device that calculates scene depth information by emitting modulated light signals and measuring the phase difference of the reflected light. The raw color image refers to the visible light band information of the scene acquired by the camera through its color sensor, with each pixel containing intensity values ​​for the red, green, and blue channels. The raw depth image refers to the scene distance information calculated by the camera using the indirect time-of-flight principle, with each pixel representing the depth value of the corresponding spatial point.

[0027] Understandably, this step involves simultaneously reading the raw color image and raw depth image captured by the indirect time-of-flight camera through the camera interface, ensuring that the two images are synchronized in time and that their pixel coordinates correspond one-to-one in space, thus providing a data foundation for subsequent multimodal information fusion.

[0028] In the specific implementation, the original color image and original depth image captured simultaneously by the iTOF camera are read. When the iTOF camera is working, it simultaneously acquires the original color image and original depth image of the scene. In practical applications, the original image may be affected by factors such as ambient light and sensor noise, resulting in a certain degree of noise and error.

[0029] Step S20: Perform instance segmentation on the original color image to obtain an instance segmentation map; It should be noted that instance segmentation is a computer vision task that separates each target instance in an image from the background and assigns it a unique identifier. An instance segmentation map is a masked image of the same size as the input image, where each pixel stores the number of the instance it belongs to. Pixels of different instances have different number values, and background pixels are numbered zero.

[0030] Understandably, this step performs instance segmentation on the original color image, identifies the boundary contours of each independent object in the image and generates a segmentation mask, which is used to distinguish the depth distribution range of different objects in subsequent steps and provides semantic context constraints for anomaly detection and denoising.

[0031] In the specific implementation, instance segmentation is performed on the RGB image. Instance segmentation separates each target instance in the image and marks the boundaries of different objects. This step uses the lightweight instance segmentation algorithm EdgeSAM to accurately identify various objects in the image and generate precise segmentation masks.

[0032] Step S30: Detect abnormal depth values ​​in the instance segmentation region to obtain a binary map of abnormal depth values; It should be noted that abnormal depth value detection refers to the process of identifying noisy pixels that deviate from the true distance based on depth statistical characteristics. An instance segmentation region refers to the set of connected pixels marked with the same instance number in the instance segmentation map. An abnormal depth value binary map is a binary image with the same size as the depth image, where the pixel values ​​at abnormal point locations are a first preset value, and the pixel values ​​at normal point locations are a second preset value.

[0033] Understandably, this step statistically analyzes the pixel depth values ​​within each instance segmentation region of the original depth image based on the instance segmentation map, calculating the mean and standard deviation of the depth values. Pixels whose depth values ​​deviate from the mean depth value by more than twice the standard deviation are initially marked as outliers. Further, the rate of change of a preset number of nearest neighbor depth values ​​for the initially marked outliers is calculated. When the rate of change of the preset number of nearest neighbor depth values ​​exceeds a preset threshold of 30%, the pixel is finally marked as an outlier. Additionally, a binary map of outlier depth values ​​is generated based on all the finally marked outliers to distinguish reliable regions from noisy regions in the depth image, providing spatial attention guidance for subsequent denoising processing.

[0034] In the specific implementation, abnormal depth value detection is performed on each instance segmentation region. Within each instance segmentation region, the presence of abnormal depth value points is determined based on the statistical characteristics of depth values ​​and the relationship between adjacent pixels. Specifically, the deviation of each pixel's depth value from the average depth value in that region is calculated. If the deviation exceeds a certain threshold, the pixel is marked as an abnormal depth value point. Simultaneously, the depth value changes of adjacent pixels are considered; if the depth values ​​of adjacent pixels change drastically and the pixel's depth value deviates from the normal range, it is also regarded as an anomaly.

[0035] Step S40: Based on the instance segmentation map and the binary map of abnormal depth values, the original depth image is subjected to diffusion model denoising processing to obtain a denoised depth image; It should be noted that diffusion model denoising refers to an iterative noise removal method based on the probability diffusion process. A denoised depth image refers to a high-quality depth image that has undergone noise suppression processing, preserving the true depth structure and removing flying point noise.

[0036] Understandably, this step uses the semantic boundary information provided by the instance segmentation map and the spatial attention mask provided by the binary map of abnormal depth values ​​as dual conditions to perform conditional diffusion model denoising on the original depth image. During the iteration process, the model is guided to prioritize the removal of abnormal point noise and protect the edge structure of instances, generating a denoised depth image with complete details and significantly reduced noise.

[0037] In the specific implementation, a diffusion model is used to denoise the depth image, conditioned on the instance segmentation map and the binary map of abnormal depth points. The diffusion model-based denoising method can remove noise while preserving image details. The instance segmentation map and the binary map of abnormal depth points are used as conditional inputs to the diffusion model, which adaptively removes noise from the depth image based on these conditions. During the denoising process, the model performs different levels of denoising processing on different regions of the depth image according to the input conditional map. More refined processing is applied to areas near instance segmentation regions and abnormal depth points to avoid destroying important image information.

[0038] In one feasible implementation, step S40 includes steps A11 to A14: Step A11: Input the original depth image, instance segmentation map, and binary map of abnormal depth values ​​into the conditional diffusion model to obtain the first noise prediction value; It should be noted that the conditional diffusion model refers to a noise prediction network that can be guided by external conditional information during the denoising process. The first noise prediction value refers to the noise component estimate predicted by the conditional diffusion model based on the input original depth image, instance segmentation map, and anomaly depth value binary map.

[0039] Understandably, this step uses the original depth image as the data to be processed, the instance segmentation map as the semantic condition, and the binary map of abnormal depth values ​​as the spatial attention mask. These are all input into the conditional diffusion model, enabling the model to perceive the depth distribution characteristics of different objects and the location of abnormal noise, and generate noise prediction values ​​corresponding to the current depth image state.

[0040] Step A12: Input the original depth image into the unconditional diffusion model to obtain the second noise prediction value; It should be noted that the unconditional diffusion model refers to a baseline network that predicts noise solely based on the features of the depth image itself, without relying on any external conditions. The second noise prediction value refers to the noise component estimate predicted by the unconditional diffusion model based solely on the content of the original depth image.

[0041] Understandably, this step inputs the original depth image into the unconditional diffusion model separately to obtain a pure data-driven noise estimate that does not depend on semantics and spatial masks. This provides an unbiased reference for subsequent weighted combination and avoids excessive influence of conditional information that could lead to distortion of the denoising results.

[0042] Step A13: Based on the preset guiding coefficient, the first noise prediction value and the second noise prediction value are weighted and combined to obtain the joint noise prediction value; It should be noted that the preset guiding coefficient is a hyperparameter used to adjust the intensity of the influence of conditional information relative to unconditional information. The joint noise prediction value is the comprehensive noise estimate obtained by linearly weighting the first noise prediction value and the second noise prediction value according to the preset guiding coefficient.

[0043] Understandably, this step controls the proportion of conditional information in the final noise prediction by using a preset guiding coefficient to control the proportion of conditional information in the instance segmentation map and the binary map of abnormal depth values. When the preset guiding coefficient is greater than one, the influence of conditional information is enhanced, and when the preset guiding coefficient is less than one, the influence of conditional information is weakened, so that the denoising process achieves a balance between semantic constraints and data authenticity.

[0044] Step A14: Perform a preset number of iterations to denoise the original depth image based on the joint noise prediction value to obtain a denoised depth image.

[0045] It should be noted that the preset number of iterations refers to the number of iterations performed by the diffusion model during the denoising process, i.e., the number of rounds in which noise is gradually removed. A denoised depth image refers to a high-quality depth image obtained after a complete iterative denoising process, in which noise is significantly suppressed and the depth structure is preserved intact.

[0046] Understandably, this step gradually subtracts noise components from the original depth image based on the joint noise prediction value during a preset number of iterations. The depth image state is updated in each iteration, and finally converges to a denoised depth image with noise removal and detail preservation.

[0047] In the specific implementation, the original depth image is denoised using a diffusion model based on the instance segmentation map and the binary map of outlier depth points, effectively removing noise while preserving image details. This invention trains a classifier-free conditional diffusion model to denoise the depth image; the combined noise prediction is as follows:

[0048] in, It is noise predicted by the conditional model. It is noise in the unconditional model prediction. An instance segmentation map representing an RGB image. A binary map representing points with abnormal depth values. It is a hyperparameter used to adjust the influence weight of conditional information. xt is the "noisy image" at step t in the diffusion process of the depth image. The diffusion model generates xt by gradually adding noise to the original depth image. The larger t is, the stronger the noise in xt.

[0049] T is the "time step" of the diffusion process, and its value ranges from 0 to 1. T is the preset total number of diffusion steps. t=0 corresponds to the original depth map, and t=T corresponds to a completely noisy map.

[0050] The depth image is denoised using a classifier-free conditional diffusion model denoising formula to obtain a denoised depth image:

[0051] Where αt is the "noise coefficient" at step t of the diffusion process (a preset sequence of positive numbers less than 1, which controls the intensity of noise added to the image at step t; the larger αt is, the less noise is added). α1 represents the cumulative product from α1 to αt, used to simplify probability calculations in the diffusion process. The larger αt is, the higher the similarity between xt and the original depth map; σt is the noise standard deviation at step t of the diffusion process; z represents the distribution following a standard normal distribution (z0). The random noise of N(0,1) is used to simulate the randomness of the diffusion process, ensuring the model's ability to generalize to different noise scenarios. For the t-th diffusion process The image at step 1 (i.e., the intermediate result obtained after denoising xt, which is iterated step by step until t=0 to obtain the denoised depth map).

[0052] Step S50: Based on the texture complexity of the original depth image, perform adaptive weighted fusion of the denoised depth image and the original depth image to obtain the target depth image.

[0053] It should be noted that texture complexity is a quantitative indicator describing the degree of drastic depth changes in a local region of an image, reflecting the structural richness of that region. The target depth image refers to the final output depth image obtained after adaptive weighted fusion, which achieves the optimal balance between denoising effect and original data fidelity.

[0054] Understandably, this step calculates the texture complexity of the original depth image, dynamically adjusts the fusion weights of the denoised depth image and the original depth image based on the abnormal point positions marked by the binary image of abnormal depth values, preserves the authenticity of the original depth data in flat areas, and adopts the accuracy of the denoised depth image in areas with complex textures, thereby generating a target depth image that both removes noise and retains effective structural information.

[0055] In the specific implementation, the final depth map (S500) is obtained based on the diffusion results and the original acquired depth data. The denoised result of the diffusion model is fused with the original depth data, and the final depth map is obtained by combining the advantages of both through methods such as weighted averaging. For example, the weights are determined based on the noise level and the denoising effect. For areas with high noise, the weight of the denoised data is appropriately increased; for areas with low noise, the weight of the original data is kept relatively high.

[0056] This solution combines the original color image and original depth image captured by the iTOF camera, and uses the RGB image instance segmentation results and abnormal depth value point detection results to assist in the denoising of the depth image, which is different from the traditional single depth image denoising method. It uses the binary image of instance segmentation map and abnormal depth value point as conditions to denoise the original depth image, and uses conditional information to improve the denoising accuracy, which is different from the conventional denoising model. It dynamically adjusts the fusion weight of the denoised depth image and the original depth data according to the texture complexity of the depth image, which is more intelligent and adaptable than the fixed weight fusion method.

[0057] In one feasible implementation, step S50 includes steps A21 to A25: Step A21: Calculate the horizontal and vertical gradients of each pixel in the original depth image using the Sobel operator; The Sobel operator is a discrete differential operator used to calculate the magnitude and direction of image gradients. It consists of two sets of 3x3 convolution kernels, used for detecting horizontal and vertical edges, respectively. The horizontal gradient refers to the rate of change of depth of a pixel in the horizontal direction, and the vertical gradient refers to the rate of change of depth of a pixel in the vertical direction.

[0058] Understandably, this step uses the horizontal and vertical convolution kernels of the Sobel operator to perform convolution operations on the original depth image, respectively, to calculate the depth change rate of each pixel in the horizontal and vertical directions, providing basic gradient information for subsequent texture complexity calculations.

[0059] Step A22: Calculate the gradient magnitude based on the horizontal and vertical gradients; It should be noted that gradient magnitude refers to the overall intensity value of pixel depth changes, reflecting the degree of depth discontinuity in a local region. Gradient magnitude is obtained by taking the square root of the sum of the squares of the horizontal and vertical gradients, and is used to quantify the complexity of the texture.

[0060] Understandably, this step performs a square root operation on the horizontal and vertical gradient values ​​of each pixel to obtain the gradient magnitude. This value is larger at depth edges and in areas with complex textures, and smaller in flat areas, providing a basis for measuring texture complexity.

[0061] Step A23: Average the gradient magnitude within a preset local window to obtain the texture complexity of each pixel; It should be noted that the preset local window refers to a square neighborhood centered on the current pixel, used for statistical analysis of local texture information; its size is determined by preset parameters. Texture complexity refers to the average gradient magnitude within the local window, used to eliminate the influence of abrupt gradient changes in a single pixel and obtain a stable local texture description.

[0062] Understandably, this step averages the gradient magnitude of each pixel across all pixels within a preset local window to obtain the texture complexity of that pixel. This value is higher in textured edge regions and lower in flat regions, and is used to distinguish the structural features of different regions.

[0063] Step A24: Input the texture complexity into the preset activation function to obtain the fusion weight value; It should be noted that the preset activation function refers to a non-linear function that maps texture complexity to fusion weights between zero and one. The fusion weight values ​​are coefficients calculated based on texture complexity and used to control the blending ratio between the denoised depth image and the original depth image.

[0064] Understandably, this step inputs the texture complexity into a preset activation function and converts the texture complexity into a fusion weight value through non-linear mapping. This weight value increases monotonically with the texture complexity, achieving an adaptive adjustment where complex texture areas tend to use denoising results and flat areas tend to retain the original data.

[0065] Step A25: At the abnormal point locations marked in the binary map of abnormal depth values, the denoised depth image and the original depth image are weighted and fused according to the fusion weight value to obtain the target depth image.

[0066] It should be noted that the outlier location refers to the coordinate position of the pixel value in the binary image of the outlier depth value, i.e., the region where the depth outlier pixel is located. Weighted fusion refers to the calculation process of linearly interpolating and combining two input images according to the fusion weight value.

[0067] Understandably, this step only performs the fusion operation at the abnormal point locations marked by the binary image of abnormal depth values. The denoised depth image and the original depth image are weighted and combined according to the fusion weight value. In flat areas with low texture complexity, the original depth image has a higher proportion, and in complex areas with high texture complexity, the denoised depth image has a higher proportion, generating a target depth image that removes noise while retaining effective structural information.

[0068] In its implementation, the adaptive fusion strategy dynamically adjusts the fusion weights based on the local features of the depth image. In regions with complex textures, the denoised depth image may be more reliable; in flat regions, the original depth data may be more reliable. Texture complexity. It is a measure based on gradient magnitude. Gradient magnitude reflects the degree of change in pixel values ​​in an image; typically, regions with complex textures have larger gradient magnitudes. Texture complexity can be measured by calculating the average gradient magnitude of local regions in an image. Let's consider a depth image. At pixel The horizontal gradient at that point is The vertical gradient is Then the gradient magnitude for:

[0069] The horizontal and vertical gradients can be calculated using the Sobel operator.

[0070] For a pixel Centered Local region, texture complexity It can be defined as the average value of the gradient magnitude within this region:

[0071] The Sigmoid function is used to calculate the weights of the denoised depth data:

[0072] Finally, based on the binary image of the abnormal depth points, the abnormal points in the original depth image are adaptively weighted and fused to obtain the target depth image:

[0073] Among them, D denoised The depth data after denoising the diffusion model, i.e., the output denoised result, has less noise but may lose some details. D original The raw depth data acquired by the iTOF camera is detailed but contains noise.

[0074] Furthermore, after step S50, steps A31 to A35 are also included: Step A31: Convert the instance segmentation logic, abnormal depth value detection logic, diffusion model denoising logic, and adaptive weighted fusion logic into a hardware description language to obtain the algorithm hardware description code; It should be noted that a hardware description language is a text-based design language used to describe the structure and behavior of digital circuits. Algorithm hardware description code refers to the code file formed by mapping the aforementioned processing logic to register-transfer level descriptions or synthesizable behavioral level descriptions; that is, hardware description language code.

[0075] Understandably, this step abstracts the algorithm flow of instance segmentation, abnormal depth value detection, diffusion model denoising, and adaptive weighted fusion into hardware. It uses a hardware description language to describe the data path, control state machine, memory interface, and computing unit of each functional unit, generating algorithm hardware description code that can be implemented on digital circuits.

[0076] Step A31 includes: The instance splitting processing logic is converted into the first functional unit code; The anomaly detection and handling logic is converted into the code of the second functional unit. The diffusion model denoising logic is converted into third functional unit code; The adaptive fusion processing logic is converted into the fourth functional unit code; The first functional unit code, the second functional unit code, the third functional unit code, and the fourth functional unit code are interconnected through a bus interface to obtain the algorithm hardware description code.

[0077] It should be noted that the first functional unit code refers to the hardware description code describing the parameter storage and inference calculation process of the instance segmentation model. The second functional unit code refers to the hardware description code describing the logic of statistical calculation and anomaly detection. The third functional unit code refers to the hardware description code describing the logic of noise prediction and iterative control. The fourth functional unit code refers to the hardware description code describing the logic of gradient calculation, weight calculation, and fusion.

[0078] Understandably, this step converts each algorithm logic into independent functional unit code. Each functional unit code includes input / output interfaces, internal registers, operational logic, and control states. The functional units are interconnected through a bus interface to form a complete data processing link, realizing the modular design and integration of the algorithm hardware architecture.

[0079] Step A32: Perform logic synthesis on the algorithm hardware description code to convert it into a gate-level netlist; It should be noted that logic synthesis refers to the process of using electronic design automation (EDA) tools to convert hardware description language code into gate-level circuit descriptions based on standard cell libraries. A gate-level netlist is a circuit description file composed of basic units such as logic gates and flip-flops, and their connections.

[0080] Understandably, this step uses logic synthesis tools to parse, optimize, and map the algorithm hardware description code, converting the high-level behavioral description into a gate-level circuit implementation. At the same time, logic optimization, timing optimization, and area optimization are performed to generate a gate-level netlist that meets the preset synthesis constraints.

[0081] Step A33: Perform placement and routing on the gate-level netlist to obtain the physical layout; It should be noted that placement and routing refers to the process of assigning the positions of logic cells in the gate-level netlist on the chip's physical plane and completing the interconnection between the cells. Physical layout refers to a graphical design file containing all geometric, material, and electrical connection information of the chip; that is, the physical layout.

[0082] Understandably, this step involves physically implementing the gate-level netlist. First, each logic unit is placed in the chip plane, taking into account timing, power consumption, and heat dissipation factors. Then, signal connections between units are completed through multi-layer metal wiring to generate a physical layout that meets design and electrical rules.

[0083] Step A34: Perform functional simulation and timing analysis on the physical layout to verify the results; It should be noted that functional simulation refers to the process of checking the logical functionality correctness of the physical layout. Timing analysis and verification refers to the process of checking the physical layout for meeting timing constraints such as signal transmission delay, clock frequency, and setup / hold time. The verification result refers to a status indicator showing whether the physical layout meets the functional and performance requirements.

[0084] Understandably, this step involves performing functional simulation on the physical layout to confirm that its implemented logic functions are consistent with the algorithm hardware description code. At the same time, static timing analysis is performed to confirm that all timing paths meet the preset clock frequency requirements, generating a result indicating whether the verification passed or failed.

[0085] Step 35: After the verification result indicates that the verification is successful, perform integrated circuit design to obtain the chip, and embed the chip into the indirect time-of-flight camera.

[0086] It should be noted that integrated circuit design refers to the complete engineering process of chip manufacturing, packaging, and testing based on a verified physical layout. A chip is an integrated circuit entity formed by fabricating circuitry on a semiconductor wafer. Embedding refers to the electrical connection and mechanical installation of the chip with components such as optical sensors, image sensors, and control processors in an indirect time-flying camera.

[0087] Understandably, this step initiates the chip manufacturing process after the verification result indicates that the verification has passed. The physical layout is delivered to the wafer fab for manufacturing processes such as photolithography, etching, and diffusion. After packaging and testing, the functional chip is obtained. The chip is then installed on the circuit board of the indirect time-of-flight camera and connects with the camera sensor and controller through a data interface to work collaboratively, thereby achieving real-time noise reduction processing of depth images.

[0088] In the specific implementation, the above depth image processing algorithms are described using a hardware description language (HDL), such as Verilog or VHDL. Taking Verilog as an example, the functional modules such as image reading, instance segmentation, anomaly detection, denoising, and depth map generation are respectively converted into corresponding Verilog modules.

[0089] HDL code is converted into a gate-level netlist using logic synthesis tools (such as Synopsys Design Compiler). During logic synthesis, the code is optimized based on the target chip's technology library, including optimizing the number of logic gates and reducing circuit latency. Simultaneously, appropriate synthesis constraints, such as clock frequency and area constraints, are set to meet the chip's performance requirements.

[0090] Placement and routing tools (such as Cadence Encounter) are used to convert the gate-level netlist into a physical layout. During the placement phase, each logic unit (such as registers and logic gates) is placed appropriately on the chip, taking into account factors such as signal transmission delay, power consumption, and heat dissipation. During the routing phase, the connections between the units are completed to ensure correct signal transmission. Multiple iterations and optimizations are required during the placement and routing process to improve the chip's performance and reliability.

[0091] Before chip manufacturing, comprehensive verification and testing are conducted. Simulation tools (such as ModelSim) are used to perform functional simulations of the designed circuit to verify whether it meets the expected functional requirements. Simultaneously, timing analysis is performed to ensure that the circuit meets timing requirements under various operating conditions. After chip manufacturing is completed, hardware testing is performed, embedding the chip into the iTOF camera system to process the actually acquired images and verify the effectiveness of depth image processing.

[0092] The designed and tested chip is embedded into the iTOF camera for integration. At the same time, system-level debugging and optimization are performed to ensure that the entire camera system can work stably and efficiently.

[0093] Through the specific implementation methods described above, this invention can effectively process depth images acquired by an iTOF camera, improving the quality and measurement accuracy of the depth images to meet the needs of different application scenarios. Furthermore, through dedicated integrated circuit design and chip embedding, high efficiency and real-time performance of depth image processing are achieved, providing strong support for the application of iTOF cameras in more fields.

[0094] Application-Specific Integrated Circuit (ASIC) Design and Chip Embedding: A flowchart of chip operation is shown below. Figure 2 As shown, the process begins with the image reading module (S100) acquiring raw depth and color data. Next, the raw color data is fed into the instance segmentation module (S200) to obtain an instance segmentation map. The raw depth data and instance segmentation map are then fed into the anomaly detection module (S300) to obtain a binary map of anomalous depth points. This binary map of anomalous depth points, the instance segmentation map, and the raw depth data are then fed into the denoising module (S400) as input conditions. The denoising module then transmits the data to the depth map generation module (S500), ultimately yielding denoised depth data. The image reading module reads the raw color and raw depth images simultaneously acquired by the iTOF camera. This module connects to the iTOF camera via a camera interface, enabling it to quickly and accurately acquire image data and transmit it to subsequent processing modules. The instance segmentation module performs instance segmentation on the RGB image. This module uses the aforementioned instance segmentation algorithm to process the input RGB image and output a segmentation mask for each object. The anomaly detection module is responsible for detecting anomalous depth points in each instance segmentation region. This module detects pixels within the instance segmentation region based on a set depth value deviation threshold and rules governing depth value changes between adjacent pixels, marking outlier depth value points. The denoising module uses the instance segmentation map and the binary image of the outlier depth value points as conditions to perform diffusion model denoising on the depth image. This module includes a pre-trained diffusion model capable of efficiently denoising the depth image based on the input conditional map. The depth map generation module generates the final depth map based on the diffusion results and the original acquired depth data. This module uses a fusion algorithm to fuse the denoised result with the original depth data to generate a final high-quality depth map.

[0095] This embodiment provides a method for denoising depth images from an indirect time-of-flight (iTOF) camera. Addressing the issues of noise interference leading to decreased image quality and limited depth estimation accuracy in practical applications, iTOF cameras utilize artificial intelligence algorithms to dynamically adjust and optimize strategies based on real-time acquired camera data. This effectively improves the denoising performance and depth estimation accuracy of iTOF cameras, enabling the acquisition of high-quality images and accurate depth information in complex and varied scenarios. The optimized images and depth data can then be used to expand the application scope of iTOF cameras in fields such as robot vision and 3D reconstruction.

[0096] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S20 includes steps S201 to S205: Step S201: Input the original color image into the preset instance segmentation model for inference to obtain the initial segmentation mask; It should be noted that the preset instance segmentation model refers to a pre-trained and configured deep learning network model used to identify different object instances in an image and generate segmentation results. This model is implemented based on the lightweight instance segmentation algorithm EdgeSAM (Efficient Decoupled Segment Anything Model), which can reduce computational complexity while ensuring segmentation accuracy. The initial segmentation mask refers to the unprocessed binary or class mask image directly output by the model, where each pixel is labeled with its corresponding instance number, but may contain noise, holes, or small false detection areas.

[0097] Understandably, this step inputs the original color image into a preset instance segmentation model to perform forward inference calculation. The model outputs the identified object instances and their corresponding pixel masks through feature extraction, pixel classification, and mask generation processes, thus obtaining an initial segmentation mask containing all detected instances, providing basic data for subsequent mask optimization.

[0098] Step S202: Perform an opening operation on the initial segmentation mask to remove isolated noise regions and obtain the first mask; It should be noted that opening is a morphological image processing method that includes two sub-operations: erosion and dilation. It is used to eliminate isolated noise points or small connected regions in a binary image. Isolated noise regions refer to small clusters of pixels separated from the main object in the initial segmentation mask due to model prediction errors; these regions do not belong to the real object. The first mask is the optimized mask obtained after opening, where isolated noise regions have been eliminated, but the object mask edges may be slightly shrunken.

[0099] Understandably, this step performs a morphological opening operation on the initial segmentation mask. First, an erosion operation is used to eliminate isolated noise regions smaller than the structuring element. Then, a dilation operation is used to restore the main shape of the object mask, resulting in a first mask with noise interference removed, thus improving the purity of the mask.

[0100] Step S203: Perform a closing operation on the first optimized mask to fill the holes inside the mask and obtain the second mask; It should be noted that the first optimized mask is the first mask obtained in step S202. Closing operation is another morphological image processing method, comprising two sub-operations: dilation followed by erosion, used to fill in the empty regions inside objects in a binary image. The holes inside the mask refer to pixel regions within the object instance region of the first mask that are not correctly segmented due to occlusion or uneven reflection, appearing as background pixels inside the mask. The second mask is the optimized mask obtained after the closing operation, where the holes inside the object have been filled and the mask edges are smoothed.

[0101] Understandably, this step performs a morphological closing operation on the first mask. First, the object mask boundary is expanded by dilation to fill the internal voids, and then the mask outline is restored by erosion to obtain a second mask with a complete interior and smooth edges, ensuring the continuity and integrity of the object mask.

[0102] Step S204: Perform connected component analysis on the second optimized mask, remove regions with an area smaller than a preset area threshold, and obtain the third mask; It should be noted that the second optimized mask is the second mask obtained in step S203. Connected component analysis refers to an image processing technique that identifies regions composed of all connected pixels in a binary image and calculates the attributes of each region. The preset area threshold is a pre-set lower limit value for the number of pixels used to filter out false targets that are too small. The third mask is the optimized mask obtained after connected component analysis, in which false detection regions with an area smaller than the preset area threshold have been removed, and the mask that retains the real object instances is preserved.

[0103] Understandably, this step performs connected component analysis on the second mask, identifies all independent connected regions and counts the pixel area of ​​each region, determines regions with an area smaller than a preset area threshold as false detections and removes them, retains the masks of real objects with areas that meet the requirements, and obtains the third mask after filtering out false targets, thereby improving the credibility of the segmentation results.

[0104] Step S205: Perform Gaussian filtering on the third mask to smooth the mask edges and obtain the instance segmentation map.

[0105] It should be noted that the third mask is the optimized mask obtained in step S204. Gaussian filtering is a linear smoothing filtering method that smooths edges and reduces jagged contours by applying a weighted average to the mask image. The instance segmentation map refers to the final segmentation mask obtained after the complete post-processing workflow, in which object boundaries are accurate, continuous, and smooth, and can be directly used for subsequent depth value detection and denoising.

[0106] Understandably, this step performs Gaussian filtering on the third mask, uses a Gaussian kernel to perform smooth convolution on the mask edges, eliminates discrete jagged edges, makes the transition of the segmentation mask edges natural and the positioning accurate, and obtains a high-quality instance segmentation map suitable as a conditional input.

[0107] In the implementation, the preprocessed RGB image is input into the loaded EdgeSAM model for inference. During inference, EdgeSAM extracts and analyzes features of objects in the image, predicting the instance segmentation mask for each object. The obtained segmentation mask is then optimized to improve segmentation accuracy. This includes removing small noise regions, filling mask holes, connected component analysis and filtering, and edge optimization.

[0108] In the segmented mask, small noise regions are usually isolated pixels or small connected regions caused by model prediction errors or interference factors in the image. These small regions may affect subsequent analysis and processing, so they need to be removed. The "opening" operation in morphological operations can effectively achieve this. The opening operation first performs erosion on the image, followed by dilation. Erosion shrinks the object's boundaries, removing isolated pixels; dilation expands the object's boundaries, restoring the original object's approximate shape.

[0109] Holes are regions within an object's mask that are not segmented into the object itself, potentially leading to an incorrect representation of the object's shape. These holes are filled using a "closing" operation, which involves first dilating and then eroding. Dilating expands the object's boundaries, filling in small holes, while erosion restores the expanded boundaries to a more accurate position.

[0110] Besides morphological operations, connected component analysis can be used to further optimize the mask. Connected component labeling algorithms can identify all connected regions in the mask and obtain statistical information for each region. Based on this statistical information, connected regions that meet the criteria can be filtered out, while small or outlier regions that do not meet the criteria can be removed.

[0111] To make the edges of the segmentation mask smoother and more accurate, image filtering algorithms can be used. Gaussian filtering is a commonly used method that smooths the image by applying a weighted average, reducing jagged edges.

[0112] This embodiment provides a method for denoising depth images from an indirect time-of-flight camera. An initial segmentation mask is obtained by inferring from the original color image using a preset instance segmentation model. Then, an opening operation is performed to remove isolated noise, a closing operation to fill internal holes, connected component analysis to remove small-area false targets, and Gaussian filtering to smooth edges. This results in a high-quality instance segmentation map with accurate, continuous, and smooth boundaries. This effectively eliminates noise interference and structural defects introduced by model prediction errors, improves the purity and integrity of the segmentation mask, provides reliable semantic boundary constraints for subsequent depth anomaly detection and conditional denoising, enhances the protection of real object edges during depth processing, and reduces the risk of misjudgment of depth values ​​due to inaccurate segmentation.

[0113] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S30 includes steps S301 to S304: Step S301: Statistically analyze the pixel depth values ​​of each instance segmentation map to obtain the mean depth value and the standard deviation of the depth value; It should be noted that an instance segmentation region refers to a set of connected pixels marked with the same instance number in the instance segmentation map, with each region corresponding to an independent object. The depth mean is the arithmetic mean of the depth values ​​of all pixels within a single instance segmentation region, reflecting the overall distance level of that region. The depth standard deviation is the degree of dispersion of the depth values ​​of all pixels within a single instance segmentation region relative to the mean, measuring the magnitude of the fluctuation in the depth distribution.

[0114] Understandably, this step iterates through each instance segmentation region divided by the instance segmentation map, sums up the depth values ​​of all pixels in the corresponding region of the original depth image, calculates the average value to obtain the mean depth value, and calculates the average of the squared differences between each depth value and the mean, and takes the square root to obtain the standard deviation of the depth value, providing a statistical benchmark for anomaly detection.

[0115] Step S302: Mark pixels whose mean depth value deviation is greater than twice the standard deviation of depth value as preliminary outliers; It should be noted that preliminary outliers refer to candidate pixels suspected of being noise in the first screening. The deviation of the mean depth value is the absolute difference between the actual depth value of a pixel and the mean depth value. Twice the standard deviation of the depth values ​​is a threshold limit obtained by multiplying the standard deviation of the depth values ​​by two, used to measure the degree to which the depth deviates from a reasonable range.

[0116] Understandably, in this step, the pixel depth value is compared with the mean depth value within each instance segmentation region. When the absolute difference exceeds twice the standard deviation of the depth value, the pixel depth value is determined to deviate significantly from the statistical characteristics of the region, and it is marked as a preliminary outlier, thus completing the initial screening of outliers based on statistical distribution.

[0117] Step S303: Determine the rate of change of the nearest neighbor depth value of the preliminary anomaly point. When the rate of change of the nearest neighbor depth value is greater than the preset rate of change threshold, mark the pixel as the final anomaly point. It should be noted that the nearest neighbor depth value is the k-nearest neighbor pixel, and the nearest neighbor depth value change rate refers to the average relative depth difference between the initial outlier and its surrounding preset number of neighboring pixels, reflecting the local depth continuity. The preset change rate threshold is a pre-set percentage limit used to distinguish between true edge jumps and noise flypoints. The final outlier refers to the depth-abnormal pixel confirmed through dual judgment.

[0118] Understandably, this step calculates the rate of change of the depth values ​​of a preset number of neighboring pixels for the initial outlier. When the rate of change is greater than the preset rate of change threshold (i.e., 30%), it indicates that the depth value of the point is significantly different from the neighborhood and does not conform to the continuous change characteristics of the object surface. It is then upgraded to the final outlier to eliminate false detections caused by statistical deviations due to the true edge of the object.

[0119] Step S304: Generate a binary map of anomaly depth values ​​based on each final anomaly point.

[0120] It should be noted that the abnormal depth value binary image is a binary image with the same size as the depth image, where the pixel value of the final abnormal point position is the first preset value and the pixel value of the normal point position is the second preset value, which is used to spatially mark all depth abnormal pixels.

[0121] Understandably, this step iterates through all instance segmentation regions, assigns the pixel positions marked as final outliers to the first preset value in the blank binary map, and assigns the remaining positions to the second preset value, generating an outlier depth value binary map that fully covers all depth outlier regions, providing accurate spatial attention guidance for subsequent diffusion model denoising.

[0122] In the specific implementation, the steps for detecting abnormal depth values ​​in each instance segmented region include: The principle for detecting outliers in depth images is that the depths of segmented regions within the same instance should be similar.

[0123] For each instance segmented region, calculate the mean, standard deviation, and other statistics of the depth values ​​within the region. Iterate through all pixels within the instance segmented region, accumulate the depth values ​​and record the number of pixels, and then calculate the mean and standard deviation.

[0124] Based on the calculated statistics, a depth value deviation threshold is set. This invention sets the threshold to twice the standard deviation of the mean. Pixels within the instance segmentation region are traversed, and the deviation of each pixel's depth value from the mean is calculated. If the deviation exceeds the threshold, it is initially marked as an outlier. Simultaneously, the depth value changes of this pixel relative to its k neighboring pixels are checked. The depth values ​​of neighboring pixels are obtained, and the rate of change of their depth values ​​is calculated.

[0125] in, This represents the depth value of the current pixel. This represents the set of k nearest neighbors of the current pixel. If the rate of change exceeds a certain percentage (the default is 30%), then the pixel is ultimately determined to be an abnormal depth value.

[0126] Based on the results of identifying abnormal depth points, a binary image of the abnormal depth points is generated. In the binary image, the pixel value corresponding to the abnormal depth point is set to 1, and the values ​​of other pixels are set to 0.

[0127] This embodiment provides a method for denoising depth images from an indirect time-of-flight camera. It achieves initial anomaly screening based on the depth distribution of the instance region by statistically analyzing the mean and standard deviation. Secondary verification is performed using the nearest neighbor rate of change, effectively distinguishing between true depth transition edges and noise flypoints, avoiding misclassification of object geometric edges as anomalies and improving anomaly detection accuracy. Furthermore, a spatial mask in the form of a binary image is generated, enabling subsequent denoising processing to accurately locate noise positions. Strong denoising is applied to anomaly regions, while weak denoising or preservation is applied to normal regions, achieving differentiated processing and maximizing the preservation of effective depth structural details while removing flypoint noise.

[0128] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the indirect time-flying camera depth image denoising method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0129] This application also provides a denoising device for depth images from an indirect time-flying camera; please refer to [reference needed]. Figure 5 The indirect time-of-flight camera depth image denoising device includes: Image reading module 10 is used to acquire the original color image and original depth image captured by the indirect time-of-flight camera; Instance segmentation module 20 is used to perform instance segmentation on the original color image to obtain an instance segmentation map; The depth value detection module 30 is used to detect abnormal depth values ​​in the instance segmentation region and obtain a binary map of abnormal depth values. The diffusion denoising module 40 is used to perform diffusion model denoising on the original depth image based on the instance segmentation map and the binary map of abnormal depth values ​​to obtain a denoised depth image. The adaptive fusion module 50 is used to adaptively weight and fuse the denoised depth image and the original depth image according to the texture complexity of the original depth image to obtain the target depth image.

[0130] The indirect time-of-flight camera depth image denoising apparatus provided in this application employs the indirect time-of-flight camera depth image denoising method described in the above embodiments, and can solve the technical problem of how to improve the denoising effect and depth estimation accuracy of indirect time-of-flight camera depth images. Compared with the prior art, the beneficial effects of the indirect time-of-flight camera depth image denoising apparatus provided in this application are the same as those of the indirect time-of-flight camera depth image denoising method provided in the above embodiments, and other technical features in the indirect time-of-flight camera depth image denoising apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0131] In one embodiment, the instance segmentation module 20 is further configured to input the original color image into a preset instance segmentation model for inference to obtain an initial segmentation mask; The initial segmentation mask is subjected to an opening operation to remove isolated noise regions, resulting in the first mask. Perform a closing operation on the first optimized mask to fill the holes inside the mask, and obtain the second mask; Perform connectivity analysis on the second optimized mask, remove regions with an area smaller than a preset area threshold, and obtain the third mask. Gaussian filtering is applied to the third mask to smooth the mask edges and obtain the instance segmentation map.

[0132] In one embodiment, the depth value detection module 30 is further configured to perform statistical analysis on the pixel depth values ​​of each instance segmentation map to obtain the mean depth value and the standard deviation of the depth value. Pixels whose mean depth value deviates from twice the standard deviation of the depth value are marked as preliminary outliers. Determine the rate of change of the nearest neighbor depth value of the initial anomaly point, and mark the pixel as the final anomaly point when the rate of change of the nearest neighbor depth value is greater than the preset rate of change threshold; Generate a binary map of anomaly depth values ​​based on each final anomaly point.

[0133] In one embodiment, the diffusion denoising module 40 is further configured to input the original depth image, instance segmentation map and abnormal depth value binary map into the conditional diffusion model to obtain a first noise prediction value; The original depth image is input into the unconditional diffusion model to obtain the second noise prediction value; Based on the preset guidance coefficient, the first noise prediction value and the second noise prediction value are weighted and combined to obtain the joint noise prediction value; The original depth image is denoised by performing a preset number of iterations based on the joint noise prediction values ​​to obtain a denoised depth image.

[0134] In one embodiment, the adaptive fusion module 50 is further configured to calculate the horizontal and vertical gradients of each pixel in the original depth image using the Sobel operator; Calculate the gradient magnitude based on the horizontal and vertical gradients; The gradient magnitude is averaged within a preset local window to obtain the texture complexity of each pixel. The texture complexity is input into a preset activation function to obtain the fusion weight value; At the anomaly points marked in the binary map of anomaly depth values, the denoised depth image and the original depth image are weighted and fused according to the fusion weight value to obtain the target depth image.

[0135] In one embodiment, the adaptive fusion module 50 is further configured to convert the instance segmentation logic, the abnormal depth value detection logic, the diffusion model denoising logic, and the adaptive weighted fusion logic into a hardware description language to obtain the algorithm hardware description code; The algorithm hardware description code is subjected to logic synthesis processing and converted into a gate-level netlist; Perform placement and routing on the gate-level netlist to obtain the physical layout; Functional simulation and timing analysis were performed on the physical layout to verify the results. After the verification results indicate that the verification is successful, integrated circuit design is carried out to obtain a chip, which is then embedded into an indirect time-of-flight camera.

[0136] In one embodiment, the adaptive fusion module 50 is further configured to convert the instance segmentation processing logic into first functional unit code; The anomaly detection and handling logic is converted into the code of the second functional unit. The diffusion model denoising logic is converted into third functional unit code; The adaptive fusion processing logic is converted into the fourth functional unit code; The first functional unit code, the second functional unit code, the third functional unit code, and the fourth functional unit code are interconnected through a bus interface to obtain the algorithm hardware description code.

[0137] This application provides an indirect time-of-flight camera depth image denoising device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the indirect time-of-flight camera depth image denoising method in the first embodiment described above.

[0138] The following is for reference. Figure 6 This document illustrates a structural schematic diagram of an indirect time-of-flight camera depth image denoising device suitable for implementing embodiments of this application. The indirect time-of-flight camera depth image denoising device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast acquisition devices, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The indirect time-of-flight camera depth image denoising device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0139] like Figure 6As shown, the indirect time-of-flight camera depth image denoising device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the indirect time-of-flight camera depth image denoising device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the indirect time-of-flight camera depth image denoising device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows an indirect time-of-flight camera depth image denoising device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0140] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0141] The indirect time-of-flight camera depth image denoising device provided in this application employs the indirect time-of-flight camera depth image denoising method described in the above embodiments, and can solve the technical problem of how to improve the denoising effect and depth estimation accuracy of indirect time-of-flight camera depth images. Compared with the prior art, the beneficial effects of the indirect time-of-flight camera depth image denoising device provided in this application are the same as those of the indirect time-of-flight camera depth image denoising method provided in the above embodiments, and other technical features in this indirect time-of-flight camera depth image denoising device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0144] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the indirect time-of-flight camera depth image denoising method in the above embodiments.

[0145] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0146] The aforementioned computer-readable storage medium may be included in the indirect time-of-flight camera depth image denoising device; or it may exist independently and not be assembled into the indirect time-of-flight camera depth image denoising device.

[0147] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the indirect time-of-flight camera depth image denoising device, the indirect time-of-flight camera depth image denoising device performs the following actions: acquires the original color image and the original depth image captured by the indirect time-of-flight camera; performs instance segmentation on the original color image to obtain an instance segmentation map; performs abnormal depth value detection on the instance segmentation region to obtain an abnormal depth value binary map; performs diffusion model denoising processing on the original depth image based on the instance segmentation map and the abnormal depth value binary map to obtain a denoised depth image; and performs adaptive weighted fusion of the denoised depth image and the original depth image according to the texture complexity of the original depth image to obtain a target depth image.

[0148] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0151] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described indirect time-of-flight camera depth image denoising method. This solves the technical problem of how to improve the denoising effect and depth estimation accuracy of indirect time-of-flight camera depth images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the indirect time-of-flight camera depth image denoising method provided in the above embodiments, and will not be repeated here.

[0152] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the indirect time-of-flight camera depth image denoising method described above.

[0153] The computer program product provided in this application can solve the technical problem of how to improve the denoising effect and depth estimation accuracy of indirect time-of-flight camera depth images. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the indirect time-of-flight camera depth image denoising method provided in the above embodiments, and will not be repeated here.

[0154] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for denoising depth images from an indirect time-of-flight camera, characterized in that, The method includes: Acquire raw color and raw depth images captured by the indirect time-of-flight camera; The original color image is segmented into instances to obtain an instance segmentation map; Anomaly depth value detection is performed on the segmented region of the instance to obtain a binary map of the anomaly depth value; Based on the instance segmentation map and the abnormal depth value binary map, the original depth image is subjected to diffusion model denoising to obtain a denoised depth image. Based on the texture complexity of the original depth image, the denoised depth image and the original depth image are adaptively weighted and fused to obtain the target depth image.

2. The method as described in claim 1, characterized in that, The step of performing instance segmentation on the original color image to obtain an instance segmentation map includes: The original color image is input into a preset instance segmentation model for inference to obtain an initial segmentation mask; The initial segmentation mask is subjected to an opening operation to remove isolated noise regions, resulting in a first mask. Perform a closing operation on the first optimized mask to fill the internal holes of the mask and obtain the second mask; Perform connectivity analysis on the second optimized mask, remove regions with an area smaller than a preset area threshold, and obtain the third mask; The third mask is subjected to Gaussian filtering to smooth the mask edges and obtain the instance segmentation map.

3. The method as described in claim 1, characterized in that, The step of performing abnormal depth value detection on the instance segmentation map to obtain an abnormal depth value binary map includes: The pixel depth values ​​of each instance segmentation map are statistically analyzed to obtain the mean depth value and the standard deviation of the depth value; Pixels whose mean depth value deviates from twice the standard deviation of the depth value are marked as preliminary outliers. Determine the rate of change of the nearest neighbor depth value of the preliminary anomaly point, and when the rate of change of the nearest neighbor depth value is greater than a preset rate of change threshold, mark the pixel point as the final anomaly point; Generate a binary map of anomaly depth values ​​based on each of the final anomaly points.

4. The method as described in claim 1, characterized in that, The step of performing diffusion model denoising on the original depth image based on the instance segmentation map and the abnormal depth value binary map to obtain a denoised depth image includes: The original depth image, the instance segmentation map, and the binary map of abnormal depth values ​​are input into the conditional diffusion model to obtain the first noise prediction value. The original depth image is input into the unconditional diffusion model to obtain the second noise prediction value; Based on a preset guiding coefficient, the first noise prediction value and the second noise prediction value are weighted and combined to obtain a joint noise prediction value. The original depth image is denoised by performing a preset number of iterations based on the joint noise prediction value to obtain a denoised depth image.

5. The method as described in claim 1, characterized in that, The step of adaptively weighting and fusing the denoised depth image and the original depth image based on the texture complexity of the original depth image to obtain the target depth image includes: The horizontal and vertical gradients of each pixel in the original depth image are calculated using the Sobel operator. Calculate the gradient magnitude based on the horizontal gradient and the vertical gradient; The gradient magnitude is averaged within a preset local window to obtain the texture complexity of each pixel. The texture complexity is input into a preset activation function to obtain the fusion weight value; At the anomaly point locations marked in the binary map of the anomaly depth values, the denoised depth image and the original depth image are weighted and fused according to the fusion weight value to obtain the target depth image.

6. The method as described in claim 1, characterized in that, The method further includes: The instance segmentation logic, abnormal depth value detection logic, diffusion model denoising logic, and adaptive weighted fusion logic are converted into a hardware description language to obtain the algorithm hardware description code. The algorithm hardware description code is subjected to logical synthesis processing and converted into a gate-level netlist; The gate-level netlist is subjected to layout and routing processing to obtain the physical layout; Functional simulation and timing analysis were performed on the physical layout to verify the results. After the verification result indicates that the verification is successful, an integrated circuit is designed to obtain a chip, which is then embedded into an indirect time-of-flight camera.

7. The method as described in claim 6, characterized in that, The step of converting the instance segmentation logic, the abnormal depth value detection logic, the diffusion model denoising logic, and the adaptive weighted fusion logic into a hardware description language to obtain the algorithm hardware description code includes: The instance splitting processing logic is converted into the first functional unit code; The anomaly detection and handling logic is converted into the code of the second functional unit. The diffusion model denoising logic is converted into third functional unit code; The adaptive fusion processing logic is converted into the fourth functional unit code; The first functional unit code, the second functional unit code, the third functional unit code, and the fourth functional unit code are interconnected through a bus interface to obtain the algorithm hardware description code.

8. A noise reduction device for depth images from an indirect time-of-flight camera, characterized in that, The device includes: The image reading module is used to acquire the raw color image and raw depth image captured by the indirect time-of-flight camera; The instance segmentation module is used to perform instance segmentation on the original color image to obtain an instance segmentation map; The depth value detection module is used to detect abnormal depth values ​​in the segmented region of the instance and obtain a binary map of abnormal depth values. The diffusion denoising module is used to perform diffusion model denoising on the original depth image based on the instance segmentation map and the abnormal depth value binary map to obtain a denoised depth image. An adaptive fusion module is used to adaptively weight and fuse the denoised depth image and the original depth image according to the texture complexity of the original depth image to obtain a target depth image.

9. A device for denoising depth images from an indirect time-of-flight camera, characterized in that, The device includes: a memory, a processor, and an indirect time-of-flight camera depth image denoising program stored in the memory and executable on the processor, the indirect time-of-flight camera depth image denoising program being configured to implement the steps of the indirect time-of-flight camera depth image denoising method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores an indirect time-of-flight camera depth image denoising program, which, when executed by a processor, implements the steps of the indirect time-of-flight camera depth image denoising method as described in any one of claims 1 to 7.

Images