Camera image data processing method and system based on noise reduction technology

Abstract

The present invention discloses a camera image data processing method and system based on noise reduction technology, which relates to the field of camera data processing technology. The method comprises the following steps: collecting multi-phase charge quantity data and converting it into a voltage signal matrix, generating a multi-frequency measurement tensor after frequency grouping processing; performing phase compensation and windowing on the extended frequency domain tensor, then performing an inverse Fourier transform, enhancing the time domain resolution through interpolation, and suppressing transient noise using median filtering to generate a transient time domain image; extracting time domain waveform features from the transient time domain image, segmenting confidence regions using confidence thresholds, and combining multi-peak detection with random forests to distinguish valid reflection signals, thereby outputting a depth probability map; performing channel-space dual attention weighting on the depth probability map, modeling neighborhood correlation using graph convolution, and implementing hybrid coding to output a noise-resistant depth map. The present invention implements hybrid coding using an uncertainty matrix, reducing data redundancy while ensuring depth map accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of camera data processing, and in particular to a camera image data processing method and system based on noise reduction technology. Background Art

[0002] In recent years, indirect time-of-flight (iToF) sensors have been widely used in the fields of three-dimensional reconstruction, autonomous driving, and robot navigation due to their real-time depth perception capabilities. iToF technology calculates the distance to the target by emitting a modulated light signal and detecting the phase delay of the reflected signal. Traditional methods usually use a single modulation frequency, combined with four-phase sampling to resolve depth information, and reduce interference through ambient light suppression algorithms. As application scenarios become more complex, researchers have proposed multi-frequency modulation technology to balance short-range resolution and long-range measurement range. For example, high- and low-frequency alternating emission modes are used, combined with time delay offsets to adapt to the charge integration characteristics of different frequencies. In addition, noise modeling techniques (such as Gaussian-Poisson mixed noise separation) and non-uniformity correction matrices are introduced to improve the consistency of sensor data.

[0003] However, existing technologies still have significant shortcomings: first, the joint processing of multi-frequency signals lacks a dynamic optimization architecture, resulting in inefficient cross-band feature fusion, especially in long-distance multipath interference scenarios, where high-frequency aliasing and low-frequency phase ambiguity problems coexist; second, the existing frequency domain-time domain conversion method has insufficient compensation for the phase offset of the sensor hardware, resulting in a decrease in the contrast of the time domain pulse signal. At the same time, traditional window functions (such as rectangular windows) are prone to introduce spectral leakage, affecting the peak positioning accuracy of weak reflection signals. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a camera image data processing method based on noise reduction technology to solve the problems of decreased depth measurement accuracy and insufficient signal-to-noise ratio caused by multi-frequency aliasing, phase offset and noise interference in complex scenes.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a camera image data processing method based on noise reduction technology, which includes collecting multi-phase charge quantity data and converting it into a voltage signal matrix, and generating a multi-frequency measurement tensor after frequency grouping processing;

[0008] The multi-frequency measurement tensor is input into a convolutional neural network with an encoder-decoder structure, and an adversarial training mechanism is used to generate an extended frequency domain tensor covering all target frequencies.

[0009] Phase compensation and windowing are performed on the extended frequency domain tensor, followed by inverse Fourier transform. The time domain resolution is enhanced by interpolation and transient noise is suppressed by median filtering to generate a transient time domain image.

[0010] Extract time domain waveform features from transient time domain images, segment confidence regions using confidence thresholds, and combine multi-peak detection with random forests to distinguish valid reflection signals, outputting a depth probability map.

[0011] Channel-spatial dual attention weighting is performed on the depth probability map, graph convolution is used to model neighborhood correlation and hybrid coding is implemented to output a noise-resistant enhanced depth map.

[0012] As a preferred solution of the camera image data processing method based on noise reduction technology of the present invention, wherein: generating a multi-frequency measurement tensor includes the following steps:

[0013] The indirect time-of-flight sensor alternately outputs sinusoidal modulation waveforms of different frequencies;

[0014] At each frequency, the integrating capacitor is controlled to start the charge and discharge cycle at multiple phase points, and the charge amount data at each phase point is collected;

[0015] The charge data of each phase point are arranged according to the spatial dimension and spliced ​​into the original charge matrix, which is converted into a voltage signal matrix and then reorganized into a multi-frequency measurement tensor by frequency grouping.

[0016] As a preferred solution of the camera image data processing method based on noise reduction technology of the present invention, wherein: the inputting of the multi-frequency measurement tensor into the convolutional neural network of the encoder-decoder structure includes the following steps:

[0017] The encoder compresses the spatial dimension and expands the number of channels through multi-stage strided convolution, and combines skip connections to preserve intermediate features of different scales;

[0018] The decoder restores the spatial resolution step by step through transposed convolution and concatenates it with the intermediate features of different scales saved by the encoder to generate the final feature map;

[0019] In the final feature map, complex components covering all target frequencies are generated through extended frequency domain extrapolation and adversarial training.

[0020] As a preferred solution of the camera image data processing method based on noise reduction technology of the present invention, wherein: performing phase compensation and windowing processing on the extended frequency domain tensor includes the following steps:

[0021] Load the pre-calibrated sensor frequency response parameter matrix and perform phase compensation on the complex tensor covering all target frequencies;

[0022] A window function is applied to the phase-compensated complex component based on the Hanning window weight to suppress high-frequency oscillations, and the real and imaginary components are merged to generate a frequency domain tensor after windowing.

[0023] As a preferred solution of the camera image data processing method based on noise reduction technology of the present invention, the confidence threshold segmentation and multi-peak detection include the following steps:

[0024] Divide high and low confidence areas according to the peak confidence heat map;

[0025] Perform multi-order derivative analysis on the time domain waveform in the low confidence area to calculate the curvature value and screen the candidate peaks;

[0026] The candidate peaks are probabilistically classified using a pre-trained random forest model, and the effective peaks are determined based on the time priority principle.

[0027] As a preferred solution of the camera image data processing method based on noise reduction technology of the present invention, wherein: the channel-space dual attention weighting includes the following steps:

[0028] Extract direct and indirect light paths from the depth probability map to generate a multimodal tensor, and perform channel-wise weighting to generate a spatial feature map;

[0029] The spatial feature map is doubly weighted by spatial attention weights, and the neighborhood features are aggregated by graph convolution to output the optimized depth map.

[0030] As a preferred solution of the camera image data processing method based on noise reduction technology described in the present invention, the hybrid coding refers to calculating the uncertainty matrix based on the optimized indirect light path probability map, dividing the image into sub-blocks of predetermined size, performing lossless coding on high-confidence sub-blocks, and performing lossy compression on low-confidence sub-blocks.

[0031] In a second aspect, the present invention provides a camera image data processing system based on noise reduction technology, comprising: a multi-frequency acquisition module that collects multi-phase charge quantity data and converts it into a voltage signal matrix, and generates a multi-frequency measurement tensor after frequency grouping processing;

[0032] The frequency domain expansion module inputs the multi-frequency measurement tensor into a convolutional neural network with an encoder-decoder structure and uses an adversarial training mechanism to generate an extended frequency domain tensor covering all target frequencies;

[0033] The time domain noise reduction module performs phase compensation and windowing on the extended frequency domain tensor, then performs inverse Fourier transform, enhances the time domain resolution through interpolation, and suppresses transient noise using median filtering to generate a transient time domain image.

[0034] The signal classification module extracts time domain waveform features from transient time domain images, segments confidence regions using confidence thresholds, and combines multi-peak detection with random forests to distinguish valid reflection signals, outputting a depth probability map.

[0035] The depth optimization module performs channel-spatial dual attention weighting on the depth probability map, uses graph convolution to model neighborhood correlation and implements hybrid coding to output a noise-resistant and enhanced depth map.

[0036] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the camera image data processing method based on noise reduction technology as described in the first aspect of the present invention is implemented.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the camera image data processing method based on noise reduction technology as described in the first aspect of the present invention is implemented.

[0038] The beneficial effects of the present invention are as follows: a Gaussian-Poisson mixed noise model is generated based on the quantum efficiency term and the Box-Muller transform, and the noise ratio is dynamically allocated in combination with the sensor dark field calibration data to effectively simulate the real noise distribution; an extended frequency domain tensor is generated through frequency domain extrapolation adversarial training, and the frequency domain resolution is improved to the upper limit of the extrapolation frequency by combining the phase compensation matrix and the Hanning window, ensuring that the time domain waveform after the inverse Fourier transform has high contrast and low ringing artifacts; a multi-peak detection mechanism guided by the confidence heat map is adopted, and effective reflections and noise are distinguished through curvature analysis and the random forest model, with the earliest arriving peak being given priority; hybrid coding is implemented based on the uncertainty matrix, lossless details are retained in high-confidence areas, and DCT dynamic quantization is used to compress noise energy in low-confidence areas, reducing data redundancy while ensuring the accuracy of the depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 The flowchart of the camera image data processing method based on noise reduction technology.

[0041] Figure 2 Generate a schematic for a multi-frequency measurement tensor.

[0042] Figure 3Schematic diagram of the encoder-decoder processing flow.

[0043] Figure 4 Schematic diagram of confidence threshold segmentation and multi-peak detection. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0047] Reference Figures 1 to 4 , is an embodiment of the present invention, which provides a camera image data processing method based on noise reduction technology, comprising the following steps:

[0048] S1. Configure the optical signal transmission module of the indirect time-of-flight (iToF) sensor to alternately output 20 MHz and 100 MHz sinusoidal modulation waveforms, and set the modulated light duration of each frequency to 5 ms.

[0049] At a modulation frequency of 20MHz, the integrating capacitor is controlled to start the charge and discharge cycle at four phase points: 0°, 90°, 180°, and 270°. The delay offset corresponding to each phase point is 0ns, 12.5ns, 25ns, and 37.5ns, and the charge data of each pixel at the four phase points is collected. At a modulation frequency of 100MHz, the phase point control process is repeated, and the delay offset is adjusted to 0ns, 2.5ns, 5ns, and 7.5ns, and the charge data of the corresponding phase points is collected.

[0050] It should be noted that high frequency (100MHz) can improve short-range resolution, and low frequency (20MHz) can expand the long-range measurement range (reduce phase aliasing); 0°, 90°, 180°, and 270° phase sampling can eliminate ambient light interference, extract complete sinusoidal period information, and ensure depth solution accuracy; the delay offset adaptation frequency can match the sensor charge integration characteristics and maximize signal utilization.

[0051] The four phase charge data at 20 MHz and 100 MHz are arranged into 120 × 160 × 4 sub-tensors according to the spatial dimensions, where the third dimension is in the order of 0°, 90°, 180°, and 270°. The 20 MHz and 100 MHz sub-tensors are concatenated along the third dimension to generate an original charge matrix with dimensions of 120 × 160 × 8. The charge matrix is ​​converted into a voltage signal matrix based on the charge-to-voltage conversion coefficient of the indirect time-of-flight sensor.

[0052] The voltage signals are extracted by frequency grouping: the 20 MHz frequency group contains channels 1-4, and the 100 MHz frequency group contains channels 5-8, which are reorganized into a 120 × 160 × 4 × 2 tensor. The voltage signals of the two frequencies are separated along the fourth dimension to generate the original voltage signal matrix with a final dimension of 120 × 160 × 4.

[0053] It should be noted that the 20MHz and 100MHz data are separated by frequency grouping, retaining the independence between frequencies and facilitating optimized processing for different frequency bands.

[0054] The charge data of each pixel at four phase angles under 20MHz and 100MHz modulation frequencies were extracted from the original voltage signal matrix with a dimension of 120×160×4. The same transformation was performed on the charge data of the 20MHz frequency group and the charge data of the 100MHz frequency group, and the quantum efficiency terms were calculated respectively. A random noise value conforming to the standard Gaussian distribution was independently generated for each phase angle using the Box-Muller transform method, and the random noise value was superimposed with the quantum efficiency term according to frequency and phase. The superimposed four-phase data of the 20MHz and 100MHz frequency groups were reorganized into an intermediate photon counting matrix in the original channel order. The third dimension of the intermediate photon counting matrix was compressed to retain only the valid channel data corresponding to the 0° and 90° phases, and the invalid phase channels were zeroed, finally generating an initial photon counting matrix with a dimension of 120×160×4.

[0055] It should be noted that the zeroing process performed on the invalid phase channel is to suppress the 180° and 270° phase noise (which is greatly affected by the non-ideality of the circuit), retain the 0° and 90° valid signals, and improve the signal-to-noise ratio.

[0056] Traverse each pixel position and phase channel of the initial photon counting matrix to extract the expected number of photons at the current position; generate a uniformly distributed random number r∈[0,1] for each expected number of photons, and use the linear congruential generator algorithm to generate a random sequence seed with a fixed value to ensure repeatability.

[0057] The noise distribution characteristics of the indirect time-of-flight sensor in dark field (no light input) and uniform illumination conditions are calibrated to balance the contributions of the two noise types. The noise generation method is selected by setting the random number threshold to 0.3:

[0058] When r<0.3, Gaussian noise generation is performed, by calculating the standard deviation and using Box-Muller transformation to generate standard normal distribution variables and synthesize Gaussian noise values;

[0059] When r≥0.3, Poisson noise generation is performed, and the inverse transformation method is used to generate Poisson noise values ​​through cumulative distribution function comparison;

[0060] The Gaussian noise value and the Poisson noise value are logically combined according to conditional selection to obtain the noise photon count of each pixel; a saturation truncation operation is performed on the noise photon count that exceeds the maximum photon counting capacity of the sensor; and all truncated noise photon values ​​are reorganized in the dimensional order of the initial photon counting matrix to generate a noise photon counting matrix.

[0061] It should be noted that the noise of the sensor under no-light conditions (dark field) is mainly composed of thermal electron emission and circuit noise, and its statistical characteristics follow a Gaussian distribution. Through measured dark field data, it is found that the noise accounts for about 30% of the total noise energy. Under light conditions, the randomness of the arrival of photons leads to Poisson noise, and the intensity is proportional to the square root of the signal strength. When the signal strength is within the normal working range, Poisson noise accounts for about 70%. Therefore, the random number threshold is set to 0.3.

[0062] The noise photon counting matrix is ​​separated into four sub-matrices according to the phase channel order, corresponding to the phase angles of 0°, 90°, 180° and 270° respectively; the 0° and 180° phase sub-matrix data of the 20MHz and 100MHz frequencies are subtracted to generate the real part, and the 90° and 270° phase sub-matrix data are subtracted to generate the imaginary part; the real and imaginary parts of the 20MHz and 100MHz frequencies are respectively combined into complex form to form complex component matrices containing two channels; the 20MHz and 100MHz complex component matrices are spliced ​​along the channel dimension to generate the original complex tensor containing four channels.

[0063] It should be noted that the calculation of the real and imaginary parts (0°, 90°, 180°, 270°) can eliminate the DC offset, extract the AC signal component, and enhance the contrast of the effective reflection signal.

[0064] Load a pre-calibrated non-uniformity correction matrix from non-volatile memory.

[0065] It is further explained that the non-uniformity correction matrix is ​​generated by the indirect time-of-flight sensor in the uniform illumination calibration experiment to compensate for the pixel-level response differences (such as quantum efficiency deviation and circuit gain fluctuation) and improve the spatial consistency of the data.

[0066] Traverse each spatial pixel point of the original complex tensor and extract the complex component data of the four channels in each spatial pixel point; perform correction operation on the real channel of 20MHz frequency and multiply the complex component data by the correction matrix value at the corresponding position; repeat the same multiplication correction operation on the 20MHz imaginary part, 100MHz real part and imaginary part channels in turn; reassemble the corrected four channel data into an intermediate tensor in the original order; verify whether the real and imaginary part values ​​of all channels of the intermediate tensor exceed the range of the analog-to-digital converter, saturate and truncate the excess part, and output the corrected intermediate tensor.

[0067] Traverse each spatial position and channel index of the corrected intermediate tensor and extract the real and imaginary complex components corresponding to the 20 MHz and 100 MHz frequencies. Calculate the modulus of the real and imaginary parts of each complex component and use a fast square root algorithm to obtain the square root of the corresponding sum of squares. Record all modulus calculation results and determine the global maximum modulus value through a parallel reduction algorithm. Normalize the real and imaginary parts of each complex component based on the global maximum modulus value. Recombine the normalized real and imaginary parts in the original channel order to generate a normalized multi-frequency measurement tensor containing the 20 MHz and 100 MHz phase components.

[0068] S2. Extract four-channel data for each spatial position from the normalized multi-frequency measurement tensor; initialize the three-dimensional convolution kernel of the input convolution layer and configure the weight parameters; perform a convolution operation on the input four-channel data to generate 64-channel intermediate eigenvalues; process the intermediate eigenvalues ​​through a nonlinear activation function and suppress negative outputs to obtain an activated feature map; perform channel-level normalization on the activated feature map to stabilize the data distribution; verify the normalized feature map, maintain the original spatial resolution and expand the number of channels to form an initial feature map with a dimension of 120×160×64.

[0069] It should be noted that by fusing the real and imaginary components of multiple frequencies (20MHz / 100MHz) with a three-dimensional convolution kernel, cross-band joint features are extracted and the ability to model inter-frequency correlation is enhanced; the number of channels is expanded to 64, which improves the feature expression capability while maintaining the original spatial resolution and avoiding early information loss.

[0070] Extract each 3×3 local region from the initial feature map of dimension 120×160×64, covering four channels of input data; initialize 128 independent 3×3 convolution kernels, where the weight matrix of the convolution kernel is initialized by He normal distribution and the bias term is initialized to zero;

[0071] A convolution operation with a stride of 2 is performed on each 60×80 output position: every other pixel is sampled in the horizontal and vertical directions, and the boundary integrity is maintained by padding 1, and a 60×80×128 feature map is output; a ReLU activation function is applied to the output 60×80×128 feature map to suppress negative values ​​and retain positive activations; the mean and variance of 128 channels are calculated along the batch dimension, and batch normalization is applied for processing; the processed feature map with a dimension of 60×80×128 is saved as skip connection 1 for use in the first upsampling layer of the decoder.

[0072] Taking the 60×80×128 feature map of skip connection 1 as input, initialize 256 3×3 convolution kernels. The parameter initialization method is the same as the first downsampling. Perform convolution operation with stride 2 and padding 1, and output a feature map of 30×40×256 dimensions. Perform ReLU activation and batch normalization on the 30×40×256 feature map in sequence, save the result as skip connection 2, and pass it to the second upsampling layer of the decoder.

[0073] Taking the 30×40×256 feature map of skip connection 2 as input, initialize 512 3×3 convolution kernels; perform convolution operations with the same parameters, and output a feature map of dimension 15×20×512; apply ReLU activation and batch normalization to the feature map of dimension 15×20×512, and save the final downsampling result as skip connection 3.

[0074] It should be noted that through three stride-2 convolutions, the spatial dimension is compressed step by step (4x → 16x → 64x downsampling), the receptive field is expanded, and the global scene structure (such as large-scale object contours) is captured; the number of channels is doubled (64 → 128 → 256 → 512), and the abstraction ability of high-level semantic features (such as material, motion blur) is enhanced; the skip connection preservation is to retain intermediate features of different scales for the decoder to fuse multi-resolution information and restore details when upsampling.

[0075] The skip connection 3 of the third downsampling output is used as input to extract the feature vector of each spatial position; 512 3×3 dilated convolution kernels are initialized with parameters of dilation rate 2 and padding 2;

[0076] A dilated convolution operation is performed on the feature vector of each spatial position: at each 15×20 position, a 3×3 neighborhood is sampled with an interval of 2 pixels, actually covering a 7×7 physical area; the eigenvalue of each output position is calculated by dot multiplication and accumulation, and the output dimension is a feature map of 15×20×512.

[0077] It should be noted that the dilated convolution is performed without reducing the resolution (15×20×512), covering a 7×7 physical area, capturing long-range dependencies (such as long-distance multipath interference patterns), avoiding the loss of spatial information caused by traditional downsampling, and maintaining sensitivity to small targets.

[0078] Initialize 256 2×2 transposed convolution kernels, initialize the weights by bilinear interpolation, and set the bias term to zero; perform a transposed convolution operation with a stride of 2 on the feature map of dimension 15×20×512: insert zero values ​​at each 15×20 position and perform convolution, increase the spatial resolution to 30×40, reduce the number of channels to 256, and output a feature map of dimension 30×40×256.

[0079] Initialize 128 2×2 transposed convolution kernels and perform a transposed convolution operation with a stride of 2 on the feature map of dimension 30×40×256: restore the resolution to 60×80, reduce the number of channels to 128, output a feature map of dimension 60×80×128, and concatenate it with the encoder skip connection 1 along the channel dimension to generate a fused feature map of dimension 60×80×256.

[0080] Initialize 64 2×2 transposed convolution kernels, perform a transposed convolution operation with a step size of 2 on the fused feature map of dimension 60×80×256, restore the original resolution 120×160, reduce the number of channels to 64, and output a feature map of dimension 120×160×64. It is then concatenated with the feature map of dimension 120×160×64 output by the input convolution layer along the channel dimension to generate a final feature map of dimension 120×160×128.

[0081] The feature vector of each spatial position is extracted from the final feature map with a dimension of 120×160×128; 40 independent 1×1 convolution kernels are initialized, and a fully connected convolution operation is performed on the feature vector of each spatial position: the 128-dimensional feature vector is point-multiplied and accumulated with the 40 convolution kernels to generate a 40-channel output, and the output channels are arranged in ascending order of frequency to generate an extended frequency domain tensor containing multi-frequency real and imaginary components; a loss function is constructed based on the absolute error between the extended frequency domain tensor containing multi-frequency real and imaginary components and the true value Fourier coefficients to guide the frequency domain extrapolation; a discriminator network is designed to analyze the data distribution difference between the extended frequency domain tensor and the true value; the adversarial training mechanism is used to optimize the reconstruction accuracy of the high-frequency components of the extended frequency domain tensor, and finally an extended frequency domain tensor covering all target frequencies is output.

[0082] S3. Extract the real and imaginary data of the frequency channel from each spatial position of the extended frequency domain tensor covering the full target frequency; load the pre-calibrated sensor frequency response parameter matrix, which stores the phase compensation factor for each frequency point; and perform a phase compensation operation on the channel index corresponding to each sensor frequency response parameter matrix.

[0083] To further explain, the phase compensation operation refers to calculating the compensated complex component based on the extracted real and imaginary data of each frequency channel, eliminating the phase error introduced by the hardware, and enhancing the contrast of the pulse signal after the frequency domain-time domain conversion.

[0084] It should be noted that the purpose of loading the frequency response parameter matrix is ​​to compensate for the inherent phase offset of the sensor at different frequency points (such as circuit delay and quantum efficiency difference), ensure the phase consistency of multi-band signals, and improve the alignment accuracy of full-frequency domain data.

[0085] The real and imaginary data of the compensated complex components are rewritten into an extended frequency domain tensor covering all target frequencies in the order of frequency channels to generate a phase-aligned frequency domain tensor.

[0086] According to the frequency range, multiple discrete frequency points are generated; the Hanning window weight is calculated for each discrete frequency point; the phase-aligned frequency domain tensor is separated according to the odd and even index, where the odd channels are the real part and the even channels are the imaginary part; the separated frequency domain tensor is matched with the Hanning window weight of the corresponding frequency, and the window function weight is applied to the real and imaginary components of each frequency channel to suppress high-frequency oscillations. The real and imaginary data after windowing are merged and the high-frequency attenuation effect is verified to generate the frequency domain tensor after windowing.

[0087] It should be noted that the calculation of the Hanning window weight is to suppress the spectrum leakage caused by the truncation of the frequency domain data, smooth the high-frequency oscillation, and reduce the ringing artifacts in the time domain waveform; the real and imaginary components are windowed independently to retain the orthogonality of the frequency domain signal and avoid the complex signal distortion introduced by the window weight.

[0088] The real and imaginary data of the frequency channel are extracted from each pixel of the windowed frequency domain tensor and arranged into a complex sequence. High-frequency zero-padding is performed before and after the complex sequence to form frequency domain data of standard length. A fast inverse Fourier transform is performed on the frequency domain data of standard length to generate a time domain waveform. The total time domain duration, time interval and time point are set according to the time domain waveform, and the corresponding pulse starting point and the corresponding waveform attenuation end point are checked to generate the initial time domain energy.

[0089] It should be noted that high-frequency zero-padding expansion can improve the resolution of frequency domain data, increase the number of time domain waveform interpolation points, and make the generated time domain pulse more accurate; the inverse Fourier transform is to convert the frequency domain extended signal into a time domain waveform.

[0090] According to the extrapolated frequency upper limit, the Nyquist sampling theorem is applied to determine the minimum time interval; a cubic spline interpolation function is constructed for the time domain waveform of each pixel of the initial time domain energy, and a new time network is generated with the initial time point as the node, outputting the interpolated time domain tensor; based on the interpolated time domain tensor, the effective reflection signal interval and the noise-dominated attenuation tail are determined, and the previous fixed time point of the time domain waveform of each pixel is intercepted to generate a dynamically adjusted time domain tensor.

[0091] It should be noted that the application of the Nyquist sampling theorem is based on the extrapolation of the upper frequency limit to determine the minimum time interval to avoid aliasing distortion of the time domain waveform; cubic spline interpolation is to smooth the initial time domain waveform, improve the time resolution, and enhance the detectability of weak signal peaks.

[0092] Read the dynamically adjusted time domain tensor and traverse all spatial positions and time points; perform conditional judgment on each dynamically adjusted time domain tensor: if the light signal intensity value at each spatial position and time point is less than zero, the dynamically adjusted time domain tensor is zero, otherwise the light signal intensity value of the position before the conditional judgment is performed is retained; count the number of negative value points that are reset to zero in the dynamically adjusted time domain tensor, and calculate the proportion of negative values; extract the light signal intensity values ​​of all pixels at the initial moment from the non-negative processed time domain tensor, traverse all spatial positions, find the maximum initial light signal intensity value and perform global scaling correction, verify the corrected light signal intensity constraint and energy conservation characteristics, and output a transient time domain waveform that conforms to physical rationality.

[0093] It should be noted that by excluding the noise-dominated attenuation tail (such as time domain data ≥80ns), the effective reflection range is retained, and computational redundancy and noise interference are reduced; the negative photon counts are forced to zero to ensure that the time domain waveform conforms to physical reality (the photoelectric signal cannot be negative), and abnormal pulses caused by computational errors are eliminated; global scaling correction is performed to avoid excessive differences in light intensity between different pixels (such as strongly reflective objects and weak signal backgrounds), prevent numerical overflow, and balance the dynamic range.

[0094] Each pixel time domain waveform is extracted from the transient time domain waveform to construct continuous sampling points of the sliding window; the out-of-bounds positions at the start and end of each pixel time domain waveform are filled with mirror images, and the continuous intensity values ​​of the sampling points in the window are collected at each out-of-bounds position after filling, and the middle value is taken as the intermediate result after median filtering denoising; all pixel positions and time points are traversed, and the intermediate result after denoising is replaced with the median filtering result to generate a denoised waveform; the mirror points are extended at both ends of each time domain waveform of the denoised waveform, and the extended mirror points are multiplied and accumulated with the Gaussian kernel points at each position to obtain a transient time domain image.

[0095] It should be noted that by suppressing the instantaneous noise in the time domain waveform (such as thermal noise burrs), the steep edge of the true peak is retained and the peak positioning accuracy is improved; by eliminating the truncation effect at the beginning and end of the time domain waveform, the boundary noise is reduced and a continuous and smooth transient time domain image is generated.

[0096] S4. Reshape the time series of each pixel point of the time domain waveform in the transient time domain image into a spatial-temporal two-dimensional structure to generate an input tensor; input the input tensor sequentially into three convolutional layers to enhance local feature expression, and output a feature map with a resolution of 30×40×64; apply global average pooling and a fully connected layer to the feature map with a resolution of 30×40×64 to generate a peak confidence heat map.

[0097] The time series of the time domain waveform in the transient image is decoupled from the spatial dimension to form an input tensor, which makes it easier for the convolution layer to capture the dynamic change characteristics of the time domain. The three-layer convolution layer is used to extract local features and compress the spatial dimension, enhance the key patterns of the time domain waveform (such as the steepness of the main peak and the position of the secondary peak), and suppress random noise.

[0098] The confidence value of each pixel is extracted from the peak confidence heat map and a confidence threshold is set. All spatial positions are traversed and the confidence value of each pixel is compared with the confidence threshold. If the confidence value is greater than or equal to the confidence threshold, it is marked as a high confidence area; if the confidence value is less than the confidence threshold, it is marked as a low confidence area.

[0099] It is further explained that the confidence threshold is determined by statistically analyzing the distribution of confidence values ​​corresponding to the true peak position during the uniform reflective surface calibration phase. 0.7 is an empirical balance point that can retain high-reliability depth information (such as the main peak of the direct light path) while avoiding excessive filtering of weak signals (such as distant low-reflectivity targets).

[0100] For high-confidence areas, the corresponding time domain waveform is extracted from the transient time domain image, the maximum position is searched along the time axis, and the peak time and amplitude are recorded;

[0101] For low-confidence areas, the corresponding time domain waveform is extracted from the transient time domain image, and zero-filling and Gaussian convolution operations are performed to generate smoothed time domain waveform data; based on the smoothed time domain waveform data, the first-order derivative of each time point is calculated by the central difference method to describe the change in waveform slope; based on the first-order derivative result, the second-order derivative is further calculated by the central difference method to reflect the change in the concavity and convexity of the waveform; the first-order derivative and the second-order derivative are substituted into the curvature formula to calculate the curvature value of each time point, and the size of the curvature value represents the local curvature degree of the waveform; scan along the time axis, the sign change point of the curvature value, mark the zero-crossing position of the curvature sign change, count the length of the continuous zero-crossing interval, combine the amplitude threshold to eliminate invalid peaks, correct the time domain waveform data, and finally screen out the candidate peaks.

[0102] To further explain, the amplitude threshold is based on the noise data collected by the sensor during the dark field (no light input) calibration phase. The mean and standard deviation of the time domain waveform amplitude are calculated, and the amplitude threshold of the effective signal is determined through a uniform illumination calibration experiment.

[0103] Load the pre-trained random forest model, which contains 100 decision trees, each with a maximum depth of 10, and supports multi-classification probability output; input the candidate peaks generated after multi-peak detection in the low-confidence area into the random forest model, and the random forest outputs the probability of each candidate peak; set the effective depth threshold, filter out the candidate peaks with an output probability greater than or equal to the effective depth threshold, and mark them as valid peaks. If no candidate peak meets the effective depth threshold, it is marked as an invalid detection; when there are multiple candidate peaks with an output probability greater than or equal to the effective depth threshold at the same time, extract the corresponding time, select the peak that arrives earliest as the valid peak, and integrate all valid peaks to form a depth probability map; in the depth probability map, record the direct light path depth as channel 1, the indirect light path probability as channel 2, and mark the invalid detection area.

[0104] It is further explained that the effective depth threshold is determined by combining the ROC curve of the calibration experiment with the multipath suppression requirements to ensure a balance between detection sensitivity and anti-interference capability while retaining a high reliability peak.

[0105] It should be noted that the candidate peak features (amplitude ratio, time interval, and area proportion) are used for multi-classification probability prediction to distinguish the main peak of the direct light path, the secondary peak of the multipath, and the noise. The direct light path signal (the earliest arriving peak) takes precedence over the multipath reflection, which conforms to the laws of physical propagation and reduces interference in dynamic scenes.

[0106] S5. Extract the direct light path depth value of each pixel from channel 1 of the depth probability map, extract the indirect light path probability value of each pixel from channel 2 of the depth probability map D, traverse each pixel of the transient time domain image, and extract the maximum amplitude from the time domain waveform; stack the direct light path depth value, indirect light path probability value and maximum amplitude in sequence along the channel dimension to generate a multimodal feature tensor, where channel 3 is the maximum amplitude.

[0107] It should be noted that the maximum amplitude reflects signal strength, helps distinguish true reflections (high amplitude) from noise (low amplitude), and improves anti-interference capabilities.

[0108] Global average pooling is performed on each channel of the multimodal feature tensor, the global mean of each channel is calculated, and a channel importance vector is generated; the channel importance vector is converted into channel attention weights through a fully connected layer and a Sigmoid activation function; the multimodal feature tensor is weighted channel by channel based on the channel attention weights to generate a channel-weighted feature tensor; the channel-weighted feature tensor is averaged along the channel dimension to generate a spatial feature map, which is input into the convolutional layer, and the spatial attention weights are generated through convolution and Sigmoid activation; the channel-weighted feature tensor is doubly weighted by combining the channel and spatial attention weights to generate an enhanced multimodal feature tensor.

[0109] It should be noted that global average pooling and Sigmoid activation are used to quantify the importance of each channel (depth, probability, amplitude) and suppress the interference of redundant or low signal-to-noise ratio channels. Spatial feature maps are generated based on channel weighted features to highlight key areas (such as the edges of highly reflective objects and multipath high-incidence areas) and weaken the contribution of uniform background.

[0110] Each pixel of the enhanced multimodal feature tensor is defined as a graph node, a 3×3 neighborhood connection relationship is constructed and the node attributes are bound; the neighborhood features are aggregated and activated through the first layer of graph convolution to generate high-dimensional intermediate features; the high-dimensional intermediate features are reconstructed by dimensionality reduction using the second layer of graph convolution, the original number of channels is restored and the optimized deep feature map is output.

[0111] It should be noted that the feature vector of each pixel is associated with the neighboring pixels to model local spatial correlation (such as object continuity and spatial propagation pattern of multipath reflection); through the aggregation and dimensionality reduction of double-layer graph convolution, the spatial consistency of the depth map is improved and isolated noise points and outliers are reduced.

[0112] Depth channels are separated from the optimized depth feature map: channel 1 is the optimized depth map, and channel 2 is the optimized indirect light path probability map; the uncertainty matrix is ​​extracted from the optimized indirect light path probability map of channel 2, divided into 8×8 sub-blocks, and the average uncertainty of each block is calculated; in the uniform illumination calibration stage, the uncertainty threshold is set to 0.2 by analyzing the depth error corresponding to different uncertainties; when the average uncertainty is less than or equal to 0.2, it is regarded as a high-confidence sub-block, and the corresponding area is extracted from the optimized depth map of channel 1 and lossless coding is performed; when the average uncertainty is greater than 0.2, it is regarded as a low-confidence sub-block, and the corresponding area of ​​the optimized depth map of channel 1 is losslessly compressed, and DCT transform, dynamic quantization and run-length coding are performed; the lossless coding block and the lossy compression block are reorganized into a mixed code stream according to the spatial position, the block type mark and quantization parameter are added, and the block is encapsulated to output the noise-resistant enhanced depth map.

[0113] It should be noted that the uncertainty is calculated based on the optimized indirect light probability map to quantify the depth reliability of each area; high-confidence blocks retain lossless details to avoid compression distortion, and low-confidence blocks use lossy compression to suppress noise energy.

[0114] This embodiment further provides a camera image data processing system based on noise reduction technology, including:

[0115] The multi-frequency acquisition module collects multi-phase charge data and converts it into a voltage signal matrix, generating a multi-frequency measurement tensor after frequency grouping processing;

[0116] The frequency domain expansion module inputs the multi-frequency measurement tensor into a convolutional neural network with an encoder-decoder structure and uses an adversarial training mechanism to generate an extended frequency domain tensor covering all target frequencies;

[0117] The time domain noise reduction module performs phase compensation and windowing on the extended frequency domain tensor, then performs inverse Fourier transform, enhances the time domain resolution through interpolation, and suppresses transient noise using median filtering to generate a transient time domain image.

[0118] The signal classification module extracts time domain waveform features from transient time domain images, segments confidence regions using confidence thresholds, and combines multi-peak detection with random forests to distinguish valid reflection signals, outputting a depth probability map.

[0119] The depth optimization module performs channel-spatial dual attention weighting on the depth probability map, uses graph convolution to model neighborhood correlation and implements hybrid coding to output a noise-resistant and enhanced depth map.

[0120] This embodiment also provides a computer device, which is suitable for the camera image data processing method based on noise reduction technology, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the camera image data processing method based on noise reduction technology proposed in the above embodiment.

[0121] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0122] This embodiment further provides a storage medium having a computer program stored thereon. When the program is executed by a processor, the method for processing camera image data based on noise reduction technology as proposed in the above embodiment is implemented. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0123] In summary, the present invention generates a Gaussian-Poisson mixed noise model based on the quantum efficiency term and the Box-Muller transform, dynamically allocates the noise ratio in combination with the sensor dark field calibration data, and effectively simulates the real noise distribution; generates an extended frequency domain tensor through frequency domain extrapolation adversarial training, and combines the phase compensation matrix and the Hanning window to increase the frequency domain resolution to the upper limit of the extrapolation frequency, ensuring that the time domain waveform after the inverse Fourier transform has high contrast and low ringing artifacts; adopts a multi-peak detection mechanism guided by the confidence heat map, distinguishes effective reflections from noise through curvature analysis and the random forest model, and gives priority to the earliest arriving peak; implements hybrid coding based on the uncertainty matrix, retains lossless details in high-confidence areas, and uses DCT dynamic quantization to compress noise energy in low-confidence areas, reducing data redundancy while ensuring the accuracy of the depth map.

[0124] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A camera image data processing method based on noise reduction technology, characterized by: include, Collect multi-phase charge data and convert it into a voltage signal matrix, and generate a multi-frequency measurement tensor after frequency grouping processing; The multi-frequency measurement tensor is input into a convolutional neural network with an encoder-decoder structure, and an adversarial training mechanism is used to generate an extended frequency domain tensor covering all target frequencies. Phase compensation and windowing are performed on the extended frequency domain tensor, followed by inverse Fourier transform. The time domain resolution is enhanced by interpolation and transient noise is suppressed by median filtering to generate a transient time domain image. Extract time domain waveform features from transient time domain images, segment confidence regions using confidence thresholds, and combine multi-peak detection with random forests to distinguish valid reflection signals, outputting a depth probability map. Channel-spatial dual attention weighting is performed on the depth probability map, graph convolution is used to model neighborhood correlation and hybrid coding is implemented to output a noise-resistant enhanced depth map.

2. The camera image data processing method based on noise reduction technology according to claim 1, wherein: The generating of the multi-frequency measurement tensor comprises the following steps: The indirect time-of-flight sensor alternately outputs sinusoidal modulation waveforms of different frequencies; At each frequency, the integrating capacitor is controlled to start the charge and discharge cycle at multiple phase points, and the charge amount data at each phase point is collected; The charge data of each phase point are arranged according to the spatial dimension and spliced ​​into the original charge matrix, which is converted into a voltage signal matrix and then reorganized into a multi-frequency measurement tensor by frequency grouping.

3. The camera image data processing method based on noise reduction technology according to claim 2, wherein: The method of inputting the multi-frequency measurement tensor into the convolutional neural network of the encoder-decoder structure includes the following steps: The encoder compresses the spatial dimension and expands the number of channels through multi-stage strided convolution, and combines skip connections to preserve intermediate features of different scales; The decoder restores the spatial resolution step by step through transposed convolution and concatenates it with the intermediate features of different scales saved by the encoder to generate the final feature map; In the final feature map, complex components covering all target frequencies are generated through extended frequency domain extrapolation and adversarial training.

4. The camera image data processing method based on noise reduction technology according to claim 3, wherein: The phase compensation and windowing processing performed on the extended frequency domain tensor comprises the following steps: Load the pre-calibrated sensor frequency response parameter matrix and perform phase compensation on the complex tensor covering all target frequencies; A window function is applied to the phase-compensated complex component based on the Hanning window weight to suppress high-frequency oscillations, and the real and imaginary components are merged to generate a frequency domain tensor after windowing.

5. The camera image data processing method based on noise reduction technology according to claim 4, characterized in that: The confidence threshold segmentation and multi-peak detection includes the following steps: Divide high and low confidence areas according to the peak confidence heat map; Perform multi-order derivative analysis on the time domain waveform in the low confidence area to calculate the curvature value and screen the candidate peaks; The candidate peaks are probabilistically classified using a pre-trained random forest model, and the effective peaks are determined based on the time priority principle.

6. The camera image data processing method based on noise reduction technology according to claim 5, characterized in that: The channel-spatial dual attention weighting The following steps are included: Extract direct and indirect light paths from the depth probability map to generate a multimodal tensor, and perform channel-by-channel weighting to generate a spatial feature map; The spatial feature map is doubly weighted by spatial attention weights, and the neighborhood features are aggregated by graph convolution to output the optimized depth map.

7. The camera image data processing method based on noise reduction technology according to claim 6, characterized in that: The hybrid coding refers to calculating the uncertainty matrix based on the optimized indirect light path probability map, dividing the sub-blocks into predetermined sizes, performing lossless coding on high-confidence sub-blocks, and performing lossy compression on low-confidence sub-blocks.

8. A camera image data processing system based on noise reduction technology, based on the camera image data processing method based on noise reduction technology according to any one of claims 1 to 7, characterized in that: include, The multi-frequency acquisition module collects multi-phase charge data and converts it into a voltage signal matrix, generating a multi-frequency measurement tensor after frequency grouping processing; The frequency domain expansion module inputs the multi-frequency measurement tensor into a convolutional neural network with an encoder-decoder structure and uses an adversarial training mechanism to generate an extended frequency domain tensor covering all target frequencies; The time domain noise reduction module performs phase compensation and windowing on the extended frequency domain tensor, then performs inverse Fourier transform, enhances the time domain resolution through interpolation, and suppresses transient noise using median filtering to generate a transient time domain image. The signal classification module extracts time domain waveform features from transient time domain images, segments confidence regions using confidence thresholds, and combines multi-peak detection with random forests to distinguish valid reflection signals, outputting a depth probability map. The depth optimization module performs channel-spatial dual attention weighting on the depth probability map, uses graph convolution to model neighborhood correlation and implements hybrid coding to output a noise-resistant and enhanced depth map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the camera image data processing method based on noise reduction technology according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the camera image data processing method based on noise reduction technology according to any one of claims 1 to 7 are implemented.

Images