4D millimeter wave radar and camera information fusion image enhancement method and system based on deep learning

By performing spatiotemporal registration of radar point clouds and camera images and fuzzy weight-guided cross-modal attention fusion under a unified time base, the problems of geometric mismatch and artifact amplification in multimodal fusion are solved, and high-precision and robust image enhancement effects are achieved.

CN120726274AInactive Publication Date: 2025-09-30YANCHENG INST OF TECH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202510840383.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-23
Publication Date
2025-09-30
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing technologies have problems in multimodal fusion, such as geometric mismatch, artifact amplification, and lack of dynamic credibility guidance, making it difficult to achieve high-precision and robust image enhancement in complex environments.

Method used

By acquiring radar point clouds and camera images under a unified time reference, using differentiable projection and fuzzy C-Means-ICP collaborative algorithm for spatiotemporal registration, generating a confidence matrix, and guiding cross-modal attention fusion with fuzzy weights in a convolution-transformer-diffusion combined network, combined with self-supervised optimization technology, the image enhancement effect is improved.

Benefits of technology

It achieves sub-pixel mapping accuracy of millimeter-wave radar point clouds in image space, alleviates artifact accumulation and information redundancy interference in low-visibility scenarios, and improves image clarity and robustness in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120726274A_ABST
    Figure CN120726274A_ABST
Patent Text Reader

Abstract

The invention provides a 4D millimeter wave radar and camera information fusion image enhancement method and system based on deep learning, and relates to the technical field of image enhancement. According to the method, a confidence coefficient matrix is generated through space-time registration, micro projection and fuzzy C-Means-ICP fusion, a radar feature body guided by fuzzy weight is constructed, cross-modal attention fusion and self-supervised optimization are realized in a convolution-converter-diffusion network, and the image enhancement quality and robustness in a complex environment are effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to a 4D millimeter-wave radar and camera information fusion image enhancement method and system based on deep learning. Background Art

[0002] In the fields of intelligent driving, security monitoring, and complex environment perception, 4D millimeter-wave radar (which simultaneously outputs four-dimensional information: azimuth, pitch, range, and radial velocity) is becoming an important complement to visual sensors due to its all-day, all-weather capability and insensitivity to interference from fog, rain, and dust. Meanwhile, high-definition RGB / IR cameras remain the preferred method for obtaining high-resolution texture and semantic details. Currently, a significant amount of research is focused on "multimodal fusion," but mainstream methods only perform feature concatenation or simple weighting at the back end, making it difficult to fully exploit the complementary and redundant information of each modality.

[0003] With the success of deep neural network architectures such as Transformer and diffusion models in cross-modal tasks, researchers have begun to explore more fine-grained spatial-temporal alignment mechanisms (such as spatiotemporal contrastive learning and cross-modal attention) to build a unified perception stack; at the same time, for complex weather and low-light scenes, adaptive credibility modeling and uncertainty suppression combined with fuzzy set theory has become a new direction to improve fusion robustness.

[0004] There are three major deficiencies in existing technologies:

[0005] (1) Multi-scale geometric mismatch: radar point clouds are sparse and have large scale dynamic changes, making it difficult to ensure pixel-level registration using traditional projection mapping;

[0006] (2) Local artifact amplification: In scenes with fog or highly reflective metal, camera imaging has low contrast and radar echoes contain spurious signals; simple feature stacking easily leads to noise accumulation;

[0007] (3) Lack of dynamic credibility evaluation - Existing deep fusion networks are trained with a single loss function, ignoring the signal-noise differences of different modalities in each spatiotemporal segment, and are unable to attenuate unreliable information during inference. Summary of the Invention

[0008] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a 4D millimeter-wave radar and camera information fusion image enhancement method and system based on deep learning, which effectively solves the problems of geometric mismatch, artifact amplification and lack of dynamic credibility guidance in multimodal fusion, and significantly improves the accuracy and robustness of image enhancement in complex environments.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A deep learning-based 4D millimeter-wave radar and camera information fusion image enhancement method, comprising:

[0011] Simultaneously acquire millimeter-wave radar point clouds containing azimuth, elevation, range, and radial velocity information, along with camera images at the corresponding time, under a unified time base. Use timestamps and extrinsic parameters to perform preliminary spatiotemporal registration.

[0012] The radar point cloud is mapped to the camera pixel coordinate system through the differentiable projection and fuzzy C-Means-ICP collaborative algorithm, and a confidence matrix with a value range of 0 to 1 is generated;

[0013] According to the membership function, the confidence matrix is ​​fused with the depth-velocity tensor to obtain a radar feature volume with fuzzy weights;

[0014] In a convolution-transformer-diffusion combined network, fuzzy weights are used to guide cross-modal attention fusion of the radar feature volume and visual features to output an enhanced image.

[0015] performing self-supervised optimization of the convolution-transformer-diffusion combination network using contrast, structural similarity, and blur regularization loss between the enhanced image and a multi-exposure reference image;

[0016] Real-time point clouds and images are input into the convolution-transformer-diffusion combination network to obtain enhanced results.

[0017] Preferably, a millimeter-wave radar point cloud containing azimuth, elevation, range, and radial velocity information and a camera image at the corresponding moment are simultaneously acquired under a unified time reference, and a preliminary spatiotemporal registration is performed using timestamps and external parameters, including:

[0018] A unified high-precision time synchronization module is embedded in the preset millimeter-wave radar and camera systems respectively, and a pulse synchronization signal is used to ensure that the timestamps of the data collected by the millimeter-wave radar and the camera system are consistent;

[0019] Collecting a radar point cloud of each frame, and extracting target points with azimuth, pitch angle, radial distance and velocity information from the radar point cloud;

[0020] While acquiring the radar point cloud, triggering the camera system to acquire an image frame matching its timestamp, and recording the timestamp information of the acquisition moment of the camera system;

[0021] Based on the pre-calibrated extrinsic parameters, the point cloud in the millimeter-wave radar coordinate system is converted to the coordinate system of the camera system to achieve preliminary spatiotemporal alignment; the extrinsic parameters include the rotation matrix and translation vector between the camera and the radar.

[0022] Preferably, the radar point cloud is mapped to the camera pixel coordinate system through a microprojection and fuzzy C-Means-ICP collaborative algorithm, and a confidence matrix with a value range of 0 to 1 is generated, including:

[0023] Based on pre-calibrated extrinsic parameters, each 3D point in the radar point cloud is converted from the radar coordinate system to the camera coordinate system to obtain the corresponding 3D spatial position;

[0024] Perform perspective projection on the point cloud in the camera coordinate system using the camera intrinsic parameter matrix to obtain the two-dimensional continuous coordinate position of each point on the image plane, forming a projection point set of the radar point cloud in the image coordinate system;

[0025] Mapping each two-dimensional coordinate in the projection point set to a nearest pixel grid point in the image according to the floating point position of the two-dimensional coordinate, and recording the corresponding pixel position, depth value, and radial velocity value to generate a sparse radar image feature map;

[0026] Based on the radar image feature map, a fuzzy C-Means clustering algorithm is used to comprehensively consider the projection point density, depth gradient and velocity continuity within the pixel area, perform fuzzy classification on each projection pixel and calculate the membership score;

[0027] Extracting an image edge map or a corner feature map from the camera image, and establishing initial spatial matching point pairs with the projected pixels in the image plane;

[0028] Based on the initial spatial matching point pairs, an improved iterative closest point algorithm with a fuzzy weight correction term is applied to minimize the weighted Euclidean distance error between radar projection pixels and image feature points, thereby achieving sub-pixel projection position correction of the radar point cloud in the image space;

[0029] The membership score and the correction residual of each projection position are fused to assign a normalized confidence value to each image pixel, and the confidence matrix with a value range of 0 to 1 is constructed to characterize the fusion credibility between the radar point cloud and the image.

[0030] Preferably, according to the membership function, the confidence matrix is ​​fused with the depth-velocity tensor to obtain a radar feature volume with fuzzy weights, including:

[0031] The confidence value corresponding to each image pixel position in the confidence matrix is ​​used as a fuzzy inference input to construct a fuzzy membership function with an input interval of [0, 1]. The membership function uses a differentiable expression based on the logistic function to represent the credibility level of the radar information in the pixel area.

[0032] Based on the image coordinates, the confidence matrix is ​​paired with the depth value and radial velocity value of the radar image projection point in the image space obtained previously by pixel position to construct a three-channel tensor, where the first channel is the depth value, the second channel is the velocity value, and the third channel is the fuzzy membership score of the corresponding pixel;

[0033] According to the fuzzy membership function, the depth value and the speed value of each pixel are weighted to generate a fuzzy depth map and a fuzzy speed map that are integrated with confidence weights;

[0034] The blurred depth map, blurred velocity map and original confidence matrix are spliced ​​in the channel dimension to generate a radar feature volume with a complete structure; the radar feature volume simultaneously encodes the depth, velocity and blurred confidence information for each pixel in the image space.

[0035] Preferably, in a convolution-transformer-diffusion combined network, the radar feature body and the visual feature are guided to be cross-modal attentionally fused with fuzzy weights to output an enhanced image, including:

[0036] Inputting the radar feature volume into the first feature encoding branch, and using a multi-scale convolution module to extract radar local spatial structure information while retaining the distinguishing representations of depth, velocity, and confidence channels;

[0037] The camera image at the corresponding timestamp is input into the second feature encoding branch, and the hierarchical visual Transformer is used to extract the visual global context features and generate multi-scale image semantic embeddings;

[0038] A cross-modal multi-head attention mechanism is implemented on the feature layer output by the two branches, where the confidence channel in the radar feature volume is used as the attention guidance weight to guide the distribution of attention weights and enhance the fusion strength between visual features and credible radar information.

[0039] The multimodal features after attention fusion are input into the diffusion residual enhancement module, and feature layer iterative denoising and reconstruction are performed under noise scheduling control to generate enhanced image feature maps;

[0040] The enhanced image feature map is mapped back to the image space through a learnable decoder, and is aligned and corrected with the original image, and finally the enhanced image with enhanced contrast and details is output.

[0041] Preferably, the convolution-transformer-diffusion combination network is self-supervisedly optimized using the contrast, structural similarity and blur regularization loss between the enhanced image and the multi-exposure reference image, comprising:

[0042] Acquire a multi-exposure reference image that is consistent with the input image at the same time, and construct a high dynamic range reference image based on a local exposure feature fusion strategy to serve as a contrast supervision target for the enhanced image;

[0043] Pair the enhanced image generated by the network with the multi-exposure reference image at the pixel level and calculate a contrast loss term; the contrast loss is based on a bidirectional feature difference metric that encourages the enhanced image to converge to the reference image in terms of color, edge, and brightness distribution;

[0044] Calculating the structural similarity loss between the enhanced image and the reference image, and using a structural similarity index as a structure preservation indicator to measure the fidelity of the enhanced image in terms of details, texture, and contour structure;

[0045] Extracting the confidence score and fuzzy membership response corresponding to each pixel position in the enhanced image, constructing a fuzzy regularization term, and suppressing over-enhancement or artifact generation in the fuzzy area by minimizing the gradient oscillation in the low-confidence area;

[0046] The contrast loss, the structural similarity loss and the fuzzy regularization term are weighted and summed according to set weights to form a joint loss function;

[0047] Taking the joint loss function as the optimization objective, end-to-end self-supervised training is performed on the convolution-transformer-diffusion combination network, and the network parameters are iteratively updated until convergence.

[0048] Preferably, the convolution-transformer-diffusion combination network comprises:

[0049] a set of parallel input coding layers, including a first convolutional coding branch and a second transformer coding branch; the first convolutional coding branch is used to receive the radar feature volume and extract local spatial features using a multi-scale convolutional block, while maintaining the independence of confidence channels during feature extraction; the second transformer coding branch is used to receive the original camera image and extract global contextual semantic information using a hierarchical visual Transformer network;

[0050] A cross-modal fusion module, including a cross-channel multi-head attention layer, is used to align the radar features output by the first convolutional encoding branch with the visual features output by the second transformer encoding branch in the spatial dimension, and uses the confidence channel in the radar feature volume as an attention weight guidance factor to enhance the multimodal information coupling capability of high-confidence areas;

[0051] The diffusion image reconstruction module consists of multiple cascaded noise scheduling layers and residual fusion layers, which are used to perform layer-by-layer denoising and detail recovery based on the fused features to reconstruct high-quality enhanced image feature maps.

[0052] A learnable decoder structure is used to restore the image feature map output by the diffusion image reconstruction module to a standard image format, complete spatial alignment with the original image, and output the final enhanced image.

[0053] A deep learning-based 4D millimeter-wave radar and camera information fusion image enhancement method, comprising:

[0054] The time synchronization acquisition and initial registration unit is used to simultaneously acquire the millimeter-wave radar point cloud containing azimuth, pitch, range and radial velocity information and the camera image at the corresponding time under a unified time reference, and complete the initial spatiotemporal registration using timestamps and external parameters;

[0055] A differentiable mapping and registration confidence calculation unit is used to map the radar point cloud to the camera pixel coordinate system through a differentiable projection and fuzzy C-Means-ICP collaborative algorithm, and generate a confidence matrix with a value range of 0 to 1;

[0056] A fuzzy weight construction unit is used to fuse the confidence matrix with the depth-velocity tensor according to a membership function to obtain a radar feature volume with a fuzzy weight;

[0057] a cross-modal fusion and enhancement network unit, configured to guide cross-modal attention fusion of the radar feature volume and the visual feature using fuzzy weights in a convolution-transformer-diffusion combination network, and output an enhanced image;

[0058] a self-supervised optimization training unit for performing self-supervised optimization on the convolution-transformer-diffusion combination network by utilizing the contrast, structural similarity, and blur regularization loss between the enhanced image and the multi-exposure reference image;

[0059] A real-time inference and output unit is used to input real-time point clouds and images into the convolution-transformer-diffusion combination network to obtain enhanced results.

[0060] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0061] The present invention achieves sub-pixel mapping accuracy of millimeter-wave radar point clouds in image space by constructing a collaborative registration mechanism that combines microprojection with fuzzy clustering, overcoming the geometric scale mismatch problem existing in traditional multimodal fusion. It introduces a fuzzy membership function to construct a confidence matrix, which is then fused with depth-velocity information to construct a radar feature body with dynamic credibility guidance capabilities, effectively alleviating artifact accumulation and information redundancy interference in low-visibility scenarios. At the same time, it adopts a deep network structure that combines convolution, transformer, and diffusion models, and uses confidence as a guiding term for cross-modal attention fusion, which improves the image detail enhancement capability while achieving adaptive information coupling between different modalities. Finally, by combining the contrast supervision of multi-exposure images with a fuzzy regularization training mechanism, the clarity, structural fidelity, and perceptual robustness of the fused image under complex lighting conditions are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0064] Figure 2 A schematic diagram of the system structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] The purpose of the present invention is to provide a 4D millimeter-wave radar and camera information fusion image enhancement method and system based on deep learning, which effectively solves the problems of geometric mismatch, artifact amplification and lack of dynamic credibility guidance in multimodal fusion, and significantly improves the accuracy and robustness of image enhancement in complex environments.

[0067] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Figure 1 A flow chart of the method provided in the embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a 4D millimeter wave radar and camera information fusion image enhancement method based on deep learning, comprising:

[0069] Step 100: Acquire a millimeter-wave radar point cloud containing azimuth, elevation, range, and radial velocity information and a camera image at the corresponding time simultaneously under a unified time reference, and perform preliminary spatiotemporal registration using timestamps and extrinsic parameters;

[0070] Step 200: Map the radar point cloud to the camera pixel coordinate system through the differentiable projection and fuzzy C-Means-ICP collaborative algorithm, and generate a confidence matrix with a value range of 0 to 1;

[0071] Step 300: According to the membership function, the confidence matrix is ​​fused with the depth-velocity tensor to obtain a radar feature volume with fuzzy weights;

[0072] Step 400: In a convolution-transformer-diffusion combined network, fuzzy weights are used to guide cross-modal attention fusion of radar features and visual features, and an enhanced image is output;

[0073] Step 500: Perform self-supervised optimization on the convolution-transformer-diffusion combination network using the comparison between the enhanced image and the multi-exposure reference image, structural similarity, and blur regularization loss;

[0074] Step 600: Input the real-time point cloud and image into the convolution-transformer-diffusion combination network to obtain the enhanced result.

[0075] Preferably, a millimeter-wave radar point cloud containing azimuth, elevation, range, and radial velocity information and a camera image at the corresponding moment are simultaneously acquired under a unified time reference, and a preliminary spatiotemporal registration is performed using timestamps and external parameters, including:

[0076] A unified high-precision time synchronization module is embedded in the preset millimeter-wave radar and camera systems respectively, and a pulse synchronization signal is used to ensure that the timestamps of the data collected by the millimeter-wave radar and the camera system are consistent;

[0077] Collecting a radar point cloud of each frame, and extracting target points with azimuth, pitch angle, radial distance and velocity information from the radar point cloud;

[0078] While acquiring the radar point cloud, triggering the camera system to acquire an image frame matching its timestamp, and recording the timestamp information of the acquisition moment of the camera system;

[0079] Based on the pre-calibrated extrinsic parameters, the point cloud in the millimeter-wave radar coordinate system is converted to the coordinate system of the camera system to achieve preliminary spatiotemporal alignment; the extrinsic parameters include the rotation matrix and translation vector between the camera and the radar.

[0080] In this embodiment, to achieve unified spatiotemporal alignment of the millimeter-wave radar point cloud and camera imagery, a strictly synchronized data acquisition baseline must first be established. To this end, high-precision clock synchronization modules are integrated into the hardware architectures of both the millimeter-wave radar and the camera, respectively, and both sensors are driven for acquisition via a centralized pulse trigger signal. This embodiment connects a unified clock signal to the trigger input ports of both the radar and the camera, enabling both to complete sampling and receive the same timestamp under the same pulse edge trigger. This ensures consistent acquisition time during the data generation phase, avoiding frame offset and interpolation errors introduced by later software alignment.

[0081] After achieving time synchronization, this embodiment reads the raw point cloud data acquired by each frame of millimeter-wave radar data and selects echo target points containing azimuth, elevation, radial distance, and radial velocity information as a valid point cloud set. To ensure the reversibility and stability of the spatial projection calculation, this embodiment removes invalid clutter points and radar points with confidence levels below a set threshold during the extraction process. The target points are then reordered according to their position in the body coordinate system to construct a structured spatiotemporal point cloud frame. Simultaneously, upon receiving the synchronization trigger signal, the camera completes image frame acquisition consistent with the point cloud frame's timestamp and saves the acquisition time information for subsequent spatiotemporal correlation operations.

[0082] To achieve preliminary registration of the millimeter-wave radar point cloud to the camera image coordinate system, this embodiment uses the external parameters obtained through calibration during the deployment phase, including the rotation matrix and translation vector of the camera relative to the radar, to construct a three-dimensional rigid transformation model. The specific operation is: each three-dimensional point in the radar point cloud is transformed in posture through the rotation matrix and then added with the translation vector to obtain its corresponding spatial position in the camera coordinate system, and the original radar physical property information corresponding to this position is retained. This step forms the basis for the physical mapping of the radar point in the camera's perspective and is a prerequisite for subsequent differentiable projection, confidence matrix calculation, and cross-modal attention fusion. Its accuracy directly affects the geometric consistency of the final fused image and the target contour alignment effect.

[0083] Preferably, the radar point cloud is mapped to the camera pixel coordinate system through a microprojection and fuzzy C-Means-ICP collaborative algorithm, and a confidence matrix with a value range of 0 to 1 is generated, including:

[0084] Based on pre-calibrated extrinsic parameters, each 3D point in the radar point cloud is converted from the radar coordinate system to the camera coordinate system to obtain the corresponding 3D spatial position;

[0085] Perform perspective projection on the point cloud in the camera coordinate system using the camera intrinsic parameter matrix to obtain the two-dimensional continuous coordinate position of each point on the image plane, forming a projection point set of the radar point cloud in the image coordinate system;

[0086] Mapping each two-dimensional coordinate in the projection point set to a nearest pixel grid point in the image according to the floating point position of the two-dimensional coordinate, and recording the corresponding pixel position, depth value, and radial velocity value to generate a sparse radar image feature map;

[0087] Based on the radar image feature map, a fuzzy C-Means clustering algorithm is used to comprehensively consider the projection point density, depth gradient and velocity continuity within the pixel area, perform fuzzy classification on each projection pixel and calculate the membership score;

[0088] Extracting an image edge map or a corner feature map from the camera image, and establishing initial spatial matching point pairs with the projected pixels in the image plane;

[0089] Based on the initial spatial matching point pairs, an improved iterative closest point algorithm with a fuzzy weight correction term is applied to minimize the weighted Euclidean distance error between radar projection pixels and image feature points, thereby achieving sub-pixel projection position correction of the radar point cloud in the image space;

[0090] The membership score and the correction residual of each projection position are fused to assign a normalized confidence value to each image pixel, and the confidence matrix with a value range of 0 to 1 is constructed to characterize the fusion credibility between the radar point cloud and the image.

[0091] In this embodiment, to achieve high-precision mapping of millimeter-wave radar point clouds in image space, each 3D point in the radar point cloud is first transformed from the radar coordinate system to the camera coordinate system based on the extrinsic parameters obtained during the calibration phase, obtaining its spatial position from the camera's perspective. These extrinsic parameters, consisting of a rotation matrix and a translation vector, are used to perform rigid spatial transformations, ensuring that the point cloud structure maintains a consistent spatial reference before cross-modal projection. To maintain accuracy, this embodiment retains the depth and radial velocity attributes of the original radar points during the transformation process, which serve as the input for subsequent fuzzy credibility modeling.

[0092] After completing the coordinate transformation, this embodiment performs perspective projection on the radar point cloud in the camera coordinate system based on the camera's intrinsic parameter matrix, obtaining the continuous two-dimensional coordinate position of each point on the image plane. To adapt the image resolution, this embodiment maps these floating-point projection coordinates to the nearest integer pixel grid. The original radar depth and velocity values ​​corresponding to each pixel position are recorded in the mapping result to construct a sparse radar image feature map. This feature map, representing the spatial distribution of radar information on the image plane, is characterized by low density, non-uniformity, and a multi-scale range.

[0093] To improve the fusion reliability of the projection area, this embodiment introduces a fuzzy C-Means clustering algorithm based on the above-mentioned feature map. It comprehensively evaluates the projection point density distribution, depth gradient changes, and velocity continuity within the pixel neighborhood, performs fuzzy clustering on the projection pixels, and calculates the fuzzy membership score of each pixel to reflect its local projection reliability. Subsequently, an edge map or corner map is extracted from the original image to construct the initial spatial point pairs between the radar projection pixels. This embodiment further introduces an improved iterative closest point algorithm that integrates fuzzy membership weights to minimize the weighted Euclidean distance error based on the initial point pairs, achieve sub-pixel position correction of the radar point cloud in the image space, and record the projection offset residuals before and after correction.

[0094] Finally, this embodiment performs a normalized weighted fusion of the fuzzy membership scores and the position correction residuals to calculate a fusion confidence value for each projected pixel in the image plane, constructing a confidence matrix ranging from 0 to 1. This matrix measures the reliability of the spatial matching of radar points in the image coordinate system and serves as an important foundation for subsequent modality fusion attention guidance and weighted reconstruction.

[0095] Preferably, according to the membership function, the confidence matrix is ​​fused with the depth-velocity tensor to obtain a radar feature volume with fuzzy weights, including:

[0096] The confidence value corresponding to each image pixel position in the confidence matrix is ​​used as a fuzzy inference input to construct a fuzzy membership function with an input interval of [0, 1]. The membership function uses a differentiable expression based on the logistic function to represent the credibility level of the radar information in the pixel area.

[0097] Based on the image coordinates, the confidence matrix is ​​paired with the depth value and radial velocity value of the radar image projection point in the image space obtained previously by pixel position to construct a three-channel tensor, where the first channel is the depth value, the second channel is the velocity value, and the third channel is the fuzzy membership score of the corresponding pixel;

[0098] According to the fuzzy membership function, the depth value and the speed value of each pixel are weighted to generate a fuzzy depth map and a fuzzy speed map that are integrated with confidence weights;

[0099] The blurred depth map, blurred velocity map and original confidence matrix are spliced ​​in the channel dimension to generate a radar feature volume with a complete structure; the radar feature volume simultaneously encodes the depth, velocity and blurred confidence information for each pixel in the image space.

[0100] In this embodiment, to construct a radar feature with fuzzy credibility guidance capabilities, the confidence value corresponding to each image pixel position in the confidence matrix is ​​first used as the fuzzy inference input to construct a fuzzy membership function model with a continuous interval of [0,1]. The function uses a differentiable expression based on the logistic function and is defined as μ(x) = 1 / (1 + e^(-α(x-β))), where α and β are shape control parameters used to adjust the function response sensitivity and center offset. This embodiment adaptively adjusts the function parameters so that the fuzzy response exhibits a high slope characteristic in the medium and low confidence intervals, thereby enhancing the credibility discrimination ability of the boundary fuzzy area, and the output of this function is used as the fuzzy membership score of each pixel.

[0101] Next, this embodiment performs pixel-level pairing of the aforementioned confidence matrix with the depth value and radial velocity value at each pixel position in the registered radar image feature map to construct a three-channel tensor. Specifically, using pixels as index units, the depth value is filled in the first channel, the velocity value is filled in the second channel, and the above-mentioned fuzzy membership score is filled in the third channel. To ensure numerical consistency, this embodiment normalizes the depth and velocity channels before tensor construction so that their numerical ranges are consistent with the fuzzy score, facilitating subsequent point-to-point weighted fusion. This three-channel tensor preliminarily expresses the spatial, motion, and confidence attributes of each pixel position and is the basic structure for radar fuzzy fusion modeling.

[0102] Subsequently, this embodiment performs a fuzzy weighting operation on each pixel value in the depth channel and velocity channel based on the constructed fuzzy membership function. That is, the original depth and velocity are scaled by the membership value to suppress the interference effect of low-confidence areas. After weighting, a fuzzy depth map and a fuzzy velocity map are obtained respectively. Finally, the fuzzy depth map, the fuzzy velocity map, and the original confidence matrix are spliced ​​in the channel dimension to form a structurally complete radar feature volume. This radar feature volume implements multidimensional radar attribute encoding for each pixel in the image space, combining geometric information, dynamic information, and confidence credibility, providing high-quality input for subsequent cross-modal attention fusion.

[0103] Preferably, in a convolution-transformer-diffusion combined network, the radar feature body and the visual feature are guided to be cross-modal attentionally fused with fuzzy weights to output an enhanced image, including:

[0104] Inputting the radar feature volume into the first feature encoding branch, and using a multi-scale convolution module to extract radar local spatial structure information while retaining the distinguishing representations of depth, velocity, and confidence channels;

[0105] The camera image at the corresponding timestamp is input into the second feature encoding branch, and the hierarchical visual Transformer is used to extract the visual global context features and generate multi-scale image semantic embeddings;

[0106] A cross-modal multi-head attention mechanism is implemented on the feature layer output by the two branches, where the confidence channel in the radar feature volume is used as the attention guidance weight to guide the distribution of attention weights and enhance the fusion strength between visual features and credible radar information.

[0107] The multimodal features after attention fusion are input into the diffusion residual enhancement module, and feature layer iterative denoising and reconstruction are performed under noise scheduling control to generate enhanced image feature maps;

[0108] The enhanced image feature map is mapped back to the image space through a learnable decoder, and is aligned and corrected with the original image, and finally the enhanced image with enhanced contrast and details is output.

[0109] In this embodiment, in order to achieve deep fusion and guidance enhancement of radar and visual information, the constructed radar feature body is first input into the first encoding branch of the convolution-transformer-diffusion combination network. This branch uses a multi-scale convolution module to extract local features from the three-channel input respectively, maintaining the independence and complementarity of the three channels of depth, speed and confidence in feature representation. By using convolution kernels of different scales (for example, 3×3 and 5×5) to perceive the same input area in parallel, and adding normalization and nonlinear activation operations after each layer, this embodiment can retain microstructural details such as target edges, depth mutations and confidence mutations, and improve the expression accuracy of low-density radar points in image space.

[0110] At the same time, this embodiment inputs the camera image consistent with the radar point cloud timestamp into the second encoding branch, which is built based on a hierarchical visual Transformer model and relies on segmentation and multi-head self-attention mechanisms to extract global context information. The Transformer encoding branch divides the image into fixed-scale block regions and introduces position encoding to represent their spatial relationships. It then achieves expression compression of semantic fields, texture continuity, and spatial structures in the image through multi-layer residual stacking and attention transfer. The multi-scale image semantic embedding output by this branch can fully represent the significant structure in the visual channel and provide clear semantic guidance for subsequent fusion.

[0111] At the feature layer output by the two branches, this embodiment employs a cross-modal multi-head attention mechanism to fuse radar and visual features. During the fusion process, the confidence channel within the radar feature volume serves as an explicit guidance weight and acts as a bias factor in the normalized distribution of attention weights. This guides the Transformer module to strengthen visual attention in high-confidence radar regions while suppressing uncertainty transfer in low-confidence regions, thereby forming an attention weight map with differentiated signal fusion capabilities. The fusion result forms a unified multimodal fusion feature, which is then input into the diffusion residual enhancement module. This module incorporates a multi-layer noise scheduling controller and residual convolution units to simulate the inverse diffusion process, gradually reconstructing image details. Finally, the fused feature is mapped back to the image space via a learnable upsampling decoder, outputting an enhanced image with enhanced brightness, contrast, and structure. This image not only significantly enhances low-visibility or blurred-edge regions but also serves as feedback input for self-supervised optimization, further improving network parameter convergence.

[0112] Preferably, the convolution-transformer-diffusion combination network is self-supervisedly optimized using the contrast, structural similarity and blur regularization loss between the enhanced image and the multi-exposure reference image, comprising:

[0113] Acquire a multi-exposure reference image that is consistent with the input image at the same time, and construct a high dynamic range reference image based on a local exposure feature fusion strategy to serve as a contrast supervision target for the enhanced image;

[0114] Pair the enhanced image generated by the network with the multi-exposure reference image at the pixel level and calculate a contrast loss term; the contrast loss is based on a bidirectional feature difference metric that encourages the enhanced image to converge to the reference image in terms of color, edge, and brightness distribution;

[0115] Calculating the structural similarity loss between the enhanced image and the reference image, and using a structural similarity index as a structure preservation indicator to measure the fidelity of the enhanced image in terms of details, texture, and contour structure;

[0116] Extracting the confidence score and fuzzy membership response corresponding to each pixel position in the enhanced image, constructing a fuzzy regularization term, and suppressing over-enhancement or artifact generation in the fuzzy area by minimizing the gradient oscillation in the low-confidence area;

[0117] The contrast loss, the structural similarity loss and the fuzzy regularization term are weighted and summed according to set weights to form a joint loss function;

[0118] Taking the joint loss function as the optimization objective, end-to-end self-supervised training is performed on the convolution-transformer-diffusion combination network, and the network parameters are iteratively updated until convergence.

[0119] In this embodiment, to optimize the performance of the fusion network without labeled supervision, a multi-exposure image sequence acquired simultaneously with the input image is used to construct a reference target for the enhanced image. This reference target is generated by fusion of several images at different exposure levels using an exposure-weighted fusion method. The fusion process weights each image based on local contrast and brightness distribution, preserving both dark details and highlight information, resulting in a reference image with high dynamic range. This embodiment uses this reference image as the supervised control target for the enhanced image, ensuring physical consistency in terms of illumination restoration and edge preservation in the enhanced result, addressing the structural distortion and artifact amplification issues associated with traditional single-exposure enhancement training.

[0120] During the supervision process, this embodiment first performs pixel-level registration on the enhanced image output by the network and the multi-exposure reference image, and constructs a contrast loss function based on the color channel mean difference, edge direction gradient and local brightness deviation to guide the network output to move closer to the reference image in terms of color distribution and overall brightness trend. At the same time, this embodiment further calculates the structural similarity loss and uses a structural similarity index to measure the local texture similarity between the enhanced image and the reference image at multiple scales to enhance the expressiveness of the enhanced results in terms of detail, texture and contour fidelity. The structural similarity calculation window covers different scales from 3×3 to 11×11, and introduces a covariance suppression term between image channels to improve local contrast robustness.

[0121] Taking into account that radar images are prone to over-enhancement, virtual edges or background bleaching in low-density or low-confidence areas, this embodiment constructs a fuzzy regularization term during the training process to perform spatial constraints on the fuzzy response area in the enhanced image. Specifically, the confidence score of each image pixel position and its corresponding fuzzy membership response are extracted, and by calculating its first-order gradient amplitude and local fluctuation range in the spatial domain, a regularization loss function is constructed to penalize the amplification of high-frequency oscillation components in such areas. In this embodiment, the contrast loss, structural similarity loss and fuzzy regularization term are weighted and summed according to empirical proportions, and used as the final joint loss function to drive the parameter optimization of the convolution-transformer-diffusion combination network, so that it can achieve high-fidelity, low-artifact, and clear-structured image enhancement effects in different scenarios.

[0122] Preferably, the convolution-transformer-diffusion combination network comprises:

[0123] a set of parallel input coding layers, including a first convolutional coding branch and a second transformer coding branch; the first convolutional coding branch is used to receive the radar feature volume and extract local spatial features using a multi-scale convolutional block, while maintaining the independence of confidence channels during feature extraction; the second transformer coding branch is used to receive the original camera image and extract global contextual semantic information using a hierarchical visual Transformer network;

[0124] A cross-modal fusion module, including a cross-channel multi-head attention layer, is used to align the radar features output by the first convolutional encoding branch with the visual features output by the second transformer encoding branch in the spatial dimension, and uses the confidence channel in the radar feature volume as an attention weight guidance factor to enhance the multimodal information coupling capability of high-confidence areas;

[0125] The diffusion image reconstruction module consists of multiple cascaded noise scheduling layers and residual fusion layers, which are used to perform layer-by-layer denoising and detail recovery based on the fused features to reconstruct high-quality enhanced image feature maps.

[0126] A learnable decoder structure is used to restore the image feature map output by the diffusion image reconstruction module to a standard image format, complete spatial alignment with the original image, and output the final enhanced image.

[0127] In this embodiment, to achieve differential structural modeling and feature fusion of radar and visual information, a convolution-transformer-diffusion combined network is constructed, which includes a parallel coding layer, a cross-modal fusion module, a diffusion image reconstruction module, and a learnable decoder structure. The parallel coding layer includes a first convolutional coding branch and a second transformer coding branch. The first convolutional coding branch receives the constructed three-channel radar feature volume and extracts local spatial structural information from the depth, velocity, and confidence channels through a series of multi-scale convolution blocks. The confidence channel is explicitly maintained as an independent channel feature expression during the convolution process for weight guidance of the subsequent attention mechanism. The second transformer coding branch receives the original camera image and, through a hierarchical visual Transformer structure, uses a multi-head self-attention mechanism to extract the global contextual semantic features of the image step by step, while retaining the residual connections between layers to enhance the structure preservation capability.

[0128] This embodiment further incorporates a cross-modal fusion module between the outputs of the two encoding branches. This module employs a cross-channel multi-head attention structure, using the confidence channels in the radar feature volume as guiding weights to perform selective feature emphasis and spatial registration on the visual semantic feature map, thereby enhancing the information coupling strength in high-confidence regions. The fused multimodal features are then fed into a diffuse image reconstruction module, which consists of a multi-level noise scheduling control layer and residual convolution units. A layer-by-layer feature inversion strategy is employed to perform multiple rounds of denoising and structural completion on the fused representation, generating an image feature map with rich content and clear texture. Finally, the reconstructed image feature map is upsampled to the original image size using a learnable decoder structure and spatially aligned with the initial camera intrinsic parameters, resulting in a final enhanced image with high contrast, edge clarity, and realistic texture restoration.

[0129] Corresponding to the above method, such as Figure 2As shown, this embodiment also provides a 4D millimeter-wave radar and camera information fusion image enhancement method based on deep learning, including:

[0130] The time synchronization acquisition and initial registration unit is used to simultaneously acquire the millimeter-wave radar point cloud containing azimuth, pitch, range and radial velocity information and the camera image at the corresponding time under a unified time reference, and complete the initial spatiotemporal registration using timestamps and external parameters;

[0131] A differentiable mapping and registration confidence calculation unit is used to map the radar point cloud to the camera pixel coordinate system through a differentiable projection and fuzzy C-Means-ICP collaborative algorithm, and generate a confidence matrix with a value range of 0 to 1;

[0132] A fuzzy weight construction unit is used to fuse the confidence matrix with the depth-velocity tensor according to a membership function to obtain a radar feature volume with a fuzzy weight;

[0133] a cross-modal fusion and enhancement network unit, configured to guide cross-modal attention fusion of the radar feature volume and the visual feature using fuzzy weights in a convolution-transformer-diffusion combination network, and output an enhanced image;

[0134] a self-supervised optimization training unit for performing self-supervised optimization on the convolution-transformer-diffusion combination network by utilizing the contrast, structural similarity, and blur regularization loss between the enhanced image and the multi-exposure reference image;

[0135] A real-time inference and output unit is used to input real-time point clouds and images into the convolution-transformer-diffusion combination network to obtain enhanced results.

[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0137] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A 4D millimeter-wave radar and camera information fusion image enhancement method based on deep learning, characterized in that: include: Simultaneously acquire millimeter-wave radar point clouds containing azimuth, elevation, range, and radial velocity information, along with camera images at the corresponding time, under a unified time base. Use timestamps and extrinsic parameters to perform preliminary spatiotemporal registration. The radar point cloud is mapped to the camera pixel coordinate system through the differentiable projection and fuzzy C-Means-ICP collaborative algorithm, and a confidence matrix with a value range of 0 to 1 is generated; According to the membership function, the confidence matrix is ​​fused with the depth-velocity tensor to obtain a radar feature volume with fuzzy weights; In a convolution-transformer-diffusion combined network, fuzzy weights are used to guide cross-modal attention fusion of the radar feature volume and visual features to output an enhanced image. performing self-supervised optimization of the convolution-transformer-diffusion combination network using contrast, structural similarity, and blur regularization loss between the enhanced image and a multi-exposure reference image; Real-time point clouds and images are input into the convolution-transformer-diffusion combination network to obtain enhanced results.

2. The image enhancement method based on 4D millimeter-wave radar and camera information fusion based on deep learning according to claim 1, characterized in that: Simultaneously acquire millimeter-wave radar point clouds containing azimuth, elevation, range, and radial velocity information, along with camera images at the corresponding time, under a unified time base. Use timestamps and extrinsic parameters to perform preliminary spatiotemporal registration, including: A unified high-precision time synchronization module is embedded in the preset millimeter-wave radar and camera systems respectively, and a pulse synchronization signal is used to ensure that the timestamps of the data collected by the millimeter-wave radar and the camera system are consistent; Collecting a radar point cloud of each frame, and extracting target points with azimuth, pitch angle, radial distance and velocity information from the radar point cloud; While acquiring the radar point cloud, triggering the camera system to acquire an image frame matching its timestamp, and recording the timestamp information of the acquisition moment of the camera system; Based on the pre-calibrated extrinsic parameters, the point cloud in the millimeter-wave radar coordinate system is converted to the coordinate system of the camera system to achieve preliminary spatiotemporal alignment; the extrinsic parameters include the rotation matrix and translation vector between the camera and the radar.

3. The image enhancement method based on deep learning and fusion of 4D millimeter-wave radar and camera information according to claim 1 is characterized in that: The radar point cloud is mapped to the camera pixel coordinate system through the differentiable projection and fuzzy C-Means-ICP collaborative algorithm, and a confidence matrix with a value range of 0 to 1 is generated, including: Based on pre-calibrated extrinsic parameters, each 3D point in the radar point cloud is converted from the radar coordinate system to the camera coordinate system to obtain the corresponding 3D spatial position; Perform perspective projection on the point cloud in the camera coordinate system using the camera intrinsic parameter matrix to obtain the two-dimensional continuous coordinate position of each point on the image plane, forming a projection point set of the radar point cloud in the image coordinate system; Mapping each two-dimensional coordinate in the projection point set to a nearest pixel grid point in the image according to the floating point position of the two-dimensional coordinate, and recording the corresponding pixel position, depth value, and radial velocity value to generate a sparse radar image feature map; Based on the radar image feature map, a fuzzy C-Means clustering algorithm is used to comprehensively consider the projection point density, depth gradient and velocity continuity within the pixel area, perform fuzzy classification on each projection pixel and calculate the membership score; Extracting an image edge map or a corner feature map from the camera image, and establishing initial spatial matching point pairs with the projected pixels in the image plane; Based on the initial spatial matching point pairs, an improved iterative closest point algorithm with a fuzzy weight correction term is applied to minimize the weighted Euclidean distance error between radar projection pixels and image feature points, thereby achieving sub-pixel projection position correction of the radar point cloud in the image space; The membership score and the correction residual of each projection position are fused to assign a normalized confidence value to each image pixel, and the confidence matrix with a value range of 0 to 1 is constructed to characterize the fusion credibility between the radar point cloud and the image.

4. The image enhancement method based on deep learning and fusion of 4D millimeter-wave radar and camera information according to claim 1 is characterized in that: According to the membership function, the confidence matrix is ​​fused with the depth-velocity tensor to obtain a radar feature volume with fuzzy weights, including: The confidence value corresponding to each image pixel position in the confidence matrix is ​​used as a fuzzy inference input to construct a fuzzy membership function with an input interval of [0, 1]. The membership function uses a differentiable expression based on the logistic function to represent the credibility level of the radar information in the pixel area. Based on the image coordinates, the confidence matrix is ​​paired with the depth value and radial velocity value of the radar image projection point in the image space obtained previously by pixel position to construct a three-channel tensor, where the first channel is the depth value, the second channel is the velocity value, and the third channel is the fuzzy membership score of the corresponding pixel; According to the fuzzy membership function, the depth value and the speed value of each pixel are weighted to generate a fuzzy depth map and a fuzzy speed map that are integrated with confidence weights; The blurred depth map, blurred velocity map and original confidence matrix are spliced ​​in the channel dimension to generate a radar feature volume with a complete structure; the radar feature volume simultaneously encodes the depth, velocity and blurred confidence information for each pixel in the image space.

5. The image enhancement method based on deep learning and fusion of 4D millimeter-wave radar and camera information according to claim 1 is characterized in that: In the convolution-transformer-diffusion combined network, the radar feature body and the visual features are guided by fuzzy weights to perform cross-modal attention fusion and output an enhanced image, including: Inputting the radar feature volume into the first feature encoding branch, and using a multi-scale convolution module to extract radar local spatial structure information while retaining the distinguishing representations of depth, velocity, and confidence channels; The camera image at the corresponding timestamp is input into the second feature encoding branch, and the hierarchical visual Transformer is used to extract the visual global context features and generate multi-scale image semantic embeddings; A cross-modal multi-head attention mechanism is implemented on the feature layer output by the two branches, where the confidence channel in the radar feature volume is used as the attention guidance weight to guide the distribution of attention weights and enhance the fusion strength between visual features and credible radar information. The multimodal features after attention fusion are input into the diffusion residual enhancement module, and feature layer iterative denoising and reconstruction are performed under noise scheduling control to generate enhanced image feature maps; The enhanced image feature map is mapped back to the image space through a learnable decoder, and is aligned and corrected with the original image, and finally the enhanced image with enhanced contrast and details is output.

6. The image enhancement method based on deep learning and fusion of 4D millimeter-wave radar and camera information according to claim 1, characterized in that: The convolution-transformer-diffusion combination network is self-supervisedly optimized using the contrast, structural similarity, and blur regularization loss between the enhanced image and the multi-exposure reference image, including: Acquire a multi-exposure reference image that is consistent with the input image at the same time, and construct a high dynamic range reference image based on a local exposure feature fusion strategy to serve as a contrast supervision target for the enhanced image; Pair the enhanced image generated by the network with the multi-exposure reference image at the pixel level and calculate a contrast loss term; the contrast loss is based on a bidirectional feature difference metric that encourages the enhanced image to converge to the reference image in terms of color, edge, and brightness distribution; Calculating the structural similarity loss between the enhanced image and the reference image, and using a structural similarity index as a structure preservation indicator to measure the fidelity of the enhanced image in terms of details, texture, and contour structure; Extracting the confidence score and fuzzy membership response corresponding to each pixel position in the enhanced image, constructing a fuzzy regularization term, and suppressing over-enhancement or artifact generation in the fuzzy area by minimizing the gradient oscillation in the low-confidence area; The contrast loss, the structural similarity loss and the fuzzy regularization term are weighted and summed according to set weights to form a joint loss function; Taking the joint loss function as the optimization objective, end-to-end self-supervised training is performed on the convolution-transformer-diffusion combination network, and the network parameters are iteratively updated until convergence.

7. The image enhancement method based on deep learning and fusion of 4D millimeter-wave radar and camera information according to claim 1, characterized in that: The convolution-transformer-diffusion combination network includes: a set of parallel input coding layers, including a first convolutional coding branch and a second transformer coding branch; the first convolutional coding branch is used to receive the radar feature volume and extract local spatial features using a multi-scale convolutional block, while maintaining the independence of confidence channels during feature extraction; the second transformer coding branch is used to receive the original camera image and extract global contextual semantic information using a hierarchical visual Transformer network; A cross-modal fusion module, including a cross-channel multi-head attention layer, is used to align the radar features output by the first convolutional encoding branch with the visual features output by the second transformer encoding branch in the spatial dimension, and uses the confidence channel in the radar feature volume as an attention weight guidance factor to enhance the multimodal information coupling capability of high-confidence areas; The diffusion image reconstruction module consists of multiple cascaded noise scheduling layers and residual fusion layers, which are used to perform layer-by-layer denoising and detail recovery based on the fused features to reconstruct high-quality enhanced image feature maps. A learnable decoder structure is used to restore the image feature map output by the diffusion image reconstruction module to a standard image format, complete spatial alignment with the original image, and output the final enhanced image.

8. A 4D millimeter-wave radar and camera information fusion image enhancement method based on deep learning, characterized in that: include: The time synchronization acquisition and initial registration unit is used to simultaneously acquire the millimeter-wave radar point cloud containing azimuth, pitch, range and radial velocity information and the camera image at the corresponding time under a unified time reference, and complete the initial spatiotemporal registration using timestamps and external parameters; A differentiable mapping and registration confidence calculation unit is used to map the radar point cloud to the camera pixel coordinate system through a differentiable projection and fuzzy C-Means-ICP collaborative algorithm, and generate a confidence matrix with a value range of 0 to 1; A fuzzy weight construction unit is used to fuse the confidence matrix with the depth-velocity tensor according to a membership function to obtain a radar feature volume with a fuzzy weight; a cross-modal fusion and enhancement network unit, configured to guide cross-modal attention fusion of the radar feature volume and the visual feature using fuzzy weights in a convolution-transformer-diffusion combination network, and output an enhanced image; a self-supervised optimization training unit for performing self-supervised optimization on the convolution-transformer-diffusion combination network by utilizing the contrast, structural similarity, and blur regularization loss between the enhanced image and the multi-exposure reference image; A real-time inference and output unit is used to input real-time point clouds and images into the convolution-transformer-diffusion combination network to obtain enhanced results.

Citation Information

Cited By

  • Dam global hidden danger detection method based on radar image interpretation and visual large model

    CN122023341A

  • Stream refinement feature design method and device for 4D millimeter wave radar point cloud

    CN122135330A