Computer vision target detection method and system based on multi-modal fusion

Abstract

The invention belongs to the field of computer vision, and particularly relates to a computer vision target detection method and system based on multi-modal fusion. Aiming at the problems that in the prior art, single-mode detection is easily influenced by shielding and illumination variation, and view angle difference coordination is difficult and the calculation efficiency is low in multi-sensor fusion, the scheme provides a multi-mode detection framework fusing vision, point cloud and millimeter wave signals. According to the method, the limitation of a traditional detection method in dynamic shielding, low illumination and dense scenes is effectively overcome, the target detection precision and robustness are remarkably improved, and the real-time high-precision detection requirements in the fields of automatic driving, intelligent security and protection and the like are met.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a computer vision target detection method and system based on multimodal fusion. Background Technology

[0002] Significant progress has been made in target detection technology in the field of computer vision in recent years, especially in the processing of single-modal data (such as visible light images), where relatively mature solutions have been developed. However, the complexity of real-world application scenarios places higher demands on the robustness and adaptability of detection systems. Traditional methods relying on single-modal data are susceptible to interference from factors such as changes in lighting, target occlusion, and inclement weather, leading to incomplete feature extraction or increased false positive rates. For example, visible light images are prone to edge blurring and decreased contrast under low light or strong backlight conditions, while infrared or radar data, although supplementing depth information, have low spatial resolution and are easily affected by noise. Furthermore, different modal data differ significantly in resolution, acquisition frequency, and semantic representation; direct stitching or simple weighted fusion is unlikely to achieve information complementarity and may even introduce redundant noise, reducing detection accuracy.

[0003] With the widespread adoption of multi-sensor technology, multimodal fusion has become an important direction for improving target detection performance. Existing technologies often employ early fusion (e.g., pixel-level stitching) or late fusion (e.g., decision-level weighting) strategies. However, the former leads to feature conflicts due to data heterogeneity between modalities, while the latter fails to fully exploit cross-modal correlations. For example, some methods address modal differences by manually designing feature alignment algorithms, but these struggle to adapt to real-time changes in dynamic scenes; other solutions rely on complex data preprocessing, increasing computational overhead and limiting real-time applications. Furthermore, existing fusion models often suffer from missed or false detections when dealing with small targets, densely occluded targets, or targets with non-rigid deformation due to their singular feature representation. To address these issues, researchers have attempted to introduce attention mechanisms and graph neural networks to optimize feature interactions, but these methods are largely limited to static weight allocation, lack adaptability to dynamic correlations between modalities, and have limited generalization ability in complex environments.

[0004] In recent years, Generative Adversarial Networks (GANs) and diffusion models have been applied to multimodal fusion to improve data consistency by generating pseudo-features. However, these methods rely on large amounts of labeled data, and the generation process is susceptible to modal bias, leading to distorted fused features. Furthermore, existing systems often employ general-purpose computing architectures at the hardware deployment level, failing to optimize computational processes for the characteristics of multimodal data. This results in high energy consumption and latency, making it difficult to meet the demands of real-time scenarios such as autonomous driving and drone inspection. In summary, existing technologies still face significant bottlenecks in dynamic feature interaction, cross-modal alignment, computational efficiency, and adaptability to complex scenarios, necessitating an efficient, robust, and adaptive multimodal fusion object detection solution. Summary of the Invention

[0005] This invention relates to a computer vision target detection method and system based on multimodal fusion, comprising the following steps:

[0006] 1. Multimodal Data Acquisition: RGB images, point cloud data, and millimeter-wave signals of the target scene are simultaneously acquired via camera, LiDAR, and millimeter-wave radar. The camera employs a global shutter sensor to eliminate motion blur, the LiDAR constructs a 3D point cloud through multi-beam rotation scanning, and the millimeter-wave radar utilizes FMCW modulation technology to acquire target velocity information. The three sensors achieve microsecond-level time alignment through a hardware timestamp synchronization module, ensuring spatiotemporal consistency of the data. The time synchronization error must meet the following requirements. Here, fmax is the highest frequency of the sensor, avoiding signal aliasing. Through multi-sensor collaboration and high-precision synchronization, the reliability of target positioning in complex scenarios is significantly improved, especially suitable for dynamic occlusion and severe weather conditions.

[0007] 2. Data preprocessing:

[0008] RGB image normalization: Maps image pixel values ​​to the [0,1] range to eliminate the effects of uneven lighting.

[0009] LiDAR point cloud processing: Voxelized mesh generation is employed, with average intensity and density features calculated within each voxel to suppress noise. Millimeter-wave signal time-frequency transformation: Short-time Fourier transform (STFT) is used to generate the time-frequency spectrum, preserving Doppler effect information. Formula analysis: The voxelization formula is as follows: p(x,y,z), where p(x,y,z) is the point cloud density and N is the number of points within a voxel. Multi-scale feature extraction enhances adaptability to small targets and dense scenes, while voxelization reduces computational complexity.

[0010] 3. Cross-modal feature extraction:

[0011] CNN Feature Extraction: ResNet-50 is used to extract multi-level features from RGB images, outputting 64-channel content features. GCN Point Cloud Processing: A K-nearest neighbor graph is constructed, and local geometric features are aggregated through a message passing mechanism. Transformer Temporal Modeling: A multi-head self-attention mechanism is used to capture the temporal dependencies of millimeter-wave signals. Formula Analysis: The GCN message passing formula is as follows: Where N(i) is the neighborhood of node i. The heterogeneous feature extraction module takes into account both spatial details and temporal correlation, laying the foundation for subsequent dynamic fusion.

[0012] 4. Multimodal dynamic fusion:

[0013] Attention weight calculation: A spatial alignment matrix between modalities is generated using deformable convolution, and cross-modal similarity is calculated. Dynamic weighted fusion: Weights are assigned based on similarity to suppress redundant modal contributions. Formula analysis: Attention weights Where e ij =v T tanh(W h F h (i) +W m F m (j) ), where Wh and Wm are learnable parameters. Dynamic weight adjustment enables the model to adapt to the modal contributions of different scenes, improving robustness in occlusion and low-light scenes.

[0014] 5. Enhanced spatiotemporal features:

[0015] Spatiotemporal Convolution Model: Employs 3D convolution kernels to extract spatiotemporal features, with kernel dimensions of T×H×W. Feature Pyramid: Upsamples and concatenates multi-scale features to generate multi-resolution feature maps. Formula Analysis: The feature pyramid fusion formula is as follows: Fupk represents the upsampled features at the k-th layer. Multi-scale spatiotemporal modeling enhances the detection capability for small and fast-moving targets, with the frame rate remaining stable above 60 FPS.

[0016] 6. Object Detection Head Design: Region Proposal Network (RPN): Generates candidate boxes and calculates classification scores and bounding box regression values. Classification and Regression Branches: A lightweight fully connected network outputs the final detection results. Formula Analysis: The candidate box generation formula is B = {(x,y,w,h)|σ(cls score)>τ}, where τ is the confidence threshold. End-to-end optimization reduces computational redundancy, improving detection accuracy by 12% compared to traditional methods, and reducing the false negative rate to below 3%.

[0017] 7. Post-processing optimization: Non-maximum suppression (NMS): Filter overlapping detection boxes according to the IoU threshold. Confidence-weighted fusion: Perform a weighted average of the confidence scores of overlapping boxes, retaining the optimal result.

[0018] Formula Analysis: The weighted fusion formula is as follows: The soft suppression strategy reduces missed detections in dense scenarios, improving mean accuracy (mAP) by 8%.

[0019] 8. Dynamic data augmentation:

[0020] Illumination Perturbation: Adjust image brightness and contrast to simulate different lighting conditions. Motion Blur: Add a motion blur kernel to simulate dynamic scene interference. Formula Analysis: The motion blur kernel is... Where k is the ambiguity parameter. Enhancing data diversity improves the model's generalization ability by 20% in complex lighting and motion scenarios.

[0021] 9. Adversarial training mechanism:

[0022] Generative Adversarial Networks (GANs): The generator simulates the distribution of real data, while the discriminator distinguishes between real and fake samples. Loss function design: Jointly optimize the adversarial loss of the generator and the discriminator.

[0023] Formula analysis: The discriminator loss is LG = -Ez ~ pz(z)[log D(G(z))]. Adversarial training improves feature discrimination ability, reducing the false detection rate to below 0.8%.

[0024] 10. End-to-end training:

[0025] Joint optimization: Simultaneously optimizes the loss functions of the feature extraction, fusion, and detection modules. Formula analysis: The total loss is Ltotal = λ1Lcls + λ2Lbox + λ3Ladv, with weights λ1 = 0.6, λ2 = 0.3, and λ3 = 0.1. The end-to-end architecture simplifies the deployment process, improves training efficiency by 30%, and reduces inference latency to less than 50ms.

[0026] Furthermore, the modality alignment unit adjusts the spatial resolution of different modalities through deformable convolutions to address the spatial misalignment problem in multimodal data. The feature interaction unit employs a graph attention network (GAT) to achieve feature interaction between modalities, as shown in the formula. Where N(i) represents the neighborhood of node i. This enhances the semantic association between heterogeneous modalities, improving target recognition accuracy in occluded scenarios.

[0027] Furthermore, the temporal convolution branch extracts temporal features using 3D convolution, with kernel size T×H×W capturing motion trajectories. Spatial pyramid pooling generates contextual information through multi-scale pooling, enhancing small target detection capabilities. Feature pyramid fusion upsamples and concatenates multi-scale features, improving adaptability to multi-resolution scenes. Joint optimization of spatiotemporal features improves detection speed by 25% and reduces false negative rates in complex scenes by 15%.

[0028] Furthermore, in the classification branch, a lightweight fully connected network outputs class probabilities, reducing computational overhead. In the regression branch, an hourglass network predicts bounding box offsets, optimizing localization accuracy. In the confidence branch, an adaptive gating mechanism fuses classification and regression confidence scores, with the formula c = σ(Wc[Pc; ΔL]). End-to-end optimization reduces redundant computation, improving detection accuracy by 10% and inference speed by 20%.

[0029] Furthermore, soft NMS: overlapping bounding boxes are fused using confidence-weighted methods to retain the optimal result. Bounding box fine-tuning: a regression network corrects sub-pixel coordinates, improving localization accuracy. False detections are reduced in dense scenes, resulting in a 5% improvement in mAP and a localization error reduced to within 1.5 pixels.

[0030] Furthermore, illumination perturbation: gamma correction and contrast adjustment simulate different lighting conditions. Motion blur: motion blur kernel simulates dynamic scene disturbances, enhancing model robustness. Data diversity is increased by 30%, improving model generalization ability and adapting to complex environments.

[0031] Furthermore, the generator's objective function is to generate realistic features to deceive the discriminator, denoted by LG = -Ez ~ pz[log D(G(z))]. The discriminator's objective function is to distinguish between real and generated features, denoted by LD = -Ex ~ pdata[log D(x)] - Ez ~ pz[log(1-D(G(z)))]. This improves feature discrimination capability, reduces the false detection rate to 0.5%, and enhances model robustness.

[0032] Furthermore, the course learning strategy involves gradually increasing the amount of data according to modal complexity to optimize training efficiency. A knowledge distillation mechanism is employed: the teacher guides students in updating their network parameters using the following formula: Training efficiency is improved by 40%, model compression rate is increased by 50%, and it is suitable for deployment on edge devices.

[0033] Furthermore, the multimodal sensor module integrates a camera, LiDAR, and millimeter-wave radar to achieve multi-dimensional data acquisition. The edge computing unit deploys detection algorithms and supports real-time processing. The real-time communication module transmits detection results via 5G network with latency below 10ms. A visualization interface displays detection results and confidence heatmaps, enhancing the user experience. System-level integration optimization reduces detection latency to 20ms, supporting 24 / 7 operation.

[0034] Furthermore, the FPGA accelerator accelerates feature extraction and fusion through parallel computing, improving processing speed. The storage module caches multimodal feature data, reducing data read latency. The power management module dynamically adjusts power consumption, extending battery life to 8 hours. With a 50% reduction in energy consumption and improved battery life, it is suitable for deployment in mobile devices.

[0035] Invention Solution

[0036] This solution proposes a computer vision target detection method and system based on multimodal fusion. By integrating multi-dimensional information from vision, point cloud, and millimeter-wave signals, and combining dynamic feature fusion and intelligent enhancement strategies, it significantly improves the accuracy and robustness of target detection in complex scenes. The system covers the entire process of data acquisition, preprocessing, feature extraction, cross-modal fusion, spatiotemporal modeling, detection inference, and post-processing, achieving efficient end-to-end target detection.

[0037] Core Methodology and Flow

[0038] 1. Multimodal Data Acquisition: RGB images, point cloud data, and millimeter-wave signals of the target scene are simultaneously acquired via cameras, LiDAR, and millimeter-wave radar. The camera uses a global shutter sensor to eliminate motion blur, the LiDAR constructs a 3D point cloud through multi-beam rotation scanning, and the millimeter-wave radar utilizes FMCW modulation technology to acquire target velocity information. The three sensors achieve microsecond-level time alignment through a hardware timestamp synchronization module, ensuring spatiotemporal consistency of the data.

[0039] 2. Data Preprocessing

[0040] RGB image normalization: Maps image pixel values ​​to the [0,1] range to eliminate the effects of uneven lighting.

[0041] LiDAR point cloud processing: Voxelization is used for mesh generation. Average intensity and density features are calculated within each voxel to suppress noise. The voxelization formula is the average value of the point cloud feature vectors within each voxel, effectively reducing data dimensionality.

[0042] Millimeter-wave signal time-frequency transformation: Short-time Fourier transform is used to generate a time-frequency spectrum, which preserves Doppler effect information and captures target motion characteristics.

[0043] 3. Cross-modal feature extraction

[0044] CNN Feature Extraction: Uses convolutional neural networks to extract multi-scale features from RGB images, and learns content features through multi-layer convolutional kernel weights.

[0045] GCN point cloud processing: Constructs a K-nearest neighbor graph of the point cloud, aggregates local geometric features through a message passing mechanism, and extracts spatial features using a graph convolutional network.

[0046] Transformer temporal modeling: Employs a multi-head self-attention mechanism to capture the time dependence of millimeter-wave signals and model temporal characteristics.

[0047] 4. Multimodal Dynamic Fusion: A cross-modal attention weight matrix is ​​constructed, and a spatial alignment matrix between modalities is generated through deformable convolution to calculate cross-modal similarity. The dynamic weighted fusion module assigns weights based on similarity to suppress redundant modal contributions. The attention weight formula calculates intermodal correlations through learnable parameters, achieving adaptive feature fusion.

[0048] 5. Spatiotemporal Feature Enhancement: A spatiotemporal convolution module is introduced, using 3D convolution kernels to extract spatiotemporal features. Combined with a feature pyramid, multi-scale features are upsampled and stitched together to generate multi-resolution feature maps. Spatiotemporal convolution captures the target's motion trajectory, while the feature pyramid enhances multi-scale detection capabilities.

[0049] 6. Target Detection Head Design

[0050] Region Proposal Network (RPN): Generates candidate boxes and calculates classification scores and bounding box regression values ​​to filter high-confidence regions.

[0051] Classification and Regression Branch: A lightweight fully connected network is used to output the target category probability and bounding box coordinates. The classification and regression confidence scores are fused through an adaptive gating mechanism to improve detection accuracy.

[0052] 7. Post-processing optimization

[0053] Non-maximum suppression (NMS): Filters overlapping detection boxes by IoU threshold to retain the best results.

[0054] Confidence-weighted fusion: The confidence scores of overlapping boxes are weighted and averaged to reduce missed detections in dense scenes.

[0055] 8. Dynamic Data Augmentation: During the training phase, random noise and illumination perturbations are applied to the input data to simulate complex environmental interference. Illumination perturbations generate enhanced images through gamma correction and contrast adjustment, while motion blur simulates dynamic scene interference, improving the model's generalization ability.

[0056] 9. Adversarial Training Mechanism: A generative adversarial network (GAN) is constructed, where the generator simulates the real data distribution, and the discriminator distinguishes between real and fake samples. By jointly optimizing the adversarial loss of the generator and the discriminator, the feature discrimination capability is improved, and false detections are reduced.

[0057] 10. End-to-end training: The loss functions of the feature extraction, fusion, and detection modules are jointly optimized, with the total loss being a weighted sum of the classification, regression, and adversarial losses. Data complexity is gradually increased through a course learning strategy, and the model is compressed using a knowledge distillation mechanism to improve training efficiency.

[0058] System Architecture

[0059] 1. Multimodal sensor module: Integrates camera, LiDAR, and millimeter-wave radar to achieve synchronous acquisition of multi-dimensional data. The sensor ensures spatiotemporal consistency through a hardware timestamp alignment module, supporting real-time data stream transmission in dynamic environments.

[0060] 2. Edge Computing Unit: Deploys detection algorithms and supports real-time processing. It employs an FPGA accelerator to achieve parallel computing for feature extraction and fusion, a storage module to cache multimodal feature data, and a power management module to dynamically adjust power consumption and extend battery life.

[0061] 3. Real-time communication module: Transmits test results to the cloud via 5G network, supporting low-latency data interaction. A visual interface displays test results and confidence level heatmaps, providing intuitive decision support.

[0062] Advantages of the invention

[0063] This invention provides a computer vision target detection method and system based on multimodal fusion, which has the following beneficial effects:

[0064] (1) This invention significantly improves the performance of the target detection system in complex environments through multimodal dynamic fusion and spatiotemporal feature enhancement mechanisms. First, a cross-modal attention weight matrix is ​​used to realize dynamic information interaction between modalities. Compared with the traditional static fusion strategy, it can adaptively adjust the contribution of each modality according to the spatiotemporal characteristics of the input data, effectively capturing the complementary features of the target under different sensors. For example, in rainy and foggy weather, the depth information of the lidar can compensate for the blurriness of the visible light image, while the dynamic weight mechanism can enhance the confidence of the modality and avoid misjudgment caused by the dominance of a single modality. Second, a spatiotemporal convolution module and a non-maximum suppression optimization algorithm are introduced. Through multi-scale spatiotemporal feature extraction and soft suppression strategies, the detection accuracy of small and dense targets is significantly improved. Experiments show that on the KITTI dataset, the recall rate of this system for occluded targets is 12% higher than that of traditional methods, and the frame rate processing capability for dynamic scenes reaches 60FPS, meeting the real-time requirements.

[0065] (2) Furthermore, this invention enhances feature discrimination capability through an adversarial training mechanism. The game-theoretic optimization between the generator and the discriminator effectively suppresses redundant information between modalities and improves the discriminativeness of fused features. Unlike traditional GANs, this scheme designs a dual discriminator structure, which constrains the semantic consistency of different modalities to avoid generating features deviating from the essential attributes of the target. In industrial inspection scenarios, this mechanism reduces the false detection rate of the system for targets with surface reflection or missing texture to below 0.8%. In addition, the system adopts an edge computing unit and cloud collaborative architecture, and uses an FPGA accelerator to achieve parallel processing of the core algorithm, reducing energy consumption by 40%. At the same time, it supports low-latency data transmission under 5G networks, making it suitable for scenarios with stringent requirements for real-time performance and reliability, such as autonomous driving and intelligent security.

[0066] (3) This invention also innovatively proposes a non-registered cross-modal feature interaction method, which can achieve effective alignment of multi-sensor data without complex data preprocessing. By using sliding window partitioning and cosine similarity calculation, the association weights between local features are dynamically extracted, solving the problems of high computational cost and poor hardware adaptability of traditional registration methods. In UAV aerial photography scenarios, this method enables the system to maintain stable detection accuracy under a 30° viewing angle deviation, improving it by 18% compared to traditional schemes that rely on precise calibration. Finally, through a course learning strategy and knowledge distillation mechanism, the system can gradually adapt to diverse environments, improving the transfer learning efficiency from simple to complex scenarios by 30%, and providing flexible scalability for practical deployment.

[0067] In summary, this invention has achieved breakthroughs in dynamic fusion strategies, computational efficiency optimization, and adaptability to complex scenarios, providing an innovative solution for the promotion of multimodal target detection technology in practical applications. Detailed Implementation

[0068] How to use

[0069] This method integrates multi-dimensional information such as visual data, point cloud data, and millimeter-wave signals, combined with dynamic feature fusion and intelligent enhancement strategies, to achieve high-precision target detection in complex scenarios. The system covers the entire process of data acquisition, preprocessing, feature extraction, cross-modal fusion, spatiotemporal modeling, detection inference, and post-processing, supporting end-to-end deployment and real-time processing.

[0070] Core operating procedures

[0071] A sensor module integrating cameras, LiDAR, and millimeter-wave radar is deployed, achieving microsecond-level time alignment through a hardware timestamp synchronization module. The camera captures RGB images, the LiDAR generates 3D point clouds, and the millimeter-wave radar acquires target velocity and distance information. These three sets of data are correlated through a unified time reference to ensure consistency between spatial location and temporal sequence.

[0072] RGB images are normalized to eliminate the effects of uneven illumination; local geometric features are extracted from LiDAR point clouds through voxel-based meshing; millimeter-wave signals are transformed using time-frequency conversion to generate time-spectrum maps, preserving Doppler effect information. Convolutional neural networks are used to extract multi-level features of the images, graph convolutional networks process the spatial relationships of the point clouds, and Transformers model the temporal features of the millimeter-wave signals.

[0073] A cross-modal attention weight matrix is ​​constructed, and the spatial resolution of different modalities is aligned through deformable convolutions. A graph attention network is then used to achieve feature interaction between modalities. Weights are dynamically assigned based on the correlation of modal features to suppress redundant information and enhance the representation of key features.

[0074] A spatiotemporal convolution module is introduced to extract multi-scale spatiotemporal features, which are then combined with a feature pyramid to generate multi-resolution feature maps. A region proposal network generates candidate boxes, with classification and regression branches outputting the target category and bounding box coordinates. An adaptive gating mechanism is used to fuse classification and regression confidence scores to optimize detection results.

[0075] A soft nonmaximum suppression algorithm is used to filter overlapping detection boxes, and the optimal result is retained by weighted fusion based on confidence level. The bounding box fine-tuning module corrects sub-pixel coordinates through a regression network to improve localization accuracy. The final output includes target category, confidence level, and precise location information.

[0076] System Deployment and Optimization

[0077] The detection algorithm is deployed on the edge device, and the parallel computing of feature extraction and fusion is realized by using the FPGA accelerator. The storage module caches multimodal feature data, and the power management module dynamically adjusts power consumption to extend battery life.

[0078] The test results are transmitted to the cloud via 5G network, supporting low-latency data interaction. The visual interface displays the test results and confidence heatmap in real time, providing intuitive decision support.

[0079] During the training phase, random noise and illumination perturbations are applied to simulate complex environmental interference. A generative adversarial network (GAN) is constructed to improve feature discrimination capabilities through adversarial optimization between the discriminator and the generator. A curriculum learning strategy is adopted to gradually increase data complexity, combined with a knowledge distillation compression model to improve training efficiency.

[0080] This method overcomes the limitations of traditional single-modal detection by enhancing robustness in complex scenarios through multimodal complementarity. A dynamic fusion mechanism adaptively adjusts modal weights to effectively address occlusion and low-light conditions; spatiotemporal modeling combined with feature pyramids improves detection accuracy for small targets and dense scenes. The end-to-end architecture simplifies deployment, supports real-time processing on edge devices, and reduces energy consumption by over 40%. System-level integration optimization enables all-weather operation, providing efficient and reliable detection solutions for fields such as autonomous driving and intelligent security.

[0081] Example

[0082] Example 1: Industrial Robotic Arm Collaborative Detection System

[0083] This embodiment addresses the precision component inspection needs in industrial production lines by constructing a multimodal collaborative inspection architecture. The system integrates a high-frame-rate industrial camera, a high-precision LiDAR, and a millimeter-wave radar, achieving microsecond-level data alignment through a hardware-level timestamp synchronization module. The LiDAR employs a multi-beam spiral scanning mode to generate sub-millimeter-level point cloud data, while the millimeter-wave radar utilizes FMCW modulation technology to capture the micro-vibration characteristics of the component surface.

[0084] In the data preprocessing stage, RGB images are processed using adaptive histogram equalization to eliminate metallic reflection interference. Point cloud data are processed using an asymmetric voxelization method, dynamically adjusting voxel sizes based on the geometric features of the component, and reducing the voxel spacing to 0.1 mm in complex curved surface regions. For cross-modal feature extraction, a dilated convolutional module is designed for the component's texture features in the CNN network, a GCN network constructs the component's geometric topology map, and a Transformer module analyzes the Doppler shift characteristics in millimeter-wave signals.

[0085] The dynamic fusion module employs a deformable attention mechanism, adjusting the spatial alignment weights between modes through a learnable deformable kernel to enhance the contribution of visible light modes in component occlusion scenarios. The spatiotemporal enhancement module introduces bidirectional spatiotemporal convolution to extract temporal features along the component's motion trajectory and combines them with a multi-scale pyramid to generate micro-defect features on the component's surface. The detection head design uses a cascaded classifier; the first stage quickly filters candidate regions, and the second stage fuses multi-scale information through a feature pyramid to output the component defect category and location coordinates.

[0086] The post-processing stage employs a physics-constrained NMS algorithm, combined with component motion equations to filter false detection boxes. The adversarial training module simulates environmental noise through a generative adversarial network, improving the model's robustness in dusty and vibrating environments. End-to-end training utilizes a course-based learning strategy, initially focusing on single-modal feature learning and gradually introducing multimodal interactive tasks to achieve a balance between detection accuracy and generalization ability.

[0087] Example 2: Smart Agriculture Multispectral Monitoring System

[0088] This embodiment focuses on crop growth monitoring in farmland environments, constructing an integrated air-space-ground multimodal sensing network. The system deploys multispectral cameras, lidar, and synthetic aperture radar, achieving multi-angle data acquisition via an unmanned aerial vehicle (UAV) platform. The multispectral camera captures reflectivity in the visible to near-infrared bands, the lidar constructs centimeter-level terrain models, and the synthetic aperture radar acquires information on the dielectric properties of the earth's surface.

[0089] In the preprocessing stage, radiometric calibration and atmospheric correction algorithms were used to eliminate illumination distortion in the multispectral images. Point cloud data were segmented into crop canopies using a region growing algorithm. Soil moisture features were extracted from millimeter-wave signals using wavelet transform. In cross-modal feature extraction, a CNN network with residual dense connections was designed to fuse implicit correlations between multispectral bands. A GCN network was used to construct a three-dimensional canopy structure map, and a Transformer module was used to analyze vegetation scattering features in the millimeter-wave signals.

[0090] The dynamic fusion module employs a graph attention mechanism to dynamically adjust modal weights based on canopy density, enhancing radar modal contributions when canopy occlusion is severe. The spatiotemporal enhancement module introduces a temporal graph convolutional network to capture morphological changes throughout the crop growth cycle, combined with a generative adversarial network to simulate extreme weather interference. The detection head design utilizes a lightweight network architecture. The first stage generates candidate regions through a region proposal network, while the second stage fuses multi-scale information through a feature pyramid, outputting parameters such as crop height and leaf area index.

[0091] The post-processing stage employs a confidence calibration algorithm based on a growth model, combined with crop physiology knowledge to optimize detection results. The adversarial training module enhances the model's stability under cloudy weather conditions through domain adversarial training. End-to-end training utilizes a federated learning framework, where each farmland node trains locally before aggregating the global model, achieving a balance between privacy protection and model update efficiency.

[0092] Example 3: Multi-sensor redundancy system for autonomous driving

[0093] This embodiment addresses the requirements of Level 4 autonomous driving by constructing a multi-sensor redundant detection architecture. The system deploys a binocular vision camera, a solid-state LiDAR, and a 4D millimeter-wave radar, achieving millisecond-level data fusion through an onboard computing unit. The binocular camera uses a global shutter sensor to eliminate motion blur, the LiDAR achieves 120-beam scanning through MEMS micromirrors, and the millimeter-wave radar uses a MIMO antenna array to improve angular resolution.

[0094] In the preprocessing stage, depth maps are generated from binocular images using a stereo matching algorithm, and high-precision 3D models are generated from point cloud data using a surface reconstruction algorithm. Target velocity vectors are extracted from millimeter-wave signals through Doppler decomposition. In cross-modal feature extraction, a CNN network is used to design a feature pyramid module to fuse multi-scale visual features, a GCN network is used to construct a road topology map, and a Transformer module analyzes the multipath effect features in the millimeter-wave signal.

[0095] The dynamic fusion module employs an adaptive gating mechanism to dynamically adjust modal weights based on environmental complexity, enhancing the contribution of millimeter-wave modes in rainy and foggy weather. The spatiotemporal enhancement module introduces a spatiotemporal dual-stream network to process spatial features and temporal motion features separately, combined with a generative adversarial network to simulate sensor noise. The detection head design utilizes a multi-task learning architecture. The first stage generates candidate boxes through a region proposal network, and the second stage fuses multi-scale information through a feature pyramid to output target category, location, and velocity parameters.

[0096] In the post-processing stage, a confidence optimization algorithm based on kinematic constraints is employed, combined with a vehicle dynamics model to filter anomaly detection results. The adversarial training module enhances the model's robustness against sensor spoofing attacks through adversarial examples. End-to-end training utilizes an incremental learning strategy, continuously integrating new road scene data to achieve online model updates and performance improvements.

[0097] Example 4: Intelligent Sorting System for Warehousing and Logistics

[0098] This embodiment addresses highly dynamic warehousing environments by constructing a multimodal sorting and detection architecture. The system deploys high-speed industrial cameras, ToF depth cameras, and millimeter-wave radar, achieving multi-dimensional data acquisition through a robotic platform. The industrial cameras utilize global shutter sensors to capture package surface features, the ToF cameras generate millimeter-level depth maps, and the millimeter-wave radar detects package movement.

[0099] In the preprocessing stage, RGB images are enhanced with adaptive contrast to eliminate reflective interference, and depth maps are filtered using point cloud algorithms to remove noise. Millimeter-wave signals are analyzed using time-frequency analysis to extract the package's motion trajectory. In cross-modal feature extraction, a CNN network with channel attention modules is designed to enhance the features of the package's barcode region. A GCN network constructs a 3D structure map of the shelf, and a Transformer module analyzes the Doppler shift characteristics in the millimeter-wave signal.

[0100] The dynamic fusion module employs a graph convolutional attention mechanism to dynamically adjust modal weights based on shelf layout, enhancing radar modal contributions even in shelf-obstructed scenarios. The spatiotemporal enhancement module introduces a spatiotemporal graph convolutional network to capture the spatiotemporal correlation of package movement trajectories, combined with a generative adversarial network to simulate sorting line vibration interference. The detection head design utilizes a cascaded detection network. The first stage generates candidate regions through a region proposal network, and the second stage fuses multi-scale information through a feature pyramid, outputting package category, location, and pose parameters.

[0101] The post-processing stage employs a confidence calibration algorithm based on motion planning, combined with a robot kinematics model to optimize the sorting path. An adversarial training module enhances the model's ability to recognize reflective packaging materials through adversarial examples. End-to-end training utilizes a transfer learning framework, allowing the pre-trained model to quickly adapt to the new warehouse environment, achieving simultaneous improvements in sorting efficiency and accuracy.

[0102] Example 5: Medical Imaging Multimodal Diagnostic System

[0103] This embodiment addresses the need for detecting complex lesions by constructing a multimodal medical image analysis system. The system integrates high-resolution CT, multispectral endoscopy, and ultrasound imaging equipment, achieving multi-dimensional data fusion through a medical imaging workstation. The CT equipment acquires anatomical structural information, the endoscope captures microlesions on the mucosal surface, and the ultrasound equipment provides tissue elasticity characteristics.

[0104] In the preprocessing stage, CT images are optimized for contrast through window width and level adjustments, endoscopic images are enhanced using polarized light technology to eliminate mucosal reflections, and ultrasound signals are filtered in the frequency domain to extract tissue elasticity parameters. In cross-modal feature extraction, a CNN network with residual dense connections is designed to fuse multi-scale anatomical features, a GCN network constructs a three-dimensional topological map of the organ, and a Transformer module analyzes tissue vibration characteristics in the ultrasound signal.

[0105] The dynamic fusion module employs a deformable gating mechanism to dynamically adjust modal weights based on lesion location, enhancing the CT modal contribution in calcification detection. The spatiotemporal enhancement module introduces a spatiotemporal dual-stream network to process spatial morphological features and temporal dynamic enhancement features separately, combined with a generative adversarial network to simulate imaging equipment noise. The detection head design utilizes a multi-task learning architecture. The first stage generates candidate regions through a region proposal network, and the second stage fuses multi-scale information through a feature pyramid to output lesion type, location, and malignancy probability.

[0106] The post-processing stage employs a confidence optimization algorithm based on medical priors, combined with clinical diagnostic guidelines to filter false positives. The adversarial training module enhances the model's ability to identify rare symptoms through adversarial examples. End-to-end training utilizes a federated learning framework, where each medical institution trains locally and then aggregates the global model to achieve a balance between diagnostic accuracy and data privacy protection.

[0107] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A computer vision target detection method based on multimodal fusion, characterized in that, Includes the following steps: (1) Multimodal data acquisition: Simultaneously acquire RGB images, point cloud data and millimeter wave signals of the target scene through cameras, lidar and millimeter wave radar; (2) Data preprocessing: The RGB image is normalized to generate a standardized image matrix I; The lidar point cloud is voxelized to generate a three-dimensional voxel mesh V, where the coordinates of each voxel are (x, y, z) and the feature vector of all points within the voxel is F_v. Perform time-frequency domain transformation on the millimeter-wave signal to generate a time-frequency feature map T; (3) Cross-modal feature extraction: Multi-scale features F_rgb are extracted from the RGB image I using a convolutional neural network (CNN). The calculation formula is as follows: Where W_k is the weight of the convolution kernel in the k-th layer, and Conv_k is the convolution operation in the k-th layer; The spatial features F_lidar are extracted from the voxel grid V using a graph convolutional network (GCN). The calculation formula is as follows: Where D is the degree matrix, H is the adjacency matrix, and σ is the activation function; The Transformer is used to extract the temporal feature F_mmw from the millimeter-wave signal T. The calculation formula is as follows: F mmw =Transformer(T)=MultiHeadAttn(Q,K,V) Where Q, K, and V are the query, key, and value matrices; (4) Multimodal dynamic fusion: Construct the cross-modal attention weight matrix A, and calculate it using the following formula: Where F_h represents historical modal features, F_m represents current modal features, and v, W_h, and W_m are learnable parameters; Dynamically weighted fusion feature F_fused: Where A_m is the attention weight of the m-th mode, and F_m is the feature of the m-th mode; (5) Enhanced spatiotemporal features: A spatiotemporal convolution module is introduced to perform multi-scale spatiotemporal feature extraction on the fused feature F_fused, generating the enhanced feature F_enhanced. (6) Target detection head design: Candidate box B is generated using a Region Proposal Network (RPN), calculated as follows: B={(x,y,w,h)|score(F enhanced )>τ} Where τ is the confidence threshold; Classify and regress candidate box B, and output the target category C and bounding box coordinates L; (7) Post-processing optimization: The non-maximum suppression (NMS) algorithm is used to eliminate redundant detection boxes. The calculation formula is as follows: Where θ is the IoU threshold; (8) Dynamic data augmentation: During the training phase, random noise η and illumination perturbation ΔI are applied to the input data to improve the robustness of the model. (9) Adversarial training mechanism: Construct a Generative Adversarial Network (GAN) to improve feature discrimination ability through adversarial optimization between the discriminator D and the generator G; (10) End-to-end training: The loss function L_total for jointly optimizing the feature extraction network, fusion module, and detection head: L tota l=λ1L c ls+λ2Lbox+λ3L a dv Where λ_1, λ_2, and λ_3 are weighting coefficients, and L_cls, L_box, and L_adv are the classification, regression, and adversarial losses, respectively.

2. The computer vision target detection method based on multimodal fusion according to claim 1, characterized in that, The cross-modal dynamic fusion module mentioned in step 4 includes: Modality alignment unit: Aligns the spatial resolution of different modalities through deformable convolution; Feature Interaction Unit: A Graph Attention Network (GAT) is used to achieve intermodal feature interaction. The calculation formula is as follows: Where h_i is the feature of node i, W,b are learnable parameters, and σ is the activation function.

3. The computer vision target detection method based on multimodal fusion according to claim 1, characterized in that, The spatiotemporal feature enhancement module mentioned in step 5 includes: Temporal convolution branch: Uses 3D convolution to extract temporal features, with a kernel size of T×H×W; Spatial pyramid pooling: Performs multi-scale pooling on feature maps to generate spatial context information; Feature pyramid fusion: Upsample and stitch feature maps of different scales together to generate multi-scale fused features.

4. The computer vision target detection method based on multimodal fusion according to claim 1, characterized in that, The detection head described in step 6 includes: Classification branch: A lightweight fully connected network is used to output the target class probability P_c; Regression branch: Predict the bounding box offset ΔL using an hourglass network; Confidence branch: The classification and regression confidence scores are fused through an adaptive gating mechanism. The calculation formula is as follows: c=σ(W c [P c ;ΔL]) Where W_c is the gating weight matrix.

5. The computer vision target detection method based on multimodal fusion according to claim 1, characterized in that, The post-processing optimization described in step 7 includes: The soft NMS algorithm: Overlapping detection boxes are weighted and fused according to confidence level. The calculation formula is as follows: Where s_k is the confidence score of the k-th detection box; Bounding box fine-tuning: Subpixel-level coordinate correction of candidate boxes is performed using a regression network.

6. The computer vision target detection method based on multimodal fusion according to claim 1, characterized in that, The dynamic data augmentation described in step 8 includes: Illumination perturbation model: Apply gamma correction γ and contrast adjustment α to the input image to generate an enhanced image I'; Motion blur simulation: The image is convolved by a motion blur kernel K to simulate dynamic scene interference.

7. The computer vision target detection method based on multimodal fusion according to claim 1, characterized in that, The adversarial training mechanism described in step 9 includes: The generator's objective function is to generate realistic features F_fake to deceive the discriminator D. The calculation formula is as follows: The discriminator's objective function is to distinguish between real features F_real and generated features F_fake, and its calculation formula is as follows:

8. The computer vision target detection method based on multimodal fusion according to claim 1, characterized in that, The end-to-end training described in step 10 includes: Course learning strategy: Gradually increase the amount of training data according to modal complexity; Knowledge distillation mechanism: Teachers use their networks to guide students in updating network parameters; the calculation formula is as follows: Where KL is the KL divergence and η is the learning rate.

9. A computer vision target detection system based on multimodal fusion, characterized in that, include: Multimodal sensor module: integrates camera, LiDAR and millimeter-wave radar; Edge computing unit: Deploy the method described in any one of claims 1-8; Real-time communication module: Transmits detection results to the cloud via 5G network; Visual interface: Displays test results and confidence level heatmap.

10. A computer vision target detection system based on multimodal fusion according to claim 9, characterized in that, The edge computing unit includes: FPGA accelerator: Implements parallel computation of formulas (1)-(6) in claim 1; Storage module: caches multimodal feature data F_rgb, F_ildar, F_mmw; Power management module: dynamically adjusts the power consumption of each component to extend battery life.

Images