A multi-modal fusion detection method and device

Abstract

The application discloses a multi-modal fusion detection method and device, belonging to the technical field of network and artificial intelligence. In the method, the image modal data and the auxiliary modal data such as depth / infrared / structured sensing of the object to be detected are collected first, and the multi-modal input data with alignment and quality enhancement is obtained through preprocessing, then the local spatial features are extracted through the CNN channel, the global context features are modeled through the Transformer channel, the multi-modal feature fusion is completed through the double-channel feature fusion layer, the detection result is output by the detection head network, and the display / alarm is realized. The method combines the feature extraction advantages of CNN and Transformer, realizes the collaborative use of multi-modal data through feature layer depth fusion, effectively improves the precision of target detection and abnormal identification in complex scenes, reduces the misjudgment and missed judgment caused by single modal noise, realizes the real-time feedback of the detection result, and improves the intelligence and robustness of the detection system.

Description

Technical Field

[0001] This application relates to the fields of network and artificial intelligence technology, and in particular to a multimodal fusion detection method and apparatus. Background Technology

[0002] In existing technologies, detection and monitoring for production, security, or equipment operation mostly rely on manual inspection and single-modal image detection methods. The collected image data is mostly obtained through manual observation or identification based on simple thresholds and traditional machine learning algorithms. Such methods are poorly adaptable to complex environments, weak feature targets, or multi-interference scenarios, and have many technical defects.

[0003] The data dimensions are limited, relying solely on visible light images for detection, failing to fully utilize multi-source information such as depth maps, infrared images, and structured sensor data, resulting in low data utilization efficiency. Feature modeling capabilities are also limited; convolutional neural networks can only extract local texture and edge features, lacking the ability to perceive long-distance dependencies and overall semantic scenes. Directly using Transformer structures performs poorly in fine-grained texture and small defect recognition, failing to effectively extract both local and global features. Manual monitoring has significant drawbacks; traditional methods heavily rely on operator experience, requiring staff to repeatedly observe monitoring screens or detection results, making them highly susceptible to fatigue and subjective bias, leading to frequent missed and false detections. Furthermore, a systematic multimodal fusion mechanism is lacking. While some existing methods introduce multi-source data, the data is only simply weighted or concatenated at the decision-making level, failing to achieve deep information interaction and complementarity at the feature level, thus hindering the realization of the synergistic advantages of multimodal data. These are examples of the main shortcomings.

[0004] Therefore, there is an urgent need for a multimodal detection method that can fully utilize auxiliary modal data on the basis of image modality and introduce a dual-channel fusion structure of CNN and Transformer in the feature layer to improve detection accuracy and robustness in complex scenes, while reducing the cost of manual intervention. Summary of the Invention

[0005] To address the problems of existing detection technologies that rely on single-modal data and human experience, resulting in low data utilization efficiency, limited local and global feature modeling capabilities, poor detection accuracy and robustness, and susceptibility to false negatives and missed detections, this invention provides a multimodal fusion detection method. The technical solution is as follows: On the one hand, a multimodal fusion detection method is provided, the method comprising: S101. During the target detection or monitoring process, at least one first modal image data of the object to be detected and at least one second modal auxiliary data corresponding to the first modal image data are collected. The second modal auxiliary data includes at least one of depth map, infrared map and / or structured sensing data. S102. Preprocess the first modal image data and the second modal auxiliary data to obtain aligned and enhanced quality multimodal input data; S103. Input the multimodal input data into the convolutional neural network (CNN) channel and the Transformer channel respectively. Extract the local spatial features of the first modality image data through the CNN channel to obtain the first modality features. Perform global context modeling on the second modality auxiliary data through the Transformer channel to obtain the second modality features. S104. Input the first modal feature and the second modal feature into the dual-channel feature fusion layer to perform multimodal feature fusion and obtain the fused feature representation; S105. Input the fused feature representation into the detection head network to obtain the multi-class detection result and / or anomaly determination result of the object to be detected; S106. Display and / or alarm based on the detection results.

[0006] Optionally, the acquisition device for acquiring the first modal image data is fixed at a preset position in the production line or monitoring scene, and the preset position meets the requirements for continuous high-definition capture of images and field of view coverage.

[0007] Optionally, the preprocessing described in step S102 includes: Image enhancement processing is performed on the first modality image data; Noise filtering is performed on the first modality image data and / or the second modality auxiliary data; A joint segmentation strategy combining multimodal data is adopted to segment the region of the object to be detected from the background and extract the region of interest. A dynamic alignment algorithm based on modal feature correlation is used to spatially align and scale-normalize the multimodal data in the region of interest to obtain the multimodal input data.

[0008] Optionally, the CNN channel includes multiple convolutional layers, pooling layers, and normalization layers, each convolutional layer containing multiple convolutional kernels, used to generate multi-scale local spatial feature maps based on the input first modality image data; The Transformer channel includes a multi-head self-attention layer and a feedforward network layer, which are used to perform global context modeling on the second modality auxiliary data and generate semantic feature maps.

[0009] Optionally, the dual-channel feature fusion layer includes a feature alignment module, a cross-modal attention module, a collaborative verification module, and a fusion reconstruction module. The multimodal feature fusion process includes: The feature alignment module maps the first modal feature and the second modal feature to a unified dimensional space; the cross-modal attention module uses one modal feature as the query vector and the other modal feature as the key vector, dynamically calculates and adjusts the modal weights of the query vector and the key vector, and dynamically adjusts the weight allocation ratio of each attention head in the multi-head attention according to the correlation index or confidence index of the first modal feature and the second modal feature, and obtains the cross-modal interaction feature through multi-head attention calculation; The collaborative verification module performs mutual verification during the multimodal fusion process. In the fusion stage, it performs consistency comparison based on the abnormal scores output by each modality feature. When the difference between the two scores exceeds the preset difference threshold, the cross-modal consistency verification mechanism is triggered to eliminate misjudgment interference caused by single modal noise. The fusion reconstruction module splices, weights, or performs residual fusion of the verified cross-modal interaction features, and reconstructs the fused features through a multilayer perceptron to obtain the fused feature representation.

[0010] Optionally, the detection head network includes at least one fully connected layer and a softmax classification layer and / or a sigmoid discriminant layer; The softmax classification layer is used to output multi-class classification results, and the sigmoid discriminant layer is used to output binary classification anomaly detection results.

[0011] Optionally, the step S106 of displaying and / or alarming based on the detection results includes: The monitoring interface displays the first modal image of the object to be detected and the detection result; in response to the detection result meeting the preset abnormal conditions, it issues an audible and visual alarm signal and / or sends an alarm message to the host computer.

[0012] Optionally, the method further includes: The fused feature representation and detection results are stored as a historical sample dataset; Based on the historical sample dataset, the CNN channels, the Transformer channels, and the dual-channel feature fusion layer are retrained online or offline; wherein, the retraining process includes: A modality reliability weight adjustment mechanism is introduced to dynamically update the fusion weights based on the quality of each modality data in historical samples; convolution kernel pruning is performed on the CNN channels, and attention sparsification is performed on the Transformer channels to continuously improve the multimodal fusion detection performance and inference real-time performance.

[0013] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer program, which is loaded and executed by a processor to implement the multimodal fusion detection method as described above.

[0014] The method in this embodiment, through the acquisition and deep fusion of multimodal data, combined with the feature extraction advantages of CNN and Transformer dual channels, achieves comprehensive capture of local spatial features and global contextual features, breaking through the detection limitations of single-modal data and single network structure, and improving adaptability to complex environments, weak feature targets and multi-interference scenarios; at the same time, through real-time display and alarm of detection results, the detection process is made intelligent and real-time, reducing manual intervention, lowering the probability of missed detection and false detection, and effectively improving the accuracy and robustness of target detection and anomaly recognition. Attached Figure Description

[0015] Figure 1 A flowchart of a multimodal fusion detection method is shown. Figure 2 This diagram illustrates a dual-channel feature extraction and fusion network structure. Figure 3 This diagram illustrates a system application scenario and closed-loop optimization mechanism. Figure 4 A hardware structure diagram of an electronic device that performs this method is shown. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0017] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0018] Example 1 This embodiment provides a multimodal fusion detection method, the method including: S101. During the target detection or monitoring process, at least one first modal image data of the object to be detected and at least one second modal auxiliary data corresponding to the first modal image data are collected. The second modal auxiliary data includes at least one of depth map, infrared map and / or structured sensor data.

[0019] Among them, the first modal image data is the basic visual data for target detection, which can be collected by equipment such as industrial cameras and network cameras. The second modal auxiliary data supplements the state information of the object to be detected from different physical dimensions. The depth map can reflect the spatial depth characteristics of the target, the infrared map can capture the thermal distribution characteristics of the target, and the structured sensor data can include ultrasonic sensor signals, vibration sensor signals, production parameters and other data. The collaborative acquisition of multimodal data provides a comprehensive data foundation for subsequent feature fusion and accurate detection, avoiding the limitations of a single modal data dimension.

[0020] S102. Preprocess the first modality image data and the second modality auxiliary data to obtain aligned and enhanced quality multimodal input data.

[0021] Preprocessing is a crucial preliminary step in multimodal fusion detection. It eliminates data noise and unifies data structure through a series of processes, taking into account the acquisition characteristics of different modal data. This enables different modal data to have the basic conditions for fusion, ensuring the accuracy of subsequent feature extraction and solving the problems of quality differences and spatial scale mismatch caused by acquisition equipment and transmission process in multimodal data.

[0022] S103. Input the multimodal input data into the convolutional neural network (CNN) channel and the Transformer channel respectively. Extract the local spatial features of the first modality image data through the CNN channel to obtain the first modality features. Perform global context modeling on the second modality auxiliary data through the Transformer channel to obtain the second modality features.

[0023] Among them, the CNN channel targets the visual characteristics of the first modality image data, enabling the extraction of fine-grained features such as image edges, textures, and local defects. The Transformer channel targets the semantic and relational characteristics of the second modality auxiliary data, enabling the modeling of long-range data dependencies and the overall scene context. The parallel feature extraction method of the two channels makes up for the limitations of a single network in feature modeling, and achieves comprehensive capture of local and global features.

[0024] S104. Input the first modal feature and the second modal feature into the dual-channel feature fusion layer to perform multimodal feature fusion and obtain the fused feature representation.

[0025] The dual-channel feature fusion layer is the core of achieving deep interaction of multimodal features. It is not a simple feature splicing, but a feature layer fusion process that achieves information complementarity between the first and second modal features. This allows the fused features to simultaneously contain local visual details and global contextual information of the object to be detected, solving the problem of insufficient information utilization in existing technologies where multimodal data is simply fused at the decision layer.

[0026] S105. Input the fused feature representation into the detection head network to obtain the multi-class detection results and / or anomaly determination results of the object to be detected.

[0027] Among them, the fusion feature represents the effective information of multimodal data. The detection head network performs targeted classification and discrimination processing on the fusion feature. It can output the specific category result of the object to be detected or simply determine whether there is an anomaly, according to the actual detection needs, so as to achieve flexible adaptation of the detection task and improve the relevance and effectiveness of the detection results.

[0028] S106. Display and / or alarm based on the detection results.

[0029] Among them, the display and alarm of the test results enable the real-time feedback of the test system, allowing operators to intuitively obtain test information. At the same time, it can issue alarms in a timely manner when abnormal situations occur, solving the problem of lag in traditional manual inspection, realizing real-time monitoring of the test target, and facilitating timely intervention measures.

[0030] The technical effects of this embodiment include: by acquiring and deeply fusing multimodal data, and combining the feature extraction advantages of CNN and Transformer dual channels, it achieves comprehensive capture of local spatial features and global contextual features, breaking through the detection limitations of single-modal data and single network structure, and improving adaptability to complex environments, weak feature targets, and multi-interference scenarios; at the same time, by displaying and alarming the detection results in real time, it realizes the intelligence and real-time nature of the detection process, reduces manual intervention, lowers the probability of missed detection and false detection, and effectively improves the accuracy and robustness of target detection and anomaly recognition.

[0031] This embodiment corresponds to Figure 1 , Figure 1 The method can be seen to provide an overall process from multimodal data acquisition to online alarm and model optimization, demonstrating the connection between steps S101 to S106, and corresponding to... Figure 2 , Figure 2 The network structure for feature extraction and fusion of CNN channels and Transformer channels is shown, reflecting the core processes from S103 to S104.

[0032] Example 2 According to the multimodal fusion detection method of Embodiment 1, the acquisition device for acquiring first modal image data is fixed at a preset position in the production line or monitoring scene, and the preset position meets the requirements for continuous high-definition capture of images and field of view coverage.

[0033] The fixed configuration of the acquisition device ensures continuous and stable acquisition of image data in the first modality, avoiding problems such as image blurring and loss of field of view caused by device movement. The selection of the preset position is based on continuous high-definition capture and field of view coverage. The specific position can be determined according to the production line layout and the monitoring range of the monitoring scene to ensure that the acquired image data can completely and clearly reflect the state of the object to be detected, providing high-quality basic data for subsequent preprocessing and feature extraction. At the same time, continuous image acquisition can realize dynamic monitoring of the detection target, adapting to the needs of continuous monitoring scenarios such as production process and equipment operation.

[0034] The technical effects of this embodiment include: by limiting the location and acquisition requirements of the first modality image data acquisition device, the quality and continuity of basic visual data acquisition are ensured, and image data failure caused by acquisition equipment or acquisition location problems is avoided. This provides high-quality visual data support for the entire process of subsequent multimodal fusion detection, ensuring the foundation of detection accuracy. At the same time, the fixed acquisition location is adapted to the detection needs of fixed scenarios such as industrial production lines and security monitoring, improving the practical application adaptability of the method.

[0035] Example 3 According to the multimodal fusion detection method of Embodiment 1, the preprocessing in step S102 includes: performing image enhancement processing on the first modality image data; performing noise filtering processing on the first modality image data and / or the second modality auxiliary data; using a joint segmentation strategy combining multimodal data to segment the target object region from the background and extract the region of interest; and using a dynamic alignment algorithm based on modal feature correlation to perform spatial alignment and scale normalization on the multimodal data in the region of interest to obtain multimodal input data.

[0036] Image enhancement processing targets the first modality image data and employs operations such as histogram equalization, contrast enhancement, and sharpening to improve visual clarity, strengthen target features, and reduce background interference. Noise removal processing utilizes Gaussian filtering and median filtering for image data and corresponding signal denoising methods for structured sensor data to eliminate noise interference generated during acquisition and transmission, thereby improving data purity. A joint segmentation strategy combining multimodal data leverages boundary information from depth maps and thermal distribution features from infrared images to guide region segmentation of the first modality image, accurately separating the target object from complex backgrounds. The extracted regions of interest focus on the core detection, reducing interference from irrelevant background data in subsequent feature extraction. A dynamic alignment algorithm based on modal feature correlation adaptively calibrates against subtle motion deviations and transmission delays from different sensors, while simultaneously normalizing the scale of multimodal data to achieve spatial and feature scale uniformity, providing the foundation for multimodal data fusion.

[0037] The technical effects of this embodiment include: through multi-step, targeted preprocessing operations, the quality and structure of multimodal data are enhanced and unified, effectively eliminating data noise, background interference, and spatial scale mismatch between modalities, allowing subsequent feature extraction to focus on the effective features of the object to be detected, thus improving the accuracy and efficiency of feature extraction; at the same time, the combined segmentation and dynamic alignment processing methods fully utilize the complementarity of multimodal data, improving the preprocessing effect and laying a high-quality data foundation for dual-channel feature extraction and multimodal fusion.

[0038] Example 4 According to the multimodal fusion detection method in Example 1, the CNN channel includes multiple convolutional layers, pooling layers, and normalization layers. Each convolutional layer contains multiple convolutional kernels, which are used to generate multi-scale local spatial feature maps based on the input first modality image data. The Transformer channel includes a multi-head self-attention layer and a feedforward network layer, which are used to perform global context modeling on the second modality auxiliary data and generate semantic feature maps.

[0039] In this system, the convolutional layers in the CNN channel perform convolution operations on the first modality image data using multiple different convolutional kernels, thereby extracting local features at different scales of the image. The pooling layer reduces the dimensionality of the features and decreases the computational load, while the normalization layer improves the training and inference efficiency of the network. The synergistic effect of the multi-layer structure ultimately generates multi-scale local spatial feature maps, fully capturing fine-grained features such as edges, textures, and local defects of the image. The multi-head self-attention layer in the Transformer channel can simultaneously model the global dependencies of the second modality auxiliary data from multiple dimensions, capturing the long-range correlations and overall semantic information of the data. The feedforward network layer performs non-linear transformations on the output of the self-attention layer to enhance feature expression. The final semantic feature map can reflect the global context state of the object to be detected.

[0040] The technical effects of this embodiment include: by limiting the network structure and function of the CNN channel and the Transformer channel, the two channels can leverage their respective advantages in local feature extraction and global context modeling, achieving accurate capture of features from different dimensions of multimodal data and compensating for the shortcomings of a single network structure in feature modeling; the generation of multi-scale local spatial feature maps and semantic feature maps provides a rich and effective feature foundation for subsequent deep fusion of multimodal features, ensuring that the fused features can simultaneously contain local details and global correlation information, thus improving the accuracy of subsequent detection results. This embodiment corresponds to... Figure 2 , Figure 2 The diagram clearly shows the network structure where the CNN channel extracts local detail features through convolutional / pooling layers, and the Transformer channel extracts global contextual features through a self-attention mechanism, intuitively reflecting the feature extraction process of the two channels.

[0041] Example 5 According to the multimodal fusion detection method in Example 1, the dual-channel feature fusion layer includes a feature alignment module, a cross-modal attention module, a collaborative verification module, and a fusion reconstruction module. The multimodal feature fusion process includes: The feature alignment module maps the first and second modal features to a unified dimensional space. The cross-modal attention module uses one modal feature as the query vector and the other modal feature as the key vector, dynamically calculates and adjusts the modal weights of the query vector and the key vector, and dynamically adjusts the weight distribution ratio of each attention head in the multi-head attention based on the correlation index or confidence index of the first and second modal features, thus obtaining the cross-modal interaction features through multi-head attention calculation. The collaborative verification module performs mutual verification during the multi-modal fusion process. In the fusion stage, it performs consistency comparison based on the abnormal scores output by each modal feature. When the difference between the two scores exceeds the preset difference threshold, the cross-modal consistency verification mechanism is triggered to eliminate the misjudgment interference caused by single modal noise. The fusion reconstruction module concatenates, weights, or performs residual fusion on the verified cross-modal interaction features, and reconstructs the fused features through a multilayer perceptron to obtain the fused feature representation.

[0042] The feature alignment module unifies the dimensions of features from different modalities through linear mapping or convolution operations, solving the problem of dimensionality differences between the output features of CNN channels and Transformer channels, and providing a foundation for cross-modal feature interaction. The cross-modal attention module can dynamically adjust modal weights according to the signal-to-noise ratio of the input multimodal data, and optimize the configuration of the number of multi-head attention heads for different feature distributions. For example, more attention heads are allocated to CNN features with complex local textures to achieve differentiated attention to features from different modalities. Through multi-head attention calculation, the two-way features achieve deep information interaction and complementarity, and the generated cross-modal interactive features contain effective information from both local and global features. The collaborative verification module is a key verification link for multimodal fusion. Through cross-modal secondary verification, it avoids feature distortion caused by noise, physical occlusion, and other problems in single-modal data, effectively eliminating misjudgment interference and improving the reliability of fused features. The fusion reconstruction module integrates cross-modal interactive features through splicing, weighting, or residual fusion, and then achieves feature reconstruction through nonlinear transformation of a multilayer perceptron, strengthening the expressive power of the fused features. The final generated fused feature representation can comprehensively reflect the core features of multimodal data.

[0043] The technical effects of this embodiment include: through a multi-module collaborative dual-channel feature fusion layer, deep fusion of multimodal features at the feature layer is achieved, rather than simple feature splicing, allowing full information interaction and complementarity between the first and second modal features; feature alignment solves the problem of dimensional differences between modalities; cross-modal attention achieves differentiated feature focus; collaborative verification effectively reduces misjudgments caused by single-modal noise; fusion reconstruction strengthens feature expression; and the final generated fused feature representation has more comprehensive and reliable feature information, providing core support for accurate detection by the subsequent detection head network, and significantly improving the accuracy and robustness of multimodal fusion detection.

[0044] Example 6 According to the multimodal fusion detection method in Embodiment 1, the detection head network includes at least one fully connected layer, a softmax classification layer, and / or a sigmoid discriminant layer. The softmax classification layer outputs multi-class classification results, and the sigmoid discriminant layer outputs binary anomaly determination results. Specifically, the fully connected layer integrates and reduces the dimensionality of the fused feature representation, mapping high-dimensional fused features to low-dimensional features suitable for the classification / discrimination task. The softmax classification layer calculates the multi-class probability distribution of the fused features, outputting the probability values ​​of the detected object belonging to different categories, thereby achieving the identification and classification of multi-class defects and multi-type targets. The sigmoid discriminant layer outputs the probability value of whether the detected object is abnormal, and binary classification of abnormal / normal can be achieved based on a preset threshold. The synergistic effect of the fully connected layer and the classification / discrimination layer allows the detection head network to be flexibly configured according to the actual detection task requirements.

[0045] The technical effects of this embodiment include: by limiting the structure and function of the detection head network, the detection head network can adapt to different detection task requirements, enabling both multi-class classification detection and simple anomaly binary classification judgment, thus improving the flexibility and practical application adaptability of the multimodal fusion detection method; at the same time, the combination of the fully connected layer and the dedicated classification / discrimination layer achieves accurate parsing of the fusion feature representation, which can fully mine the category and anomaly information in the fusion features, improving the accuracy and relevance of the detection results.

[0046] Example 7 According to the multimodal fusion detection method of Embodiment 1, step S106 involves displaying and / or alarming based on the detection results, including: displaying the first modal image of the object to be detected and the detection results on the monitoring interface; and issuing an audible and visual alarm signal and / or sending an alarm message to the host computer in response to the detection results meeting preset abnormal conditions.

[0047] The monitoring interface can be built using graphical interface development tools or web technology to synchronously display the first modal image and the corresponding detection results, allowing operators to intuitively and clearly obtain the visual state of the object to be detected and the detection conclusions, thus realizing the visualization of detection information. Preset abnormal conditions can be set according to the needs of the actual detection scenario, such as abnormal probability thresholds, specific defect categories, etc. When the detection results meet the conditions, the system can issue an audible and visual alarm signal through an audible and visual alarm device, and at the same time send alarm information to the host computer or management platform through the network, realizing multi-terminal feedback of abnormal information.

[0048] The technical effects of this embodiment include: visualizing the detection results to provide an intuitive presentation of the detection information, reducing the difficulty for operators to view the data; and providing timely alerts for abnormal situations through a dual feedback mechanism of audible and visual alarms and upper-level computer alarms, solving the problem of lag in traditional manual inspections, enabling operators to respond quickly to abnormal situations and take timely intervention measures, thus improving the intelligence and practicality of the detection system. This embodiment corresponds to... Figure 3 , Figure 3 The output results of the detection model can be seen in the process of online monitoring / alarm, which demonstrates the application position of the core function of this embodiment in the overall system.

[0049] Example 8 According to the multimodal fusion detection method of Embodiment 1, the method further includes: storing the fused feature representation and detection results as a historical sample dataset; and retraining the CNN channel, Transformer channel and dual-channel feature fusion layer online or offline based on the historical sample dataset.

[0050] The retraining process includes: introducing a modality reliability weight adjustment mechanism to dynamically update the fusion weights based on the quality of each modality data in historical samples; and performing convolution kernel pruning on CNN channels and attention sparsification on Transformer channels to continuously improve multimodal fusion detection performance and inference real-time performance.

[0051] The fusion feature representation includes the core feature information of multimodal data, and the detection results include labeled information. Storing both as a historical sample dataset provides rich labeled samples for model retraining, enabling continuous accumulation of sample data. Online retraining is adaptable to real-time detection scenarios, while offline retraining can perform in-depth optimization on a large number of accumulated samples. The retraining scope covers the core networks of feature extraction and feature fusion, ensuring performance optimization throughout the model process. The modality reliability weight adjustment mechanism can dynamically update the fusion weights of multimodal features based on quality indicators such as the signal-to-noise ratio and completeness of each modality data in historical samples, allowing higher-quality modality data to play a greater role in fusion and improving the model's adaptability. Pruning convolutional kernels in CNN channels can remove redundant convolutional kernels, reducing network computation. Attention sparsification processing in Transformer channels can reduce ineffective attention computation, achieving network lightweighting and improving inference real-time performance while ensuring detection performance.

[0052] The technical effects of this embodiment include: through the construction of historical sample datasets and online / offline retraining of the model, continuous iterative optimization of the multimodal fusion detection model is achieved, enabling the model to adapt to new detection scenarios and changes in data distribution, maintaining long-term detection accuracy; the modal reliability weight adjustment mechanism improves the model's adaptability to multimodal data of different qualities, further reducing false positives and false negatives; the lightweight processing of CNN channels and Transformer channels solves the problem of limited computing power of edge devices in industrial fields, improves the real-time performance of model inference, and makes the method more suitable for practical application scenarios with high real-time requirements, such as industrial production and security monitoring. This embodiment corresponds to... Figure 1 and Figure 3 , Figure 1 The model optimization process can be seen in the text. Figure 3 The closed-loop mechanism of continuous iterative optimization achieved by the detection model through anomaly feedback can be clearly seen, fully reflecting the core process of this embodiment.

[0053] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0054] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multimodal fusion detection method, characterized in that, The method includes: S101. During the target detection or monitoring process, at least one first modal image data of the object to be detected and at least one second modal auxiliary data corresponding to the first modal image data are collected. The second modal auxiliary data includes at least one of depth map, infrared map and / or structured sensing data. S102. Preprocess the first modal image data and the second modal auxiliary data to obtain aligned and enhanced quality multimodal input data; S103. Input the multimodal input data into the convolutional neural network (CNN) channel and the Transformer channel respectively. Extract the local spatial features of the first modality image data through the CNN channel to obtain the first modality features. Perform global context modeling on the second modality auxiliary data through the Transformer channel to obtain the second modality features. S104. Input the first modal feature and the second modal feature into the dual-channel feature fusion layer to perform multimodal feature fusion and obtain the fused feature representation; S105. Input the fused feature representation into the detection head network to obtain the multi-class detection result and / or anomaly determination result of the object to be detected; S106. Display and / or alarm based on the detection results.

2. The multimodal fusion detection method according to claim 1, characterized in that, The acquisition device for acquiring the first modality image data is fixed at a preset position in the production line or monitoring scene, and the preset position meets the requirements for continuous high-definition capture of images and field of view coverage.

3. The multimodal fusion detection method according to claim 1, characterized in that, The preprocessing described in step S102 includes: Image enhancement processing is performed on the first modality image data; Noise filtering is performed on the first modality image data and / or the second modality auxiliary data; A joint segmentation strategy combining multimodal data is adopted to segment the region of the object to be detected from the background and extract the region of interest. A dynamic alignment algorithm based on modal feature correlation is used to spatially align and scale-normalize the multimodal data in the region of interest to obtain the multimodal input data.

4. The multimodal fusion detection method according to claim 1, characterized in that, The CNN channel includes multiple convolutional layers, pooling layers, and normalization layers. Each convolutional layer contains multiple convolutional kernels, which are used to generate multi-scale local spatial feature maps based on the input first modality image data. The Transformer channel includes a multi-head self-attention layer and a feedforward network layer, which are used to perform global context modeling on the second modality auxiliary data and generate semantic feature maps.

5. The multimodal fusion detection method according to claim 1, characterized in that, The dual-channel feature fusion layer includes a feature alignment module, a cross-modal attention module, a collaborative verification module, and a fusion reconstruction module. The multimodal feature fusion process includes: The feature alignment module maps the first modal feature and the second modal feature to a unified dimensional space; The cross-modal attention module uses one modal feature as the query vector and the other modal feature as the key vector, dynamically calculates and adjusts the modal weights of the query vector and the key vector, and dynamically adjusts the weight allocation ratio of each attention head in the multi-head attention based on the correlation index or confidence index of the first modal feature and the second modal feature, and obtains the cross-modal interaction features through multi-head attention calculation. The collaborative verification module performs mutual verification during the multimodal fusion process. In the fusion stage, it performs consistency comparison based on the abnormal scores output by each modality feature. When the difference between the two scores exceeds the preset difference threshold, the cross-modal consistency verification mechanism is triggered to eliminate misjudgment interference caused by single modal noise. The fusion reconstruction module splices, weights, or performs residual fusion of the verified cross-modal interaction features, and reconstructs the fused features through a multilayer perceptron to obtain the fused feature representation.

6. The multimodal fusion detection method according to claim 1, characterized in that, The detection head network includes at least one fully connected layer and a softmax classification layer and / or a sigmoid discriminant layer; The softmax classification layer is used to output multi-class classification results, and the sigmoid discriminant layer is used to output binary classification anomaly detection results.

7. The multimodal fusion detection method according to claim 1, characterized in that, The step S106, which involves displaying and / or alarming based on the detection results, includes: The monitoring interface displays the first modal image of the object to be detected and the detection result; in response to the detection result meeting the preset abnormal conditions, it issues an audible and visual alarm signal and / or sends an alarm message to the host computer.

8. The multimodal fusion detection method according to claim 1, characterized in that, The method further includes: The fused feature representation and detection results are stored as a historical sample dataset; Based on the historical sample dataset, the CNN channels, the Transformer channels, and the dual-channel feature fusion layer are retrained online or offline; wherein, the retraining process includes: A modality reliability weight adjustment mechanism is introduced to dynamically update the fusion weights based on the quality of each modality data in historical samples; convolution kernel pruning is performed on the CNN channels, and attention sparsification is performed on the Transformer channels to continuously improve the multimodal fusion detection performance and inference real-time performance.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the multimodal fusion detection method as described in any one of claims 1 to 8.

Images