An industrial anomaly detection method based on multi-modal data compression
By employing a multimodal data compression method, the problems of high dimensionality and high noise in industrial anomaly detection are solved, achieving accurate detection and optimization of computing resources, adapting to complex scenarios, and improving detection efficiency and applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 湖南工商大学
- Filing Date
- 2025-11-06
- Publication Date
- 2026-07-14
Smart Images
Figure CN121479446B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal information processing and industrial anomaly detection technology, and particularly relates to an industrial anomaly detection method based on multimodal data compression. Background Technology
[0002] In the field of multimodal information processing and industrial anomaly detection, industrial production scenarios exhibit significant unique characteristics due to their strong temporal sequence, high complexity, and stringent safety constraints. Production processes operate continuously, data is dynamically generated over time with close temporal correlations, and anomalies often lurk within subtle fluctuations at the millisecond timescale. The complex operating environment involves multiple sources of interference, such as equipment vibration and environmental noise, and missed anomaly detection can directly trigger serious consequences such as production stoppages and safety accidents, placing stringent demands on the real-time performance and accuracy of detection technologies. Industrial anomaly detection technology based on multimodal data compression can, theoretically, overcome the limitations of single-modal feature capture by integrating multimodal data such as video and audio, comprehensively uncovering anomalies in equipment operating status and environmental parameter changes.
[0003] In practical applications, the inherent limitations of existing methods have become a key obstacle to the implementation of industrial anomaly detection technology. In the feature compression stage, the high dimensionality and strong noise of industrial data conflict with the single-factor screening mechanism of most methods, easily leading to the loss of core information such as key equipment process parameters and abnormal vibration characteristics. When relying on attention mechanisms, the high computational complexity caused by massive industrial-grade time-series data far exceeds the computing power of terminal equipment, reducing the feasibility of algorithm deployment. In the cross-modal alignment stage, the deep semantic correlation between equipment vibration audio and fault images is significantly affected by operating conditions, making rigid alignment difficult to adapt. Unidirectional attention mechanisms are prone to causing the loss of modal information such as temperature and acoustics, and static weights cannot respond to production line switching and operating condition fluctuations, resulting in mismatched anomaly features. In the feature fusion stage, the causes of industrial anomalies are diverse, and static strategies such as direct splicing and fixed weights cannot dynamically adjust the modal contributions of vibration, temperature, etc., based on scenarios such as equipment startup and steady-state operation, weakening effective features. In the anomaly detection stage, subtle differences in industrial anomalies (such as vibration spectrum changes in early bearing wear) exceed the perception range of existing classification mechanisms, leading to accumulated risks of missed detections and misjudgments, which may cause production line shutdowns and safety accidents. These problems not only limit detection accuracy but also increase terminal deployment costs due to high computing power requirements, seriously hindering the intelligent upgrading of industry. Summary of the Invention
[0004] To address the aforementioned technical issues, this invention proposes an industrial anomaly detection method based on multimodal data compression. Through multi-stage optimization, it achieves accurate detection of industrial anomalies while balancing feature processing efficiency and semantic integrity. This reduces the computational resource requirements of terminal devices and enhances applicability in complex scenarios.
[0005] To achieve the above objectives, this invention provides an industrial anomaly detection method based on multimodal data compression, comprising:
[0006] Feature extraction is performed on video and audio data to obtain feature representations for each modality;
[0007] The extracted features are tokenized and compressed based on multi-criteria contribution scores.
[0008] Multimodal semantic alignment is achieved by fusing bidirectional cross-attention with dynamic weights;
[0009] Adaptive fusion is performed using a dynamic deep neural network to obtain the final fused features;
[0010] Anomaly detection based on confidence-driven classification mechanism.
[0011] Optionally, the process of feature extraction from video and audio data includes:
[0012] After processing, the video data is input into the ResNet encoder, and after processing, the audio data is input into the VGT-Net encoder.
[0013] Temporal modeling is performed on the video and audio data processed by the encoder using a temporal convolutional network.
[0014] Obtain feature representations of the video modality and the audio modality.
[0015] Optionally, the feature compression process based on multi-criteria contribution scoring includes:
[0016] Calculate the feature activation degree of the token, which consists of activation strength and activation entropy;
[0017] Calculate the similarity between tokens;
[0018] Based on the feature activation degree and the similarity degree, a score is calculated using a multi-criteria feature contribution function;
[0019] A dual-threshold collaborative screening mechanism is adopted to filter tokens based on the score and the similarity.
[0020] The filtered tokens are merged using a binary soft matching method.
[0021] Optionally, the process of achieving multimodal semantic alignment through bidirectional cross-attention and dynamic weight fusion includes:
[0022] Perform dimensional unification on the tokenized multimodal features;
[0023] A bidirectional cross-attention mechanism is introduced to calculate the attention weights for video-to-audio alignment and audio-to-video alignment.
[0024] By using dynamic weight fusion, the original features and the cross-attention weighted features are summed to generate semantically aligned features.
[0025] The training process is constrained by a loss function that includes reconstruction loss and alignment loss.
[0026] Optionally, the adaptive fusion process using dynamic deep neural networks includes:
[0027] Based on the aligned features, a dynamic attention mechanism is used to filter core tokens;
[0028] Multiply the attention weight matrix element-wise with the transformed feature tensor;
[0029] Calculate the cosine similarity between the completed modes;
[0030] By combining the temporal dynamics and cross-modal correlation of the completion features, weights reflecting the contribution of each modality are generated;
[0031] The completed features are then fused according to dynamic weights to generate the final fused features.
[0032] Optionally, the process of generating weights that reflect the contribution of each modality includes:
[0033] Temporal features are obtained by capturing long-distance temporal correlations of completed features through a bidirectional long short-term memory network;
[0034] The temporal features are concatenated with cross-modal similarity to form a comprehensive input;
[0035] The overall modal attention score for each modality is calculated based on the comprehensive input and then normalized to obtain the dynamic weight.
[0036] Optionally, the process of detecting abnormal behavior based on a confidence-driven classification mechanism includes:
[0037] The fused features are input into a neural network composed of fully connected layers, and an anomaly confidence score is generated by an activation function.
[0038] Abnormal events are determined by setting thresholds;
[0039] Abnormal and normal sequences are divided into feature groups to form high-confidence abnormal feature groups, low-confidence abnormal feature groups, high-confidence normal feature groups, and low-confidence normal feature groups;
[0040] Optimization was performed using the InfoNCE loss function.
[0041] Optionally, the optimization process using the InfoNCE loss function aims to make the similarity between high-confidence anomalous features and low-confidence anomalous features higher than the similarity between high-confidence anomalous features and low-confidence normal features.
[0042] Technical effects of the invention: The present invention discloses an industrial anomaly detection method based on multimodal data compression. By performing feature extraction, tokenization and compression of video and audio data, semantic alignment through bidirectional cross-attention and dynamic weight fusion, adaptive fusion of dynamic deep neural networks, and confidence-driven classification detection, multimodal redundant data compression is achieved. While ensuring feature processing efficiency and semantic integrity, the accuracy of industrial anomaly detection is improved, and the requirements for terminal equipment computing resources are reduced. Attached Figure Description
[0043] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0044] Fig. 1 This is a flowchart illustrating an industrial anomaly detection method based on multimodal data compression according to an embodiment of the present invention.
[0045] Fig. 2 This is a schematic diagram of the model structure of an industrial anomaly detection method based on multimodal data compression according to an embodiment of the present invention. Detailed Implementation
[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0047] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0048] like Figs. 1-2 As shown, this embodiment provides an industrial anomaly detection method based on multimodal data compression, including:
[0049] Feature extraction is performed on video and audio data to obtain feature representations for each modality;
[0050] The extracted features are tokenized and compressed based on multi-criteria contribution scores.
[0051] Multimodal semantic alignment is achieved by fusing bidirectional cross-attention with dynamic weights;
[0052] Adaptive fusion is performed using a dynamic deep neural network to obtain the final fused features;
[0053] Anomaly detection based on confidence-driven classification mechanism.
[0054] Furthermore, the process of feature extraction from video and audio data includes:
[0055] After processing, the video data is input into the ResNet encoder, and after processing, the audio data is input into the VGT-Net encoder.
[0056] Temporal modeling is performed on the video and audio data processed by the encoder using a temporal convolutional network.
[0057] Obtain feature representations of the video modality and the audio modality.
[0058] Furthermore, the process of feature compression based on multi-criteria contribution scoring includes:
[0059] Calculate the feature activation degree of the token, which consists of activation strength and activation entropy;
[0060] Calculate the similarity between tokens;
[0061] Based on the feature activation degree and the similarity degree, a score is calculated using a multi-criteria feature contribution function;
[0062] A dual-threshold collaborative screening mechanism is adopted to filter tokens based on the score and the similarity.
[0063] The filtered tokens are merged using a binary soft matching method.
[0064] Furthermore, the process of achieving multimodal semantic alignment through bidirectional cross-attention and dynamic weight fusion includes:
[0065] Perform dimensional unification on the tokenized multimodal features;
[0066] A bidirectional cross-attention mechanism is introduced to calculate the attention weights for video-to-audio alignment and audio-to-video alignment.
[0067] By using dynamic weight fusion, the original features and the cross-attention weighted features are summed to generate semantically aligned features.
[0068] The training process is constrained by a loss function that includes reconstruction loss and alignment loss.
[0069] Furthermore, the process of adaptive fusion using dynamic deep neural networks includes:
[0070] Based on the aligned features, a dynamic attention mechanism is used to filter core tokens;
[0071] Multiply the attention weight matrix element-wise with the transformed feature tensor;
[0072] Calculate the cosine similarity between the completed modes;
[0073] By combining the temporal dynamics and cross-modal correlation of the completion features, weights reflecting the contribution of each modality are generated;
[0074] The completed features are then fused according to dynamic weights to generate the final fused features.
[0075] Furthermore, the process of generating weights that reflect the contribution of each modality includes:
[0076] Temporal features are obtained by capturing long-distance temporal correlations of completed features through a bidirectional long short-term memory network;
[0077] The temporal features are concatenated with cross-modal similarity to form a comprehensive input;
[0078] The overall modal attention score for each modality is calculated based on the comprehensive input and then normalized to obtain the dynamic weight.
[0079] Furthermore, the process of detecting abnormal behavior based on a confidence-driven classification mechanism includes:
[0080] The fused features are input into a neural network composed of fully connected layers, and an anomaly confidence score is generated by an activation function.
[0081] Abnormal events are determined by setting thresholds;
[0082] Abnormal and normal sequences are divided into feature groups to form high-confidence abnormal feature groups, low-confidence abnormal feature groups, high-confidence normal feature groups, and low-confidence normal feature groups;
[0083] Optimization was performed using the InfoNCE loss function.
[0084] Furthermore, the optimization process using the InfoNCE loss function aims to make the similarity between high-confidence anomalous features and low-confidence anomalous features higher than the similarity between high-confidence anomalous features and low-confidence normal features.
[0085] Specifically, the implementation process of this embodiment includes:
[0086] Step 1: Extract features from video and audio data to obtain feature representations for each modality;
[0087] Step 2: Tokenize the extracted features and compress them based on multi-criteria contribution scores;
[0088] Step 3: Achieve multimodal semantic alignment through bidirectional cross-attention and dynamic weight fusion;
[0089] Bidirectional cross-attention is an attention mechanism in multimodal semantic alignment that allows two modalities (such as video and audio) to pay attention to each other's key information: the video modality calculates the association weights with the audio features and focuses on the parts of the audio that are relevant to it; the audio modality also pays attention to the corresponding key features in the video. Through bidirectional interaction, accurate cross-modal semantic alignment is achieved, avoiding information skew or loss.
[0090] Step 4: Adaptive fusion is performed using a dynamic deep neural network to obtain the final fused features;
[0091] Dynamic deep neural networks are deep learning models that can adaptively adjust the depth of their network structure. Their core principle is to change the number of network layers in real time based on the complexity of the input data or the requirements of the task. These networks do not rely on a fixed hierarchical structure but dynamically decide whether to activate or skip certain layers through gating mechanisms and other methods. This reduces redundant computation while maintaining feature extraction capabilities, making them particularly suitable for scenarios with large differences in input data or limited computing resources, thus improving the model's efficiency and adaptability.
[0092] Step 5: Detect abnormal behavior based on a confidence-driven classification mechanism.
[0093] Step one specifically includes:
[0094] Step 1.1: Input video subsequences, process the video data and input it into the ResNet encoder, and process the audio data and input it into the VGT-Net encoder;
[0095] Step 1.2: Perform time-series modeling on the encoded video and audio data using TCN;
[0096] Step 1.3: Obtain the feature representations of the video and audio modalities. The video modal feature representation is as follows: ,in For video feature dimensions, The temporal length of video features; audio modal features are represented as... ,in For audio feature dimensions, The audio feature time sequence length.
[0097] In this embodiment, the process of creating the dataset is as follows:
[0098] First, the publicly available video dataset MVTec-3D AD and its corresponding audio dataset were obtained. The video data was extracted into frame sequences at a fixed frame rate, and the frame image size was standardized to 128×128 pixels. The audio data was converted into waveform signals at a sampling rate of 16kHz, then divided into frames (20ms frame length, 10ms overlap rate), and the Mel-spectral features (80 dimensions) were calculated. The training and test sets of video and audio were divided in a 7:3 ratio, while maintaining a consistent distribution of industrial scene categories during the division.
[0099] Step two specifically includes:
[0100] Step 2.1: Tokenize the feature vectors of the video and audio modalities, converting them into token form, as shown in the following formula:
[0101] ;
[0102] ;
[0103] Step 2.2: The formula for Feature Activation (FAD) is as follows:
[0104] ;
[0105] AI represents activation strength, reflecting the high intensity of the token's feature response. Tokens with high AI values correspond to regions with strong model responses, calculated using the L2 norm. ;
[0106] AE stands for Activation Entropy, which measures the concentration of feature distribution. It penalizes scattered tokens (such as background noise) by using information entropy to retain semantic key tokens that are concentrated in the distribution. The formula is as follows:
[0107] ;
[0108] Step 2.3: Similarity calculation, used to measure the degree of similarity between tokens and reduce redundant information, is determined by cosine similarity, as shown in the following formula:
[0109] ;
[0110] Step 2.4: The Multi-Criterion Feature Contribution Function (MCDF) is as follows:
[0111] ;
[0112] in It is the feature activation degree, which represents the importance of the token; It is the inhibition coefficient, used to adjust the weight of similarity on the final score. The smaller the value, the more tolerant the model is of redundancy, and the more important the tokens are retained. This represents the degree of similarity between tokens.
[0113] Step 2.5: Employ a dual-threshold collaborative screening mechanism, with the following judgment formula:
[0114] and ;
[0115] The higher the score calculated by the Multi-Criterion Feature Contribution (MCDF), the higher the overall retention value of the corresponding token. This embodiment adopts a dual-threshold collaborative screening mechanism, which ensures that the retained tokens have both high importance and low redundancy through dual-condition filtering. It not only filters out important features through the MCDF threshold, but also removes tokens that are highly duplicated with the selected set through the similarity threshold, so as to achieve accurate retention of key information.
[0116] In this embodiment, the Adam optimizer is used for training. The initial value is set to 1.0. Set to 0.5. Setting it to 0.4 terminates training when there is no significant performance improvement in multiple rounds of testing on the test set, yielding the optimized coefficients and threshold.
[0117] Step 2.6: The filtered tokens are merged using a binary soft matching method.
[0118] Step three specifically includes:
[0119] Step 3.1: Perform dimensional unification on the tokenized multimodal features;
[0120] Step 3.2: Introduce a bidirectional cross-attention mechanism. The formula for aligning video with audio is as follows:
[0121] ;
[0122] ;
[0123] ;
[0124] ;
[0125] Step 3.3: Generate semantic alignment features through dynamic weight fusion. The formula for video fusion features is as follows:
[0126] ;
[0127] ;
[0128] Step 3.4: To ensure that the features learned by the model satisfy the dual objectives of preserving the original semantics and maintaining cross-modal semantic consistency, the training process is constrained by a loss function, as follows:
[0129] ;
[0130] in To reconstruct the loss, we focus on feature restoration at the non-missing locations to avoid the fusion process destroying the original effective semantics. The formula is as follows:
[0131] ;
[0132] To align the loss, contrastive learning is used to force semantically relevant cross-modal features to aggregate in the embedding space and irrelevant features to disperse, achieving deep alignment of multimodal semantics, as shown in the following formula:
[0133] ;
[0134] A gradient feedback mechanism for parameter optimization is constructed by fusing reconstruction loss and alignment loss. For learnable fusion weights such as α1 and β1, the cross-modal semantic bias and original feature reconstruction error calculated by the loss function are transformed into gradient update signals for the weight parameters through backpropagation, thereby dynamically adjusting the weights α1 and β1. In this embodiment, the initial value of the weight coefficient λ is set to 0.7. During training, λ is fine-tuned based on model convergence and validation set performance to determine the optimal value for fusion.
[0135] Step four specifically includes:
[0136] Step 4.1: For the aligned features, core tokens are selected using a dynamic deep neural network, and an attention matrix is generated based on the aligned features, as shown in the following formula:
[0137] ;
[0138] ;
[0139] Dynamic deep neural networks ( The depth of the tensor is determined by the characteristics of the input data and is not a fixed value. It consists of multiple fully connected layers and a gating mechanism. During initialization, it receives the input dimension, output dimension, and maximum depth parameter. Based on the maximum depth, it generates linear layers and corresponding gating networks to determine whether to proceed to the next layer. During forward propagation, the tensor undergoes a linear transformation and ReLU activation in the current layer. The gating value is calculated using sigmoid. If the threshold is exceeded, it recursively proceeds to the next layer; otherwise, it returns the output. The last layer uses PReLU activation to improve the flexibility of expression.
[0140] Step 4.2: Multiply the attention weight matrix element-wise with the transformed video and audio tensors to assign greater attention weights to important information, as shown in the following formula:
[0141] ;
[0142] ;
[0143] Step 4.3: Calculate the cosine similarity between the completed modalities to quantify semantic synergy. The formula is as follows:
[0144] ;
[0145] Step 4.4: Combining the temporal dynamics and cross-modal correlations of the completed features, generate weights reflecting the contribution of each mode. Use BiLSTM to capture long-distance temporal correlations, as shown in the following formula:
[0146] ;
[0147] ;
[0148] Step 4.5: Concatenate the temporal features with the cross-modal similarity to form a comprehensive input, as shown in the following formula:
[0149] ;
[0150] ;
[0151] Step 4.6: Calculate the overall modal attention score for video and audio modalities, and normalize it using the following formula:
[0152] ;
[0153] ;
[0154] ;
[0155] ;
[0156] Step 4.7: Fuse the completed features according to dynamic weights to generate the final fused features, as shown in the following formula:
[0157] ;
[0158] Step five specifically includes:
[0159] Step 5.1: Input the fused features into the neural network consisting of L fully connected layers. The formula for calculating the output of the Lth layer is as follows:
[0160] ;
[0161] Step 5.2: [The sentence is incomplete and requires more context to be translated accurately.] The coarse-grained anomaly confidence score p is generated by inputting the sigmoid function, as shown in the following formula:
[0162] ;
[0163] Step 5.3: Set a threshold Perform an exception event determination when p> When it is determined to be an abnormal event, it is recorded as: Conversely, a normal event is denoted as... ;
[0164] Step 5.4: Divide the abnormal and normal sequences into feature groups to form high-confidence abnormal feature groups. Low-confidence anomaly feature group High-confidence normal characteristic group and low-confidence normal feature group ;
[0165] Step 5.5: Optimize using the InfoNCE loss function, as shown in the following formula:
[0166] ;
[0167] In this embodiment, the temperature coefficient The initial value is set to 0.07, and adjusted every 10 training epochs based on the convergence of the loss function: if the improvement in fine-grained anomaly detection accuracy is not significant, [the value will be adjusted]. Reduce by a factor of 1.2; if the loss fluctuates significantly during training, Increase by 1.2 times. Anomaly detection threshold. Initially set to 0.5, updated every 5 training epochs based on the ROC curve of the test set: if the false negative rate is higher than 5%, Reduce by 0.05; if the false positive rate is higher than 10%, Improve by 0.05. If there is no significant performance improvement after three consecutive rounds on the test set, fix the final value. and Value, complete parameter optimization.
[0168] This invention discloses an industrial anomaly detection method based on multimodal data compression. By performing feature extraction, tokenization and compression of video and audio data, semantic alignment through bidirectional cross-attention and dynamic weight fusion, adaptive fusion of dynamic deep neural networks, and confidence-driven classification detection, multimodal redundant data compression is achieved. This method improves the accuracy of industrial anomaly detection while ensuring feature processing efficiency and semantic integrity, and reduces the requirements for computing resources of terminal equipment.
[0169] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. An industrial anomaly detection method based on multimodal data compression, characterized in that, include: Feature extraction is performed on video and audio data to obtain feature representations for each modality; The extracted features are tokenized and compressed based on multi-criteria contribution scores. The process of feature compression based on multi-criteria contribution scoring includes: Calculate the feature activation degree of the token, which consists of activation intensity and activation entropy. The activation intensity reflects the high feature response intensity of the token, and the activation entropy is used to measure the concentration of the feature distribution. Calculate the similarity between tokens; Based on the feature activation degree and the similarity degree, a score is calculated using a multi-criteria feature contribution function; A dual-threshold collaborative screening mechanism is adopted to filter tokens based on the score and the similarity. The filtered tokens are then merged using a binary soft matching method. Multimodal semantic alignment is achieved by fusing bidirectional cross-attention with dynamic weights; The process of achieving multimodal semantic alignment through bidirectional cross-attention and dynamic weight fusion includes: Perform dimensional unification on the tokenized multimodal features; A bidirectional cross-attention mechanism is introduced to calculate the attention weights for video-to-audio alignment and audio-to-video alignment. By using dynamic weight fusion, the original features and the cross-attention weighted features are summed to generate semantically aligned features. The training process is constrained by a loss function that includes reconstruction loss and alignment loss; Adaptive fusion is performed using a dynamic deep neural network to obtain the final fused features; The process of adaptive fusion using dynamic deep neural networks includes: Based on the aligned features, a dynamic attention mechanism is used to filter core tokens; Multiply the attention weight matrix element-wise with the transformed feature tensor; Calculate the cosine similarity between the completed modes; By combining the temporal dynamics and cross-modal correlation of the completion features, weights reflecting the contribution of each modality are generated; The completed features are then fused according to dynamic weights to generate the final fused features. Abnormal behavior detection based on confidence-driven classification mechanism; The process of detecting abnormal behavior based on a confidence-driven classification mechanism includes: The fused features are input into a neural network composed of fully connected layers, and an anomaly confidence score is generated by an activation function. Abnormal events are determined by setting thresholds; Abnormal and normal sequences are divided into feature groups to form high-confidence abnormal feature groups, low-confidence abnormal feature groups, high-confidence normal feature groups, and low-confidence normal feature groups; Optimization is performed using the InfoNCE loss function; The optimization process using the InfoNCE loss function aims to make the similarity between high-confidence anomalous features and low-confidence anomalous features higher than the similarity between high-confidence anomalous features and low-confidence normal features.
2. The industrial anomaly detection method based on multimodal data compression as described in claim 1, characterized in that, The process of feature extraction from video and audio data includes: After processing, the video data is input into the ResNet encoder, and after processing, the audio data is input into the VGT-Net encoder. Temporal modeling is performed on the video and audio data processed by the encoder using a temporal convolutional network. Obtain feature representations of the video modality and the audio modality.
3. The industrial anomaly detection method based on multimodal data compression as described in claim 1, characterized in that, The process of generating weights that reflect the contribution of each mode includes: Temporal features are obtained by capturing long-distance temporal correlations of completed features through a bidirectional long short-term memory network; The temporal features are concatenated with cross-modal similarity to form a comprehensive input; The overall modal attention score for each modality is calculated based on the comprehensive input and then normalized to obtain the dynamic weight.