Multimodal Data Fusion Method and System for an Intelligent Weak Current Monitoring System

Abstract

The present invention relates to the technical field of intelligent weak current data processing, and specifically to a multi-modal data fusion method and system for an intelligent weak current monitoring system. Specifically, it includes: collecting multi-modal weak current data; performing two-stage denoising to eliminate noise interference and retain edge details, and then generating discriminant items through standardization and hashing to ensure data validity; adopting a cross-modal adaptive attention mechanism to dynamically allocate weights for image and temporal features, and combining a residual fusion strategy to capture interactive details between modalities to achieve efficient complementary fusion of images and sensor temporal data; generating a comprehensive anomaly score through a joint decision-making mechanism of the reconstruction error of a deep autoencoder and the anomaly probability of a classification network, and combining threshold determination to achieve highly robust anomaly detection. The present invention effectively improves the anomaly detection accuracy and system reliability in weak current monitoring scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent weak current data processing, and in particular to a multi-modal data fusion method and system for an intelligent weak current monitoring system. Background Art

[0002] With the rapid development of intelligence and information technology, weak-current monitoring systems have been widely used in security, industrial monitoring, smart cities and other fields. However, traditional weak-current monitoring systems mainly rely on single-modal data, such as video surveillance or sensor data, which limits the system's perception capabilities in complex environments and is easily affected by environmental noise, lighting changes, equipment failures and other factors, thereby reducing the reliability and accuracy of monitoring.

[0003] A Chinese invention patent with announcement number CN117477767B discloses a method and system for monitoring the operation of a weak-current intelligent system. The method includes: collecting multiple weak-current intelligent units and corresponding multiple intelligent tasks within the weak-current intelligent system; collecting task feature information, and obtaining multiple privacy parameters and multiple computing power parameters through a task analysis and design module; based on cloud computing and edge computing, allocating the design processing positions of multiple intelligent tasks to obtain a weak-current business architecture space, optimizing the weak-current business architecture in combination with multiple privacy parameters and multiple processing efficiency requirement information, obtaining an optimal weak-current business architecture, and designing and operating the weak-current intelligent system; the technical problem in the prior art of low data processing efficiency and poor operation monitoring effect of the weak-current intelligent system due to excessive data volume is solved, and the technical effect of improving the data processing efficiency and operation safety of the weak-current intelligent system is achieved.

[0004] There is great heterogeneity between different modal data. How to efficiently integrate multiple data sources and perform intelligent analysis is an important challenge facing the current weak current monitoring system. The fusion effect of existing technologies in multimodal data fusion is difficult to meet the efficiency, accuracy and real-time requirements of intelligent monitoring. Therefore, there is an urgent need for an intelligent analysis method that can deeply integrate multiple types of weak current monitoring data to improve the stability, accuracy and intelligence level of the monitoring system, thereby achieving more efficient safety warning and decision support. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the background technology and to propose a multi-modal data fusion method and system for an intelligent weak current monitoring system.

[0006] The technical solution of the present invention is a multi-modal data fusion method for an intelligent weak current monitoring system, comprising the following specific implementation steps:

[0007] S1. Collect and summarize weak current data;

[0008] S2, adopt a two-stage denoising method, use adaptive wavelet threshold denoising, and then enhance the edge preservation effect through anisotropic diffusion filtering, standardize and normalize different sensor data, and generate identification items based on hash coding;

[0009] S3. The standard completeness of the identification standard data is determined by using convolutional neural networks to extract image features and long short-term memory networks to capture temporal features. Data enhancement is performed through affine transformation and random perturbations. The cross-modal adaptive attention mechanism is used to dynamically allocate weights to achieve preliminary feature weighted fusion. The residual strategy is introduced to strengthen nonlinear complementarity and generate multimodal fusion features.

[0010] S4. An anomaly detection method combining unsupervised reconstruction error and supervised classification is used to input the multimodal fusion features into a deep autoencoder, map them to a low-dimensional latent space through the encoder, and reconstruct the original features by the decoder to calculate the reconstruction error; the multimodal fusion features are classified using linear transformation and softmax function, the anomaly probability is calculated, a comprehensive anomaly score is generated, and an anomaly judgment is made based on the set threshold, and the judgment result is output;

[0011] S5. Display the processed weak current data and abnormality detection results through the user interface.

[0012] Preferably, the implementation process of the two-stage denoising method is as follows:

[0013] S21, using the adaptive wavelet threshold denoising method, the weak current data is decomposed into a set of wavelet coefficients W through the multi-scale wavelet transform operator Tr, and the signal is decomposed into different scales and directions;

[0014] S22. When denoising each wavelet coefficient w∈W, define an adaptive soft threshold T:

[0015] ;

[0016] ;

[0017] Among them, σ local represents the local noise standard deviation; N represents the total number of wavelet coefficients at the current scale; α represents the adaptive factor; median() represents the median function;

[0018] S23, process each wavelet coefficient:

[0019] ;

[0020] Where w' represents the processed wavelet coefficient; sign() represents the sign function;

[0021] S24, performing inverse wavelet transform Tr on the coefficient set W' after adaptive threshold processing −1 , and get the signal I after preliminary denoising w :I w =Tr −1 (W');

[0022] Among them, Tr −1 represents the inverse wavelet transform operator;

[0023] S25. In the preliminary denoising result I w An improved anisotropic diffusion filter is introduced to dynamically control the diffusion intensity based on local gradient information, an adaptive diffusion coefficient is used to smooth the noise and retain the edge information, and the explicit iterative method is used for numerical discretization.

[0024] Preferably, the implementation process of the improved anisotropic diffusion filtering is as follows:

[0025] S31. Construct a basic model of anisotropic diffusion:

[0026] ;

[0027] Where I(x,y,t) represents the intensity at position (x,y) and iteration time t; represents the gradient operator; represents the divergence operator; c(x,y,t) represents the diffusion coefficient;

[0028] S32. Calculate the adaptive diffusion coefficient:

[0029] ;

[0030] In the formula, Represents the initial denoising result I at position (x, y) and iteration time t w The gradient amplitude of ; k represents the contrast parameter;

[0031] S33. Use explicit iterative method to discretize the continuous diffusion process:

[0032] ;

[0033] In the formula, I n (x,y) represents the signal at the nth iteration; c n (x,y) represents the diffusion coefficient at position (x,y) of the nth iteration; Represents the time step parameter.

[0034] Preferably, the implementation process of identifying the sub-items is as follows:

[0035] S41, converting the standard data data into a binary sequence bdata, and calculating the code C=H(bdata) of the standard data data;

[0036] Wherein, H represents a predefined hash function;

[0037] S42, calculate the first element item Ief=Ea×F(C) mod η;

[0038] ;

[0039] Where Ea represents the predefined auxiliary generation element, Ea=P -1 mod η; F() represents the term generating function; η is the predefined first-level coding factor, η=p×q; P is the predefined first-level coding factor, P=lcm(p-1,q-1); p and q are predefined 160-bit large prime numbers; lcm() is the predefined least common multiple function;

[0040] S43, calculate the second element item Ies: ;

[0041] S44. Generate identification sub-item IS={Ief,Ies}.

[0042] Preferably, the process of identifying the completeness of the specification of the identification standard data is as follows:

[0043] S51, converting the received standard data data into a binary sequence bdata', and calculating the authentication code Ca=H(bdata') of the standard data data;

[0044] Wherein, H represents a predefined hash function;

[0045] S52, calculate the matching code Cm=(1+Ief×η)×(Ies) η ×Ea mod η 2 ;

[0046] S53. If Ca=Cm, it indicates that the standard data data is complete in specifications; otherwise, an alarm is immediately issued.

[0047] Preferably, the implementation process of the cross-modal adaptive attention mechanism is as follows:

[0048] S61, modal feature mapping and score calculation, linear mapping is performed on the features of the two modalities respectively, and the high-dimensional features are compressed to the same dimension:

[0049] s img =W img F img +b img ;

[0050] s sen =W sen F sen +b sen ;

[0051] Among them, W img With W sen Respectively represent the weight matrices used to map static and temporal features; b img and b sen represents the bias term; s img and sen They represent the initial estimation of the importance of each modal feature in the current scenario;

[0052] S62. Calculate the adaptive attention weight: normalize each modality score through the softmax function to obtain the attention weight:

[0053] ;

[0054] β=1-α;

[0055] In the formula, α represents the importance weight of image features; β represents the importance of temporal features;

[0056] S63, preliminary feature weighted fusion: Use the above weights to perform weighted averaging on the two modal features to obtain preliminary fusion features:

[0057] ;

[0058] In the formula, F fusel Represents the features after preliminary fusion.

[0059] Preferably, the implementation process of the residual strategy is:

[0060] S71. Interaction Information Extraction: Using element-wise product to capture the interaction information between two modal features:

[0061] ;

[0062] in, represents element-wise product; F inter Indicates the similarity and complementarity of two features in each dimension;

[0063] S72, residual fusion: weighted addition of preliminary fusion features and mutual information:

[0064] ;

[0065] In the formula, γ represents a trainable residual fusion coefficient; F fused Represents the multimodal fusion features of the final output.

[0066] Preferably, the implementation process of the anomaly detection method combining unsupervised reconstruction error and supervised classification is as follows:

[0067] S81. Branch 1, reconstruction module, i.e. unsupervised anomaly detection:

[0068] S8101, fusion feature F fused The input is sent to the deep autoencoder, which maps it to a low-dimensional latent space by the encoder E(), and then reconstructs the original features by the decoder D():

[0069] Z=E(F fused );

[0070] ;

[0071] Where E() represents the encoder network; D() represents the corresponding decoder; Z represents the potential feature representation;

[0072] S8102. Define the reconstruction error R as the scoring indicator for unsupervised anomaly detection:

[0073] ;

[0074] Where R represents the reconstruction error; represents the Euclidean distance;

[0075] S82. Branch 2: Anomaly classification detection, i.e., supervision strategy:

[0076] S8201, Feature Mapping and Linear Transformation: The fusion feature F fused Through a layer of linear transformation, it is mapped to the classification space:

[0077] ;

[0078] Where W c represents the classification network weight matrix; b c represents the bias term; z c represents the classification score vector;

[0079] S8202, probability output: use softmax activation function: p=softmax(z c );

[0080] Among them, p represents the predicted probability of each category, the categories are: normal p normal and abnormal p abnormal ;

[0081] S83. Fusion of the reconstruction error R and the classification anomaly probability generates a comprehensive anomaly score: S = λR + (1-λ)(1-p normal);

[0082] Among them, S represents the comprehensive anomaly score; λ represents the fusion weight, λ∈[0,1]; (1-p normal ) represents the abnormal probability.

[0083] Preferably, the joint loss function L of the anomaly detection method combining unsupervised reconstruction error and supervised classification is total Simultaneously guide the training of branch one and branch two:

[0084] L total =L rec +L cls ;

[0085] ;

[0086] ;

[0087] Where, L rec represents the reconstruction loss; L cls represents the cross entropy loss; y i is the true category label; p i is the predicted probability.

[0088] The technical solution of the present invention is a multimodal data fusion system of an intelligent weak current monitoring system, which is used to execute the multimodal data fusion method of the above-mentioned intelligent weak current monitoring system, including:

[0089] Weak current data acquisition module, used for multi-modal weak current data acquisition;

[0090] Weak current data processing module, used for denoising, standardization and normalization preprocessing of collected weak current data;

[0091] The weak current data fusion module is used to use multi-modal data fusion technology to comprehensively process and analyze multi-modal weak current data;

[0092] Intelligent analysis module, which is used to combine artificial intelligence algorithms to perform intelligent analysis on the fused data to determine whether there are any abnormalities;

[0093] The user interaction module is used to display the fusion results through the interface and provide user interaction functions.

[0094] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0095] This paper designs a multimodal data fusion method and system for an intelligent weak current monitoring system. Through multimodal data fusion, adaptive denoising, and joint anomaly detection technology, the accuracy, robustness, and safety of the weak current monitoring system are improved while taking into account computational efficiency and scalability:

[0096] (1) Highly robust noise suppression and edge preservation: Through the two-stage denoising technology of adaptive wavelet threshold denoising and anisotropic diffusion filtering, the threshold is estimated based on the local noise standard deviation and median absolute deviation, and the adaptive factor is dynamically adjusted in combination with a lightweight deep neural network. It effectively adapts to the noise characteristics of different sensors and avoids the over-smoothing or under-denoising problems caused by traditional fixed thresholds. The gradient amplitude is used to control the diffusion intensity. While suppressing noise during the iteration process, the edge and detail features in the weak current data (such as equipment contours and heat source change trajectories) are retained, solving the edge blurring problem caused by traditional filtering methods and providing high-fidelity input data for subsequent multimodal fusion.

[0097] (2) Efficient complementary fusion of multimodal data: Through the cross-modal adaptive attention mechanism and residual fusion strategy, the deep fusion of static image and time series sensor data is realized. The modal importance weights are generated based on linear mapping and softmax normalization, and the contribution of image and sensor features is automatically adjusted to improve the adaptability of fusion features to complex environments. By capturing the local interaction information of static and time series features and supplementing it to the preliminary fusion features through trainable residual coefficients, the fine-grained complementarity of multimodal data is strengthened, significantly improving the sensitivity and accuracy of anomaly detection;

[0098] (3) Improved robustness of high-precision anomaly detection: The anomaly detection mechanism that combines unsupervised reconstruction error and supervised classification is adopted to break through the limitations of a single detection method. The normal data distribution pattern is learned through a deep autoencoder, and the degree of data deviation is quantified using the reconstruction error to avoid missed detection problems caused by insufficient labeled data. The anomaly probability is output and the classification boundary is optimized. The unsupervised and supervised results are dynamically balanced through the fusion weights to maintain stable detection performance in complex scenarios (such as sudden changes in illumination and sensor drift).

[0099] (4) Data validity and systematic guarantee: A hash function-based identification item generation and verification mechanism is designed. The code is generated by hash function and combined with large prime number operations to generate meta-items to ensure that the data cannot be tampered with during transmission. The completeness of the data specifications is verified at the receiving end to prevent data contamination and improve the security of the weak current monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 This is a system architecture diagram of a multi-modal data fusion system of an intelligent weak current monitoring system proposed by the present invention;

[0101] Figure 2 This is a method flow chart of a multimodal data fusion method for an intelligent weak current monitoring system proposed by the present invention. DETAILED DESCRIPTION

[0102] Embodiment 1, as Figure 1 As shown, a multimodal data fusion system of an intelligent weak current monitoring system proposed in the present invention includes: a weak current data acquisition module, a weak current data processing module, a weak current data fusion module, an intelligent analysis module and a user interaction module.

[0103] The weak current data acquisition module performs multi-modal weak current data acquisition through a variety of sensors (including but not limited to cameras, infrared sensors, and environmental sensors);

[0104] The weak current data processing module performs denoising, standardization and normalization preprocessing on the collected weak current data to ensure the consistency and availability of the data;

[0105] The weak current data fusion module uses multimodal data fusion technology to comprehensively process and analyze data from different sensors;

[0106] The intelligent analysis module combines artificial intelligence algorithms (including but not limited to deep learning, image recognition, and pattern recognition) to perform intelligent analysis on the fused data, determine whether there are any abnormalities, and issue real-time warnings and notify users;

[0107] The user interaction module displays the fusion results through the interface and provides user interaction functions, including but not limited to alarm and history query.

[0108] Embodiment 2, as Figure 2 As shown, a multimodal data fusion method of an intelligent weak current monitoring system proposed in the present invention is applied to a multimodal data fusion system of an intelligent weak current monitoring system proposed in Example 1, and its specific implementation steps are as follows:

[0109] S1. Deploy various types of sensors, including but not limited to: cameras (to obtain image data of the environment), infrared sensors (to detect changes in heat sources), temperature and humidity sensors (to monitor environmental temperature and humidity), and motion sensors (to detect the movement of targets);

[0110] The weak current data acquisition module instructs each sensor to work in parallel and continuously collects real-time weak current data. The weak current data acquisition module summarizes the weak current data and transmits the summarized weak current data to the weak current data processing module.

[0111] S2, the weak current data processing module performs denoising, filtering and standardization operations on the original data, effectively eliminating interference noise and ensuring the consistency and availability of data. The specific implementation steps are as follows:

[0112] S21, introduce two-stage denoising processing, based on adaptive wavelet threshold denoising and spatiotemporal adaptive anisotropic diffusion filtering for denoising, specifically:

[0113] S2101, first stage, adaptive wavelet threshold denoising:

[0114] (1) Wavelet decomposition: The weak current data is decomposed into a set of wavelet coefficients W through the multi-scale wavelet transform operator Tr, and the signal is decomposed into different scales and directions, providing a basis for subsequent local noise analysis;

[0115] (2) Adaptive threshold setting: When denoising each wavelet coefficient w∈W, an adaptive soft threshold method is used, and its threshold T is defined as:

[0116] ;

[0117] ;

[0118] Among them, σ local represents the local noise standard deviation, which is estimated by the median absolute deviation method in a local window of M×M; N represents the total number of wavelet coefficients at the current scale, which is used to reflect the global noise level; α represents an adaptive factor, whose value is dynamically determined by a pre-trained lightweight deep neural network (including but not limited to a convolutional neural network) according to the statistical characteristics of the current local data, to adjust the differences in noise in different sensors or environments; median() represents the median function;

[0119] (3) Soft threshold processing: Use the soft threshold formula to process each wavelet coefficient:

[0120] ;

[0121] Where w' represents the processed wavelet coefficient; sign() represents the sign function;

[0122] (4) Inverse wavelet transform reconstruction: Perform inverse wavelet transform Tr on the coefficient set W' after adaptive threshold processing −1 (i.e., inverse wavelet transform operator), and obtain the signal I after preliminary denoising w :I w =Tr −1 (W');

[0123] S2102, the second stage, in order to further refine the denoising effect, especially in terms of retaining edge information and detail features, the initial denoising result I w An improved anisotropic diffusion filter is introduced to control the diffusion intensity using local gradient information to prevent edge blur:

[0124] (1) Basic model of anisotropic diffusion: The diffusion process satisfies the following partial differential equation:

[0125] ;

[0126] Where I(x,y,t) represents the intensity at position (x,y) and iteration time t; represents the gradient operator; represents the divergence operator; c(x,y,t) represents the diffusion coefficient, which is used to control the diffusion speed and direction;

[0127] (2) Adaptive diffusion coefficient: In order to preserve edge information while smoothing noise, the diffusion coefficient takes the following form:

[0128] ;

[0129] In the formula, Represents the initial denoising result I at position (x, y) and iteration time t w The gradient amplitude reflects the degree of local change; k represents the contrast parameter, which is set according to the local statistical characteristics and controls the sensitivity to the edge. The smaller the value, the stronger the protection for weak edges.

[0130] (3) Numerical iterative discretization: The continuous diffusion process is discretized using an explicit iterative method:

[0131] ;

[0132] In the formula, I n (x,y) represents the signal at the nth iteration; c n (x,y) represents the diffusion coefficient at position (x,y) of the nth iteration; Represents the time step parameter, which is set to <0.25 to ensure numerical stability;

[0133] Based on this: After several iterations, the refined denoised signal I' is finally obtained, which effectively suppresses noise interference while maintaining edge and detail features;

[0134] S22, standardize the data of different sensors, unify the dimensions of the data, make the data of different sensors comparable without bias, and normalize the data so that the data are within the same quantitative range, avoid the deviation of the final result caused by the excessive or too small values ​​of some sensors, and obtain standard data data;

[0135] S23, generating identification items for the standard data data, the generation process of the identification items is as follows:

[0136] S2301, converting the standard data data into a binary sequence bdata;

[0137] S2302, calculate the code C=H(bdata) of the standard data data;

[0138] Wherein, H represents a predefined hash function;

[0139] S2303, calculate the first element item Ief=Ea×F(C) mod η;

[0140] ;

[0141] Where Ea represents the predefined auxiliary generation element, Ea=P -1 mod η; F() represents the term generating function; η is the predefined first-level coding factor, η=p×q; P is the predefined first-level coding factor, P=lcm(p-1,q-1); p and q are predefined 160-bit large prime numbers; lcm() is the predefined least common multiple function;

[0142] S2304, calculate the second element item Ies:

[0143] ;

[0144] S2305, generate identification sub-item IS={Ief,Ies};

[0145] S24, transmitting {identification sub-item IS={Ief,Ies}, standard data data} to the weak current data fusion module.

[0146] S3, weak current data fusion module is based on standard data (image or sensor data), adopts hybrid network structure to realize feature extraction, uses data enhancement to improve data diversity and model generalization ability, and targets different modal data (including but not limited to image feature F img With the sensor timing characteristics F sen ), build a cross-modal adaptive attention fusion mechanism and residual fusion strategy to achieve more accurate feature complementation and decision support, specifically:

[0147] S31, receiving {identification item IS={Ief,Ies}, standard data data}, extracting identification item IS={Ief,Ies} and standard data data, and in order to ensure the completeness of the received standard data data, identifying the received standard data data, the identification process is as follows:

[0148] S3101, converting the received standard data data into a binary sequence bdata';

[0149] S3102, calculate the authentication code Ca=H(bdata') of the received standard data data;

[0150] Wherein, H represents a predefined hash function;

[0151] S3103, calculate the matching code Cm=(1+Ief×η)×(Ies) η ×Ea mod η 2 ;

[0152] S3104, if Ca=Cm, it indicates that the received standard data data is complete in specification, that is, the authentication is passed; otherwise, an alarm is immediately issued;

[0153] S32. Extract features of each modality (including but not limited to static images and time series sensor data) based on standard data data, use a dedicated network for different modal data and use enhancement strategies, so that the subsequent fusion module can obtain more diverse and refined features, and reduce the uncertainty caused by the influence of a single modality. The specific process is as follows:

[0154] S3201, analyzing the modal data in the standard data data, and extracting the image data and sensor data in the standard data data;

[0155] S3202, Extract local structure and edge features from image data using convolutional neural network (CNN) img ;

[0156] S3203, for sensor time series data, use long short-term memory network (LSTM) to capture time evolution characteristics F sen ;

[0157] S3204, in order to make the network better cope with environmental changes, perform affine transformation and random perturbation on the standard data data, and define the data enhancement operator , using rotation, scaling, and translation operations to generate diverse data:

[0158] I ang =A(data,θ);

[0159] Among them, I ang represents the enhanced data; θ represents a set of enhancement parameters (including but not limited to the rotation angle θ t , scaling factor θ s , translation vector θ t ), these parameters can be randomly drawn from a preset distribution (a normal distribution with a mean of 0);

[0160] Based on this, we use convolutional networks and LSTM to extract static and temporal features for different modalities, ensuring that each type of data can capture fine-grained information. We also use the data enhancement module to generate diversified samples. These enhanced samples can share weights in the same network, thereby improving the model's ability to adapt to changes in different environments. Multi-scale features (outputs at different levels) provide rich information for subsequent multi-modal fusion, ensuring that the subsequent fusion module can select the most discriminative features from them.

[0161] S33, based on the obtained static image feature F img With the timing characteristics F sen For efficient fusion, we combine cross-modal adaptive attention fusion and residual fusion strategies. The two complement each other, realizing dynamic weight allocation and supplementing the fine-grained interaction information between the two modalities. Specifically:

[0162] S3301, Cross-modal Adaptive Attention Fusion:

[0163] (1) Modal feature mapping and score calculation: Linearly map the features of the two modalities respectively and compress the high-dimensional features to the same dimension for subsequent calculations:

[0164] s img =W img F img +b img ;

[0165] s sen =W sen F sen +b sen ;

[0166] Among them, W img With W sen They represent the weight matrices used to map static and temporal features, respectively, and their sizes are preset according to the input feature dimension and the target fusion dimension; b img and b sen represents the bias term; s img and sen They represent the initial estimation of the importance of each modal feature in the current scenario;

[0167] (2) Calculate adaptive attention weights: Normalize the scores of each modality through the softmax function to obtain the attention weights so that the contributions of the two modalities are automatically adjusted:

[0168] ;

[0169] β=1-α;

[0170] In the formula, α represents the importance weight of image features; β represents the importance of temporal features;

[0171] (3) Preliminary feature weighted fusion: Use the above weights to perform weighted averaging on the two modal features to obtain preliminary fusion features:

[0172] ;

[0173] In the formula, F fusel It represents the features after preliminary fusion, retaining the main information of each modality and dynamically adjusting their contributions according to the weights;

[0174] S3302. To fully capture the nonlinear complementary relationship between static and temporal features, a residual fusion strategy is designed:

[0175] (1) Mutual information extraction: The element-wise product is used to capture the mutual information between the two modal features. This operation can highlight the parts that change together:

[0176] ;

[0177] in, represents element-wise product; F inter Indicates the similarity and complementarity of two features in each dimension;

[0178] (2) Residual fusion: The preliminary fusion features are weighted and added to the interactive information, and the residual structure is used to ensure that information is not lost while enhancing the expression of nonlinear features:

[0179] ;

[0180] In the formula, γ represents a trainable residual fusion coefficient, which is used to balance the contribution of preliminary fusion features and interactive information; F fused The multimodal fusion features representing the final output not only contain the main information of each modality, but also make full use of the interaction details between the two;

[0181] Based on this: First, the scores of each modal feature are obtained through linear mapping, and then the dynamic weights are obtained through softmax to reflect the reliability of each modal data in the current scene, and the weights are used to weightedly fuse the two modal features to obtain the preliminary feature representation F fusel , then calculate the element-wise product F of the two modal features inter , capturing the complementary information of the two in local details, and weighting F inter Join F fusel In the above example, we obtain the final fusion feature F fused , realize nonlinear supplementation of information and detail enhancement;

[0182] S34, the fusion feature F fused Transmitted to the intelligent analysis module.

[0183] S4, the intelligent analysis module realizes accurate detection of anomalies in multimodal data by combining the decision-making mechanism of unsupervised reconstruction error and supervised classification, that is, anomaly detection is realized through two parallel branches, and the outputs are integrated to generate the final anomaly decision, specifically:

[0184] S41, branch 1, reconstruction module (unsupervised anomaly detection):

[0185] S4101, fusion feature F fused The input is sent to the deep autoencoder, which maps it to a low-dimensional latent space by the encoder E(), and then the decoder D() reconstructs the original features. The process is as follows:

[0186] Z=E(F fused );

[0187] ;

[0188] Where E() represents the encoder network, which uses a multi-layer fully connected or convolutional layer structure, and its parameters are learned from the training data; D() represents the corresponding decoder; Z represents the potential feature representation, which captures F fused The main mode of

[0189] S4102. Define the reconstruction error R as the scoring indicator for unsupervised anomaly detection:

[0190] ;

[0191] In the formula, R represents the reconstruction error, which reflects the degree to which the data deviates from the normal mode; represents the Euclidean distance;

[0192] Based on this: the reconstruction error R provides an unsupervised detection indicator for subsequent anomaly determination. If the reconstruction error of normal data is low, the reconstruction error of abnormal data is high;

[0193] S42, Branch 2, Anomaly Classification Detection (Supervisory Strategy):

[0194] S4201, feature mapping and linear transformation: fusion feature F fused Through a layer of linear transformation, it is mapped to the classification space:

[0195] ;

[0196] Where W c represents the classification network weight matrix, the size of which is set according to the fusion feature dimension and the number of categories; b c represents the bias term; z c represents the classification score vector;

[0197] S4202. Probability Output: Use the softmax activation function to normalize the output of the linear transformation into class probabilities: p = softmax(z c );

[0198] where p represents the predicted probabilities of each class. In this embodiment, the classes are normal p normal and abnormal p abnormal ;

[0199] S43. Combine the reconstruction error R and the classification anomaly probability to generate a comprehensive anomaly score: S = λR + (1 - λ)(1 - p normal );

[0200] where S represents the comprehensive anomaly score; λ represents the fusion weight, which determines the relative importance of the unsupervised reconstruction error and the supervised classification result, and is obtained from cross - validation or online adaptive adjustment, λ ∈ [0, 1]; (1 - p normal ) represents the anomaly probability, and the larger it is, the more likely the data is abnormal;

[0201] S44. Compare the comprehensive anomaly score S with a set threshold to make an anomaly determination:

[0202] Case 1: If S > THSH, then it is determined as a severe anomaly;

[0203] Case 2: If THSL ≤ S ≤ THSH, then it is determined as a minor anomaly;

[0204] Case 3: If S < THSL, then it is determined as normal;

[0205] Accordingly: Converting the comprehensive score into a clear decision is the basis for the system to finally output an anomaly warning;

[0206] S45. Define the joint loss function L total , and at the same time guide the training of Branch 1 and Branch 2 to ensure their coordinated progress, making the anomaly detection system more accurate as a whole:

[0207] L total = L rec + L cls ;

[0208] ;

[0209] ;

[0210] In the formula, L rec represents the reconstruction loss, ensuring that the auto - encoder learns the normal data pattern; L cls represents the cross - entropy loss, which is used for supervised classification; y i is the true class label; p i is the predicted probability;

[0211] Based on this, the joint loss enables the coordinated optimization of branches 1 and 2, which jointly improves the overall performance of anomaly detection. By simultaneously training the autoencoder and the classification network in the overall network, all network parameters are adjusted through joint loss back propagation, ensuring close collaboration between modules, so that the reconstruction error and classification output can promote each other and achieve global optimal anomaly detection performance.

[0212] S46: Transmit the determination result to the user interaction module.

[0213] S5. The user interaction module displays the processed data and anomaly detection results through the user interface for users to view and provide feedback in real time. Specifically:

[0214] (1) Data visualization: Display the monitoring results after multimodal data fusion through a graphical interface, provide alarm information and historical records, and intuitively present the monitoring status through images and charts;

[0215] (2) Alarm and response mechanism: Based on the judgment results of the event, hierarchical alarms are performed, prompt messages are issued for minor anomalies, and emergency response mechanisms are triggered for severe anomalies (including but not limited to telephone notifications and SMS reminders).

[0216] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A multimodal data fusion method for an intelligent weak current monitoring system, characterized in that: The specific implementation steps include the following: S1. Collect and summarize weak current data; S2, adopt a two-stage denoising method, use adaptive wavelet threshold denoising, and then enhance the edge preservation effect through anisotropic diffusion filtering, standardize and normalize different sensor data, and generate identification items based on hash coding; S3, identify the completeness of the standard data, use convolutional neural network to extract static image features and long short-term memory network to capture temporal features, and enhance the data through affine transformation and random perturbation, and use cross-modal adaptive attention mechanism to dynamically allocate weights to achieve preliminary feature weighted fusion, and obtain the preliminary fused feature F fusel , and introduce the residual strategy to strengthen nonlinear complementarity and generate multimodal fusion features; The implementation process of the residual strategy is: A1. Interaction Information Extraction: Use element-wise product to capture the interaction information between the two modal features: ； in, represents element-wise product; F img Represents static image features; F sen Indicates the time series characteristics; F inter Indicates the similarity and complementarity of two features in each dimension; A2. Residual fusion: Weighted addition of preliminary fusion features and mutual information: ； In the formula, γ represents a trainable residual fusion coefficient; F fused Represents the multimodal fusion features of the final output; S4. An anomaly detection method combining unsupervised reconstruction error and supervised classification is used to input the multimodal fusion features into a deep autoencoder, map them to a low-dimensional latent space through the encoder, and reconstruct the original features by the decoder to calculate the reconstruction error; the multimodal fusion features are classified using linear transformation and softmax function, the anomaly probability is calculated, a comprehensive anomaly score is generated, and an anomaly judgment is made based on the set threshold, and the judgment result is output; S5. Display the processed weak current data and abnormality detection results through the user interface.

2. The multimodal data fusion method of an intelligent weak current monitoring system according to claim 1 is characterized in that: The implementation process of the two-stage denoising method is as follows: S21, using the adaptive wavelet threshold denoising method, the weak current data is decomposed into a set of wavelet coefficients W through the multi-scale wavelet transform operator Tr, and the signal is decomposed into different scales and directions; S22. When denoising each wavelet coefficient w∈W, define an adaptive soft threshold T: ； ； Among them, σ local represents the local noise standard deviation; N represents the total number of wavelet coefficients at the current scale; α represents the adaptive factor; median() represents the median function; S23, process each wavelet coefficient: ； Where w' represents the processed wavelet coefficient; sign() represents the sign function; S24, performing inverse wavelet transform Tr on the coefficient set W' after adaptive threshold processing −1 , and get the signal I after preliminary denoising w :I w =Tr −1 (W'); Among them, Tr −1 represents the inverse wavelet transform operator; S25. In the preliminary denoising result I w An improved anisotropic diffusion filter is introduced to dynamically control the diffusion intensity based on local gradient information, an adaptive diffusion coefficient is used to smooth the noise and retain the edge information, and the explicit iterative method is used for numerical discretization.

3. The multimodal data fusion method of an intelligent weak current monitoring system according to claim 2 is characterized in that: The implementation process of the improved anisotropic diffusion filter is as follows: S31. Construct a basic model of anisotropic diffusion: ； Where I(x,y,t) represents the intensity at position (x,y) and iteration time t; represents the gradient operator; represents the divergence operator; c(x,y,t) represents the diffusion coefficient; S32. Calculate the adaptive diffusion coefficient: ； In the formula, Represents the initial denoising result I at position (x, y) and iteration time t w The gradient amplitude; k represents the contrast parameter; S33. Use explicit iterative method to discretize the continuous diffusion process: ； In the formula, I n (x,y) represents the signal at the nth iteration; c n (x,y) represents the diffusion coefficient at position (x,y) of the nth iteration; Represents the time step parameter.

4. The multimodal data fusion method of an intelligent weak current monitoring system according to claim 1, characterized in that: The implementation process of the identification sub-item is as follows: S41, converting the standard data data into a binary sequence bdata, and calculating the code C=H(bdata) of the standard data data; Wherein, H represents a predefined hash function; S42, calculate the first element item Ief=Ea×F(C) mod η; ； Where Ea represents the predefined auxiliary generation element, Ea=P -1 mod η; F() represents the term generating function; η is the predefined first-level coding factor, η=p×q; P is the predefined first-level coding factor, P=lcm(p-1,q-1); p and q are predefined 160-bit large prime numbers; lcm() is the predefined least common multiple function; S43, calculate the second element item Ies: ; S44. Generate identification sub-item IS={Ief,Ies}.

5. The multimodal data fusion method of an intelligent weak current monitoring system according to claim 4 is characterized in that: The process of identifying the completeness of the specification of the identification standard data is as follows: S51, converting the received standard data data into a binary sequence bdata', and calculating the authentication code Ca=H(bdata') of the standard data data; Wherein, H represents a predefined hash function; S52, calculate the matching code Cm=(1+Ief×η)×(Ies) η ×Ea mod η 2 ; S53. If Ca=Cm, it indicates that the standard data data is complete in specifications; otherwise, an alarm is immediately issued.

6. The multimodal data fusion method of an intelligent weak current monitoring system according to claim 1, characterized in that: The implementation process of the cross-modal adaptive attention mechanism is as follows: S61, modal feature mapping and score calculation, linear mapping is performed on the features of the two modalities respectively, and the high-dimensional features are compressed to the same dimension: s img =W img F img +b img ; s sen =W sen F sen +b sen ; Among them, W img With W sen Respectively represent the weight matrices used to map static image features and temporal features; b img and b sen represents the bias term; s img and sen They represent the initial estimation of the importance of each modal feature in the current scenario; S62. Calculate the adaptive attention weight: normalize each modality score through the softmax function to obtain the attention weight: ； β=1-α； In the formula, α represents the importance weight of image features; β represents the importance of temporal features; S63, preliminary feature weighted fusion: Use the above weights to perform weighted averaging on the two modal features to obtain preliminary fusion features: ； In the formula, F fusel Represents the features after preliminary fusion.

7. The multimodal data fusion method of an intelligent weak current monitoring system according to claim 1, characterized in that: The implementation process of the anomaly detection method combining unsupervised reconstruction error and supervised classification is as follows: S81. Branch 1, reconstruction module, i.e. unsupervised anomaly detection: S8101, fusion feature F fused Input to the deep autoencoder, the encoder E() maps it to a low-dimensional latent space, and the decoder D() reconstructs the original features: Z = E(F fused ); ; Where E() represents the encoder network; D() represents the corresponding decoder; Z represents the potential feature representation; S8102. Define the reconstruction error R as the scoring indicator for unsupervised anomaly detection: ； Where R represents the reconstruction error; represents the Euclidean distance; S82. Branch 2: Anomaly classification detection, i.e., supervision strategy: S8201, Feature Mapping and Linear Transformation: The fusion feature F fused Through a layer of linear transformation, it is mapped to the classification space: ； Where W c represents the classification network weight matrix; b c represents the bias term; z c represents the classification score vector; S8202, probability output: use softmax activation function: p=softmax(z c ); Among them, p represents the predicted probability of each category, the categories are: normal p normal and abnormal p abnormal ; S83. Fusion of the reconstruction error R and the classification anomaly probability generates a comprehensive anomaly score: S = λR + (1-λ)(1-p normal ); Among them, S represents the comprehensive anomaly score; λ represents the fusion weight, λ∈[0,1]; (1-p normal ) represents the abnormal probability.

8. The multimodal data fusion method of the intelligent weak current monitoring system according to claim 7 is characterized in that: The joint loss function L of the anomaly detection method combining unsupervised reconstruction error and supervised classification total Simultaneously guide the training of branch one and branch two: L total =L rec +L cls ; ； ； Where, L rec represents the reconstruction loss; L cls represents the cross entropy loss; y i is the true category label; p i is the predicted probability.

9. A multimodal data fusion system of an intelligent weak current monitoring system, which is used to execute a multimodal data fusion method of an intelligent weak current monitoring system according to any one of claims 1 to 8, characterized in that: include: Weak current data acquisition module, used for multi-modal weak current data acquisition; Weak current data processing module, used for denoising, standardization and normalization preprocessing of collected weak current data; The weak current data fusion module is used to use multi-modal data fusion technology to comprehensively process and analyze multi-modal weak current data; Intelligent analysis module, which is used to combine artificial intelligence algorithms to perform intelligent analysis on the fused data to determine whether there are any abnormalities; The user interaction module is used to display the fusion results through the interface and provide user interaction functions.

Images