Deep learning based multi-source monitoring data fusion method

Abstract

The application discloses a multi-source monitoring data fusion method based on deep learning, comprising: standardizing and processing in space-time dimension alignment of multi-source monitoring data, constructing a multi-scale data subsequence, inputting a multi-scale space-time feature interaction network based on a Mamba structure to extract and interact multi-scale space-time features, constructing a cross-modal data topology graph structure, dynamically calculating and adjusting the attention weight of the nodes and edges of the topology graph by using an adaptive cross-modal graph attention mechanism, generating a dynamically optimized cross-modal fusion feature, and performing collaborative feature decoding to output a fusion result. The application improves the multi-source data fusion precision and generalization ability in complex monitoring scenarios, and improves the fusion feature expression ability and decision reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and multi-source data fusion, and in particular to a multi-source monitoring data fusion method based on deep learning. Background Art

[0002] With the rapid development of artificial intelligence and sensing technologies, multi-source monitoring data fusion has been widely applied in fields such as industrial monitoring, environmental monitoring, intelligent transportation, and smart city development. By comprehensively analyzing and utilizing heterogeneous data from different sensors, multi-source monitoring data fusion can effectively improve the overall performance and decision-making accuracy of monitoring systems. Current mainstream data fusion methods typically involve multiple key steps, including data preprocessing, feature extraction, and fusion strategy design. Among these, data-driven methods, such as deep learning, have become a research hotspot in the field of multi-source data fusion due to their outstanding advantages in nonlinear feature modeling and automatic feature representation.

[0003] Specifically, deep learning fusion methods in the existing technology are mainly implemented based on frameworks such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and self-attention mechanisms. Among them, CNN mainly realizes spatial feature extraction through spatial dimension convolution operations, while RNN focuses more on processing data with obvious time series characteristics. In recent years, due to its good feature capture ability, the self-attention mechanism has been gradually introduced into the multi-source data fusion framework to improve the modeling ability of feature correlation relationships. However, the above methods still have limitations in processing the spatiotemporal correlation characteristics of multi-source data in complex scenarios. The main manifestation is that it is difficult to effectively express spatiotemporal features of different scales. In particular, when it comes to the interactive fusion of cross-modal data, there is a lack of effective dynamic adaptability mechanisms, which limits the feature expression ability after fusion.

[0004] To overcome these shortcomings, researchers have recently begun to focus on introducing multi-scale feature extraction networks in order to capture richer spatiotemporal details. For example, existing multi-scale CNN models use convolutional kernels of different sizes to extract spatial features at different scales, significantly improving the expressive power of spatial features. However, such methods still struggle to address the interactive fusion of cross-scale spatiotemporal features and are insufficient in modeling long-term, multimodal data. Furthermore, some research efforts have attempted to utilize graph neural networks (GNNs) to model feature associations in multi-source data. By abstracting the data into a graph structure for processing, they effectively improve the cross-modal feature association representation capability. However, most existing GNN methods rely solely on static graph structures for processing. In actual monitoring scenarios, they struggle to adapt to the dynamic changes in the associations between data, and cannot achieve flexible and efficient cross-modal fusion.

[0005] At the same time, the novel Mamba network architecture, a highly efficient sequence processing structure based on a state-space model (SSM), has attracted widespread attention for its outstanding performance in extracting long sequence features. Despite its outstanding performance in sequence modeling, its application in multi-source monitoring data fusion is still in its early stages of exploration. In particular, there are currently no mature technical solutions for effectively applying it to multi-scale spatiotemporal feature interaction and cross-modal data fusion. Therefore, the existing Mamba model and its variants cannot directly meet the requirements of multi-scale feature interaction and cross-modal feature fusion in complex monitoring scenarios.

[0006] Furthermore, existing cross-modal fusion methods generally employ simple feature concatenation, linear mapping, or static attention weighting mechanisms. These methods lack effective mechanisms for real-time adjustment of fusion strategies based on dynamic changes in data features, making it difficult to achieve high-precision, well-generalized fusion results in practical applications. Furthermore, existing methods are relatively limited in their approach to modeling the associated topological structure of cross-modal data, failing to effectively capture and express the nonlinear dynamic relationships between modalities, resulting in limited fusion performance.

[0007] Therefore, how to provide a multi-source monitoring data fusion method based on deep learning is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0008] One purpose of the present invention is to propose a multi-source monitoring data fusion method based on deep learning. The present invention adopts a technical route that combines the Mamba multi-scale spatiotemporal feature interaction network and the adaptive cross-modal graph attention mechanism. By first standardizing and precisely aligning the spatiotemporal dimensions of multi-source sensor data, constructing multi-scale data subsequences and extracting corresponding multi-scale spatiotemporal features, and then constructing a cross-modal data topology graph structure based on the features, and using the adaptive cross-modal graph attention mechanism to realize dynamic calculation and adjustment of node feature vectors and edge association weights in the topology graph, further obtaining dynamically optimized cross-modal fusion features, and finally outputting the fusion results through multi-channel collaborative decoding processing, thereby realizing efficient and accurate fusion of complex multi-scale and cross-modal monitoring data features. It has the technical advantages of high fusion accuracy, strong generalization ability, and excellent feature expression ability, and can significantly improve the reliability of multi-source data fusion results in complex monitoring scenarios.

[0009] The multi-source monitoring data fusion method based on deep learning according to an embodiment of the present invention includes the following steps:

[0010] S1. Collect monitoring data output by multiple sensors, and perform standardization and spatiotemporal alignment processing on the monitoring data to generate a standardized spatiotemporal data sequence;

[0011] S2. performing multi-scale division processing on the standardized spatiotemporal data sequence to construct a multi-scale data subsequence set including different time scales and spatial scales;

[0012] S3. Inputting the multi-scale data subsequence set into a multi-scale spatiotemporal feature interactive network based on the Mamba structure, extracting spatiotemporal features of different scales, and interactively processing the spatiotemporal features of different scales to generate fused multi-scale spatiotemporal features;

[0013] S4. Constructing a cross-modal data topology structure that expresses feature association relationships between different modal data based on the fused multi-scale spatiotemporal features;

[0014] S5. Using an adaptive cross-modal graph attention mechanism, dynamically calculate and adjust the attention weights between each node and its edges in the cross-modal data topology graph structure to generate dynamically optimized cross-modal fusion features;

[0015] S6. Perform collaborative feature decoding processing on the dynamically optimized cross-modal fusion features, and output a fusion result of the multi-source monitoring data.

[0016] Optionally, the S1 specifically includes:

[0017] S11, the sampling frequencies are f s =1kHz sound sensor, f e =2kHz vibration sensor and f c =100Hz temperature sensor synchronously collects sound signal, vibration signal and temperature signal, and records the spatial coordinate position of each sensor;

[0018] S12, filtering the sound signal through a weighted Chebyshev bandpass filter, where the passband range is f low =20Hz to f high =2000Hz, filter order is n=5, passband ripple is less than 0.5dB;

[0019] S13, performing short-time Fourier transform processing on the vibration signal, performing windowing processing using a Hanning window function, setting the window length to L=1024 and the window overlap rate to 50%, and extracting the spectrum energy distribution characteristics of the vibration signal;

[0020] S14, based on the bidirectional adaptive sliding window, the temperature signal is smoothed in real time, where the forward sliding window length is T fw = 30s, the backward sliding window length is T bw = 30s, and then a dynamic threshold based on three times the standard deviation is used to trim abnormal temperature signal data points;

[0021] S15, based on the Kriging spatial interpolation method, combining the spatial coordinate positions of the sensors, the sound signal, the vibration signal and the temperature signal are subjected to spatial dimension data interpolation alignment, wherein the spatial interpolation radius is set to r=5m, and the interpolation accuracy control threshold is ε=0.01;

[0022] S16, the sound signal, the vibration signal and the temperature signal subjected to the spatial interpolation processing are subjected to normalization processing based on the maximum-minimum value method, to obtain standard data with a normalized numerical range of 0 to 1;

[0023] S17, taking the unified clock signal of GPS time service as a reference, the normalized sound, vibration and temperature standard data are subjected to millisecond-level synchronization alignment in the time dimension, to obtain standardized space-time data sequences.

[0024] Optionally, the S2 specifically comprises:

[0025] S21, the standardized space-time data sequences are subjected to sliding window division with time window lengths of T1=10s, T2=30s and T3=60s respectively, and the window sliding step is uniformly set to Δt=1s, to generate data subsequences of different time scales;

[0026] S22, each time scale data subsequence is subjected to spatial scale grid division respectively, and the grid sizes are set to l1=1m×1m, l2=2m×2m and l3=4m×4m respectively, to obtain grid data subsequences of different spatial scales;

[0027] S23, for the data subsequence of the time scale T1, the time domain average value and the variance of the numerical data in each grid are respectively counted under the corresponding spatial grid scale l1, to obtain a first group of multi-scale data subsequences;

[0028] S24, for the data subsequence of the time scale T2, the time domain maximum value and the minimum value of the numerical data in each grid are respectively extracted under the corresponding spatial grid scale l2, to obtain a second group of multi-scale data subsequences;

[0029] S25, for the data subsequence of the time scale T3, the time domain cumulative value of the numerical data in each grid is respectively calculated under the corresponding spatial grid scale l3, to obtain a third group of multi-scale data subsequences;

[0030] S26, the first group, the second group and the third group of multi-scale data subsequences are arranged and combined in time and space order, to form a multi-scale data subsequence set containing different time scales and spatial scales.

[0031] Optionally, the multi-scale space-time feature interaction network based on the Mamba structure specifically comprises:

[0032] For the multi-scale data subsequence in the multi-scale data subsequence set Perform initial feature extraction based on the Mamba structure to obtain the spatiotemporal feature vectors of the corresponding scale

[0033]

[0034] in, Represents the spatiotemporal feature vector corresponding to the data subsequence of the i-th time scale and the j-th spatial scale, represents a multi-scale data subsequence of the i-th time scale and the j-th spatial scale, represents the intermediate feature vector obtained after one-dimensional convolution, LN(·) represents the layer normalization operation on the input vector, SSConv(·) represents the state space convolution operation on the input vector, and Conv(·) represents the one-dimensional convolution operation on the input vector;

[0035] Calculate the adaptive weights of spatiotemporal feature vectors at each scale based on global average pooling operation and linear mapping

[0036]

[0037] in, Represents the adaptive weight of the feature vector of the i-th time scale and the j-th spatial scale, Indicates the calculation of the intermediate weight value of the feature vector of the i-th time scale and the j-th spatial scale, GAP(·) represents the global average pooling operation, and Linear(·) represents the linear mapping operation;

[0038] For any two-scale spatiotemporal feature vectors and Calculate dynamic weight factors of interactions between scales

[0039]

[0040] in, represents the dynamic weight factor of the interaction between the feature vectors of the mth time scale, nth spatial scale and the ith time scale, jth spatial scale at time t+1, σ(·) represents the Sigmoid function, [·;·] represents the concatenation operation on the vectors, and Linear(·) represents the linear mapping operation;

[0041] Based on the dynamic weight factor of the interaction between scales Update each scale feature vector to obtain the updated scale feature vector

[0042]

[0043] in, represents the scale feature vector of the i-th time scale and the j-th spatial scale after being updated at time t+1, represents the interactive dynamic weight factor between the mth time scale, nth spatial scale and the ith time scale, jth spatial scale, The eigenvector representing the mth time scale and the nth spatial scale at time t;

[0044] Optionally, the S3 specifically includes:

[0045] S31, using the adaptive weight The updated scale feature vector Perform weighted fusion to obtain the fused multi-scale spatiotemporal feature vector F fusion

[0046]

[0047] Among them, F fusion represents the final fused multi-scale spatiotemporal feature vector, represents the adaptive weight of the scale feature vector corresponding to the i-th time scale and the j-th spatial scale, Represents the updated scale feature vector of the i-th time scale and the j-th spatial scale.

[0048] S32, the fused multi-scale spatiotemporal feature vector F fusion Perform linear mapping processing to obtain a dimensionality-reduced fusion feature vector with the dimension reduced from 512 to 128;

[0049] S33, dividing the dimension-reduced fusion feature vector into 8 feature sub-vectors, and calculating the query vector, key vector, and value vector corresponding to each feature sub-vector respectively;

[0050] S34. Calculate the multi-head self-attention weight matrix of each feature sub-vector based on the query vector and the key vector, and perform normalization processing to obtain the self-attention weight of each feature sub-vector;

[0051] S35, performing weighted processing on the value vector corresponding to each feature sub-vector according to the self-attention weight, to obtain 8 feature sub-vectors optimized by self-attention;

[0052] S36, performing vector concatenation and linear mapping operations on the eight self-attention optimized feature sub-vectors in sequence to obtain a fused feature vector after self-attention optimization;

[0053] S37, performing layer normalization processing on the fusion feature vector optimized by self-attention to obtain a final output of the fused multi-scale space-time feature.

[0054] Optionally, the S4 specifically includes:

[0055] S41, taking the fused multi-scale space-time feature vector as input data, defining nodes of a cross-modal data topology graph, each node representing a fusion feature vector corresponding to a monitoring data;

[0056] S42, calculating the feature correlation strength between the fusion feature vectors corresponding to any two nodes by using the Pearson correlation coefficient to obtain a node correlation strength matrix R=[r uv ] N×N , wherein r uv represents the feature correlation strength between the u-th node and the v-th node, and N represents the total number of nodes;

[0057] S43, setting a correlation strength threshold r th =0.7 according to the node correlation strength matrix, and when the correlation strength satisfies r uv ≥r th , constructing a connection edge of the topology graph between the node u and the node v, otherwise not establishing a connection edge;

[0058] S44, for each connection edge, calculating the Euclidean distance between the corresponding node fusion feature vectors, taking the reciprocal of the Euclidean distance as the edge weight value e uv , defining an edge weight matrix E=[e uv ] N×N ;

[0059] S45, performing clustering analysis on the node correlation strength matrix by using a spectral clustering algorithm to determine a clustering cluster label of the nodes, and dividing the nodes belonging to the same clustering cluster label into the same node subgraph;

[0060] S46, updating the topology graph connection structure according to the division result of the node subgraph, retaining only the connection edges between the nodes within the same subgraph, and deleting the connection edges between different subgraphs;

[0061] S47, finally determining the cross-modal data topology graph structure expressing the feature correlation relationship between different modal data according to the updated node connection structure and the edge weight matrix.

[0062] Optionally, the S5 specifically includes:

[0063] S51, taking the node feature vector and the edge weight of the cross-modal data topology graph structure as input, respectively calculating an initial attention query vector, a key vector and a value vector of each node feature vector;

[0064] S52: Calculate the initial attention weight between nodes based on the query vector and key vector of any two nodes to obtain the initial attention weight matrix between nodes in, represents the initial attention weight between node u and node v, and N represents the total number of nodes;

[0065] S53, the initial attention weight matrix A between the nodes (0) Perform element-by-element weighted adjustment of the edge weight matrix E to obtain the edge-enhanced attention weight matrix A ′ =[a ′ uv ] N×N , the adjustment rules are e uv is the weight of the edge between node u and node v;

[0066] S54, use the edge-enhanced attention weight matrix A for the eigenvalue vector of the node ′ Perform weighted aggregation processing to obtain the aggregated feature vector of the node;

[0067] S55, use the adaptive weight update formula to dynamically update the attention weight of the node, and the updated attention weight between nodes

[0068]

[0069] in, represents the attention weight between node u and node v after the k+1th round of update, and denote the aggregated feature vectors of node u and node v after the kth round of update, respectively, and φ(·,·) denotes the function for calculating the cosine similarity between feature vectors;

[0070] S56, repeat S51 to S56 until the Frobenius norm difference between the attention weight matrices of two adjacent rounds is less than the set threshold δ=10 -4 The iteration is stopped when , and the node aggregation feature vector at the end of the iteration is used as the cross-modal fusion feature for dynamic optimization.

[0071] Optionally, the S6 specifically includes:

[0072] S61, performing multi-channel collaborative decoding processing on the dynamically optimized cross-modal fusion features, and dividing the dynamically optimized cross-modal fusion features into 4 channels, with a feature sub-vector length of 32 in each channel;

[0073] S62, perform a deconvolution operation on the feature sub-vector of each channel independently, using a convolution kernel size of 3×3, a step size of s=1, and a padding of p=1 to generate a channel feature map with a spatial size of 16×16;

[0074] S63. Process each channel feature map using depthwise separable convolution, where the depthwise convolution kernel size is 3×3, the pointwise convolution kernel size is 1×1, and the convolution step size and padding are both set to s=1 and p=1;

[0075] S64, dynamically weighting the spatial position information of each channel feature map through channel-by-channel spatial attention operation;

[0076] S65, concatenate the channel feature maps processed by channel-by-channel spatial attention along the channel dimension to generate an intermediate fusion feature map with a fusion size of 16×16×128;

[0077] S66. Use a fully connected layer to perform a dimension transformation operation on the intermediate fused feature map, flatten it into a one-dimensional feature vector with a length of L = 32768 and map it to a fused feature vector with an output dimension of M = 512;

[0078] S67. Use the Softmax function to perform probability mapping on the fused feature vector and output the fusion result of the multi-source monitoring data.

[0079] The beneficial effects of the present invention are:

[0080] (1) By adopting the Mamba multi-scale spatiotemporal feature interaction network, the present invention realizes deep interaction and efficient fusion between spatiotemporal features of different scales, effectively improving the accuracy of complex multi-source monitoring data fusion and the richness of data feature expression, and enhancing the generalization performance of the model and the reliability of feature representation in multi-scale monitoring scenarios.

[0081] (2) By constructing a cross-modal data topology graph structure based on the Pearson correlation coefficient and spectral clustering algorithm, the present invention can accurately model the correlation relationship between different modal data, significantly improve the feature correlation strength and expression ability of cross-modal data fusion, and show better adaptability and stability in complex multimodal monitoring environments.

[0082] (3) In terms of dynamic fusion feature optimization, the present invention uses an adaptive cross-modal graph attention mechanism to perform real-time dynamic calculation and update of the attention weights of nodes and edges in the topological graph, effectively solving the problem in the existing technology that the fusion features cannot be adaptively adjusted as the monitoring data changes dynamically. It breaks through the adaptability bottleneck of the existing static or fixed attention mechanism in practical applications, realizes the dynamic optimization and refined regulation of the fusion features, and thus effectively improves the accuracy and generalization performance of multi-source monitoring data fusion.

[0083] (4) Through multi-channel collaborative decoding and spatial attention mechanism, the present invention can achieve efficient decoding and accurate output of dynamically optimized cross-modal fusion features, significantly improving the accuracy and stability of the fusion feature decoding process, and showing higher decision reliability in practical application fields such as complex industrial monitoring, intelligent transportation and environmental monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0085] Figure 1 This is a schematic diagram of the overall process of the multi-source monitoring data fusion method based on deep learning proposed in the present invention;

[0086] Figure 2 Schematic diagram of the Mamba multi-scale spatiotemporal feature interaction network structure of the multi-source monitoring data fusion method based on deep learning proposed in this invention;

[0087] Figure 3 Schematic diagram of the adaptive cross-modal graph attention mechanism structure of the multi-source monitoring data fusion method based on deep learning proposed in this invention. DETAILED DESCRIPTION

[0088] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0089] refer to Figure 1-Figure 3 ,The multi-source monitoring data fusion method based on deep learning, includes the following steps:

[0090] S1. Collect monitoring data output by multiple sensors, and perform standardization and spatiotemporal alignment processing on the monitoring data to generate a standardized spatiotemporal data sequence;

[0091] S2. performing multi-scale division processing on the standardized spatiotemporal data sequence to construct a multi-scale data subsequence set including different time scales and spatial scales;

[0092] S3. Inputting the multi-scale data subsequence set into a multi-scale spatiotemporal feature interactive network based on the Mamba structure, extracting spatiotemporal features of different scales, and interactively processing the spatiotemporal features of different scales to generate fused multi-scale spatiotemporal features;

[0093] S4. Constructing a cross-modal data topology structure that expresses feature association relationships between different modal data based on the fused multi-scale spatiotemporal features;

[0094] S5. Using an adaptive cross-modal graph attention mechanism, dynamically calculate and adjust the attention weights between each node and its edges in the cross-modal data topology graph structure to generate dynamically optimized cross-modal fusion features;

[0095] S6. Perform collaborative feature decoding processing on the dynamically optimized cross-modal fusion features, and output a fusion result of the multi-source monitoring data.

[0096] The monitoring data described in the present invention is specifically collected from three types of sensors: temperature sensor, vibration sensor and sound sensor, wherein the sampling frequency of the temperature sensor is set to 100Hz, the sampling frequency of the vibration sensor is set to 2000Hz, and the sampling frequency of the sound sensor is set to 1000Hz; after collecting the data, the sound signal is bandpass filtered, the passband frequency range is specifically set to 20Hz to 2000Hz, the filter order is set to 5th order, and the passband ripple is not greater than 0.5dB; the vibration signal is short-time Fourier transform processed, specifically using the Hanning window function for windowing, and the window length is set to 1024 data. The temperature signal is smoothed using a bidirectional sliding window with a specific sliding window length of 30 seconds, and abnormal data points in the temperature signal are removed using a dynamic threshold method based on three times the standard deviation. Subsequently, the sound, vibration, and temperature signals are spatially interpolated and aligned using the Kriging interpolation method, with an interpolation radius of 5 meters and an interpolation accuracy threshold of 0.01. The interpolated signal is normalized to the maximum and minimum values, and the normalized data range is 0 to 1. Finally, the GPS timing unified clock signal is used to achieve millisecond-level synchronization alignment of the modal data in the time dimension.

[0097] The present invention achieves the full extraction and precise interactive fusion of multi-scale spatiotemporal features of multi-source monitoring data by constructing a multi-scale spatiotemporal feature interaction network based on the Mamba structure and integrating it with an adaptive cross-modal graph attention mechanism. It effectively solves the problems of insufficient multi-scale data feature extraction, low efficiency of cross-modal feature fusion, and insufficient dynamic adaptability in the existing technology, significantly improves the accuracy, generalization ability and stability of the fusion results in adapting to complex monitoring environments, and has good industrial practical value.

[0098] In this embodiment, S1 specifically includes:

[0099] S11, the sampling frequencies are f s =1kHz sound sensor, f e =2kHz vibration sensor and f c =100Hz temperature sensor synchronously collects sound signal, vibration signal and temperature signal, and records the spatial coordinate position of each sensor;

[0100] S12, filtering the sound signal through a weighted Chebyshev bandpass filter, where the passband range is f low =20Hz to f high =2000Hz, filter order is n=5, passband ripple is less than 0.5dB;

[0101] S13, performing short-time Fourier transform processing on the vibration signal, performing windowing processing using a Hanning window function, setting the window length to L=1024 and the window overlap rate to 50%, and extracting the spectrum energy distribution characteristics of the vibration signal;

[0102] S14, based on the bidirectional adaptive sliding window, the temperature signal is smoothed in real time, where the forward sliding window length is T fw = 30s, the backward sliding window length is T bw = 30s, and then a dynamic threshold based on three times the standard deviation is used to trim abnormal temperature signal data points;

[0103] S15. Based on the Kriging spatial interpolation method, in combination with the spatial coordinate positions of the sensors, the sound signal, the vibration signal, and the temperature signal are interpolated and aligned in the spatial dimension, wherein the spatial interpolation radius is set to r = 5m and the interpolation accuracy control threshold is ε = 0.01;

[0104] S16, performing normalization processing based on the maximum-minimum method on the sound signal, vibration signal, and temperature signal after the spatial interpolation processing to obtain standard data with a normalized value range of 0 to 1;

[0105] S17. Using the unified clock signal of GPS timing as a benchmark, the normalized sound, vibration, and temperature standard data are synchronized and aligned at the millisecond level in the time dimension to obtain a standardized spatiotemporal data sequence.

[0106] The sound sensor model used in the present invention is CRY343, with a sensitivity of -42dB, the vibration sensor model is VS1024, with a sensitivity of 50mV / g, and the temperature sensor model is PT100, with a temperature measurement accuracy of ±0.1°C; the spatial coordinate positions of the three sensors are represented by a three-dimensional rectangular coordinate system, and the coordinate accuracy is controlled within 1cm; the weighted Chebyshev bandpass filter used in the sound signal filtering processing is specifically obtained by transforming the Butterworth prototype filter, and the specific band-stop attenuation is 40dB; the short-time Fourier transform processing The length of the generated vibration signal spectrum energy distribution feature vector is fixed at 512 points; the dynamic threshold calculation formula used for temperature signal outlier clipping is threshold T = μ ± 3σ, where μ is the window data average and σ is the window data standard deviation; the Kriging interpolation processing method specifically adopts the ordinary Kriging interpolation algorithm, the semi-variogram function adopts the exponential model, the range is set to 10m, the base value is set to 0.01, and the nugget value is 0.005; the synchronization alignment accuracy of the GPS timing signal is specifically 1ms to ensure that the modal data are strictly synchronized in the time dimension.

[0107] The present invention effectively improves the accuracy of feature extraction and the reliability of data quality of multi-source monitoring data by performing specific and precise processing and standardized alignment of space-time dimensions on sound, vibration and temperature data respectively, significantly reduces the errors caused by mismatch of data space-time dimensions in the subsequent data fusion process, and effectively enhances the accuracy, stability and adaptability of the overall data fusion method to practical applications.

[0108] In this embodiment, S2 specifically includes:

[0109] S21. Divide the standardized spatiotemporal data sequence into sliding windows with time window lengths of T1 = 10s, T2 = 30s, and T3 = 60s, and uniformly set the window sliding step length to Δt = 1s to generate data subsequences of different time scales.

[0110] S22, performing grid division of the spatial scale on the data subsequences of each time scale, with the grid sizes set to l1 = 1m × 1m, l2 = 2m × 2m, and l3 = 4m × 4m, respectively, to obtain grid data subsequences of different spatial scales;

[0111] S23, for the data subsequence with a time scale of T1, respectively counting the time domain mean and variance of the numerical data in each grid at the corresponding spatial grid scale l1, to obtain a first set of multi-scale data subsequences;

[0112] S24, for the data subsequence with a time scale of T2, extract the time domain maximum and minimum values ​​of the numerical data in each grid at the corresponding spatial grid scale l2, to obtain a second set of multi-scale data subsequences;

[0113] S25. For the data subsequence with a time scale of T3, the time domain cumulative value of the numerical data in each grid is calculated at the corresponding spatial grid scale l3 to obtain a third set of multi-scale data subsequences;

[0114] S26. Arrange and combine the first, second, and third groups of multi-scale data subsequences in temporal and spatial order to form a set of multi-scale data subsequences containing different temporal scales and spatial scales.

[0115] The multi-scale sliding window partitioning method described in the present invention uses a fixed window sliding step of 1 second. The number of data points within the window is determined by the data sampling frequency. A 10-second window contains 10 temperature data points, 10,000 vibration data points, and 10,000 sound data points; a 30-second window contains 30 temperature data points, 60,000 vibration data points, and 30,000 sound data points; and a 60-second window contains 60 temperature data points, 120,000 vibration data points, and 60,000 sound data points. During the spatial grid partitioning process, spatial regions are divided according to a fixed grid size. The specific grid boundaries start from the origin of the monitoring area (0,0) and the regions are divided by integer multiples of the grid size. When performing statistical calculations within different grids, if there are no data points in a grid area, the data statistics within the grid are defaulted to 0 to ensure that the subsequent feature extraction and fusion processes do not produce missing data anomalies. The specific calculation formulas for the time domain mean, variance, maximum, minimum, and cumulative value are implemented according to statistical standard definitions, and the results are accurate to 4 decimal places.

[0116] The present invention effectively realizes the precise organization and feature statistical calculation of multi-scale and multi-dimensional data by strictly and meticulously performing multi-scale sliding window and spatial gridding on the standardized data sequence, significantly improving the accuracy and robustness of subsequent multi-scale spatiotemporal feature interaction network processing, thereby effectively enhancing the application performance and adaptability of the overall data fusion process in complex multi-source monitoring environments.

[0117] The multi-scale spatiotemporal feature interaction network based on the Mamba structure specifically includes:

[0118] For the multi-scale data subsequence in the multi-scale data subsequence set Perform initial feature extraction based on the Mamba structure to obtain the spatiotemporal feature vectors of the corresponding scale

[0119]

[0120] in, Represents the spatiotemporal feature vector corresponding to the data subsequence of the i-th time scale and the j-th spatial scale, represents a multi-scale data subsequence of the i-th time scale and the j-th spatial scale, represents the intermediate feature vector obtained after one-dimensional convolution, LN(·) represents the layer normalization operation on the input vector, SSConv(·) represents the state space convolution operation on the input vector, and Conv(·) represents the one-dimensional convolution operation on the input vector;

[0121] Calculate the adaptive weights of spatiotemporal feature vectors at each scale based on global average pooling operation and linear mapping

[0122]

[0123] in, Represents the adaptive weight of the feature vector of the i-th time scale and the j-th spatial scale, Indicates the calculation of the intermediate weight value of the feature vector of the i-th time scale and the j-th spatial scale, GAP(·) represents the global average pooling operation, and Linear(·) represents the linear mapping operation;

[0124] For any two-scale spatiotemporal feature vectors and Calculate dynamic weight factors of interactions between scales

[0125]

[0126] in, represents the dynamic weight factor of the interaction between the feature vectors of the mth time scale, nth spatial scale and the ith time scale, jth spatial scale at time t+1, σ(·) represents the Sigmoid function, [·;·] represents the concatenation operation on the vectors, and Linear(·) represents the linear mapping operation;

[0127] Based on the dynamic weight factor of the interaction between scales Update each scale feature vector to obtain the updated scale feature vector

[0128]

[0129] in, represents the scale feature vector of the i-th time scale and the j-th spatial scale after being updated at time t+1, represents the interactive dynamic weight factor between the mth time scale, nth spatial scale and the ith time scale, jth spatial scale, The eigenvector representing the mth time scale and the nth spatial scale at time t;

[0130] In this embodiment, S3 specifically includes:

[0131] S31, using the adaptive weight The updated scale feature vector Perform weighted fusion to obtain the fused multi-scale spatiotemporal feature vector F fusion

[0132]

[0133] Among them, F fusion represents the final fused multi-scale spatiotemporal feature vector, Represents the adaptive weight of the scale feature vector corresponding to the i-th time scale and the j-th spatial scale, Represents the updated scale feature vector of the i-th time scale and the j-th spatial scale.

[0134] S32, the fused multi-scale spatiotemporal feature vector F fusion Perform linear mapping processing to obtain a dimensionality-reduced fusion feature vector with the dimension reduced from 512 to 128;

[0135] S33, dividing the dimension-reduced fusion feature vector into 8 feature sub-vectors, and calculating the query vector, key vector, and value vector corresponding to each feature sub-vector respectively;

[0136] S34. Calculate the multi-head self-attention weight matrix of each feature sub-vector based on the query vector and the key vector, and perform normalization processing to obtain the self-attention weight of each feature sub-vector;

[0137] S35, performing weighted processing on the value vector corresponding to each feature sub-vector according to the self-attention weight, to obtain 8 feature sub-vectors optimized by self-attention;

[0138] S36, performing vector concatenation and linear mapping operations on the eight self-attention optimized feature sub-vectors in sequence to obtain a fused feature vector after self-attention optimization;

[0139] S37. Perform layer normalization on the fused feature vector after self-attention optimization to obtain the final output fused multi-scale spatiotemporal features.

[0140] The initial one-dimensional convolution operation described in the present invention specifically adopts a convolution kernel size of 1×3, a number of convolution kernels of 64, a fixed step size of 1, and a padding size of 1 to ensure that the length of the feature sequence after convolution is consistent with the length of the input data subsequence; the state space convolution (SSConv) operation processes the feature vector sequence by introducing a state space matrix and combining it with the convolution operation. The state space matrix dimension is set to 64×64 and is defined in the discretization form of the classical state space model. The specific matrix elements are initialized using a random orthogonal method; in the adaptive weight calculation process, the global average pooling operation (GAP) specifically performs averaging processing on the feature vector sequence along the time dimension, and the linear mapping operation (Linear) is implemented using a fully connected network, and the network weight parameters are initialized using the Xavier uniform method; the calculation of the dynamic weight factor of the interaction between scales adopts the form of linear mapping combined with the Sigmoid activation function, where the input dimension of the linear mapping network is 128 and the output dimension is 64. The linear mapping network weight initialization method also adopts Xavier uniform initialization; during the scale feature vector update process, the dimension of each scale feature vector is uniformly set to 128 to facilitate subsequent fusion calculation processing.

[0141] By constructing a multi-scale Mamba spatiotemporal feature interaction network, the present invention significantly improves the ability to fully extract and dynamically interact with data features of different scales in complex monitoring scenarios, effectively overcoming the problem of insufficient multi-scale feature expression capability of existing single-scale networks, and significantly improving the accuracy of feature fusion, data utilization and the reliability of the overall data fusion effect, with obvious practical application value.

[0142] In this embodiment, the S4 specifically includes:

[0143] S41. Using the fused multi-scale spatiotemporal feature vector as input data, define nodes of a cross-modal data topology graph, where each node represents a fused feature vector corresponding to a type of monitoring data;

[0144] S42, use the Pearson correlation coefficient to calculate the feature correlation strength between any two nodes corresponding to the fusion feature vector, and obtain the node correlation strength matrix R = [r uv ] N×N , where r uv represents the feature association strength between the u-th node and the v-th node, and N represents the total number of nodes;

[0145] S43. Set the association strength threshold r according to the association strength matrix between nodes. th =0.7, when the correlation strength satisfies r uv ≥r th When , a connection edge of the topological graph is constructed between node u and node v, otherwise no connection edge is established;

[0146] S44. For each connecting edge, calculate the Euclidean distance between the corresponding node fusion feature vectors, and use the reciprocal of the Euclidean distance as the edge weight value e uv , define the edge weight matrix E = [e uv ] N×N ;

[0147] S45, using a spectral clustering algorithm to perform cluster analysis on the node association strength matrix, determine the cluster labels of the nodes, and divide the nodes belonging to the same cluster label into the same node subgraph;

[0148] S46. Based on the node subgraph division result, the topology graph connection structure is updated, only the connection edges between nodes belonging to the same subgraph are retained, and the connection edges between different subgraphs are deleted;

[0149] S47. Based on the updated node connection structure and edge weight matrix, a cross-modal data topology graph structure that expresses the feature association relationship between different modal data is finally determined.

[0150] In the present invention, the dimension of each node feature vector used to construct the cross-modal data topology structure is unified to 128 dimensions, and the node feature association strength is specifically calculated using the Pearson correlation coefficient method. Before calculation, the feature vector needs to be zero-meaned to ensure that the feature vector data has the same statistical benchmark; when constructing the node association relationship, the association strength threshold is specifically set to 0.7. All node pairs with an association strength greater than or equal to the threshold are connected, and those with an association strength less than the threshold are not connected; when calculating the Euclidean distance between node feature vectors to determine the edge weight, if the node feature vectors are the same, the Euclidean distance is 0. Specifically, a minimum constant of 0.0001 is added to the distance calculation result to ensure the numerical stability of the edge weight calculation process; the cluster analysis process specifically uses the standard spectral clustering method to mark the cluster clusters, the number of cluster clusters is clearly set to 3, and the node connection structure is updated according to the clustering results, retaining only the connection edges between nodes within the same cluster cluster, and deleting the connection edges between cluster clusters to obtain the final cross-modal data topology structure.

[0151] The present invention accurately constructs an adaptive cross-modal data topology structure through the calculation of inter-node feature association strength based on the Pearson correlation coefficient and the spectral clustering algorithm, effectively overcoming the problem in existing methods that node connection relationships and weights are difficult to dynamically adapt to changes in data features, significantly improving the stability, accuracy and robustness of cross-modal feature fusion, and effectively enhancing the application performance of the present invention in actual complex monitoring scenarios.

[0152] In this embodiment, the S5 specifically includes:

[0153] S51, taking the node feature vectors and edge weights of the cross-modal data topology structure as input, respectively calculating the initial attention query vector, key vector, and value vector of each node feature vector;

[0154] S52: Calculate the initial attention weight between nodes based on the query vector and key vector of any two nodes to obtain the initial attention weight matrix between nodes in, represents the initial attention weight between node u and node v, and N represents the total number of nodes;

[0155] S53, the initial attention weight matrix A between the nodes (0) Perform element-by-element weighted adjustment of the edge weight matrix E to obtain the edge-enhanced attention weight matrix A ′ =[a ′ uv ] N×N , the adjustment rules are e uv is the weight of the edge between node u and node v;

[0156] S54, use the edge-enhanced attention weight matrix A for the eigenvalue vector of the node ′ Perform weighted aggregation processing to obtain the aggregated feature vector of the node;

[0157] S55, use the adaptive weight update formula to dynamically update the attention weight of the node, and the updated attention weight between nodes

[0158]

[0159] in, represents the attention weight between node u and node v after the k+1th round of update, and denote the aggregated feature vectors of node u and node v after the kth round of update, respectively, and φ(·,·) denotes the function for calculating the cosine similarity between feature vectors;

[0160] S56, repeat S51 to S56 until the Frobenius norm difference between the attention weight matrices of two adjacent rounds is less than the set threshold δ=10 -4 The iteration is stopped when , and the node aggregation feature vector at the end of the iteration is used as the cross-modal fusion feature for dynamic optimization.

[0161] The calculation of the node initial attention query vector, key vector and value vector described in the present invention is respectively implemented through a linear mapping network, the input and output dimensions of the mapping network are fixed to 128 dimensions, and the network parameters are initialized using the Xavier method; the calculation of the initial attention weight matrix between nodes is to perform a dot product operation on the query vector and the key vector, and then normalize it using the Softmax function; the adjustment of the edge weight enhanced attention matrix is ​​specifically implemented by multiplying the corresponding elements of the initial attention weight matrix and the edge weight matrix one by one; the weighted aggregation processing of node features is specifically performed through matrix multiplication of the edge enhanced attention matrix and the node eigenvalue vector; in the dynamic update operation of the adaptive attention weight, the cosine similarity between the eigenvectors is specifically calculated as the dot product of the corresponding eigenvectors divided by the product of the modulus of each vector; the iterative termination condition of the attention weight update is specifically set to stop the iteration when the Frobenius norm difference between two adjacent attention weight matrices is less than 0.0001, and the final number of iterations is usually not more than 20 times in actual implementation to ensure convergence and computational efficiency.

[0162] The present invention realizes the dynamic adjustment of node feature weights in the cross-modal data fusion process through an adaptive cross-modal graph attention mechanism, effectively solving the problem that the existing static attention method cannot adapt to the changes of data features, significantly improving the accuracy and stability of cross-modal fusion features, and making the data fusion results more robust and generalizable in complex monitoring environments.

[0163] In this embodiment, S6 specifically includes:

[0164] S61, performing multi-channel collaborative decoding processing on the dynamically optimized cross-modal fusion features, and dividing the dynamically optimized cross-modal fusion features into 4 channels, with a feature sub-vector length of 32 in each channel;

[0165] S62, perform a deconvolution operation on the feature sub-vector of each channel independently, using a convolution kernel size of 3×3, a step size of s=1, and a padding of p=1 to generate a channel feature map with a spatial size of 16×16;

[0166] S63. Process each channel feature map using depthwise separable convolution, where the depthwise convolution kernel size is 3×3, the pointwise convolution kernel size is 1×1, and the convolution step size and padding are both set to s=1 and p=1;

[0167] S64, dynamically weighting the spatial position information of each channel feature map through channel-by-channel spatial attention operation;

[0168] S65, concatenate the channel feature maps processed by channel-by-channel spatial attention along the channel dimension to generate an intermediate fusion feature map with a fusion size of 16×16×128;

[0169] S66. Use a fully connected layer to perform a dimension transformation operation on the intermediate fused feature map, flatten it into a one-dimensional feature vector with a length of L = 32768 and map it to an output fused feature vector with a dimension of M = 512;

[0170] S67. Use the Softmax function to perform probability mapping on the fused feature vector and output the fusion result of the multi-source monitoring data.

[0171] The multi-channel collaborative decoding described in the present invention is specifically achieved by first decomposing the dynamically optimized cross-modal fusion features into 4 sub-vectors of length 32, and then independently performing a deconvolution operation on each sub-vector; the deconvolution operation adopts a fixed convolution kernel size of 3×3, the output feature map size is 16×16, and the convolution kernel parameters are initialized with random normal; the depth-separable convolution processing is specifically composed of depth convolution and point convolution in sequence, the number of convolution kernel channels of the depth convolution is the same as the number of channels of the input feature map, both are 1 channel, and the number of convolution kernel channels of the point convolution is equal to the number of output channels number; the channel-by-channel spatial attention operation first performs maximum pooling and mean pooling operations on the feature maps of each channel, and then uses a network containing two fully connected layers to perform linear mapping. After that, the Sigmoid function is used to calculate the spatial attention weight of each channel feature map; after the intermediate fusion feature map is processed by spatial attention, it is spliced ​​along the channel dimension to form a fusion feature map of size 16×16×128, and finally mapped to a one-dimensional fusion feature vector of length 512 through a fully connected layer, and the fusion result is mapped and output through the Softmax function.

[0172] The present invention realizes the refined decoding and efficient processing of cross-modal fusion features by adopting multi-channel collaborative decoding and channel-by-channel spatial attention mechanism, effectively solving the problems of insufficient decoding accuracy and limited expression ability of fusion features in the existing technology, significantly improving the accuracy and stability of the final fusion result, and meeting the high-precision and high-robustness data fusion requirements in actual complex monitoring environments.

[0173] Example 1:

[0174] In order to verify the feasibility of the present invention, the present invention was applied to the equipment safety monitoring environment of Company B, a medium-sized chemical enterprise. The company is mainly engaged in the production of chemical raw materials, and the production process involves a large amount of temperature monitoring, equipment operation vibration monitoring and environmental sound anomaly detection. Due to the complexity of the chemical production environment, the diverse sources of equipment operation status data and the obvious spatiotemporal correlation characteristics, the traditional single-modal monitoring method is unable to accurately and timely detect potential safety hazards, and equipment anomalies are frequently missed or falsely reported. Especially in the second half of last year, the company experienced several consecutive production equipment failure accidents. Although no personal injury was ultimately caused, they all affected normal production to varying degrees, resulting in reduced production efficiency and significant economic losses.

[0175] To solve the above problems, the enterprise decided to introduce the multi-source monitoring data fusion method based on Mamba multi-scale spatio-temporal feature interaction network and adaptive cross-modal graph attention mechanism. Based on the existing monitoring equipment of the enterprise, high-precision PT100 temperature sensors, VS1024 vibration sensors and CRY343 sound sensors were first deployed. These sensors were installed at different positions of key production equipment such as reaction kettles, mixers and compressors, and real-time acquisition of temperature, vibration and sound signals during equipment operation was realized. These signals were transmitted to the central server through a special data acquisition module and real-time preprocessing was performed, including signal filtering, outlier rejection, time and space dimension standardization and accurate alignment to ensure data quality.

[0176] After standardization, the system uses a multi-scale sliding window method to divide the data subsequence based on the scheme of the present application, generating a data window every 1 second. Among them, 10 seconds, 30 seconds and 60 seconds are used as time windows to statistically process temperature, vibration and sound signals, respectively, to construct a set of data subsequences of different time and space scales, ensuring that subsequent analysis fully covers short-term fluctuations and long-term trend characteristics. By constructing a Mamba multi-scale spatio-temporal feature interaction network, the system can efficiently extract spatio-temporal features at different scales and use a dynamic interaction mechanism between scales to perform deep fusion and generate high-quality fusion feature vectors.

[0177] Subsequently, the extracted multi-scale fusion features are further input into a cross-modal data topology graph constructed based on the Pearson correlation coefficient and spectral clustering algorithm. By setting the node correlation strength threshold to 0.7, the system automatically identifies and establishes the correlation between different modal data, and then uses an adaptive cross-modal graph attention mechanism to dynamically adjust the attention weights of each node and edge in real time, continuously generating dynamically optimized cross-modal fusion features. This dynamic feature adjustment mechanism enables the system to adapt to data fluctuations and environmental changes in real time, more accurately identifying abnormal states.

[0178] During the continuous operation after system deployment, from January to March 2024, the company's actual production environment was monitored and analyzed for three consecutive months, and relevant data on the implementation effect was recorded. The specific data is shown in Table 1:

[0179] Table 1: Comparison of B Company's Equipment Safety Monitoring Implementation Effect

[0180]

[0181] According to the data in Table 1, before the solution of the present invention was deployed, Company B had an average of 6 missed equipment failures per month, as many as 14 false alarms, and an advance warning time of only 3 minutes, which resulted in very limited response time for on-site staff. After deploying the solution of the present invention, the accuracy of system monitoring data increased from 82.7% to 98.6%, the false alarm rate decreased from 9.5% to 1.2%, and zero missed failures were achieved. The advance warning time was increased to an average of 26 minutes, allowing maintenance personnel to have sufficient response time to prevent equipment failures in advance. At the same time, the number of equipment shutdowns and maintenance also dropped from 4 times per month to 0 times, and the labor cost of monthly equipment inspections was reduced from 210 hours to 78 hours, greatly improving equipment management efficiency. In addition, the overall operating efficiency (OEE) of the equipment increased significantly by 14.6 percentage points to 92.8%, significantly improving the company's production efficiency and safety management effectiveness.

[0182] For example, in mid-February 2024, the system of the present invention detected abnormal changes in the vibration and sound data characteristics of an important production mixer. After precise analysis by the multi-scale feature interaction network and the cross-modal graph attention mechanism, the system triggered an early warning 29 minutes in advance and notified the on-site staff to conduct inspections and repairs. On-site verification showed that the mixer bearings showed early wear. If not discovered in time, it is likely to cause major equipment damage and production line shutdowns, resulting in serious economic losses. With the help of the solution of the present invention, the company successfully avoided this potential accident and significantly improved the level of equipment safety management and economic benefits.

[0183] The above specific embodiments show that the multi-source monitoring data fusion method proposed in this invention, based on the Mamba multi-scale spatiotemporal feature interaction network fused with an adaptive cross-modal graph attention mechanism, effectively solves the problems of difficulty in fusing multi-source heterogeneous data, insufficient data feature extraction, and low anomaly detection efficiency in actual production environment applications. By effectively improving data fusion precision and anomaly detection accuracy, it reduces equipment operation risks, saves maintenance costs, improves equipment operation efficiency, and significantly enhances the enterprise's production site safety management capabilities, demonstrating outstanding practical application value and broad promotion prospects.

[0184] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-source monitoring data fusion method based on deep learning, characterized by: The steps include: S1. Collect monitoring data output by multiple sensors, and perform standardization and spatiotemporal alignment processing on the monitoring data to generate a standardized spatiotemporal data sequence; S2. performing multi-scale division processing on the standardized spatiotemporal data sequence to construct a multi-scale data subsequence set including different time scales and spatial scales; S3. Inputting the multi-scale data subsequence set into a multi-scale spatiotemporal feature interactive network based on the Mamba structure, extracting spatiotemporal features of different scales, and interactively processing the spatiotemporal features of different scales to generate fused multi-scale spatiotemporal features; The multi-scale spatiotemporal feature interaction network based on the Mamba structure specifically includes: For the multi-scale data subsequence in the multi-scale data subsequence set Perform initial feature extraction based on the Mamba structure to obtain the spatiotemporal feature vectors of the corresponding scale Calculate the adaptive weights of spatiotemporal feature vectors at each scale based on global average pooling operation and linear mapping For any two-scale spatiotemporal feature vectors and Calculate dynamic weight factors of interactions between scales Based on the dynamic weight factor of the interaction between scales Update each scale feature vector to obtain the updated scale feature vector The S3 specifically includes: S31, using the adaptive weight The updated scale feature vector Perform weighted fusion to obtain the fused multi-scale spatiotemporal feature vector F fusion ; S32, the fused multi-scale spatiotemporal feature vector F fusion Perform linear mapping processing to obtain a dimensionality-reduced fusion feature vector with the dimension reduced from 512 to 128; S33, dividing the dimension-reduced fusion feature vector into 8 feature sub-vectors, and calculating the query vector, key vector, and value vector corresponding to each feature sub-vector respectively; S34. Calculate the multi-head self-attention weight matrix of each feature sub-vector based on the query vector and the key vector, and perform normalization processing to obtain the self-attention weight of each feature sub-vector; S35, performing weighted processing on the value vector corresponding to each feature sub-vector according to the self-attention weight, to obtain 8 feature sub-vectors optimized by self-attention; S36, performing vector concatenation and linear mapping operations on the eight self-attention optimized feature sub-vectors in sequence to obtain a fused feature vector after self-attention optimization; S37, performing layer normalization processing on the fused feature vector after self-attention optimization to obtain the final output fused multi-scale spatiotemporal features; S4. Constructing a cross-modal data topology structure that expresses feature association relationships between different modal data based on the fused multi-scale spatiotemporal features; The S4 specifically includes: S41. Using the fused multi-scale spatiotemporal feature vector as input data, define nodes of a cross-modal data topology graph, where each node represents a fused feature vector corresponding to a type of monitoring data; S42, use the Pearson correlation coefficient to calculate the feature correlation strength between any two nodes corresponding to the fusion feature vector, and obtain the node correlation strength matrix R = [r uv ] N×N , where r uv represents the feature association strength between the u-th node and the v-th node, and N represents the total number of nodes; S43. Set the association strength threshold r according to the association strength matrix between nodes. th =0.7, when the correlation strength satisfies r uv ≥r th When , a connection edge of the topological graph is constructed between node u and node v, otherwise no connection edge is established; S44. For each connecting edge, calculate the Euclidean distance between the corresponding node fusion feature vectors, and use the reciprocal of the Euclidean distance as the edge weight value e uv , define the edge weight matrix E = [e uv ] N×N ; S45, using a spectral clustering algorithm to perform cluster analysis on the node association strength matrix, determine the cluster labels of the nodes, and divide the nodes belonging to the same cluster label into the same node subgraph; S46. Based on the node subgraph division result, the topology graph connection structure is updated, only the connection edges between nodes belonging to the same subgraph are retained, and the connection edges between different subgraphs are deleted; S47. Finally, a cross-modal data topology structure expressing feature associations between different modal data is determined based on the updated topology connection structure and edge weight matrix. S5. Using an adaptive cross-modal graph attention mechanism, dynamically calculate and adjust the attention weights between each node and its edges in the cross-modal data topology graph structure to generate dynamically optimized cross-modal fusion features; S6. Perform collaborative feature decoding processing on the dynamically optimized cross-modal fusion features, and output a fusion result of the multi-source monitoring data.

2. The multi-source monitoring data fusion method based on deep learning according to claim 1 is characterized in that: Said S1 specifically includes: S11, the sampling frequencies are f s =1kHz sound sensor, f e =2kHz vibration sensor and f c =100Hz temperature sensor synchronously collects sound signal, vibration signal and temperature signal, and records the spatial coordinate position of each sensor; S12, filtering the sound signal through a weighted Chebyshev bandpass filter, where the passband range is f low =20Hz to f high =2000Hz, filter order is n=5, passband ripple is less than 0.5dB; S13, performing short-time Fourier transform processing on the vibration signal, performing windowing processing using a Hanning window function, setting the window length to L=1024 and the window overlap rate to 50%, and extracting the spectrum energy distribution characteristics of the vibration signal; S14, based on the bidirectional adaptive sliding window, the temperature signal is smoothed in real time, where the forward sliding window length is T fw = 30s, the backward sliding window length is T bw = 30s, and then a dynamic threshold based on three times the standard deviation is used to trim abnormal temperature signal data points; S15. Based on the Kriging spatial interpolation method, in combination with the spatial coordinate positions of the sensors, the sound signal, the vibration signal, and the temperature signal are interpolated and aligned in the spatial dimension, wherein the spatial interpolation radius is set to r = 5m and the interpolation accuracy control threshold is ε = 0.01; S16, performing normalization processing based on the maximum-minimum method on the sound signal, vibration signal, and temperature signal after the spatial interpolation processing to obtain standard data with a normalized value range of 0 to 1; S17. Using the unified clock signal of GPS timing as a benchmark, the normalized sound, vibration, and temperature standard data are synchronized and aligned at the millisecond level in the time dimension to obtain a standardized spatiotemporal data sequence.

3. The multi-source monitoring data fusion method based on deep learning according to claim 1 is characterized in that: The S2 specifically includes: S21. Divide the standardized spatiotemporal data sequence into sliding windows with time window lengths of T1 = 10s, T2 = 30s, and T3 = 60s, and uniformly set the window sliding step length to Δt = 1s to generate data subsequences of different time scales. S22, performing grid division of the spatial scale on the data subsequences of each time scale, with the grid sizes set to l1 = 1m × 1m, l2 = 2m × 2m, and l3 = 4m × 4m, respectively, to obtain grid data subsequences of different spatial scales; S23, for the data subsequence with a time scale of T1, respectively counting the time domain mean and variance of the numerical data in each grid at the corresponding spatial grid scale l1, to obtain a first set of multi-scale data subsequences; S24, for the data subsequence with a time scale of T2, extract the time domain maximum and minimum values ​​of the numerical data in each grid at the corresponding spatial grid scale l2, to obtain a second set of multi-scale data subsequences; S25. For the data subsequence with a time scale of T3, the time domain cumulative value of the numerical data in each grid is calculated at the corresponding spatial grid scale l3 to obtain a third set of multi-scale data subsequences; S26. Arrange and combine the first, second, and third groups of multi-scale data subsequences in temporal and spatial order to form a set of multi-scale data subsequences containing different temporal scales and spatial scales.

4. The multi-source monitoring data fusion method based on deep learning according to claim 3 is characterized in that: The multi-scale spatiotemporal feature interaction network based on the Mamba structure specifically includes: For the multi-scale data subsequence in the multi-scale data subsequence set Perform initial feature extraction based on the Mamba structure to obtain the spatiotemporal feature vectors of the corresponding scale in, Represents the spatiotemporal feature vector corresponding to the data subsequence of the i-th time scale and the j-th spatial scale, represents a multi-scale data subsequence of the i-th time scale and the j-th spatial scale, represents the intermediate feature vector obtained after one-dimensional convolution, LN(·) represents the layer normalization operation on the input vector, SSConv(·) represents the state space convolution operation on the input vector, and Conv(·) represents the one-dimensional convolution operation on the input vector; Calculate the adaptive weights of spatiotemporal feature vectors at each scale based on global average pooling operation and linear mapping in, Represents the adaptive weight of the feature vector of the i-th time scale and the j-th spatial scale, Indicates the calculation of the intermediate weight value of the feature vector of the i-th time scale and the j-th spatial scale, GAP(·) represents the global average pooling operation, and Linear(·) represents the linear mapping operation; For any two-scale spatiotemporal feature vectors and Calculate dynamic weight factors of interactions between scales in, represents the dynamic weight factor of the interaction between the feature vectors of the mth time scale, nth spatial scale and the ith time scale, jth spatial scale at time t+1, σ(·) represents the Sigmoid function, [·;·] represents the concatenation operation on the vectors, and Linear(·) represents the linear mapping operation; Based on the dynamic weight factor of the interaction between scales Update each scale feature vector to obtain the updated scale feature vector in, represents the scale feature vector of the i-th time scale and the j-th spatial scale after being updated at time t+1, represents the interactive dynamic weight factor between the mth time scale, nth spatial scale and the ith time scale, jth spatial scale, Represents the eigenvector of the mth time scale and the nth spatial scale at the tth moment.

5. The multi-source monitoring data fusion method based on deep learning according to claim 1 is characterized in that: The S5 specifically includes: S51, taking the node feature vectors and edge weights of the cross-modal data topology structure as input, respectively calculating the initial attention query vector, key vector, and value vector of each node feature vector; S52: Calculate the initial attention weight between nodes based on the query vector and key vector of any two nodes to obtain the initial attention weight matrix between nodes in, represents the initial attention weight between node u and node v, and N represents the total number of nodes; S53, the initial attention weight matrix A between the nodes (0) Perform element-by-element weighted adjustment of the edge weight matrix E to obtain the edge-enhanced attention weight matrix A ′ =[a ′ uv ] N×N , the adjustment rules are e uv is the weight of the edge between node u and node v; S54, use edge-enhanced attention weight matrix A for node feature vectors ′ Perform weighted aggregation processing to obtain the aggregated feature vector of the node; S55, use the adaptive weight update formula to dynamically update the attention weight of the node, and the updated attention weight between nodes in, represents the attention weight between node u and node v after the k+1th round of update, and denote the aggregated feature vectors of node u and node v after the kth round of update, respectively, and φ(·,·) denotes the function for calculating the cosine similarity between feature vectors; S56, repeat S51 to S56 until the Frobenius norm difference between the attention weight matrices of two adjacent rounds is less than the set threshold δ=10 -4 The iteration is stopped when , and the node aggregation feature vector at the end of the iteration is used as the cross-modal fusion feature for dynamic optimization.

6. The multi-source monitoring data fusion method based on deep learning according to claim 1 is characterized in that: The S6 specifically includes: S61, performing multi-channel collaborative decoding processing on the dynamically optimized cross-modal fusion features, and dividing the dynamically optimized cross-modal fusion features into 4 channels, with a feature sub-vector length of 32 for each channel; S62, perform a deconvolution operation on the feature sub-vector of each channel independently, using a convolution kernel size of 3×3, a step size of s=1, and a padding of p=1 to generate a channel feature map with a spatial size of 16×16; S63. Process each channel feature map using depthwise separable convolution, where the depthwise convolution kernel size is 3×3, the pointwise convolution kernel size is 1×1, and the convolution step size and padding are both set to s=1 and p=1; S64, dynamically weighting the spatial position information of each channel feature map through channel-by-channel spatial attention operation; S65, concatenate the channel feature maps processed by channel-by-channel spatial attention along the channel dimension to generate an intermediate fusion feature map with a fusion size of 16×16×128; S66. Use a fully connected layer to perform a dimension transformation operation on the intermediate fused feature map, flatten it into a one-dimensional feature vector with a length of L = 32768 and map it to a fused feature vector with an output dimension of M = 512; S67. Use the Softmax function to perform probability mapping on the fused feature vector and output the fusion result of the multi-source monitoring data.

Images