Multi-source heterogeneous data fusion processing method and system based on deep learning

Abstract

The invention provides a multi-source heterogeneous data fusion processing method and system based on deep learning, and the method comprises the steps: obtaining a multi-source sensor data stream of an Internet of Things terminal, extracting a modal description vector of the multi-source sensor data stream, generating a meta-feature, and independently packaging each data source into a data block; processing paths are dynamically distributed according to element feature vectors in the data blocks, and parallel feature extraction branches are input into the data blocks of different modalities to perform modal feature extraction; inputting the extracted different modal features into a cross-modal semantic graph generator, extracting semantic similarity measurement values among the features through comparative learning, and obtaining a dynamically updated cross-modal knowledge graph; performing multilayer graph convolution calculation on the modal features based on a cross-modal knowledge graph to generate a joint feature tensor; according to the method, the problems of low fusion efficiency, poor adaptability, poor consistency and the like caused by multi-source, heterogeneous and strong time-varying characteristics of sensor data in an existing Internet of Things system can be effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent data processing technology, and relates to a method and system for multi-source heterogeneous data fusion processing based on deep learning. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) technology, various sensor devices are widely deployed in application scenarios such as industrial control, intelligent transportation, environmental monitoring, and smart healthcare, generating massive amounts of sensor data. This data is typically characterized by being multi-source, heterogeneous, and highly time-varying, meaning it originates from different types of sensors (such as temperature sensors, humidity sensors, image sensors, and accelerometers), and its data format, sampling frequency, accuracy level, and semantic expression methods differ significantly. Therefore, achieving effective fusion processing of multi-source heterogeneous data in IoT systems has become a key step in improving the system's sensing capabilities, decision-making accuracy, and intelligence level.

[0003] In IoT sensing systems, data collected by a single sensor often fails to fully reflect complex environmental conditions and is susceptible to noise interference or equipment malfunctions, leading to incomplete or unreliable information. Multi-source data fusion can combine the advantages of different sensors, improving system robustness and sensing accuracy. For example, in smart home systems, combining multi-dimensional data such as temperature, humidity, light intensity, and human activity allows for more precise adjustment of the indoor environment; in industrial predictive maintenance, integrating multiple signals such as vibration, temperature, and current helps detect potential equipment failures earlier.

[0004] However, current data fusion methods still face numerous technical challenges, especially in dynamically changing environments. How to efficiently adapt to and integrate new data sources has become a pressing issue. Most existing data fusion frameworks are based on pre-defined data structures and fusion models. When introducing new sensors or new data types, manual intervention is often required to reconfigure data interfaces, feature extraction methods, and fusion algorithm parameters. This not only increases system maintenance costs but also reduces system flexibility and real-time responsiveness. Furthermore, due to difficulties in spatiotemporal alignment and significant semantic differences between different sensors, new data sources may exhibit poor consistency, high redundancy, or conflicts with existing data, further impacting the reliability of the fusion results.

[0005] Therefore, how to provide a deep learning-driven data fusion processing method and system that can automatically identify and effectively integrate new data sources is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention proposes a multi-source heterogeneous data fusion processing method and system based on deep learning, which can effectively improve the perception and adaptation capabilities of IoT systems in complex dynamic environments.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention discloses a method for multi-source heterogeneous data fusion processing based on deep learning, comprising the following steps:

[0009] S1: Obtain multi-source sensor data streams from IoT terminals, extract modal description vectors from the multi-source sensor data streams, generate meta-features, and independently encapsulate each data source into a data block formed by combining {data stream, meta-feature}.

[0010] S2: Dynamically allocate processing paths based on the meta-feature vectors in the data block, and input parallel feature extraction branches to extract modal features for data blocks of different modalities respectively;

[0011] S3: Input the modal features corresponding to the extracted data blocks of different modalities into the cross-modal semantic graph generator, and extract the semantic similarity measure between features through comparative learning to obtain a dynamically updated cross-modal knowledge graph;

[0012] S4: Perform multi-layer graph convolution calculation on modal features based on cross-modal knowledge graph to generate joint feature tensors.

[0013] Preferably, the meta-features include at least: data type, sampling rate, dimension, and physical unit.

[0014] Preferably, in step S1, new data streams from sensor sources of the Internet of Things (IoT) terminal are received in real time.

[0015] Preferably, step S2, which uses a gating attention mechanism to dynamically allocate processing paths for the data stream in the data block, includes the following steps:

[0016] By retrieving the pre-trained embedding table, embedding mapping is performed on categorical features, and numerical features are normalized. After concatenating all features, a unified embedding vector is output through a fully connected layer.

[0017] Based on the generated unified embedding vector, the path selection gating G_path and G_param are triggered to calculate and generate the branch activation probability vector and the branch parameter vector.

[0018] The feature extraction branch into which the data stream corresponding to the current meta-feature vector enters is determined based on the branch activation probability vector.

[0019] The computational parameters of the feature extraction branch being entered are dynamically initialized based on the branch parameter vector.

[0020] Preferably, the feature extraction branch in S2 includes a time-space joint coding network based on Transformer-XL, used to extract features from the time-series signal data stream in the data block, including the following steps:

[0021] The sampling rate is parsed to calculate the time step, and the G_param parameter vector is used to generate a learnable position code as a time-aware embedding vector.

[0022] Initialize the historical perception embedding vector, concatenate the current time perception embedding vector, the corresponding time-series signal data stream, and the historical perception embedding vector, and then calculate different attention distributions in parallel based on a multi-head self-attention mechanism. Concatenate the parallel calculation results to obtain the attention output vector.

[0023] The vector segment with a preset time step at the end of the attention output vector is used as the historical perception embedding vector for the next time window to update the data, and finally the time series signal features are obtained.

[0024] Preferably, the feature extraction branch in S2 includes a 3D convolutional branch network for feature extraction of the multimedia data stream in the data block, and the meta-features further include resolution: including the following steps:

[0025] The basic convolutional kernel size is determined based on the parsed G_param parameter vector; the basic convolutional kernel size is adaptively adjusted by combining the resolution information in the meta-features to obtain a dynamically depth-separable convolutional kernel group.

[0026] The multimedia data stream is input into the dynamic depth-separable convolution kernel group for multi-level convolution processing to obtain the convolution output vector.

[0027] The convolutional output vector is fed into a dual-path attention network; the spatial attention branch generates a spatial saliency map through global average pooling and calculates the importance weight of each pixel region using the Sigmoid function; the temporal attention branch calculates the inter-frame correlation matrix, performs feature weighted fusion after Softmax normalization and the aforementioned importance weights, and finally obtains multimedia features.

[0028] Preferably, before step S3, the method further includes a step of feature alignment of data at different sampling rates using a learnable position embedding matrix:

[0029] Based on preset rules, the sampling rate in multiple meta-feature vectors is filtered to determine the global reference time axis, and data blocks from multiple data sources are mapped to a unified time grid.

[0030] A time interpolation vector is obtained based on a learnable location embedding matrix to represent the mapping relationship from different sampling rates to the global reference time axis;

[0031] Based on the time interpolation vector, a sequence of up / down sampling granularity is generated to generate meta-feature vectors corresponding to different data blocks;

[0032] The feature-aligned data stream is calculated based on the generated upsampling / downsampling granularity sequence and used as the input to S3.

[0033] Preferably, S3 includes:

[0034] Create a cross-modal knowledge graph structure, including nodes and edges; where each node represents a modal feature of a single modality, and the edges represent the semantic similarity measure between nodes;

[0035] Modal features extracted from different modal data blocks are input into the cross-modal semantic graph generator in the form of tensors;

[0036] A contrastive learning framework is used to calculate the semantic similarity metric between current modal features and construct a similarity matrix. The cross-modal knowledge graph structure is then initialized based on the similarity matrix.

[0037] New modal features are acquired in real time. If the similarity metric between the new modal features and existing nodes in the initialized cross-modal knowledge graph structure exceeds a threshold, the new features are clustered under the existing nodes to form a semantic cluster; otherwise, a new node is created.

[0038] If the similarity metric between the new node and an existing node in the initialized cross-modal knowledge graph structure is higher than threshold two, and the similarity threshold one is higher than the similarity threshold two, then an edge is added to the initialized cross-modal knowledge graph structure to connect the new node and the existing node, the weight of the edge is set to the similarity metric, and the initialized cross-modal knowledge graph structure is updated.

[0039] Preferably, S4 includes:

[0040] Extract the semantic clusters of each node from the cross-modal knowledge graph and merge the semantic clusters into a unified feature vector;

[0041] Extracting edge similarity metrics from cross-modal knowledge graphs;

[0042] For each node i, the feature vector of node i is updated by using the adjacent nodes j connected by the edge and the similarity measure between them, through stacked multi-layer GCN layers, to obtain the updated feature representation of node i.

[0043] A global aggregation operation is performed on the updated feature representations of all nodes to generate a joint feature tensor.

[0044] The present invention also provides a deep learning-based multi-source heterogeneous data fusion processing system according to the aforementioned deep learning-based multi-source heterogeneous data fusion processing method, comprising:

[0045] The data access and preprocessing module is used to acquire multi-source sensor data streams from IoT terminals, extract modal description vectors from the multi-source sensor data streams, generate meta-features, and independently encapsulate each data source into a data block formed by combining {data stream, meta-feature}.

[0046] The dynamic path allocation module is used to dynamically allocate processing paths based on the meta-feature vectors in the data block, and to extract modal features by inputting parallel feature extraction branches into data blocks of different modalities respectively.

[0047] The cross-modal semantic graph generation module is used to input the modal features corresponding to the extracted data blocks of different modalities into the cross-modal semantic graph generator. Through comparative learning, the semantic similarity measure between features is extracted to obtain a dynamically updated cross-modal knowledge graph.

[0048] The joint feature tensor generation module is used to perform multi-level graph convolution calculations on modal features based on cross-modal knowledge graphs to generate joint feature tensors.

[0049] As can be seen from the above technical solution, compared with the existing technology, it can effectively solve the problems of low fusion efficiency, poor adaptability, and poor consistency caused by the multi-source, heterogeneous, and time-varying nature of sensor data in existing Internet of Things systems. The beneficial effects of the present invention include:

[0050] This invention extracts meta-features from sensor data streams and generates metadata blocks. It then uses a gated attention mechanism to dynamically allocate processing paths and initialize parameters for feature extraction branches. This enables the system to automatically identify and adapt to data sources of different types, sampling rates, and formats, significantly improving the system's flexibility and automation capabilities when faced with new sensors or data types.

[0051] This invention employs parallel feature extraction branches, including a Transformer-XL network suitable for time-series signals and a 3D convolutional network suitable for multimedia data. It combines dynamic convolutional kernels with a dual-path attention mechanism, enabling efficient and accurate feature extraction for data of different modalities, thereby improving the robustness and generalization ability of feature representation.

[0052] This invention introduces a cross-modal semantic graph generator, which uses a contrastive learning method to extract semantic similarity metrics between modalities and dynamically constructs and updates cross-modal knowledge graphs. This effectively solves the problem of semantic differences between different modalities and enhances the system's ability to understand and integrate complex multi-source information.

[0053] This invention supports real-time access to new data sources and achieves automatic identification, clustering, and fusion of new modal features by dynamically updating the cross-modal knowledge graph structure, thereby enhancing the system's scalability and online learning capabilities and making it suitable for dynamically changing IoT application scenarios.

[0054] Therefore, this invention provides an efficient and flexible solution for multi-source heterogeneous data fusion by constructing a meta-feature-driven adaptive feature extraction mechanism, a cross-modal semantic graph modeling method, and a multi-layer graph convolution fusion strategy. It effectively overcomes the fusion bottleneck of existing technologies in dynamic environments and has significant technological progress and practical application value. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1 A flowchart of a deep learning-based multi-source heterogeneous data fusion processing method provided in an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the dynamic path allocation mechanism provided in this embodiment of the invention;

[0058] Figure 3 This is a schematic diagram of a cross-modal semantic map provided in an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The first aspect of this invention provides a method for multi-source heterogeneous data fusion processing based on deep learning, such as... Figure 1 As shown, it includes the following steps:

[0061] S1: Obtain multi-source sensor data streams from IoT terminals, extract modal description vectors from the multi-source sensor data streams, generate meta-features, and independently encapsulate each data source into a data block formed by combining {data stream, meta-feature}.

[0062] S2: Dynamically allocate processing paths based on the meta-feature vectors in the data block, and input the parallel feature extraction branches to extract modal features for data blocks of different modalities respectively;

[0063] S3: Input the modal features corresponding to the extracted data blocks of different modalities into the cross-modal semantic graph generator, and extract the semantic similarity measure between features through comparative learning to obtain a dynamically updated cross-modal knowledge graph;

[0064] S4: Perform multi-layer graph convolution calculation on modal features based on cross-modal knowledge graph to generate joint feature tensors.

[0065] In one embodiment, the meta-features include at least: data type, sampling rate, dimension, and physical unit.

[0066] In one embodiment, S1 receives new data streams from sensor sources of the Internet of Things (IoT) terminal in real time.

[0067] In one embodiment, such as Figure 2 As shown, S2 uses a gating attention mechanism to dynamically allocate processing paths to the data stream in the data block, including the following steps:

[0068] Embedding mapping for categorical feature data_type:

[0069] Look up the pre-trained embedding table: thermal_image → [0.2, -0.7, ..., 1.3] ∈ R^{128};

[0070] Normalize the numerical features (sample_rate, dim_size):

[0071] Linear scaling to the [-1,1] interval: sample_rate = 5Hz → 0.32;

[0072] After concatenating all features, a unified embedding vector E_m is output by compression through a fully connected layer.

[0073] Based on the generated unified embedding vector, the path selection gating G_path and G_param are triggered to calculate and generate the branch activation probability vector and the branch parameter vector.

[0074] The feature extraction branch into which the data stream corresponding to the current meta-feature vector enters is determined based on the branch activation probability vector.

[0075] The computational parameters of the feature extraction branch being entered are dynamically initialized based on the branch parameter vector.

[0076] In this embodiment, the steps for calculating the path selection gating G_path based on the generated unified embedding vector include:

[0077] G_path=Sigmoid(W_g·E_m+b_g)

[0078] Wherein, the weight matrix W_g∈R {2×256} G_path outputs a two-dimensional probability vector [P_conv, P_tr ans], with values ​​ranging from [0,1], used to decide which processing branch the data should be assigned to. An example is given below:

[0079] P_conv>0.7 → Activate the 3D convolution branch;

[0080] P_transformer>0.7 → Activate the Transformer-XL branch.

[0081] In this embodiment, the steps for calculating the path selection gating G_param based on the generated unified embedding vector include:

[0082] G_param = tanh(W_p·E_m)

[0083] Wherein, the weight matrix W_p∈R {k×256} The output parameter vector G_param is used to dynamically initialize the computational parameters required for the corresponding branch. An example is given below:

[0084] For 3D convolutional branch networks, the first 3 dimensions of G_param are the kernel size: [5,5,3] → 5×5×3 convolutional kernel;

[0085] For the temporal-spatial joint coding network, the last 128 dimensions of G_param are extracted as the position embedding basis vector. The input data is output as a temporal feature tensor by the Transformer-XL encoder, such as vibration signal → [T, 256].

[0086] In one embodiment, the feature extraction branch in S2 includes a Transformer-XL-based temporal-spatial joint coding network for feature extraction of the temporal signal data stream in the data block, including the following steps:

[0087] The sampling rate is parsed to calculate the time step, and the G_param parameter vector is used to generate a learnable position code as a time-aware embedding vector.

[0088] Initialize the historical perception embedding vector, concatenate the current time perception embedding vector, the corresponding time-series signal data stream, and the historical perception embedding vector, and then compute different attention distributions in parallel based on a multi-head self-attention mechanism. Concatenate the parallel computation results to obtain the attention output vector.

[0089] The vector segment with a preset time step at the end of the attention output vector is used as the historical perception embedding vector for the next time window to update the data, and finally the time series signal features are obtained.

[0090] In one embodiment, the feature extraction branch in S2 includes a 3D convolutional branch network for feature extraction of multimedia data streams in the data block, and the meta-features also include resolution, comprising the following steps:

[0091] The basic convolutional kernel size is determined based on the parsed G_param parameter vector; the basic convolutional kernel size is adaptively adjusted by combining the resolution information in the meta-features to obtain a dynamically depth-separable convolutional kernel group.

[0092] Multimedia data streams are input into a dynamically depth-separable convolutional kernel group for multi-level convolution processing to obtain convolutional output vectors.

[0093] The convolutional output vector is fed into a dual-path attention network; the spatial attention branch generates a spatial saliency map through global average pooling and calculates the importance weight of each pixel region using the Sigmoid function; the temporal attention branch calculates the inter-frame correlation matrix, performs feature weighted fusion after Softmax normalization and combining the importance weights, and finally obtains multimedia features.

[0094] In one embodiment, prior to S3, a step of feature alignment of data at different sampling rates using a learnable location embedding matrix is ​​included:

[0095] Based on preset rules, the sampling rate in multiple meta-feature vectors is filtered to determine the global reference time axis, and data blocks from multiple data sources are mapped to a unified time grid.

[0096] A time interpolation vector is obtained based on a learnable location embedding matrix to represent the mapping relationship from different sampling rates to the global reference time axis;

[0097] Generate up / down sampling granularity sequences for generating meta-feature vectors corresponding to different data blocks based on time interpolation vectors;

[0098] The feature-aligned data stream is calculated based on the generated upsampling / downsampling granularity sequence and used as the input to S3.

[0099] This embodiment improves the consistency of multi-source data in the spatiotemporal dimension by performing feature alignment processing on data with different sampling rates, constructing a unified temporal grid, and combining it with a learnable location embedding matrix for time interpolation, thereby improving the accuracy and reliability of the fusion results.

[0100] In one embodiment, S3 includes:

[0101] Create a cross-modal knowledge graph structure, including nodes and edges; where each node represents a modal feature of a single modality, and the edges represent the semantic similarity measure between nodes;

[0102] Modal features extracted from different modal data blocks are input into the cross-modal semantic graph generator in the form of tensors;

[0103] A contrastive learning framework is used to calculate the semantic similarity metric between current modal features and construct a similarity matrix. The cross-modal knowledge graph structure is then initialized based on the similarity matrix.

[0104] New modal features are acquired in real time. If the similarity metric between the new modal feature and an existing node in the initialized cross-modal knowledge graph structure exceeds a threshold, the new feature is clustered under the existing node to form a semantic cluster; otherwise, a new node is created.

[0105] If the similarity metric between the new node and an existing node in the initialized cross-modal knowledge graph structure is higher than threshold 2, and the similarity threshold 1 is higher than the similarity threshold 2, then an edge connecting the new node and the existing node is added to the initialized cross-modal knowledge graph structure, the weight of the edge is set to the similarity metric, and the initialized cross-modal knowledge graph structure is updated.

[0106] like Figure 3 As shown, the cross-modal knowledge graph structure is obtained by using vibration, temperature, and current characteristics as initial nodes and updating for newly added nodes.

[0107] In one embodiment, S4 includes:

[0108] Extract the semantic clusters of each node from the cross-modal knowledge graph and merge the semantic clusters into a unified feature vector;

[0109] Extracting edge similarity metrics from cross-modal knowledge graphs;

[0110] For each node i, the feature vector of node i is updated by using its adjacent nodes j connected by edges and the similarity metric between them, through multiple stacked GCN layers. The updated feature representation of node i is as follows:

[0111]

[0112] In the formula, v i (l) v is the feature vector of node i in the l-th layer. j (l)Let be the feature vector of node j in layer l, W be the similarity measure between node i and node j, σ be the activation function (such as ReLU), deg(i) and deg(j) be the degrees of node i and node j respectively, and neighbors(i) be the set of neighboring nodes of node i.

[0113] A global aggregation operation is performed on the updated feature representations of all nodes to generate a joint feature tensor.

[0114] This embodiment uses a multi-layer graph convolutional network to aggregate and calculate node features in a cross-modal knowledge graph, generating a joint feature tensor. This not only improves the efficiency of fusion computation but also provides high-quality fusion feature inputs for subsequent intelligent decision-making, prediction, or control tasks, thereby enhancing the real-time response capability and intelligence level of the entire system.

[0115] A second aspect of the present invention also provides a deep learning-based multi-source heterogeneous data fusion processing system according to the first aspect of the present invention, comprising:

[0116] The data access and preprocessing module is used to acquire multi-source sensor data streams from IoT terminals, extract modal description vectors from the multi-source sensor data streams, generate meta-features, and independently encapsulate each data source into a data block formed by combining {data stream, meta-feature}.

[0117] The dynamic path allocation module is used to dynamically allocate processing paths based on the meta-feature vectors in the data block, and to extract modal features by inputting parallel feature extraction branches into data blocks of different modalities respectively.

[0118] The cross-modal semantic graph generation module is used to input the modal features corresponding to the extracted data blocks of different modalities into the cross-modal semantic graph generator. Through comparative learning, the semantic similarity measure between features is extracted to obtain a dynamically updated cross-modal knowledge graph.

[0119] The joint feature tensor generation module is used to perform multi-level graph convolution calculations on modal features based on cross-modal knowledge graphs to generate joint feature tensors.

[0120] The second aspect of the present invention is used to perform all the steps of the first aspect of the present invention.

[0121] The following is an example of the specific implementation process of this invention in the field of wind power monitoring:

[0122] The system receives the gearbox's infrared video stream (5fps) and vibration signal (10kHz), which are then preprocessed as follows:

[0123] The infrared branch extracts hot spot features (local temperature 105℃, background temperature 72℃). The cosine similarity between the hot spot and the "local overheating" node is calculated to be 0.92 (added by the "hot spot-wear" edge weight of 0.85) through the cross-modal semantic graph generator, thus completing the feature mapping.

[0124] The vibration branch detected a 32Hz impact component (amplitude 4.2g), and its Euclidean distance to the "tooth surface wear" node was only 0.08 (threshold < 0.1), establishing a strong binding relationship.

[0125] The system retrieves the first-order neighbors of the "local overheating" node, obtains the fault feature vector [-0.3, 1.2] of the "cooling failure" node (edge ​​weight 0.7), and obtains the feature vector [0.8, -0.5] of the "insufficient lubrication" node (edge ​​weight 0.9). Weighted aggregation is performed, and the output features carry the semantics of insufficient lubrication.

[0126] Analysis of the cross-modal edge weights (w = 0.85) for "local overheating" and "tooth surface wear":

[0127] Construct a dual-modal fusion gating system: G = σ(0.85 × MLP([F_hotspot, F_vibration])) = 0.69; Perform gating fusion: F_fuse = 0.69 × [0.42, 0.07] + 0.31 × [-0.2, 1.8] = [0.24, 0.58], where visual features account for 69%.

[0128] After encapsulating the joint features according to the knowledge graph hierarchy, the joint feature tensor is parsed:

[0129] The tooth surface condition value of 1.25 is greater than the fault threshold of 0.8 (the historical average fault value is 1.3); the correlation matrix [0][1] = 0.9 indicates a strong correlation between temperature and wear, thus triggering a secondary warning: "Insufficient lubrication leads to progressive wear of the tooth surface". This achieves early prediction of gear tooth breakage under a multi-sensor fusion discrimination mechanism, reducing the false alarm rate to 9.8%.

[0130] In addition, this invention can be widely applied to various Internet of Things (IoT) scenarios such as intelligent transportation, industrial monitoring, smart healthcare, and smart homes. It can effectively improve the system's perception accuracy and decision-making ability in complex environmental conditions, and has good application prospects and promotion value.

[0131] The above provides a detailed description of the multi-source heterogeneous data fusion processing method and system based on deep learning provided by the present invention. Specific examples are used in this embodiment to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0132] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in these embodiments may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for fusing and processing multi-source heterogeneous data based on deep learning, characterized in that, Includes the following steps: S1: Obtain multi-source sensor data streams from IoT terminals, extract modal description vectors from the multi-source sensor data streams, generate meta-features, and independently encapsulate each data source into a data block formed by combining {data stream, meta-feature}. S2: Dynamically allocate processing paths based on the meta-feature vectors in the data block, and input parallel feature extraction branches to extract modal features for data blocks of different modalities respectively; S3: Input the modal features corresponding to the extracted data blocks of different modalities into the cross-modal semantic graph generator, and extract the semantic similarity measure between features through comparative learning to obtain a dynamically updated cross-modal knowledge graph; S4: Perform multi-layer graph convolution calculation on modal features based on cross-modal knowledge graph to generate joint feature tensors.

2. The method for multi-source heterogeneous data fusion processing based on deep learning according to claim 1, characterized in that, The meta-features include at least: data type, sampling rate, dimension, and physical unit.

3. The method for multi-source heterogeneous data fusion processing based on deep learning according to claim 1, characterized in that, In S1, new data streams from sensor sources of the Internet of Things (IoT) terminal are received in real time.

4. The method for multi-source heterogeneous data fusion processing based on deep learning according to claim 2, characterized in that, The S2 step involves dynamically allocating processing paths to the data stream within the data block using a gating attention mechanism, including the following steps: By retrieving the pre-trained embedding table, embedding mapping is performed on categorical features, and numerical features are normalized. After concatenating all features, a unified embedding vector is output through a fully connected layer. Based on the generated unified embedding vector, the path selection gating G_path and G_param are triggered to calculate and generate the branch activation probability vector and the branch parameter vector. The feature extraction branch into which the data stream corresponding to the current meta-feature vector enters is determined based on the branch activation probability vector. The computational parameters of the feature extraction branch being entered are dynamically initialized based on the branch parameter vector.

5. The method for multi-source heterogeneous data fusion processing based on deep learning according to claim 4, characterized in that, The feature extraction branch in S2 includes a time-space joint coding network based on Transformer-XL, used to extract features from the time-series signal data stream in the data block, including the following steps: The sampling rate is parsed to calculate the time step, and the G_param parameter vector is used to generate a learnable position code as a time-aware embedding vector. Initialize the historical perception embedding vector, concatenate the current time perception embedding vector, the corresponding time-series signal data stream, and the historical perception embedding vector, and then calculate different attention distributions in parallel based on a multi-head self-attention mechanism. Concatenate the parallel calculation results to obtain the attention output vector. The vector segment with a preset time step at the end of the attention output vector is used as the historical perception embedding vector for the next time window to update the data, and finally the time series signal features are obtained.

6. The method for multi-source heterogeneous data fusion processing based on deep learning according to claim 4, characterized in that, The feature extraction branch in S2 includes a 3D convolutional branch network for feature extraction of the multimedia data stream in the data block. The meta-features also include resolution, and include the following steps: The basic convolutional kernel size is determined based on the parsed G_param parameter vector; the basic convolutional kernel size is adaptively adjusted by combining the resolution information in the meta-features to obtain a dynamically depth-separable convolutional kernel group. The multimedia data stream is input into the dynamic depth-separable convolution kernel group for multi-level convolution processing to obtain the convolution output vector. The convolutional output vector is fed into a dual-path attention network; The spatial attention branch generates a spatial saliency map through global average pooling and uses the sigmoid function to calculate the importance weight of each pixel region. The temporal attention branch calculates the inter-frame correlation matrix, and after Softmax normalization, it performs feature weighted fusion in combination with the importance weights to finally obtain multimedia features.

7. The method for multi-source heterogeneous data fusion processing based on deep learning according to claim 1, characterized in that, Before step S3, the method further includes a step of feature alignment of data at different sampling rates using a learnable position embedding matrix: Based on preset rules, the sampling rate in multiple meta-feature vectors is filtered to determine the global reference time axis, and data blocks from multiple data sources are mapped to a unified time grid. A time interpolation vector is obtained based on a learnable location embedding matrix to represent the mapping relationship from different sampling rates to the global reference time axis; Based on the time interpolation vector, a sequence of up / down sampling granularity is generated to generate meta-feature vectors corresponding to different data blocks; The feature-aligned data stream is calculated based on the generated upsampling / downsampling granularity sequence and used as the input to S3.

8. The method for multi-source heterogeneous data fusion processing based on deep learning according to claim 1, characterized in that, S3 includes: Create a cross-modal knowledge graph structure, including nodes and edges; where each node represents a modal feature of a single modality, and the edges represent the semantic similarity measure between nodes; Modal features extracted from different modal data blocks are input into the cross-modal semantic graph generator in the form of tensors; A contrastive learning framework is used to calculate the semantic similarity metric between current modal features and construct a similarity matrix. The cross-modal knowledge graph structure is then initialized based on the similarity matrix. New modal features are acquired in real time. If the similarity metric between the new modal features and existing nodes in the initialized cross-modal knowledge graph structure exceeds a threshold, the new features are clustered under the existing nodes to form a semantic cluster; otherwise, a new node is created. If the similarity metric between the new node and an existing node in the initialized cross-modal knowledge graph structure is higher than threshold two, and the similarity threshold one is higher than the similarity threshold two, then an edge is added to the initialized cross-modal knowledge graph structure to connect the new node and the existing node, the weight of the edge is set to the similarity metric, and the initialized cross-modal knowledge graph structure is updated.

9. A method for multi-source heterogeneous data fusion processing based on deep learning according to claim 8, characterized in that, S4 includes: Extract the semantic clusters of each node from the cross-modal knowledge graph and merge the semantic clusters into a unified feature vector; Extracting edge similarity metrics from cross-modal knowledge graphs; For each node i, the feature vector of node i is updated by using the adjacent nodes j connected by the edge and the similarity measure between them, through stacked multi-layer GCN layers, to obtain the updated feature representation of node i. A global aggregation operation is performed on the updated feature representations of all nodes to generate a joint feature tensor.

10. A deep learning-based multi-source heterogeneous data fusion processing system according to any one of claims 1-9, characterized in that, include: The data access and preprocessing module is used to acquire multi-source sensor data streams from IoT terminals, extract modal description vectors from the multi-source sensor data streams, generate meta-features, and independently encapsulate each data source into a data block formed by combining {data stream, meta-feature}. The dynamic path allocation module is used to dynamically allocate processing paths based on the meta-feature vectors in the data block, and to extract modal features by inputting parallel feature extraction branches into data blocks of different modalities respectively. The cross-modal semantic graph generation module is used to input the modal features corresponding to the extracted data blocks of different modalities into the cross-modal semantic graph generator. Through comparative learning, the semantic similarity measure between features is extracted to obtain a dynamically updated cross-modal knowledge graph. The joint feature tensor generation module is used to perform multi-head cross-attention calculation on modal features based on cross-modal knowledge graphs to generate joint feature tensors.

Images