Data fusion characterization method based on multi-modal features

Abstract

The invention discloses a data fusion characterization method based on multi-modal features, and relates to the technical field of data processing, and the method comprises the steps: collecting multi-modal data, carrying out the preprocessing and preliminary feature extraction, and generating a preliminary feature vector set; according to the preliminary feature vector set, taking each modal feature as a graph node, establishing a graph edge according to time, space and semantic relationships, initializing graph node features and graph edge weights, and constructing a dynamic graph structure; performing feature dependency relationship calculation on the dynamic graph structure by using a graph attention network to obtain an attention weight matrix, and performing weighted updating on graph node features to generate an updated graph node feature set; and inputting the updated graph node feature set into a variational auto-encoder, encoding the updated graph node feature set to generate a mean vector and a variance vector, and performing calculation through a re-parameterization sampling method to generate a potential representation set. According to the invention, the precision and interpretability of feature fusion are improved, and the representation capability of fused features is enhanced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data fusion and characterization method based on multimodal features. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal data fusion has gradually become an important research direction in information processing and intelligent decision-making. Multimodal data fusion aims to comprehensively utilize information from different sensors, data sources, or signal channels to construct more comprehensive and accurate feature representations. Existing multimodal feature fusion methods mainly include feature-level fusion, decision-level fusion, and model-level fusion approaches. By aligning and integrating features from heterogeneous data sources such as vision, audio, text, and sensor signals, a unified representation is formed to support tasks such as classification, recognition, and prediction. Multimodal fusion has achieved significant application results in fields such as intelligent monitoring, sentiment analysis, autonomous driving, and medical image analysis, improving the accuracy and robustness of tasks.

[0003] While existing technologies can achieve a certain degree of feature integration in multimodal feature fusion, they generally lack the ability to effectively model and dynamically update the structured dependencies between modalities. Most methods only use fixed weights or shallow mappings in the fusion stage, making it difficult to capture the dynamic characteristics of modal features changing over time and space, resulting in insufficient feature representation capabilities in complex tasks. Furthermore, when dealing with high-dimensional, multi-source heterogeneous data, existing methods often fail to adequately guarantee the distributional consistency of the fused representations, which restricts the performance of subsequent tasks such as classification, prediction, and anomaly detection. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a data fusion and representation method based on multimodal features to solve the problem of dynamic fusion and real-time updating of modal features.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a data fusion representation method based on multimodal features, which includes: Collect multimodal data, perform preprocessing and preliminary feature extraction, and generate a preliminary feature vector set; Based on the initial feature vector set, each modal feature is used as a graph node. Graph edges are established according to temporal, spatial, and semantic relationships. The graph node features and graph edge weights are initialized to construct a dynamic graph structure. Using a graph attention network, feature dependencies are calculated on a dynamic graph structure to obtain an attention weight matrix. The graph node features are then updated with weights to generate an updated graph node feature set. The updated graph node feature set is input into the variational autoencoder to encode the updated graph node feature set, generate mean vector and variance vector, and generate latent representation set by reparameterized sampling method; The latent representation set is input into the discriminator for adversarial training to optimize the consistency of the latent representation distribution. The parameters of the variational autoencoder are then iteratively updated through the feedback of the discriminator to obtain the optimized latent representation set. The optimized latent representation set is classified, trend predicted, and abnormal pattern identified in real time to obtain task results. These results are then used as initial reference features for the next time window to enter the iterative loop, forming a dynamically updated fusion representation.

[0007] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for generating the preliminary feature vector set are as follows: Noise suppression, missing value imputation, and anomaly removal are performed on the multimodal data to generate a cleaned multimodal dataset; The cleaned multimodal dataset is normalized and modality-specific enhancement is performed, and preliminary feature vectors for each modality are extracted to generate feature sets for each modality. The feature sets of each modality are aligned and their dimensions are unified. The correlation between modalities is enhanced by temporal, spatial and semantic relationships between modalities, resulting in a feature set with enhanced relationships. The feature set for relationship enhancement is filtered according to modality importance and context relevance, and redundant and noisy features are removed to generate a preliminary feature vector set.

[0008] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for constructing the dynamic graph structure are as follows: Each modal feature in the initial feature vector set is mapped to a graph node to generate an initial node set; Calculate the temporal, spatial, and semantic relationships between modalities based on the initial node set, establish graph edges, and generate a preliminary graph structure; Normalize and enhance the features of each graph node in the initial graph structure to generate an initial node feature set; The weights of the graph edges are initialized based on temporal, spatial, and semantic relationships to form a weighted dynamic graph structure; Local structural optimization is performed on the weighted dynamic graph structure to generate a new dynamic graph structure.

[0009] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for obtaining the attention weight matrix are as follows: The dynamic graph structure is input into the graph attention network, and the features of the neighboring nodes of each graph node are aggregated and the attention coefficients are calculated to generate the initial node attention weights. The graph edge weights and the initial node attention weights are fused and weighted and normalized to generate the optimized node attention weight matrix; The optimized node attention weights are computed in parallel in different attention subspaces to generate a multi-head node attention weight set, which is then fused to obtain the attention weight matrix.

[0010] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for generating the updated graph node feature set are as follows: The features of graph nodes are mapped and combined with the features of neighboring nodes according to the attention weight matrix to generate a node neighborhood feature set; The neighborhood feature set of the nodes is normalized and its proportion is adjusted to obtain a balanced neighborhood feature set, which is then fused with the graph node features according to a set proportion to form a preliminary updated node feature representation. The initially updated node feature representation is subjected to nonlinear transformation to generate a nonlinear node feature representation. The nonlinear node feature representation is expanded in parallel in the multi-head feature space to capture the structural dependencies of different subspaces, obtain the multi-head updated node feature set, and then splice and fuse them to generate the updated graph node feature set.

[0011] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for generating the mean vector and variance vector are as follows: The updated graph node feature set is input into the encoder end of the variational autoencoder to generate a preliminary hidden representation set; The initial hidden representation set is transformed using a first nonlinear mapping and a fully connected network to generate a mean vector. The mean vector and the initial hidden representation set are combined, and a variance vector is generated through a second nonlinear mapping.

[0012] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for generating the latent representation set are as follows: Randomly perturb the mean vector and variance vector to generate a random noise vector; By combining the random noise vector with the variance vector and performing perturbation mapping, a potential perturbation representation of the node is formed. The mean vector is integrated into the node latent perturbation representation to generate the node latent representation, and then summarized to form a latent representation set.

[0013] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for obtaining the optimized latent representation set are as follows: The latent representation set is input into the discriminator, and each latent representation is scored as true or false to generate a preliminary discrimination result; The deviation between the latent representation and the true distribution is evaluated based on the discrimination results, and feedback information is generated. The feedback information is input into the variational autoencoder, and the parameters of the variational autoencoder are iteratively adjusted to optimize the distribution consistency of the latent representation and generate an optimized latent representation set.

[0014] As a preferred embodiment of the data fusion and representation method based on multimodal features described in this invention, the specific steps for obtaining the task results are as follows: Decode and map the optimized latent representation set to generate a task feature vector set; Perform category and state analysis on the task feature vector set to generate preliminary classification results; The task feature vector set and the optimized latent representation set are combined and used for prediction to generate future prediction results; The system identifies abnormal patterns and deviations in the task feature vector set and generates anomaly detection results. The preliminary classification results, future prediction results, and anomaly detection results are integrated to obtain the task results.

[0015] As a preferred embodiment of the data fusion representation method based on multimodal features described in this invention, the specific steps for forming a dynamically updated fusion representation are as follows: The task results are combined with the optimized latent representation set to form initial reference features; New node latent representations are constructed based on the initial reference features and integrated to form a new latent representation set. Simultaneously, decoding and mapping are performed to generate a task feature vector set. Within the new time window, the task feature vector set is classified, trended, and anomaly pattern identified to obtain the window task results. The window task results are merged with the new set of potential representations, the initial reference features for the next time window are updated, and the process is repeated iteratively to form a dynamically updated fused representation.

[0016] The beneficial effects of this invention are as follows: by mapping the initial feature vector set to graph nodes and establishing a weighted dynamic graph structure, and using a graph attention network to calculate feature dependencies, dynamic structured modeling and context relevance capture of multimodal features are realized, thereby improving the accuracy and interpretability of feature fusion, enhancing the representational ability of fused features, and providing a more accurate and robust feature foundation for subsequent potential representation generation and task execution. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a data fusion representation method based on multimodal features; Figure 2 A flowchart for the initial feature vector set; Figure 3 A flowchart for constructing a dynamic graph structure; Figure 4 A flowchart for generating the potential representation set. Detailed Implementation

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0022] Reference Figures 1-4 This is one embodiment of the present invention, which provides a data fusion representation method based on multimodal features, including the following steps: S1. Collect multimodal data, perform preprocessing and preliminary feature extraction, and generate a preliminary feature vector set.

[0023] S1.1 Perform noise suppression, missing value imputation, and anomaly removal on the multimodal data to generate a cleaned multimodal dataset.

[0024] Specifically, the multimodal data acquisition process includes simultaneously acquiring multiple types of data from different sensory sources, such as visual data (e.g., images and videos), sound data (e.g., speech signals and ambient sounds), sensor data (e.g., temperature, pressure, acceleration, and displacement), and text data (e.g., logs and descriptive information). Kalman filtering is used to suppress noise in the multimodal data, removing high-frequency random disturbances and smoothing the data curves to obtain noise-suppressed multimodal data. Missing values ​​are then detected according to the time series, and linear interpolation is used to impute these missing values, generating complete multimodal data. Anomaly removal based on quantiles is then performed on the completed multimodal data, identifying and removing outliers that exceed reasonable ranges, resulting in a cleaned multimodal dataset.

[0025] S1.2 Normalize and modality-specific enhancement processes on the cleaned multimodal dataset, and extract the preliminary feature vectors for each modality to generate feature sets for each modality.

[0026] Specifically, the cleaned multimodal datasets are normalized. For example, visual data is normalized by pixel normalization or zero-mean normalization, audio data is normalized by frame energy normalization, sensor data is normalized by Z-score normalization, and text representation vectors are normalized by L2 norm normalization. Modality-specific enhancement processing is then implemented. For example, visual data is subjected to histogram equalization or convolutional filtering, audio data is extracted by Mel-frequency cepstral coefficients, sensor data is subjected to bandpass filtering or temporal sliding smoothing, and text data is segmented to extract BERT model sentence vectors. Preliminary feature vectors are extracted from the enhanced multimodal data, and the preliminary feature vectors under the same modality are aggregated to form feature sets for each modality.

[0027] S1.3 Align and unify the dimensions of each modality feature set, and enhance the correlation between modalities through temporal, spatial and semantic relationships to generate a feature set with enhanced relationships.

[0028] Specifically, the feature sets of each modality are aligned through time synchronization and resampling, and sampling rate differences are handled by dynamic time warping; then, spatial registration is performed based on geometric calibration parameters to map different modalities to a unified coordinate system; linear projection is used to unify dimensions, and a unified dimensional feature set is generated through L2 standardization; modal relationship weights are calculated based on cosine similarity or mutual information, and weighted fusion is performed on the unified dimensional feature set to output a feature set with enhanced relationships.

[0029] S1.4. The feature set for relationship enhancement is filtered according to modal importance and context relevance, and redundant and noisy features are removed to generate a preliminary feature vector set.

[0030] Specifically, the cleaned multimodal dataset is processed using a principal component analysis contribution rate and information entropy weighted calculation method to generate modal importance. The temporal and spatial dependencies of multimodal features are analyzed using a sliding window mutual information calculation method and an autocorrelation coefficient average calculation method to generate contextual relevance. Modal importance and contextual relevance are combined to obtain a feature score set. A signal-to-noise ratio (SNR) calculation method is used to identify and remove noisy features from the feature score set, generating a noise-removed feature set. A Pearson correlation coefficient calculation method is used to generate a feature correlation matrix, identifying redundant feature pairs with correlation coefficients exceeding a set threshold. Features with high scores are retained from these redundant feature pairs; for example, the correlation coefficient between temperature and humidity is 0.92, identifying them as redundant feature pairs, and only the one with the higher score is retained, forming a redundant feature set. The redundant feature set is then sorted according to its comprehensive feature score using a sorting method, and features with high scores are selected and combined to generate a preliminary feature vector set.

[0031] S2. Based on the preliminary feature vector set, each modal feature is used as a graph node. Graph edges are established according to time, space and semantic relationships. The graph node features and graph edge weights are initialized to construct a dynamic graph structure.

[0032] S2.1 Map each modal feature in the initial feature vector set to a graph node to generate an initial node set.

[0033] Specifically, each modal feature in the initial feature vector set is transformed into a corresponding node vector using existing feature mapping methods, such as vector mapping methods, to form a node feature. A unique identifier is assigned to each node feature, and node attribute records are established according to the modal category to generate an initial node set containing the node vectors and attribute information of all modal features.

[0034] S2.2 Calculate the temporal, spatial and semantic relationships between modalities based on the initial node set, establish graph edges and generate a preliminary graph structure.

[0035] Specifically, based on the initial node set, time series similarity calculation methods are used to calculate the temporal relationship between nodes, such as using dynamic time warping to calculate temporal similarity; spatial distance calculation methods are used to calculate the spatial relationship between initial nodes, such as using Euclidean distance to calculate spatial similarity; semantic similarity calculation methods are used to calculate the semantic relationship between initial nodes, such as vector similarity calculation based on cosine similarity; time similarity, spatial similarity, and semantic similarity are integrated into a comprehensive similarity using a weighted method, and the connection relationship between initial nodes is determined based on the comprehensive similarity, establishing graph edges to form a preliminary graph structure, including the initial node set and the set of edges with similarity weights connecting each initial node.

[0036] S2.3 Normalize and enhance the features of each graph node in the preliminary graph structure to generate an initial node feature set.

[0037] Specifically, for each graph node feature in the initial graph structure, a normalization calculation method is used, such as using min-max normalization to map the feature values ​​to the range of 0 to 1. Then, a feature enhancement calculation method is used to process the normalized graph node features, such as using local weighted regression to increase the discriminative power of the features and generate enhanced features. Finally, the normalized and enhanced graph node features are sequentially summarized to form the initial node feature set. S2.4 Initialize the graph edge weights according to time, space and semantic relationships to form a weighted dynamic graph structure.

[0038] Specifically, based on the initial node set and initial node feature set, the similarity between any two graph nodes in time series is calculated using a temporal similarity calculation method, such as obtaining temporal similarity weights through mutual information calculation; the spatial distance between any two graph nodes is calculated using a spatial similarity calculation method, such as obtaining spatial similarity weights through Euclidean distance calculation; the semantic relevance between any two graph nodes is calculated using a semantic similarity calculation method, such as obtaining semantic similarity weights through word vector similarity calculation; the temporal similarity weights, spatial similarity weights, and semantic similarity weights are then weighted and fused to calculate a comprehensive graph edge weight, and the comprehensive graph edge weights between each graph node are assigned a value to form a weighted dynamic graph structure.

[0039] S2.5 Perform local structural optimization on the weighted dynamic graph structure to generate a new dynamic graph structure.

[0040] Specifically, based on the weighted dynamic graph structure, a local structure optimization method is used to process each graph node and its adjacent edges. The neighborhood structure density of each graph node is calculated using a local neighborhood density calculation method. Edges with lower weights are removed using an edge weight comparison method, and the graph edge set is updated. The importance of graph nodes is evaluated based on graph node importance calculation methods, such as a sorting method based on node degree centrality, and the connection relationships of graph nodes are adjusted to optimize the local structure. The optimized graph node and edge set is then reconstructed to generate the dynamic graph structure.

[0041] S3. Using a graph attention network, feature dependencies are calculated on the dynamic graph structure to obtain the attention weight matrix. The graph node features are then updated with weights to generate the updated graph node feature set.

[0042] S3.1 Input the dynamic graph structure into the graph attention network, perform feature aggregation and calculate attention coefficients for the neighboring nodes of each graph node, and generate preliminary node attention weights.

[0043] Specifically, the dynamic graph structure is input into the graph attention network. For each graph node, a node feature vector is extracted, and a feature linear transformation method is used to map the node features to a unified feature space. Based on the graph structure information, a neighbor node sampling method is used to determine the set of neighbor nodes for each graph node. For the feature vectors of each graph node and its neighbor nodes, an attention coefficient calculation method, such as a dot product attention method, is used to calculate the attention coefficient, expressed as: ; in, This represents the attention coefficient between graph node i and its neighboring graph node j. Let i represent the original feature vector of graph node i. Let represent the original eigenvector of neighboring graph node j, W represent the eigenlinear transformation matrix, d represent the dimension of the feature space, i represent the node number, and j represent the node numbers of the neighboring graph nodes of node i. Indicates the transpose operation; The attention coefficients are normalized using the Softmax normalization method to generate preliminary node attention weights.

[0044] S3.2. The graph edge weights and the initial node attention weights are fused and weighted and normalized to generate the optimized node attention weight matrix.

[0045] Specifically, a weighted fusion method is used to add the graph edge weight matrix and the initial node attention weight matrix by weighting them according to preset weight coefficients to generate a fused weight matrix. A normalization method, such as Softmax normalization, is used to normalize each element of the fused weight matrix to obtain a weight normalization matrix. The weight normalization matrix is ​​then output as the optimized node attention weight matrix.

[0046] Among them, the preset weight coefficient is a value determined in advance based on the fusion requirements to balance the contribution ratio of graph edge weights and initial node attention weights.

[0047] S3.3. The optimized node attention weights are computed in parallel in different attention subspaces to generate a multi-head node attention weight set, and then fused to obtain the attention weight matrix.

[0048] Specifically, the optimized node attention weights are divided into multiple attention subspaces based on the number of attention heads. A parallel computing method is used to independently perform weighted feature aggregation in each attention subspace to generate node attention weights for multiple subspaces. A concatenation method is then used to fuse the node attention weights of multiple subspaces to obtain an attention weight matrix.

[0049] The number of attention heads refers to the number of independent attention subspaces computed in parallel in a graph attention network, used to capture the dependencies and diverse information of different feature subspaces.

[0050] S3.4. Map and combine the features of graph nodes with the features of neighboring nodes according to the attention weight matrix to generate a node neighborhood feature set.

[0051] Specifically, based on the corresponding weight values ​​in the attention weight matrix, a weighted summation method is used to fuse the initial node features with the features of neighboring nodes; vector normalization is applied to the weighted fused feature vector to ensure that the scale of the node neighborhood features is consistent; the normalized weighted fused features are combined according to the node order to generate a node neighborhood feature set.

[0052] S3.5 Normalize and adjust the scale of the node neighborhood feature set to obtain a balanced neighborhood feature set, and merge it with the graph node features according to a set scale to form a preliminary updated node feature representation.

[0053] Specifically, Z-Score normalization is used to normalize the node neighborhood feature set to obtain a normalized neighborhood feature set; the normalized neighborhood feature set is scaled according to a set ratio coefficient using a scaling method to obtain a balanced neighborhood feature set; a weighted fusion method is used to fuse the balanced neighborhood feature set with the graph node features according to a set fusion ratio to generate a preliminary updated node feature representation.

[0054] It should also be noted that the set ratio coefficient is a numerical parameter used to control the contribution of the balanced neighborhood feature set and graph node features in the fusion process. It is usually determined by empirical values, cross-validation or optimization algorithms to ensure that the fused graph node features retain the important information of the neighborhood features while taking into account the expressive power of the original node features. Setting the fusion ratio is a parameter used to determine the weight of different feature sources during the feature fusion process of graph nodes. Its function is to balance the influence of neighborhood features and the node's own features. It is usually obtained through empirical setting or optimization calculation based on validation data.

[0055] S3.6 Perform nonlinear transformation on the initially updated node feature representation to generate a nonlinear node feature representation.

[0056] Specifically, for the initially updated node feature representation, a nonlinear transformation is performed using a nonlinear activation function. This includes applying activation functions sequentially to each node feature vector to calculate the nonlinear mapping value and generate the corresponding nonlinear node feature vector. For example, the ReLU activation function is used to set the negative values ​​in each node feature vector to zero and retain the positive values, or the Sigmoid activation function is used to map each node feature vector to the (0,1) interval. The nonlinear feature vectors of all nodes are then aggregated to form a nonlinear node feature representation.

[0057] S3.7. The nonlinear node feature representation is expanded in parallel in the multi-head feature space to capture the structural dependencies of different subspaces, obtain the multi-head updated node feature set, and then splice and fuse them to generate the updated graph node feature set.

[0058] Specifically, the nonlinear node feature representation is copied to multiple attention subspaces. A parallel computing method is used to perform weighted mapping on each node feature vector in each subspace to capture the structural dependencies between different subspaces, generating multiple sets of subspace node feature vectors to form a multi-head updated node feature set. A concatenation method is used to concatenate the multiple sets of subspace node feature vectors according to their dimensions, and a weighted fusion method is used to normalize the concatenated feature vectors to obtain the updated graph node feature set.

[0059] It should be noted that by using graph attention networks to calculate feature dependencies on dynamic graph structures, the information transfer weights between nodes can be dynamically and adaptively allocated, significantly improving the expressive power and fusion accuracy of intermodal dependencies. Dynamic structured modeling between multimodal features is realized, enabling the explicit and dynamic characterization of complex temporal, spatial, and semantic dependencies between different modalities. Furthermore, by updating node features with weights, the expressive power and interpretability of feature fusion are improved.

[0060] S4. Input the updated graph node feature set into the variational autoencoder to encode the updated graph node feature set, generate the mean vector and variance vector, and calculate them through the reparameterized sampling method to generate the latent representation set.

[0061] S4.1 Input the updated graph node feature set into the encoder end of the variational autoencoder to generate a preliminary hidden representation set.

[0062] Specifically, the updated graph node feature set is input into the encoder of the variational autoencoder. A layer-by-layer nonlinear mapping method is used to perform linear transformation and activation function processing on the feature vector of each graph node to obtain the mean vector and variance vector of each node feature. Based on the mean vector and variance vector, latent variables are generated through random sampling to form a preliminary hidden representation vector. The set of preliminary hidden representation vectors of all graph nodes constitutes the preliminary hidden representation set.

[0063] S4.2. Through the first nonlinear mapping and the fully connected network, the initial hidden representation set is subjected to feature transformation processing to generate the mean vector.

[0064] Specifically, the initial hidden representation set is input into the first nonlinear mapping, and a nonlinear transformation is performed using an activation function to obtain the nonlinearly transformed feature representation. The nonlinearly transformed feature representation is then input into a fully connected network, and a linear transformation operation is performed using a weight matrix and a bias vector to obtain the mean vector of each graph node. For example, the ReLU function is used as the first nonlinear mapping, and a fully connected operation is performed using a weight matrix of size 128×64 to generate the mean vector.

[0065] S4.3 Combine the mean vector and the initial hidden representation set, and generate the variance vector through a second nonlinear mapping.

[0066] Specifically, the mean vector and the initial hidden representation set are concatenated element-wise to generate a concatenated feature representation. The concatenated feature representation is then input into a second nonlinear mapping and subjected to a nonlinear transformation using an activation function to obtain a nonlinearly transformed feature representation. This nonlinearly transformed feature representation is then input into a fully connected network, where a linear transformation operation is performed using a weight matrix and a bias vector to generate a variance vector for each graph node. For example, the sigmoid function can be used as the second nonlinear mapping, and a fully connected operation is performed using a weight matrix of size 128×64 to generate the variance vector.

[0067] S4.4. Randomly perturb the mean vector and variance vector to generate a random noise vector.

[0068] Specifically, based on each element of the variance vector, a random perturbation vector is generated using a standard normal distribution random number generation method, such as generating a set of random numbers with the same dimension as the variance vector; based on a weighted perturbation calculation method, the random perturbation vector and the variance vector are weighted to obtain the perturbation vector; based on a vector fusion method, the mean vector and the perturbation vector are fused to form a random noise vector.

[0069] S4.5 Combine the random noise vector with the variance vector and perform perturbation mapping to form a node potential perturbation representation.

[0070] Specifically, according to the perturbation mapping rule, each element of the variance vector is associated with the corresponding random noise vector element to form an initial perturbation pair. For example, if the variance vector is [0.1, 0.2] and the random noise vector is [0.8, -0.5], then the perturbation pairs (0.1, 0.8) and (0.2, -0.5) are formed. According to the proportional mapping, each perturbation pair is mapped to a perturbation value. Using the vector integration method, all perturbation values ​​are combined to form a perturbation mapping vector. The perturbation mapping vector is used as the node potential perturbation representation, and a node potential perturbation representation set is generated.

[0071] Among them, the perturbation mapping rule refers to a predefined method used to associate the random noise vector with the corresponding elements in the variance vector and generate perturbation values ​​through a specific mapping method.

[0072] S4.6 Integrate the mean vector into the node latent perturbation representation to generate the node latent representation, and summarize them to form a latent representation set.

[0073] Specifically, the mean vector and the node latent perturbation representation are paired according to a preset mapping relationship. Each element in the mean vector is mapped to the corresponding element in the node latent perturbation representation to form a node latent representation. All node latent representations are collected sequentially according to the node order to generate a complete set of node latent representations, forming a latent representation set. An example includes arranging each node latent representation according to the node order and uniformly encoding it into a latent representation set.

[0074] It should also be noted that the pre-defined mapping relationship refers to establishing a clear association between the feature values ​​in one feature set and the feature values ​​in another feature set based on pre-determined feature correspondence rules, so as to ensure that the integration and transformation between features are consistent and controllable.

[0075] S5. Input the latent representation set into the discriminator for adversarial training to optimize the consistency of the latent representation distribution. Then, update the parameters of the variational autoencoder iteratively through the feedback of the discriminator to obtain the optimized latent representation set.

[0076] S5.1 Input the latent representation set into the discriminator, score each latent representation for true or false, and generate preliminary discrimination results.

[0077] Specifically, each latent representation in the latent representation set is read sequentially. The read latent representation is then input into the discriminator's discriminator layer. The latent representation features are nonlinearly mapped according to a preset discriminator function to generate a discriminator feature vector. The discriminator feature vector is then input into the discriminator's output layer. A scoring operation is performed according to the preset discriminator function to obtain the true / false judgment score of the corresponding latent representation. The discriminator scores of all latent representations are recorded in sequence to form a preliminary discrimination result. For example, a discriminator score with a value between 0 and 1 is generated for each latent representation.

[0078] It should also be noted that the discriminator is constructed and judged based on the generative adversarial network, where the discriminator is a standard component used to distinguish the authenticity of real data from generated data, and generates an authenticity score based on the discrimination rules and feature maps obtained during training. The preset discriminant function is determined based on the basic principles of discriminators in existing technologies. It is used to evaluate the authenticity of the input latent representation or generated data, including the difference between the distribution characteristics of real data and the distribution characteristics of generated data. It usually adopts a probability estimation function and sets the discrimination rules in combination with the statistical characteristics of historical data or training data to ensure the effectiveness and repeatability of the discrimination process.

[0079] S5.2. Evaluate the deviation between the latent representation and the true distribution based on the discrimination results, and generate feedback information.

[0080] Specifically, based on the preliminary discrimination results and the reference values ​​of the actual data distribution, the difference between the discrimination results and the actual data distribution is quantified using a deviation assessment method. For example, the difference value is calculated using a statistical distance metric to obtain the deviation value for each potential representation. All deviation values ​​are then summarized to form a set of feedback information to describe the overall deviation of the potential representation relative to the actual distribution. For example, a deviation matrix or deviation vector is generated as feedback information.

[0081] It should also be noted that the true data distribution refers to the true probability distribution of the data collected or observed in the feature space in actual application scenarios, which is used as a reference standard for judgment and evaluation.

[0082] S5.3 Input the feedback information into the variational autoencoder, iteratively adjust the parameters of the variational autoencoder, optimize the distribution consistency of the latent representation, and generate the optimized latent representation set.

[0083] Specifically, the feedback information is input into the variational autoencoder (VAE), and the optimized graph node feature set is fed into the encoder to generate a preliminary hidden representation set. The feedback information is then combined with the preliminary hidden representation set, and the error is calculated using a preset loss function value, such as a weighted combination of reconstruction error and distribution difference, to obtain gradient information. Based on the gradient information, stochastic gradient descent is used to update the parameters of the encoder and decoder of the VAE sequentially. This process is repeated until the training rounds are completed or the loss function reaches convergence. After the last iteration, an optimized latent representation set is formed, ensuring that the optimized latent representation set contains the distribution consistency of the node latent perturbation representations and the discriminator feedback adjustment information.

[0084] It should also be noted that the specific steps for setting the default loss function value include: calculating the reconstruction error based on the training data and the generated data, for example, using the cross-entropy method to obtain the reconstruction error value; calculating the distribution difference based on the latent representation distribution and the real data distribution, for example, using the Kullback-Leibler divergence to obtain the distribution difference value; and weighting the reconstruction error and the distribution difference according to the set weights to obtain the default loss function value. It should be noted that by using a discriminator to determine whether the latent representations are true or false, and dynamically adjusting the VAE encoder parameters based on the discrimination results, the distribution of the latent representations is made closer to the real distribution. This achieves dynamic optimization and consistency improvement of the latent representation distribution, making the generated latent features more discriminative and representative, thereby enhancing the accuracy and stability of subsequent classification, prediction, and anomaly detection tasks.

[0085] S6. Perform real-time classification, trend prediction, and abnormal pattern recognition on the optimized latent representation set to obtain the task results, and use them as the initial reference features for the next time window to enter the loop iteration, forming a dynamically updated fusion representation.

[0086] S6.1 Decode and map the optimized latent representation set to generate a task feature vector set.

[0087] Specifically, the optimized latent representation set is input into the decoder of the variational autoencoder. Each latent representation in the optimized latent representation set is read sequentially, and nonlinear inverse mapping is performed according to the decoding mapping rules to generate a preliminary task feature vector corresponding to each latent representation. All preliminary task feature vectors are then summarized to form a task feature vector set. For example, each preliminary task feature vector is mapped to the example dimension in the feature space to obtain the task feature vector set.

[0088] Among them, decoding mapping rules refer to the processing method of converting latent representations into target feature representations based on preset mapping relationships.

[0089] S6.2 Perform category and state analysis on the task feature vector set to generate preliminary classification results.

[0090] Specifically, according to the preset classification rules, each task feature vector in the task feature vector set is read in sequence. Each task feature vector is input into the category discrimination method and the state discrimination method for processing. The feature distribution of the feature vector is compared with the classification conditions to generate the category label and state label of each task feature vector. The category labels and state labels of all task feature vectors are summarized to form a preliminary classification result.

[0091] The preset classification rules are based on the feature distribution and judgment conditions of the task feature vectors, and set the criteria for classifying and classifying the task feature vectors.

[0092] S6.3 Combine the task feature vector set and the optimized latent representation set and make predictions to generate future prediction results.

[0093] Specifically, the task feature vector set and the optimized latent representation set are mapped and combined according to the fusion ratio to form a prediction input vector set. Based on historical prediction data and discrimination conditions, the prediction input vector set is processed by feature mapping in the prediction model to obtain a prediction feature set. Based on the classification and trend characteristics of the prediction feature set, future prediction results are generated.

[0094] It should also be noted that a predictive model refers to a mathematical or computational method that uses existing mathematical mapping methods to perform feature mapping and analysis on input data based on existing data characteristics and historical trends, thereby predicting future data change trends or category distributions. Through parameterization, feature fusion, and mapping operations, a predictive output for the target variable is formed.

[0095] S6.4. Identify abnormal patterns and deviation behaviors in the task feature vector set and generate anomaly detection results.

[0096] Specifically, based on the mean, variance, and distribution range of the task feature vector set, the differences between each task feature vector and the mean and distribution range are compared one by one. For example, when the value of a task feature vector exceeds the mean ± two standard deviations, it is marked as an anomaly. Cosine similarity is used to calculate the similarity between task feature vectors. When the similarity result is lower than the set comparison benchmark, it is marked as deviation behavior. The marking information obtained from statistical comparison and similarity calculation is summarized to generate anomaly detection results.

[0097] It should also be noted that the established comparison benchmark refers to the pre-determined reference standard for determining whether the task feature vectors belong to the normal range when identifying abnormal patterns and deviation behaviors in the task feature vector set. The reference standard is obtained by statistically analyzing the distribution characteristics of historical task feature vector data.

[0098] S6.5 Integrate the preliminary classification results, future prediction results, and anomaly detection results to obtain the task results.

[0099] Specifically, the preliminary classification results, future prediction results, and anomaly detection results are aligned according to sample identifiers or timestamps to form an aligned result set. Each result within the aligned result set undergoes unified quantification processing: the preliminary classification results are mapped to a probability confidence vector, the future prediction results are converted into prediction scores or probability distributions, and the anomaly detection results are mapped to anomaly confidence scores. Probability normalization is then used to output a standardized score vector, resulting in a standardized score set. The standardized score set is then concatenated dimensionally to form a fusion input vector. Based on the fusion weights determined by historical performance evaluations, a weighted fusion method is used to sum the fusion input vectors and calculate the fusion confidence, generating a fusion decision vector. Based on the fusion decision vector, the category label, predicted value, and anomaly identifier are selected using the maximum confidence score and summarized as the task results.

[0100] It should also be noted that the preliminary classification results represent the category labels and corresponding probabilities obtained after classifying and analyzing the task feature vector set, which are used to reveal the current category or state of the task object; the future prediction results represent the future trend values ​​or probability distributions obtained by combining the task feature vector set with the optimized latent representation set and making predictions, which are used to characterize the possible changes of the task object in the future; and the anomaly detection results represent the abnormal patterns and deviation behaviors identified in the task feature vector set and their confidence levels, which are used to identify whether the task object has abnormal behavior that is different from the normal pattern.

[0101] S6.6 Combine the task results with the optimized latent representation set to form initial reference features.

[0102] Specifically, the task results and the optimized latent representation set are aligned according to sample identifiers or timestamps to obtain aligned data pairs. Based on the aligned data pairs, the optimized latent representation set is subjected to dimensionality unification processing, and a linear mapping method is used to map the optimized latent representation set to a feature space consistent with the numerical representation of the task results, resulting in mapped latent representation vectors. Confidence vectors are extracted based on the confidence or probability distribution in the task results, and using the confidence vectors as weights, the mapped latent representation vectors are combined with the numerical representation of the task results through weighted fusion or concatenation to obtain a fused vector set. The fused vector set is normalized and subjected to L2 norm constraints to obtain standardized initial reference features. The standardized initial reference features are then subjected to short-term moving average processing to achieve time smoothing, generating initial reference features.

[0103] S6.7 Construct new node latent representations based on the initial reference features and integrate them to form a new latent representation set. Simultaneously, perform decoding mapping to generate a task feature vector set.

[0104] Specifically, based on the initial reference features, each graph node is converted into a corresponding node latent representation vector using a linear mapping method according to the graph node identifier. The converted node latent representation vectors are then normalized using the L2 norm and arranged in node index order to serve as a candidate set of node latent representations. The candidate set is then integrated by concatenating nodes by index or by using statistical aggregation methods such as dimensional averaging or dimensional max pooling to generate a new latent representation set. Each latent representation in the new latent representation set is then subjected to nonlinear inverse mapping at the decoder of the variational autoencoder according to the decoding mapping rules, outputting the task feature vector corresponding to each latent representation and converging them into a task feature vector set.

[0105] S6.8. Within the new time window, perform category determination, trend inference, and abnormal pattern recognition on the task feature vector set to obtain the window task results.

[0106] Specifically, within the new time window, each task feature vector in the task feature vector set is read sequentially according to the time window order. The numerical attributes of each task feature vector are read one by one. Cosine similarity is used to calculate the similarity index between the task feature vector and the numerical range and feature distribution characteristics of each category in the preset classification rules. The task feature vector is assigned to the category with the largest similarity index or the smallest distance as the category to which the task feature vector belongs. After classifying the task feature vector set, the numerical change trend is calculated according to the time window arrangement order. For example, the moving average method is used to estimate the direction and magnitude of the change of the task feature vector within the time window to obtain the trend prediction result. The distribution characteristics of the task feature vector set are statistically analyzed, and the difference between each task feature vector and the overall distribution of the task feature vector set is calculated. When the difference exceeds the preset abnormal threshold, the corresponding task feature vector is marked as an abnormal pattern, forming a window task result that includes the category determination result, trend prediction result, and abnormal pattern recognition result.

[0107] It should also be noted that the preset classification rules refer to statistically analyzing the numerical range and feature distribution characteristics of each category based on the labeled category features of the historical task feature vector set, and then solidifying the statistical results as the comparison benchmark for classification judgment. The specific steps for setting an anomaly threshold are as follows: collect the numerical attributes of the feature vectors of historical tasks, and statistically analyze the distribution range and fluctuation characteristics of each numerical attribute; calculate the mean and standard deviation for each numerical attribute to form a normal distribution range; then set the anomaly threshold range based on the statistical results, such as mean ± 2 times standard deviation or mean ± 3 times standard deviation as the threshold range; mark the numerical attributes that exceed the threshold range as abnormal values ​​and use them for subsequent anomaly pattern recognition.

[0108] S6.9. Merge the window task results with the new potential representation set, update the initial reference features of the next time window, and enter the loop iterative process to form a dynamically updated fused representation.

[0109] Specifically, the window task results and each feature data in the new potential representation set are read sequentially according to the node identifier, and data with the same node identifier are paired; according to the feature fusion rules, the paired features are uniformly encoded and transformed, for example, the numerical features are standardized to the example range [0,1]; the fused features are arranged in chronological order to form the updated initial reference features; the updated initial reference features are input into the next time window as input data to carry out the next round of iterative processing, and the process is repeated until the number of time windows is reached, such as 3 to 12 time windows, to form a dynamically updated fused representation.

[0110] Among them, feature fusion rules refer to the processing criteria that integrate features with the same node identifier according to a pre-set fusion strategy and generate a unified representation by using a unified encoding conversion method.

[0111] This embodiment also provides a computer device applicable to the data fusion representation method based on multimodal features, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the data fusion representation method based on multimodal features as proposed in the above embodiment.

[0112] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0113] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the data fusion characterization method based on multimodal features as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0114] In summary, this invention achieves dynamic structured modeling and context-dependent capture of multimodal features by mapping the initial feature vector set to graph nodes and establishing a weighted dynamic graph structure, and by using a graph attention network to calculate feature dependencies. This improves the accuracy and interpretability of feature fusion, enhances the representational power of fused features, and provides a more accurate and robust feature foundation for subsequent potential representation generation and task execution.

[0115] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data fusion representation method based on multimodal features, characterized in that: include, Collect multimodal data, perform preprocessing and preliminary feature extraction, and generate a preliminary feature vector set; Based on the initial feature vector set, each modal feature is used as a graph node. Graph edges are established according to temporal, spatial, and semantic relationships. The graph node features and graph edge weights are initialized to construct a dynamic graph structure. Using a graph attention network, feature dependencies are calculated on a dynamic graph structure to obtain an attention weight matrix. The graph node features are then updated with weights to generate an updated graph node feature set. The updated graph node feature set is input into the variational autoencoder to encode the updated graph node feature set, generate mean vector and variance vector, and generate latent representation set by reparameterized sampling method; The latent representation set is input into the discriminator for adversarial training to optimize the consistency of the latent representation distribution. The parameters of the variational autoencoder are then iteratively updated through the feedback of the discriminator to obtain the optimized latent representation set. The optimized latent representation set is classified, trend predicted, and abnormal pattern identified in real time to obtain task results. These results are then used as initial reference features for the next time window to enter the iterative loop, forming a dynamically updated fusion representation.

2. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for generating the initial feature vector set are as follows: Noise suppression, missing value imputation, and anomaly removal are performed on the multimodal data to generate a cleaned multimodal dataset; The cleaned multimodal dataset is normalized and modality-specific enhancement is performed, and preliminary feature vectors for each modality are extracted to generate feature sets for each modality. The feature sets of each modality are aligned and their dimensions are unified. The correlation between modalities is enhanced by temporal, spatial and semantic relationships between modalities, resulting in a feature set with enhanced relationships. The feature set for relationship enhancement is filtered according to modality importance and context relevance, and redundant and noisy features are removed to generate a preliminary feature vector set.

3. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for constructing the dynamic graph structure are as follows. Each modal feature in the initial feature vector set is mapped to a graph node to generate an initial node set; Calculate the temporal, spatial, and semantic relationships between modalities based on the initial node set, establish graph edges, and generate a preliminary graph structure; Normalize and enhance the features of each graph node in the initial graph structure to generate an initial node feature set; The weights of the graph edges are initialized based on temporal, spatial, and semantic relationships to form a weighted dynamic graph structure; Local structural optimization is performed on the weighted dynamic graph structure to generate a new dynamic graph structure.

4. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for obtaining the attention weight matrix are as follows: The dynamic graph structure is input into the graph attention network, and the features of the neighboring nodes of each graph node are aggregated and the attention coefficients are calculated to generate the initial node attention weights. The graph edge weights and the initial node attention weights are fused and weighted and normalized to generate the optimized node attention weight matrix; The optimized node attention weights are computed in parallel in different attention subspaces to generate a multi-head node attention weight set, which is then fused to obtain the attention weight matrix.

5. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for generating the updated graph node feature set are as follows: The features of graph nodes are mapped and combined with the features of neighboring nodes according to the attention weight matrix to generate a node neighborhood feature set; The neighborhood feature set of the nodes is normalized and its proportion is adjusted to obtain a balanced neighborhood feature set, which is then fused with the graph node features according to a set proportion to form a preliminary updated node feature representation. The initially updated node feature representation is subjected to nonlinear transformation to generate a nonlinear node feature representation. The nonlinear node feature representation is expanded in parallel in the multi-head feature space to capture the structural dependencies of different subspaces, obtain the multi-head updated node feature set, and then splice and fuse them to generate the updated graph node feature set.

6. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for generating the mean vector and variance vector are as follows. The updated graph node feature set is input into the encoder end of the variational autoencoder to generate a preliminary hidden representation set; The initial hidden representation set is transformed using a first nonlinear mapping and a fully connected network to generate a mean vector. The mean vector and the initial hidden representation set are combined, and a variance vector is generated through a second nonlinear mapping.

7. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for generating the latent representation set are as follows: Randomly perturb the mean vector and variance vector to generate a random noise vector; By combining the random noise vector with the variance vector and performing perturbation mapping, a potential perturbation representation of the node is formed. The mean vector is integrated into the node latent perturbation representation to generate the node latent representation, and then summarized to form a latent representation set.

8. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The optimized latent representation set is obtained through the following steps. The latent representation set is input into the discriminator, and each latent representation is scored as true or false to generate a preliminary discrimination result; The deviation between the latent representation and the true distribution is evaluated based on the discrimination results, and feedback information is generated. The feedback information is input into the variational autoencoder, and the parameters of the variational autoencoder are iteratively adjusted to optimize the distribution consistency of the latent representation and generate an optimized latent representation set.

9. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for obtaining the task results are as follows: Decode and map the optimized latent representation set to generate a task feature vector set; Perform category and state analysis on the task feature vector set to generate preliminary classification results; The task feature vector set and the optimized latent representation set are combined and used for prediction to generate future prediction results; The system identifies abnormal patterns and deviations in the task feature vector set and generates anomaly detection results. The preliminary classification results, future prediction results, and anomaly detection results are integrated to obtain the task results.

10. The data fusion and representation method based on multimodal features as described in claim 1, characterized in that: The specific steps for forming the dynamically updated fusion representation are as follows: The task results are combined with the optimized latent representation set to form initial reference features; New node latent representations are constructed based on the initial reference features and integrated to form a new latent representation set. Simultaneously, decoding and mapping are performed to generate a task feature vector set. Within the new time window, the task feature vector set is classified, trended, and anomaly pattern identified to obtain the window task results. The window task results are merged with the new set of potential representations, the initial reference features for the next time window are updated, and the process is repeated iteratively to form a dynamically updated fused representation.

Images