A structural information enhanced multi-modal heterogeneous data fusion representation method
By constructing a stable graph structure and optimizing the hypergraph structure, combined with structural entropy regularization and soft allocation mechanisms, the problem of insufficient utilization of structural information in multimodal data fusion is solved, improving the quality of data representation and the generalization ability of the model, and is applicable to fields such as medical care, security and autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU INNOVATION RES INST OF BEIJING UNIV OF AERONAUTICS & ASTRONAUTICS
- Filing Date
- 2025-07-30
- Publication Date
- 2026-07-07
AI Technical Summary
In the fusion and processing of multimodal heterogeneous data, traditional methods are unable to fully extract data structure information, resulting in overfitting and insufficient generalization ability of the model when processing large-scale data, and poor adaptability to unstructured or complex data.
By preprocessing, feature extraction and transformation, a highly stable graph structure is constructed. Combining structural entropy regularization and soft allocation mechanisms, the hypergraph structure is optimized to improve data representation quality and discriminative power. Multiple regularization techniques are used to prevent overfitting, and model parameters are dynamically adjusted to adapt to the complex relationships of different modal data.
It improves the representation quality and discrimination ability of multimodal data, enhances the generalization ability of the model, and can more accurately process complex multimodal data, making it suitable for fields such as medical care, security and autonomous driving.
Smart Images
Figure CN120996150B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically to a method for fusion and characterization of multimodal heterogeneous data with enhanced structural information. Background Technology
[0002] With the rapid development of information technology, the generation and acquisition of data have become increasingly easier, and the types of data are becoming more and more diverse, including multiple modalities such as text, images, audio, and video. These multimodal heterogeneous data play a key role in many fields such as healthcare, security, autonomous driving, intelligent transportation, and finance. For example, in the medical field, doctors need to integrate multiple data such as patient medical records, X-rays, and CT images to make accurate diagnoses. In security monitoring, the system needs to integrate video images and sound signals to identify potential security threats. In autonomous vehicles, the vehicle needs to process visual data from cameras, ranging data from radar, and other data from vehicle sensors to make safe driving decisions.
[0003] However, the fusion and processing of multimodal heterogeneous data still faces numerous challenges. Different modalities of data possess distinct characteristics and structures, making the effective fusion of these data the biggest hurdle. Traditional data representation methods struggle to fully extract structural information from the data, preventing many large models from fully utilizing it. This is particularly problematic when processing large datasets, as overfitting can easily occur, impacting model performance and generalization ability. Furthermore, the complementarity and correlation between different modalities often require deeper analysis to effectively improve the quality and discriminative power of data representation.
[0004] In recent years, to address related issues, numerous multimodal data fusion and representation learning methods have emerged. Among these methods, graph-based learning methods have garnered widespread attention due to their highly efficient data capture capabilities. However, most of these methods rely on predefined graph structures, resulting in relatively poor adaptability to unstructured or complex data distributions. How to fully utilize structural information to enhance the discriminative and generalization abilities of data representations during the fusion process remains an unresolved issue. Therefore, inventing a technology that can efficiently fuse multimodal heterogeneous data and improve data representation levels has significant practical implications and broad application prospects. This type of technology can fundamentally improve the quality of data representation, enhance the model's discriminative ability, and strengthen its generalization ability, thereby enabling the model to be not only accurate but also stable when processing large-scale data. It can provide stronger and more efficient technical support for various complex data analysis and decision support tasks, making its application scenarios widely applicable in fields such as healthcare, security, and autonomous driving. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a multimodal heterogeneous data fusion and characterization method with enhanced structural information.
[0006] To achieve the above technology, the steps include:
[0007] S1. Obtain multimodal heterogeneous data through text, images, audio and video. Perform preprocessing operations on the multimodal heterogeneous data and then perform feature extraction and transformation operations to obtain a multimodal feature matrix in a unified feature space.
[0008] Multimodal heterogeneous data plays a crucial role in many fields such as healthcare, security, autonomous driving, and intelligent transportation. For example, in healthcare, doctors need to integrate various data such as patient medical records, X-rays, CT images, and patient verbal descriptions to make accurate diagnoses. In security monitoring, systems need to integrate video images, sound signals, and environmental audio to identify potential security threats. In autonomous vehicles, vehicles need to process visual data from cameras, ranging data from radar, and other data from vehicle sensors to make safe driving decisions.
[0009] In this invention, the preprocessing operations include: cleaning, normalizing, aligning, and handling missing values of the original input multimodal heterogeneous data;
[0010] In this invention, the feature extraction and transformation operation is performed as follows: using a modal model to extract features of each modality, and mapping different modal features to the same latent space through linear transformation; in this invention, the modal model includes: CNN to extract image features, and BERT to extract text features.
[0011] S2. Based on the multimodal feature matrix, construct a stable and interpretable graph structure using Graph Structure Enhancement (GSL) techniques. The steps are as follows:
[0012] S2.1 Calculate the inner product of the multimodal feature matrices and construct the initial similarity graph, as shown in the following expression:
[0013]
[0014] In the formula, Represents vertices and vertex The similarity between them; Represents the multimodal feature matrix; superscript Indicates transpose;
[0015] This invention constructs an initial similarity graph by calculating the inner product between data features. The inner product operation can measure the similarity between different data points, providing a foundation for the construction of subsequent graph structures. For example, when processing a dataset containing image and text data, this invention constructs an initial similarity graph by calculating the inner product of image features and text features. This inner product-based similarity graph can initially present the similarity between data points, providing a key reference for the optimization of subsequent graph structures.
[0016] S2.2 Construct an adjacency matrix based on the initial similarity graph. By sparsifying the matrix, the computational complexity is reduced, the computational efficiency is improved, and the local structural information between data points is highlighted.
[0017] The expression for constructing the adjacency matrix based on the initial similarity graph is as follows:
[0018]
[0019] In the formula, Represents the adjacency matrix; Indicates the activation function;
[0020] The sparsity reduction method is to use the K-Nearest Neighbors (KNN) algorithm to retain the k most similar neighbors of each node;
[0021] The expression for sparsification is as follows:
[0022]
[0023] In the formula, Represents a similar graph after sparsification; This represents a similarity graph before sparsification; in this invention, it is a feature similarity graph. ; Represents vertices Belongs to the distance from the vertex The former The most similar neighbors;
[0024] Sparsity processing can reduce the complexity of the graph, improve computational efficiency, and highlight the local structural information between data points;
[0025] S2.3. Post-processing operations on the expanded adjacency matrix after sparsification, including: symmetry operation, activation function processing and row normalization processing, to ensure the undirectedness of the graph and to ensure that the graph is normalized.
[0026] Symmetry operation: To ensure the graph is undirected, this invention achieves this by symmetricizing the adjacency matrix, as shown in the following expression:
[0027]
[0028] In the formula, Represents the adjacency matrix for symmetrization operations;
[0029] Activation function processing: Taking the adjacency matrix after symmetrization as input, an activation function (such as sigmoid) is applied to enhance the stability of the graph structure. The expression is as follows:
[0030]
[0031] In the formula, This represents the adjacency matrix after activation function processing;
[0032] Row normalization: The adjacency matrix after activation function processing is row normalized to ensure that the sum of the weights of each node is 1. The expression is as follows:
[0033]
[0034] In the formula, This represents the element in the x-th row and y-th column of the adjacency matrix after the activation function has been applied. This represents the summation of all elements in the x-th row of the adjacency matrix after the activation function is applied; This represents the adjacency matrix after row normalization.
[0035] Through the above graph structure enhancement steps, the present invention can effectively improve the representation learning performance of multimodal data and ensure the accuracy and integrity of the model when processing complex multimodal data. This method is particularly suitable for scenarios such as sentiment analysis, image classification, and speech recognition.
[0036] S3. Based on graph structure, construct a discriminative representation learning framework with structural entropy regularization to improve the model's generalization ability and clustering performance; the discriminative representation learning framework with structural entropy regularization includes: overfitting regularization of information entropy and category regularization of structural entropy.
[0037] The steps are as follows:
[0038] S3.1 Construct overfitting regularization rules for information entropy, including penalizing low-entropy distributions, achieving fast convergence, and preventing overfitting. The construction steps include:
[0039] S3.1.1, Penalizing Low-Entropy Distribution: A penalty is imposed on the low-entropy distribution exhibited by the model output. This penalty is introduced by calculating the entropy of the conditional distribution for the SoftMax output of the neural network and adding it as a regularization term to the loss function. The expression is as follows:
[0040]
[0041] In the formula, Represents model input Next Prediction Category The conditional probability distribution; This represents a hyperparameter for penalizing low-entropy distributions, used to control the strength of the entropy regularization term; The term represents negative entropy, used to penalize low-entropy distributions, where... The expression is:
[0042]
[0043] In the formula, Indicates the predicted category of class a;
[0044] S3.1.2 Fast Convergence and Overfitting Prevention: To achieve fast convergence and effectively prevent overfitting, this invention designs a method for dynamically adjusting the entropy regularization strength. At the beginning of training, the entropy regularization strength is relatively weak. As training progresses and the model begins to overfit, the entropy regularization strength is gradually increased. This is achieved by setting a threshold; when the entropy of the model's output distribution falls below this threshold, a stronger entropy regularization is applied. The formula can be expressed as:
[0045]
[0046] In the formula, This represents the entropy threshold; strong regularization is triggered when the actual entropy falls below this value.
[0047] S3.1.3, The deviation of the output distribution from the uniform distribution is penalized through label smoothing operation to smooth the distribution; the expression is as follows:
[0048]
[0049] In the formula, This represents the Kullback-Leibler divergence, used to measure the difference between two probability distributions; Indicates a uniform distribution;
[0050] S3.2 Construct the category regularization calculation rule for structural entropy, the expression is as follows:
[0051]
[0052] In the formula, It is the main loss function; It is a regularization coefficient used to balance the relationship between the principal loss and the structural entropy regularization term; This represents structural entropy; in this way, the model can not only learn the main features of the data, but also maintain the balance between categories, thereby improving the ability to distinguish between different categories of data and the effect of handling imbalanced data.
[0053] Following the steps outlined above, the structural entropy category regularization method can effectively guide the model to learn and obtain a better data representation, thereby improving the clustering performance of multimodal data. This method can enhance the model's ability to distinguish between different categories of data and improve the model's performance in handling imbalanced data, allowing the model to learn representations more accurately when faced with complex multimodal data.
[0054] S4. Design a structural information optimization framework and soft allocation mechanism to improve the quality of data representation and discriminative power;
[0055] This invention conceives an innovative framework for optimizing structural information, based on a soft allocation mechanism to optimize graph structure learning. This allows the model to better capture structural information in the data. The soft allocation mechanism enables the model to dynamically adjust the connection weights between nodes during the learning process, thereby better matching the structural characteristics of the data. This dynamic adjustment capability allows the model to more flexibly handle complex relationships between different modalities of data, improving the quality and discriminative power of data representation. For example, when processing a multimodal dataset covering text, images, and audio data, the model can dynamically adjust the connection weights between text features, image features, and audio features through the soft allocation mechanism, more effectively capturing the structural information between them and improving the quality and discriminative power of data representation.
[0056] The expression for the soft allocation mechanism is as follows:
[0057]
[0058] In the formula, and Represents vertices and vertex The feature vectors are derived from the row vectors of the multimodal feature matrix; Represents the inner product of eigenvectors; This represents the temperature parameter, used to control the weight distribution; Represents vertices The set of neighbors, among which, It is the vertex The index of the neighbor set; Represents vertices Neighboring vertices eigenvectors; superscript Indicates transpose; Represents a node and nodes Connection weights between them;
[0059] Using this method, the soft allocation mechanism can dynamically adjust the connection weights between nodes, allowing the model to better adapt to the structural characteristics of the data.
[0060] S5. Based on the obtained graph structure, the discriminative representation learning framework of structural entropy regularization, and the structural information optimization framework and soft allocation mechanism, construct the hypergraph structure and calculate the structural entropy of the hypergraph structure.
[0061] S5.1 Define the hypergraph structure;
[0062] Define an undirected weighted hypergraph This undirected weighted hypergraph contains a set of vertices. Hyperedge set and weight matrix The specific definition is as follows:
[0063] S5.1.1 Constructing the Vertex Set of the Hypergraph ;
[0064] The set of vertices of a hypergraph The set of vertices in the graph structure corresponds to the data samples, i.e. The row vector;
[0065] S5.1.2 Constructing the set of hyperedges of a hypergraph ;
[0066] The set of superedges of a hypergraph Each hyperedge connects a set of related hypergraph vertices. The method for constructing hyperedges is as follows:
[0067] For each vertex in the feature space of a hypergraph structure, the K-nearest neighbor algorithm is used to compute cross-modal similar neighbors. For example, in the medical field, a hyperedge includes { Images, medical records, electrocardiograms, and patient reports;
[0068] S5.1.3 Constructing the weight matrix of the hypergraph ;
[0069] Hypergraph weight matrix The weight of each hyperedge is stored using the following expression:
[0070]
[0071] In the formula, Indicates the super edge. ; Represents the two vertices of a superedge; Representing vertices in the feature space ; Represents vertices and The squared distance in the feature space; The bandwidth parameter represents the Gaussian kernel function;
[0072] S5.2 Define the cut edges and cut edge volumes of the hypergraph structure;
[0073] Volume of the vertex set Expanding the calculation, which is defined as follows: , here The degree of a vertex, a metric that helps in understanding the "size" of a set of vertices within a hypergraph, is defined as the cut edges formed with the outside:
[0074]
[0075] In the formula, Represents the volume of the vertex set The set of cut edges; for The complement of the vertex set; Representing an edge At least one endpoint is in the set middle; Representing an edge At least one endpoint is in the set In the middle; these cut edges serve to connect the vertex sets. The roles of internal and external vertices;
[0076] The expression for the cut volume is as follows:
[0077]
[0078] In the formula, The cut edge volume represents the degree of the hyperedge. This formula shows the strength of the connection between the vertex set and the rest of the hypergraph. The larger the cut edge volume, the stronger the connection between the vertex set and the rest of the hypergraph. Conversely, the larger the cut edge volume, the weaker the connection.
[0079] S5.3 Generate the incidence matrix of the hypergraph structure, with the following expression:
[0080]
[0081] Correlation Matrix It can be used to represent the relationship between a vertex and a hyperedge, when the vertex... Belongs to superedge At that time, Otherwise, it is 0;
[0082] S5.4. Based on the generated incidence matrix, construct a clique adjacency matrix to convert the higher-order relations of the hypergraph into ordinary graphs, as shown in the following expression:
[0083]
[0084] In the formula, Represents the clique-based adjacency matrix; The inverse matrix represents the hyperedge degree; the clique adjacency matrix provides a structured perspective for observing and analyzing hypergraphs.
[0085] S5.5 Calculate the hypergraph entropy and cut edge entropy of the hypergraph structure;
[0086] The expression for hypergraph entropy is as follows:
[0087]
[0088] In the formula, Represents the hypergraph entropy; Represents the hypergraph structure vertices in the hypergraph structure. The set of neighbors; Represents vertices in a hypergraph structure ; Represents vertices in a hypergraph structure and vertex soft links;
[0089] The expression for the cut edge entropy is as follows:
[0090]
[0091] In the formula, This represents a category matrix, indicating the connections between categories. This indicates the total number of categories, such as disease categories in the medical field, where... An index representing the total number of categories;
[0092] Hypergraph Entropy It can measure the complexity and information content of a hypergraph, such as cut edge entropy. This can reflect the uniformity of the cut edge distribution. These entropy values will be applied to the regularization model to improve its generalization ability, ensuring that the model performs well on the training data and maintains good performance on unseen data.
[0093] S6. Based on hypergraph structure entropy, guide multimodal unsupervised clustering;
[0094] The steps include:
[0095] S6.1, Multimodal data fusion;
[0096] This invention fuses data from different modalities such as text, images, and audio, and maps them to the same feature space through feature extraction and transformation, providing a unified data representation for subsequent clustering analysis;
[0097] S6.2, Structural entropy regularization-guided clustering;
[0098] The clustering objective function can be expressed as:
[0099]
[0100] In the formula, This represents the basic clustering loss function (such as the squared error of K-means clustering). The regularization coefficient representing the balance structure entropy term and the basic loss; Represents vertices in a hypergraph structure The structural entropy is expressed as follows:
[0101] ;
[0102] After completing the step of optimizing structural entropy, the model can capture complex structural information in the data more efficiently, thereby further improving the accuracy of cluster analysis.
[0103] S6.3 Clustering result optimization;
[0104] This invention is based on the clustering results after structural entropy regularization, and further optimizes the cluster centers and cluster labels by iterative optimization until the minimum convergence condition is met.
[0105] The iterative optimization process can be represented as:
[0106]
[0107] In the formula, Indicates the first During the nth iteration, the 1st The center vectors of each cluster; The set of data points belonging to the i-th cluster is expressed as follows:
[0108]
[0109] In the formula, Indicates the first During the nth iteration, it belongs to the... A set of data points in a cluster; Indicates the first Cluster centers at the next iteration; Indicates the number of clusters;
[0110] This invention can not only achieve efficient fusion processing of multimodal heterogeneous data, but also complete high-quality characterization work, providing more efficient technical support for various complex tasks, completing tasks efficiently and safely, and has wide applications in fields such as medical care, security, and autonomous driving.
[0111] Beneficial effects of the present invention
[0112] To achieve efficient fusion of multimodal data, this invention integrates text, images, audio, and other modalities more effectively and stably. The main purpose of this fusion method is to uncover the complementarity and correlation between different modalities, thereby improving the quality of the model's data representation and its discriminative ability. The application scenarios of this fusion method are very broad: in the medical field, combining patient medical records with medical image data can provide doctors with more comprehensive and complete information about the patient's condition; in the field of security monitoring, the fusion of video images and sound signals can more accurately identify potential security threats, thereby ensuring safety; in the field of autonomous vehicles, the integration of visual data and radar ranging data can provide accurate and useful information for safe driving decisions, thereby reducing the probability of accidents.
[0113] To optimize the utilization of data structural information and address the problem of incomplete information utilization in traditional data representation methods, this invention optimizes data structural information based on structural entropy regularization. It studies the structural entropy of feature similarity maps, mainly including the calculation of local entropy and joint entropy, and incorporates the obtained structural entropy as a regularization term into the loss function. This enables the model to perform deep learning on the structural features of the data during training, thereby improving the discriminative ability of data representation.
[0114] To enhance model generalization ability and address the common problems of overfitting and class imbalance during model training, this invention proposes a structural entropy regularization method. The core of this invention lies in optimizing data structure information. By deeply exploring the inherent structural relationships within the data, it enhances the model's ability to process and analyze data from different categories. During training, the structural entropy regularization method dynamically adjusts the model's parameters, preventing the model from overlearning local features from old training data and reducing the probability of overfitting. Furthermore, this method balances the contributions of different classes of data during model training, improving the model's performance in recognizing a minority of classes.
[0115] To improve the processing capabilities of unstructured data, this invention can automatically extract structural information from the data. In real-world scenarios, a large amount of data lacks a clearly defined graph structure, and the construction process is extremely complex. Traditional learning methods that rely on graph structures are often limited by pre-defined graph structures and struggle to adapt well to unstructured or complex data distributions. Unlike traditional methods, the method proposed in this invention possesses the ability to automatically learn data structure information, improving the efficiency of processing unstructured data, effectively overcoming traditional limitations, and providing a new solution. Attached Figure Description
[0116] Figure 1 This is a flowchart of the steps of the present invention;
[0117] Figure 2 This is an overall framework diagram of the present invention;
[0118] Figure 3 This is the multimodal unsupervised clustering diagram of the present invention. Detailed Implementation
[0119] The present invention will be further described in detail below with reference to specific embodiments.
[0120] like Figure 1 and Figure 2 As shown, a multimodal heterogeneous data fusion and representation method with enhanced structural information includes the following steps:
[0121] S1. Obtain multimodal heterogeneous data through text, images, audio and video. Perform preprocessing operations on the multimodal heterogeneous data and then perform feature extraction and transformation operations to obtain a multimodal feature matrix in a unified feature space.
[0122] Multimodal heterogeneous data plays a crucial role in many fields such as healthcare, security, autonomous driving, and intelligent transportation. For example, in healthcare, doctors need to integrate various data such as patient medical records, X-rays, CT images, and patient verbal descriptions to make accurate diagnoses. In security monitoring, systems need to integrate video images, sound signals, and environmental audio to identify potential security threats. In autonomous vehicles, vehicles need to process visual data from cameras, ranging data from radar, and other data from vehicle sensors to make safe driving decisions.
[0123] In this invention, the preprocessing operations include: cleaning, normalizing, aligning, and handling missing values of the original input multimodal heterogeneous data;
[0124] In this invention, the feature extraction and transformation operation is performed as follows: using a modal model to extract features of each modality, and mapping different modal features to the same latent space through linear transformation; in this invention, the modal model includes: CNN to extract image features, and BERT to extract text features.
[0125] S2. Based on the multimodal feature matrix, construct a stable and interpretable graph structure using Graph Structure Enhancement (GSL) techniques. The steps are as follows:
[0126] S2.1 Calculate the inner product of the multimodal feature matrices and construct the initial similarity graph, as shown in the following expression:
[0127]
[0128] In the formula, Represents vertices and vertex The similarity between them; Represents the multimodal feature matrix; superscript This indicates transpose; by calculating the inner product between feature vectors, a similarity graph reflecting the similarity between data points can be obtained. This similarity graph can initially present the similarity between data points and provide a key reference for subsequent graph structure optimization. For example, when processing a dataset that includes image and text data, the inner product between image features and text features can be calculated to construct the initial similarity graph.
[0129] This invention constructs an initial similarity graph by calculating the inner product between data features. The inner product operation can measure the similarity between different data points, providing a foundation for the construction of subsequent graph structures. For example, when processing a dataset containing image and text data, this invention constructs an initial similarity graph by calculating the inner product of image features and text features. This inner product-based similarity graph can initially present the similarity between data points, providing a key reference for the optimization of subsequent graph structures.
[0130] S2.2 Construct an adjacency matrix based on the initial similarity graph. By sparsifying the matrix, the computational complexity is reduced, the computational efficiency is improved, and the local structural information between data points is highlighted.
[0131] The expression for constructing the adjacency matrix based on the initial similarity graph is as follows:
[0132]
[0133] In the formula, Represents the adjacency matrix; This represents an activation function used to introduce non-linear characteristics, such as the sigmoid activation function.
[0134] The sparsity reduction method is to use the K-Nearest Neighbors (KNN) algorithm to retain the k most similar neighbors of each node;
[0135] The expression for sparsification is as follows:
[0136]
[0137] In the formula, Represents a similar graph after sparsification; This represents a similarity graph before sparsification; in this invention, it is a feature similarity graph. ; Represents vertices Belongs to the distance from the vertex The former The most similar neighbors;
[0138] Sparsity processing can reduce the complexity of the graph, improve computational efficiency, and highlight the local structural information between data points. For example, when processing a dataset containing 1000 data points, this invention can retain only the top 10 most similar neighbors of each node to construct a sparse similarity graph. This sparse similarity graph can reduce computational costs and more clearly present the local structural information between data points, providing a more accurate reference for subsequent graph structure optimization.
[0139] S2.3. Post-processing operations on the expanded adjacency matrix after sparsification, including: symmetry operation, activation function processing and row normalization processing, to ensure the undirectedness of the graph and to ensure that the graph is normalized.
[0140] Symmetry operation: To ensure the graph is undirected, this invention achieves this by symmetricizing the adjacency matrix, as shown in the following expression:
[0141]
[0142] In the formula, Represents the adjacency matrix for symmetrization operations;
[0143] Activation function processing: Taking the adjacency matrix after symmetrization as input, an activation function (such as sigmoid) is applied to enhance the stability of the graph structure. The expression is as follows:
[0144]
[0145] In the formula, This represents the adjacency matrix after activation function processing;
[0146] Row normalization: The adjacency matrix after activation function processing is row normalized to ensure that the sum of the weights of each node is 1. The expression is as follows:
[0147]
[0148] In the formula, This represents the element in the x-th row and y-th column of the adjacency matrix after the activation function has been applied. This represents the summation of all elements in the x-th row of the adjacency matrix after the activation function is applied; This represents the adjacency matrix after row normalization.
[0149] In addition to ensuring the undirectedness and normalization of the graph through post-processing operations, the graph structure discrimination ability is also improved through activation functions, so that the model can learn the intrinsic structural information of multimodal data more efficiently and make full use of it in actual operation.
[0150] Through the above graph structure enhancement steps, the present invention can effectively improve the representation learning performance of multimodal data and ensure the accuracy and integrity of the model when processing complex multimodal data. This method is particularly suitable for scenarios such as sentiment analysis, image classification, and speech recognition.
[0151] S3. Based on graph structure, construct a discriminative representation learning framework with structural entropy regularization to improve the model's generalization ability and clustering performance; the discriminative representation learning framework with structural entropy regularization includes: overfitting regularization of information entropy and category regularization of structural entropy.
[0152] In the field of multimodal data processing, neural network models often face the problem of overfitting. When overfitting occurs, the model performs well on the training data, but its generalization ability drops significantly on unseen data. To solve this problem, this invention employs various regularization techniques to improve model performance. By adjusting the probability distribution of the model's output, the generalization ability of the model is improved. This is based on maximizing the entropy of the model's output, which conforms to the maximum entropy principle, that is, under empirical constraints, the uniform probability distribution with the greatest uncertainty is selected.
[0153] The steps are as follows:
[0154] S3.1 Construct overfitting regularization rules for information entropy, including penalizing low-entropy distributions, achieving fast convergence, and preventing overfitting. The construction steps include:
[0155] S3.1.1, Penalizing Low-Entropy Distribution: A penalty is imposed on the low-entropy distribution exhibited by the model output. This penalty is introduced by calculating the entropy of the conditional distribution for the SoftMax output of the neural network and adding it as a regularization term to the loss function. The expression is as follows:
[0156]
[0157] In the formula, Represents model input Next Prediction Category The conditional probability distribution; This represents a hyperparameter for penalizing low-entropy distributions, used to control the strength of the entropy regularization term; The term represents negative entropy, used to penalize low-entropy distributions, where... The expression is:
[0158]
[0159] In the formula, Indicates the predicted category of class a;
[0160] S3.1.2 Fast Convergence and Overfitting Prevention: To achieve fast convergence and effectively prevent overfitting, this invention designs a method for dynamically adjusting the entropy regularization strength. At the beginning of training, the entropy regularization strength is relatively weak. As training progresses and the model begins to overfit, the entropy regularization strength is gradually increased. This is achieved by setting a threshold; when the entropy of the model's output distribution falls below this threshold, a stronger entropy regularization is applied. The formula can be expressed as:
[0161]
[0162] In the formula, This represents the entropy threshold; strong regularization is triggered when the actual entropy falls below this value.
[0163] S3.1.3, The label smoothing operation penalizes the deviation of the output distribution from the uniform distribution to smooth the distribution, encouraging the distribution not to be too sharp; the expression is as follows:
[0164]
[0165] In the formula, This represents the Kullback-Leibler divergence, used to measure the difference between two probability distributions; Indicates a uniform distribution;
[0166] S3.2 Construct the category regularization calculation rule for structural entropy, the expression is as follows:
[0167]
[0168] In the formula, It is the main loss function; It is a regularization coefficient used to balance the relationship between the principal loss and the structural entropy regularization term; This represents structural entropy; in this way, the model can not only learn the main features of the data, but also maintain the balance between categories, thereby improving the ability to distinguish between different categories of data and the effect of handling imbalanced data.
[0169] Following the steps outlined above, the structural entropy category regularization method can effectively guide the model to learn and obtain a better data representation, thereby improving the clustering performance of multimodal data. This method can enhance the model's ability to distinguish between different categories of data and improve the model's performance in handling imbalanced data, allowing the model to learn representations more accurately when faced with complex multimodal data.
[0170] S4. Design a structural information optimization framework and soft allocation mechanism to improve the quality of data representation and discriminative power;
[0171] This invention conceives an innovative framework for optimizing structural information, based on a soft allocation mechanism to optimize graph structure learning. This allows the model to better capture structural information in the data. The soft allocation mechanism enables the model to dynamically adjust the connection weights between nodes during the learning process, thereby better matching the structural characteristics of the data. This dynamic adjustment capability allows the model to more flexibly handle complex relationships between different modalities of data, improving the quality and discriminative power of data representation. For example, when processing a multimodal dataset covering text, images, and audio data, the model can dynamically adjust the connection weights between text features, image features, and audio features through the soft allocation mechanism, more effectively capturing the structural information between them and improving the quality and discriminative power of data representation.
[0172] The expression for the soft allocation mechanism is as follows:
[0173]
[0174] In the formula, and Represents vertices and vertex The feature vectors are derived from the row vectors of the multimodal feature matrix; Represents the inner product of eigenvectors; This represents the temperature parameter, used to control the weight distribution; Represents vertices The set of neighbors, among which, It is the vertex The index of the neighbor set; Represents vertices Neighboring vertices eigenvectors; superscript Indicates transpose; Represents a node and nodes Connection weights between them;
[0175] Using this method, the soft allocation mechanism can dynamically adjust the connection weights between nodes, allowing the model to better adapt to the structural characteristics of the data.
[0176] S5. Based on the obtained graph structure, the discriminative representation learning framework of structural entropy regularization, and the structural information optimization framework and soft allocation mechanism, construct the hypergraph structure and calculate the structural entropy of the hypergraph structure.
[0177] S5.1 Define the hypergraph structure;
[0178] Define an undirected weighted hypergraph This undirected weighted hypergraph contains a set of vertices. Hyperedge set and weight matrix The specific definition is as follows:
[0179] S5.1.1 Constructing the Vertex Set of the Hypergraph ;
[0180] The set of vertices of a hypergraph The set of vertices in the graph structure corresponds to the data samples, i.e. The row vector;
[0181] S5.1.2 Constructing the set of hyperedges of a hypergraph ;
[0182] The set of superedges of a hypergraph Each hyperedge connects a set of related hypergraph vertices. The method for constructing hyperedges is as follows:
[0183] For each vertex in the feature space of a hypergraph structure, the K-nearest neighbor algorithm is used to compute cross-modal similar neighbors. For example, in the medical field, a hyperedge includes { Images, medical records, electrocardiograms, and patient reports;
[0184] S5.1.3 Constructing the weight matrix of the hypergraph ;
[0185] Hypergraph weight matrix The weight of each hyperedge is stored using the following expression:
[0186]
[0187] In the formula, Indicates the super edge. ; Represents the two vertices of a superedge; Representing vertices in the feature space ; Represents vertices and The squared distance in the feature space; The bandwidth parameter represents the Gaussian kernel function;
[0188] S5.2 Define the cut edges and cut edge volumes of the hypergraph structure;
[0189] Volume of the vertex set Expanding the calculation, which is defined as follows: , here The degree of a vertex, a metric that helps in understanding the "size" of a set of vertices within a hypergraph, is defined as the cut edges formed with the outside:
[0190]
[0191] In the formula, Represents the volume of the vertex set The set of cut edges; for The complement of the vertex set; Representing an edge At least one endpoint is in the set middle; Representing an edge At least one endpoint is in the set In the middle; these cut edges serve to connect the vertex sets. The roles of internal and external vertices;
[0192] The expression for the cut volume is as follows:
[0193]
[0194] In the formula, The cut edge volume represents the degree of the hyperedge. This formula shows the strength of the connection between the vertex set and the rest of the hypergraph. The larger the cut edge volume, the stronger the connection between the vertex set and the rest of the hypergraph. Conversely, the larger the cut edge volume, the weaker the connection.
[0195] S5.3 Generate the incidence matrix of the hypergraph structure, with the following expression:
[0196]
[0197] Correlation Matrix It can be used to represent the relationship between a vertex and a hyperedge, when the vertex... Belongs to superedge At that time, Otherwise, it is 0;
[0198] S5.4. Based on the generated incidence matrix, construct a clique adjacency matrix to convert the higher-order relations of the hypergraph into ordinary graphs, as shown in the following expression:
[0199]
[0200] In the formula, Represents the clique-based adjacency matrix; The inverse matrix represents the hyperedge degree; the clique adjacency matrix provides a structured perspective for observing and analyzing hypergraphs.
[0201] S5.5 Calculate the hypergraph entropy and cut edge entropy of the hypergraph structure;
[0202] The expression for hypergraph entropy is as follows:
[0203]
[0204] In the formula, Represents the hypergraph entropy; Represents the hypergraph structure vertices in the hypergraph structure. The set of neighbors; Represents vertices in a hypergraph structure ; Represents vertices in a hypergraph structure and vertex soft links;
[0205] The expression for the cut edge entropy is as follows:
[0206]
[0207] In the formula, This represents a category matrix, indicating the connections between categories. This indicates the total number of categories, such as disease categories in the medical field, where... An index representing the total number of categories;
[0208] Hypergraph Entropy It can measure the complexity and information content of a hypergraph, such as cut edge entropy. This can reflect the uniformity of the cut edge distribution. These entropy values will be applied to the regularization model to improve its generalization ability, ensuring that the model performs well on the training data and maintains good performance on unseen data.
[0209] S6, such as Figure 3 As shown, multimodal unsupervised clustering is guided based on hypergraph structure entropy;
[0210] The steps include:
[0211] S6.1, Multimodal data fusion;
[0212] This invention fuses data from different modalities such as text, images, and audio, and maps them to the same feature space through feature extraction and transformation, providing a unified data representation for subsequent clustering analysis. For example, when processing a dataset that includes text, images, and audio data, this invention uses feature extraction and transformation to map them to the same feature space, providing a unified data representation for subsequent clustering analysis. This multimodal data fusion method can fully utilize the complementarity and correlation between different modalities of data, improving the accuracy and reliability of clustering analysis.
[0213] S6.2, Structural entropy regularization-guided clustering;
[0214] In the unsupervised clustering process, this invention introduces a structural entropy regularization term to guide the clustering process based on the optimized structural information of the data. Specifically, this invention calculates the structural entropy of multimodal data and adds it as a regularization term to the clustering objective function, encouraging a uniform state of cut edges between different categories in the clustering results, thus achieving more accurate clustering analysis. For example, when processing a multimodal dataset containing text, images, and audio data, this invention calculates the structural entropy of the multimodal data, allowing the model to better capture the structural information between text, images, and audio data, achieving more accurate clustering analysis. The clustering objective function can be expressed as:
[0215]
[0216] In the formula, This represents the basic clustering loss function (such as the squared error of K-means clustering). The regularization coefficient representing the balance structure entropy term and the basic loss; Represents vertices in a hypergraph structure The structural entropy is expressed as follows:
[0217] ;
[0218] After completing the step of optimizing structural entropy, the model can capture complex structural information in the data more efficiently, thereby further improving the accuracy of cluster analysis.
[0219] S6.3 Clustering result optimization;
[0220] This invention is based on the clustering results after structural entropy regularization. It further optimizes the cluster centers and cluster labels through iterative optimization until the minimum convergence condition is met. For example, when processing a dataset containing 1000 data points, the cluster centers and cluster labels are continuously adjusted through iterative optimization until the clustering results tend to stabilize.
[0221] The iterative optimization process can be represented as:
[0222]
[0223] In the formula, Indicates the first During the nth iteration, the 1st The center vectors of each cluster; The set of data points belonging to the i-th cluster is expressed as follows:
[0224]
[0225] In the formula, Indicates the first During the nth iteration, it belongs to the... A set of data points in a cluster; Indicates the first Cluster centers at the next iteration; This indicates the number of clusters; in this way, the cluster centers and labels are effectively adjusted, thereby improving the accuracy and stability of the clustering results.
[0226] Through the above technical solutions, this invention can not only achieve efficient fusion processing of multimodal heterogeneous data, but also complete high-quality characterization work, providing more efficient technical support for various complex tasks, completing tasks efficiently and safely, and is widely used in fields such as medical care, security, and autonomous driving.
[0227] The specific embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A method for fusion and characterization of multimodal heterogeneous data with enhanced structural information, characterized in that, Includes the following steps: S1. Obtain multimodal heterogeneous data through text, images, audio and video. Perform preprocessing operations on the multimodal heterogeneous data and then perform feature extraction and transformation operations to obtain a multimodal feature matrix with a unified feature space. S2. Based on the multimodal feature matrix, a graph structure enhancement technique is used to construct a stable and interpretable graph structure; The steps for constructing a stable and interpretable graph structure using graph structure enhancement techniques based on a multimodal feature matrix include: S2.1 Calculate the inner product of the multimodal feature matrices and construct the initial similarity graph; S2.2 Construct an adjacency matrix based on the initial similarity graph, and highlight the local structural information between data points through sparsification. S2.3 Post-processing operations on the expanded adjacency matrix after sparsification, including: symmetry operation, activation function processing and row normalization processing; The symmetrization operation ensures that the graph is undirected, which is achieved by symmetrizing the adjacency matrix; The activation function takes the adjacency matrix of the symmetry operation as input and applies the activation function to enhance the stability of the graph structure. The row normalization process involves normalizing the adjacency matrix after the activation function to ensure that the sum of the weights of each node is 1. S3. Based on graph structure, construct a discriminative representation learning framework with structural entropy regularization to improve the model's generalization ability and clustering performance. The discriminative representation learning framework for structural entropy regularization includes: overfitting regularization of information entropy and category regularization of structural entropy; The steps for constructing a discriminative representation learning framework based on graph structure and structural entropy regularization to improve the model's generalization ability and clustering performance are as follows: S3.1 Construct overfitting regularization rules for information entropy, including penalizing low-entropy distributions, achieving fast convergence, and preventing overfitting. S3.2 Constructing the category regularization calculation rules for structural entropy; S4. Design a structural information optimization framework and soft allocation mechanism to improve the quality of data representation and discriminative power; S5. Based on the obtained graph structure, the discriminative representation learning framework of structural entropy regularization, and the structural information optimization framework and soft allocation mechanism, construct the hypergraph structure and calculate the structural entropy of the hypergraph structure. S6. Based on the hypergraph structure entropy, guide multimodal unsupervised clustering to complete the multimodal heterogeneous data fusion representation with enhanced structural information.
2. The multimodal heterogeneous data fusion and characterization method with enhanced structural information according to claim 1, characterized in that, The overfitting regularization calculation rule for constructing information entropy includes penalizing low-entropy distributions, achieving fast convergence, and preventing overfitting. The steps include: S3.1.1, Penalizing Low-Entropy Distribution: Penalizing low-entropy distribution is implemented for the low-entropy distribution presented by the model output. For the SoftMax output of the neural network, the entropy of the conditional distribution is calculated and added as a regularization term to the loss function. S3.1.2 Fast Convergence and Overfitting Prevention: Set a threshold. When the entropy of the output distribution falls below this threshold, perform fast convergence and overfitting prevention operations. The expression is as follows: ； In the formula, Represents model input Next Prediction Category The conditional probability distribution; This represents a hyperparameter for penalizing low-entropy distributions, used to control the strength of the entropy regularization term; Indicates the entropy threshold; Represents the negative entropy term; S3.1.3, The deviation of the output distribution from the uniform distribution is penalized through label smoothing operation to smooth the distribution; the expression is as follows: ； In the formula, Represents model input Next Prediction Category The conditional probability distribution; This represents the Kullback-Leibler divergence, used to measure the difference between two probability distributions; It indicates a uniform distribution.
3. The multimodal heterogeneous data fusion and characterization method with enhanced structural information according to claim 1, characterized in that, The design structure information optimization framework and soft allocation mechanism are used to improve the quality of data representation and discriminative power. The expression of the soft allocation mechanism is as follows: ； In the formula, and Represents vertices and vertex The feature vectors are derived from the row vectors of the multimodal feature matrix; Represents the inner product of eigenvectors; This represents the temperature parameter, used to control the weight distribution; Represents vertices The set of neighbors, among which, It is the vertex The index of the neighbor set; Represents vertices Neighboring vertices eigenvectors; superscript Indicates transpose; Represents a node and nodes The connection weights between them.
4. The multimodal heterogeneous data fusion and characterization method with enhanced structural information according to claim 1, characterized in that, The steps for constructing a hypergraph structure and calculating its structural entropy, based on the obtained graph structure, the discriminative representation learning framework with structural entropy regularization, the structural information optimization framework, and the soft allocation mechanism, are as follows: S5.1 Define the hypergraph structure; S5.2 Define the cut edges and cut edge volumes of the hypergraph structure; S5.3, Generate the incidence matrix of the hypergraph structure; S5.4 Construct a clique adjacency matrix based on the generated incidence matrix; S5.5 Calculate the hypergraph entropy and cut edge entropy of the hypergraph structure. The expression for the hypergraph entropy is as follows: ； In the formula, Represents the hypergraph entropy; Represents the vertices in a hypergraph structure. The set of neighbors; Represents the vertices in a hypergraph structure; Represents vertices in a hypergraph structure and vertex Soft links.
5. The multimodal heterogeneous data fusion and characterization method with enhanced structural information according to claim 4, characterized in that, The hypergraph structure is defined as follows: S5.1 Define the hypergraph structure, specifically including: Define an undirected weighted hypergraph This undirected weighted hypergraph contains a set of vertices. Hyperedge set and weight matrix The specific definition is as follows: S5.1.1 Constructing the Vertex Set of the Hypergraph ; The set of vertices of a hypergraph The set of vertices in the graph structure corresponds to the data samples, i.e. The row vector; S5.1.2 Constructing the set of hyperedges of a hypergraph ; The set of superedges of a hypergraph Each hyperedge connects a set of related hypergraph vertices. The method for constructing hyperedges is as follows: For each vertex in the feature space of the hypergraph structure, the K-nearest neighbor algorithm is used to calculate cross-modal similar neighbors; S5.1.3 Constructing the weight matrix of the hypergraph ; Hypergraph weight matrix The weight of each hyperedge is stored using the following expression: ； In the formula, Indicates the super edge. ; Represents the two vertices of a superedge; Representing vertices in the feature space and ; Represents vertices and Squared distance in the feature space; This represents the bandwidth parameter of the Gaussian kernel function.
6. The multimodal heterogeneous data fusion and characterization method with enhanced structural information according to claim 1, characterized in that, The steps for guiding multimodal unsupervised clustering based on hypergraph structure entropy include: S6.1, Multimodal data fusion; S6.2, Structural entropy regularization-guided clustering, the expression is as follows: ； In the formula, Represents the basic clustering loss function; The regularization coefficient representing the balance structure entropy term and the basic loss; Represents vertices in a hypergraph structure structural entropy S6.3 Clustering result optimization; The clustering results are optimized by iteratively optimizing the cluster centers and cluster labels until the minimum convergence condition is met. The iterative optimization process can be represented as: ； In the formula, Indicates the first During the nth iteration, the 1st The center vectors of each cluster; This represents the set of data points belonging to the i-th cluster; Represents the vertices of a hypergraph structure eigenvectors.