Causal-invariant transformation-based multimodal data generalization learning method and system

Abstract

The application relates to a multi-modal data generalization learning method and system based on a causal invariant transformation. The method comprises the following steps: acquiring multi-modal data to construct a causal graph; designing a causal invariant transformation to simulate a causal relationship, keeping the causal features in the causal relationship unchanged, changing the non-causal features, and generating new training samples; fusing information of different modes; constructing a self-supervised learning framework, predicting labels of node pairs as a pre-training task, and learning feature representation and a causal structure of multi-modal data. By constructing and analyzing the causal graph, the causal invariant transformation is designed to simulate the causal relationship and generate new training samples, so that the model can learn the feature representation consistent with the causal structure, thereby maintaining stable performance on data of different distributions; the self-supervised learning framework is constructed, no additional labeling information is needed, the labels of node pairs are predicted as a pre-training task to learn the representation and the causal structure of the data, and the generalization ability of the model can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data processing, in particular to a multi-modal data generalization learning method and system based on causal invariant transformation. BACKGROUND

[0002] With the development of artificial intelligence, machine learning and deep learning technologies continue to break through, making multi-modal data generalization learning a hot research topic. The core goal of multi-modal data generalization learning is to give the model the ability to process and accurately predict out-of-distribution (OOD) multi-modal data, which has great application value in natural language processing, computer vision, and recommendation systems.

[0003] Existing multi-modal learning models often focus on single-modal feature extraction and fusion when processing data, ignoring the causal relationship between modalities. This single-modal processing strategy cannot fully utilize the complementary information between different modalities, resulting in limited generalization ability of the model in the face of OOD data; for example, some models may rely too much on keywords in text data, while ignoring emotional cues in image or audio modalities, which limits the model's comprehensive understanding and application of multi-modal information. In addition, existing technologies often use simple data augmentation or feature fusion strategies in multi-modal data processing, lacking in-depth analysis of data generation mechanisms, which leads to the learned features of the model containing a large amount of noise unrelated to the task, affecting the model's prediction performance on OOD data; at the same time, existing models often ignore the model's interpretability and stability when designing, which is crucial for building user trust and understanding model prediction results. The black-box nature of existing models makes it difficult to explain the prediction process of integrating different modal information.

[0004] Therefore, traditional multi-modal learning models often have the problem of insufficient model generalization ability when processing data. SUMMARY

[0005] Therefore, in order to solve the above technical problems, a multi-modal data generalization learning method and system based on causal invariant transformation are provided, which can improve the generalization ability of the model and enable the model to extract more rich feature representations.

[0006] A multi-modal data generalization learning method based on causal invariant transformation, the method comprising:

[0007] Obtain multi-modal data, and use causal inference technology to construct a causal graph based on the multi-modal data, and identify the causal relationship between each modality in the multi-modal data based on the causal graph;

[0008] design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, keep a causal feature in the causal relationship unchanged, change a non-causal feature, and generate a new training sample;

[0009] adopt a multi-modal feature fusion technology to fuse information of different modalities based on the training sample, and obtain a fused feature representation;

[0010] construct a self-supervised learning framework, predict a label of a node pair based on the training sample and the fused feature representation as a pre-training task based on the self-supervised learning framework, learn a feature representation and a causal structure of the multi-modal data, and complete multi-modal data generalization learning.

[0011] In one of the embodiments, multi-modal data is acquired, and a causal graph is constructed based on the multi-modal data by using a causal inference technology, including:

[0012] The multi-modal data is acquired, each random variable corresponding to the multi-modal data is determined, and a conditional independence test is performed on each random variable to determine whether there is a direct causal relationship between each random variable;

[0013] Each random variable having a direct causal relationship is taken as a node, and a causal relationship between each random variable is taken as an edge, to obtain a node set and an edge set;

[0014] The edges are oriented based on the node set, the edge set, and the judgment result of whether there is a direct causal relationship, to form a directed acyclic graph as a causal graph.

[0015] In one of the embodiments, the causal relationship between each modality in the multi-modal data is identified based on the causal graph, including:

[0016] Based on the causal graph, a causal effect function is introduced to describe the mutual influence between each node;

[0017] Granger causality test, instantaneous causality analysis, intervention, and counterfactual analysis are adopted to identify the causal relationship according to the mutual influence;

[0018] A structural equation model is used to quantify the identified causal relationship, and the causal graph is optimized and updated.

[0019] In one of the embodiments, a causal invariant transformation is designed, including:

[0020] A transformation operation and a causal feature are defined, and a causal invariant transformation is designed based on the transformation operation and the causal feature;

[0021] When the transformation operation is performed based on the causal invariant transformation, the causal feature is kept unchanged;

[0022] optimizing the causal invariant transformation, completing the causal invariant transformation design.

[0023] In one of the embodiments, a multi-modal feature fusion technology is used to fuse information of different modes based on the training samples, to obtain a fused feature representation, including:

[0024] using a feature extraction network to extract features of each mode in the training samples, to obtain extracted features;

[0025] performing feature alignment processing on the extracted features, and using a corresponding fusion strategy to fuse the aligned features, to obtain a fused feature representation.

[0026] In one of the embodiments, the feature alignment processing on the extracted features includes:

[0027] designing a feature alignment function, and using the feature alignment function to perform feature alignment processing on the extracted features through feature mapping or dimension reduction technology.

[0028] In one of the embodiments, a self-supervised learning framework is constructed, and based on the self-supervised learning framework, a label of a node pair is predicted according to the training samples and the fused feature representation as a pre-training task, to learn a feature representation and a causal structure of the multi-modal data, including:

[0029] defining an objective function of self-supervised learning, performing objective training based on the training samples and the fused feature representation, to obtain a training signal;

[0030] determining a node pair and predicting a label of the node pair, taking the predicted label of the node pair as a task;

[0031] obtaining a sample pair according to the training samples and the fused feature representation, and performing feature contrast learning;

[0032] performing joint learning coordination based on the training signal, the task and the feature contrast learning, to complete the construction of the self-supervised learning framework.

[0033] In one of the embodiments, the label of the node pair is predicted, including:

[0034] determining a data point in the multi-modal data, and obtaining a fused feature representation corresponding to the data point;

[0035] designing a label prediction network according to the data point and the fused feature representation corresponding to the data point;

[0036] using the label prediction network to predict the label of the node pair.

[0037] In one of the embodiments, sample pairs are obtained according to the training samples and the fused feature representation, and feature contrast learning is performed, including:

[0038] According to the training samples, the fused feature representation is obtained to obtain sample pairs, and the sample pairs are divided into positive sample pairs and negative sample pairs.

[0039] The distance difference between the positive sample pairs and the negative sample pairs is calculated, and the distance difference is minimized to learn the feature representation.

[0040] A multi-modal data generalization learning system based on causal invariant transformation, the system comprises:

[0041] A causal graph construction and analysis module is configured to obtain multi-modal data, and construct a causal graph according to the multi-modal data using causal inference technology, and identify causal relationships between different modalities in the multi-modal data based on the causal graph.

[0042] A causal invariant transformation design module is configured to design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, keep the causal features in the causal relationship unchanged, change the acausal features, and generate new training samples.

[0043] A multi-modal feature fusion module is configured to fuse information of different modalities based on the training samples using a multi-modal feature fusion technology to obtain a fused feature representation.

[0044] A self-supervised learning module is configured to construct a self-supervised learning framework, and learn the feature representation and causal structure of the multi-modal data based on the self-supervised learning framework according to the training samples and the fused feature representation to predict the labels of node pairs as a pre-training task, and complete multi-modal data generalization learning.

[0045] The above multi-modal data generalization learning method and system based on causal invariant transformation can make the model learn the feature representation consistent with the causal structure by constructing and analyzing the causal graph to identify the causal relationship between different modalities, designing the causal invariant transformation to simulate the causal relationship and generating new training samples, thereby maintaining stable performance on data of different distributions. By fusing information of different modalities, comprehensive feature representation that can capture common and complementary information in multi-modal data can be extracted. By constructing a self-supervised learning framework, without additional labeled information, the representation and causal structure of the data are learned by predicting the labels of node pairs as a pre-training task, which can effectively improve the generalization ability of the model and enhance the interpretability and stability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 An application environment diagram for the multi-modal data generalization learning method based on causal invariant transformation in one of the embodiments;

[0047] Figure 2 A flowchart of a multi-modal data generalization learning method based on causal invariant transformation in an embodiment;

[0048] Figure 3 A block diagram of a multi-modal data generalization learning system based on causal invariant transformation in an embodiment;

[0049] Figure 4 A structural diagram of a causal graph construction and analysis module in an embodiment;

[0050] Figure 5 A structural diagram of a causal invariant transformation design module in an embodiment;

[0051] Figure 6 A structural diagram of a multi-modal feature fusion module in an embodiment;

[0052] Figure 7 A structural diagram of a self-supervised learning module in an embodiment;

[0053] Figure 8 A multi-modal data generalization learning framework based on causal invariant transformation in an embodiment;

[0054] Figure 9 A flowchart of a medical image diagnosis system in an embodiment;

[0055] Figure 10 A flowchart of a multi-modal sentiment analysis system applied to a social media platform in an embodiment;

[0056] Figure 11 A flowchart of an intelligent traffic monitoring system in an embodiment;

[0057] Figure 12 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0059] The multi-modal data generalization learning method based on causal invariant transformation provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 Figure 1 ​As shown, the application environment includes a computer device 110. The computer device 110 can obtain multi-modal data, and construct a causal graph from the multi-modal data using a causal inference technique, identify causal relationships between different modalities in the multi-modal data based on the causal graph; the computer device 110 can design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, keep the causal features in the causal relationship unchanged, change the non-causal features, and generate new training samples; the computer device 110 can fuse information of different modalities based on the training samples using a multi-modal feature fusion technique to obtain a fused feature representation; the computer device 110 can construct a self-supervised learning framework, and learn feature representations and causal structures of the multi-modal data based on the training samples and the fused feature representation to predict labels of node pairs as a pre-training task, and complete multi-modal data generalization learning. The computer device 110 can be, but is not limited to, various personal computers, notebook computers, smart phones, robots, unmanned aerial vehicles and other devices.

[0060] In one embodiment, as Figure 2 shown, a multi-modal data generalization learning method based on causal invariant transformation is provided, including the following steps:

[0061] Step 202, obtaining multi-modal data, and constructing a causal graph from the multi-modal data using a causal inference technique, identifying causal relationships between different modalities in the multi-modal data based on the causal graph.

[0062] The construction and analysis of the causal graph of multi-modal data is the basis for multi-modal data generalization learning. In multi-modal data generalization learning, understanding the causal relationships between different modalities in the data is crucial to improving the generalization ability of the model. In this embodiment, the construction of the causal graph uses advanced causal inference techniques to accurately reflect the potential causal structure in the multi-modal data. And by identifying the causal relationships between different modalities in the multi-modal data, it can provide a theoretical basis for the subsequent design of causal invariant transformation.

[0063] In one embodiment, the multi-modal data generalization learning method based on causal invariant transformation can also include the process of constructing a causal graph, and the specific process includes: obtaining multi-modal data, determining each random variable corresponding to the multi-modal data, and performing conditional independence test on each random variable to determine whether there is a direct causal relationship between each random variable; taking each random variable with a direct causal relationship as a node, and the causal relationship between each random variable as an edge, to obtain a node set and an edge set; according to the node set, the edge set, and the judgment result of whether there is a direct causal relationship, the edges are oriented to form a directed acyclic graph as a causal graph.

[0064] A causal graph is a directed acyclic graph (DAG) in which nodes represent random variables and edges represent causal relationships between variables. The purpose of constructing a causal graph is to reveal the direct causal effects between variables, thereby providing a structural basis for causal inference.

[0065] In this embodiment, let the set of random variables be... ,in, Let each node represent a node, and d be the dimension of the variable. Construct a causal graph G=(V, E), where V is the set of nodes and E is the set of edges. The steps for constructing the causal graph may include: variable selection: determining all variables to be included in the causal graph; conditional independence testing: testing the conditional independence of variables. Perform conditional independence tests to determine if a direct causal relationship exists; graph orientation: based on the results of the conditional independence tests, orient the edges to form a directed acyclic graph.

[0066] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation may further include a process for identifying causal relationships. The specific process includes: introducing a causal effect function to describe the mutual influence between nodes based on a causal graph; using Granger causality test, instantaneous causal analysis, intervention and counterfactual analysis to identify causal relationships based on mutual influence; using structural equation modeling to quantify the identified causal relationships and optimize and update the causal graph.

[0067] In a causal graph, nodes right The direct impact can be achieved through the edges To formalize this relationship, a causal effect function f is introduced, which describes... How to influence ;in, This indicates other unobserved variables or noise.

[0068] In this embodiment, the mutual influence between nodes can be identified through Granger causality tests, transient causality analysis, intervention, and counterfactual analysis. Specifically, the Granger causality test is used to examine the causal relationship between variables in time series data; transient causality analysis is used to analyze the transient causal effect between variables; and intervention and counterfactual analysis are used to evaluate the causal effect under intervention or counterfactual conditions.

[0069] To further quantify the causal relationships in the cause-effect graph, structural equation modeling (SEM) can be used. In SEM, each variable... It can be represented as a function of its parent node: ;in, yes The set of parent nodes, is from to the causal effect coefficient, is the error term.

[0070] In this embodiment, the learning and optimization process of the causal graph can also be included. Specifically, the process of constructing the causal graph can be regarded as an optimization problem, the goal of which is to find the causal structure that best explains the data, which can be completed through two steps of searching and scoring. Among them, the search is to search for the graph structure that best fits the data distribution in all possible directed acyclic graphs (DAGs); the scoring is to use a scoring function to evaluate the fitting degree of each DAG, such as the Bayesian information criterion or the minimum description length. The optimization problem can be expressed as: ; wherein, is the optimal causal graph, is the scoring function, X is the set of random variables; G is the causal graph. The constructed causal graph not only provides a theoretical basis for the design of causal invariant transformation, but also can be used to guide model training and generalization performance optimization. Through the causal graph, the variables that have the greatest impact on model prediction can be identified, so that more attention can be given to these variables in model design.

[0071] In this embodiment, since the causal relationship of the data can change over time or other factors, the causal graph construction and analysis also includes a dynamic updating mechanism to adapt to changes in data distribution. The mathematical expression of dynamic updating can be expressed as: ; wherein, is the causal graph at time t, is the causal graph at time t+1, is the set of random variables at time t+1, is the update operation.

[0072] Among them, dynamic updating can be realized through online learning or incremental learning. Online learning means that when new data comes, the structure and parameters of the causal graph are updated gradually, instead of rebuilding the entire graph; incremental learning means that when new data is significantly different from existing data, the causal graph is adjusted locally to reflect the new data distribution.

[0073] The constructed causal graph will directly guide the model training. In the model training process, features can be selected, network structures can be designed, or regularization can be applied according to the structure of the causal graph. If the causal graph shows that a variable is the cause of another variable, appropriate weights or constraints can be given in the model. The mathematical expression of integrating the causal graph with model training can be expressed as: ; wherein, represents the optimal model, h is the model, L is the loss function, is the true label of the i-th sample, R is the regularization term, is the regularization coefficient, G is the causal graph.

[0074] By causal graph construction and analysis, the causal relationship in the data can be accurately identified and modeled. Compared with traditional multi-modal learning methods, the generalization ability of the model can be significantly improved when dealing with OOD data. And the experimental results show that in the multi-modal sentiment analysis task, the multi-modal data generalization learning method in the embodiment improves the average accuracy by about 4% and 6% on the MNIST and C-MNIST datasets compared with the existing method DANN, proving the improvement of the model generalization ability.

[0075] In step 204, a causal invariant transformation is designed to simulate the causal relationship and keep the causal features in the causal relationship unchanged, change the non-causal features, and generate new training samples.

[0076] After the causal structure of the data is determined, the next step can be to design a series of causal invariant transformations (CIT) to simulate these causal relationships. The causal invariant transformation aims to keep the causal features of the data unchanged while changing the non-causal features to generate new training samples, thereby helping the model to learn a feature representation consistent with the causal structure. Among them, the design of CIT is based on a core idea: although the observed data may change due to multiple factors, the causal structure of the data should remain unchanged.

[0077] Specifically, the design of CIT follows the following principles: causal feature preservation: the transformation should keep the causal features unchanged; diversity: the transformation should cover a variety of cases to simulate different data distributions; reversibility: the transformation should be reversible to facilitate analysis and experimental verification.

[0078] Among them, CIT can be generated by one of the following three methods: data augmentation: by applying known causal invariant transformations (such as rotation, flipping, etc.) to the original data, new training samples are generated; generative model: using a generative model (such as GANs or VAEs) to learn the latent representation of the data and generate new samples with the same causal features; domain adaptation: through domain adaptation techniques, the data is converted from one domain to another while keeping the causal features unchanged.

[0079] In one embodiment, the multi-modal data generalization learning method based on causal invariant transformation provided can further include the process of designing the causal invariant transformation, and the specific process includes: defining the transformation operation, the causal feature, and designing the causal invariant transformation according to the transformation operation and the causal feature; when the transformation operation is performed based on the causal invariant transformation, the causal feature is kept unchanged; the causal invariant transformation is optimized to complete the design of the causal invariant transformation.

[0080] where T is a transformation operation, g(O) is a causal feature, and T(g(O)) = g(O) indicates that the transformation T is causal invariant, which means that the causal feature g(O) will not change regardless of how the transformation acts on the data O.

[0081] In this embodiment, the optimization goal of CIT is to minimize the distribution difference of data before and after transformation while keeping the consistency of causal features. This can be formalized by an optimization problem as follows: ; where, represents expectation, D is a distribution difference measure, P is a probability distribution, T is a transformation operation, and L is a causal feature consistency loss, is a regularization coefficient, and Y represents a label. For example, O is the first six columns of data, and Y represents the predicted label of the seventh column. The goal of the formula is to find a transformed data T(O) that, while keeping the same causal features as the original data O, has a distribution as close as possible to that of the original data while keeping the consistency of causal features. By minimizing the sum of this two-objective function, new training samples can be generated, which change the acausal features while keeping the causal structure unchanged, thereby helping the model learn a feature representation consistent with the causal structure and improving the model's generalization ability to OOD data. The causal invariant transformation design not only generates new training samples consistent with the causal structure, but also effectively improves the model's generalization ability to OOD data through the combination with the causal graph. In this embodiment, the causal invariant transformation design not only generates new training samples consistent with the causal structure, but also effectively improves the model's generalization ability through the combination with the causal graph.

[0082] Through the causal invariant transformation design, new training samples can be generated, which change the acausal features while keeping the causal structure unchanged. This data augmentation strategy makes the model more robust when facing OOD data.

[0083] Step 206: Adopting a multi-modal feature fusion technique to fuse information from different modalities based on the training samples to obtain a fused feature representation.

[0084] In multi-modal data generalization learning, fusing features from different modalities is key to improving the model's generalization ability. Based on the new training samples generated by CIT, a multi-modal feature fusion technique can be further used to integrate information from different modalities, thereby extracting a comprehensive feature representation that can capture common and complementary information in multi-modal data, thereby improving the model's generalization ability.

[0085] In an embodiment, the provided multi-modal data generalization learning method based on causal invariant transformation can further include an information fusion process, and the specific process includes: using a feature extraction network to extract features of each modality in the training sample to obtain extracted features; performing feature alignment processing on the extracted features, and using a corresponding fusion strategy to perform feature fusion on the aligned features to obtain a fused feature representation.

[0086] First, for each modality, a specific feature extraction network can be used to extract features. For example, for image modalities, a convolutional neural network is used to extract visual features; for text modalities, a Transformer is used to extract text features. The feature extraction process can be represented as: ; wherein, is the feature representation of the i-th modality, is the input data of the i-th modality, is the feature extraction function of the i-th modality, is the parameter of the function.

[0087] In an embodiment, when performing feature alignment processing, it specifically includes: designing a feature alignment function, and using the feature alignment function to perform feature alignment processing on the extracted features through feature mapping or dimension reduction techniques.

[0088] Specifically, since the features of different modalities can have different distributions and dimensions, feature alignment is needed to ensure that they can be effectively fused. Feature alignment can be achieved through feature mapping or dimension reduction techniques, and the feature alignment process can be represented as: ; wherein, is the aligned feature representation, is the feature alignment function, is the feature representation of the i-th modality, is the parameter of the feature alignment function.

[0089] Since feature fusion is a core step in multi-modal learning, various fusion strategies can be used to integrate features of different modalities, including but not limited to weighted fusion, series fusion, and cascade fusion. The choice of fusion strategy depends on the features of different modalities and task requirements. The feature fusion process can be represented as: ; wherein, F is the fused feature representation, is the feature fusion function, is the parameter of the function, is each aligned feature representation that needs to be fused, and m is the number of modalities.

[0090] The multi-modal feature fusion module integrates information of different modalities to extract more rich and robust feature representations. On the MVSA dataset, by fusing text and image modalities, higher classification accuracy and robustness are exhibited compared to using only a single modality, thereby improving the quality of data representation.

[0091] At step 208, a self-supervised learning framework is constructed, and based on the self-supervised learning framework, a label of a node pair is predicted as a pre-training task based on the training sample and the fused feature representation, so as to learn the feature representation and the causal structure of the multi-modal data, and complete the generalization learning of the multi-modal data.

[0092] In order to effectively utilize the generated new training sample and the fused feature representation, a self-supervised learning framework can be constructed. The framework does not require additional labeled information, but learns the representation and causal structure of the data by predicting the label of the node pair as a pre-training task. In a multi-modal data set, the acquisition of labeled data is often costly and time-consuming. Self-supervised learning can effectively reduce the dependence on labeled data and improve the generalization ability of the model by utilizing the structural information of the data itself as a supervision signal.

[0093] In one embodiment, the multi-modal data generalization learning method based on causal invariant transformation further includes a process of constructing a self-supervised learning framework, and the specific process includes: defining an objective function of self-supervised learning, performing target training based on the training sample and the fused feature representation to obtain a training signal; determining a node pair and predicting a label of the node pair, taking the predicted label of the node pair as a task; obtaining a sample pair based on the training sample and the fused feature representation, and performing feature contrast learning; and performing joint learning coordination based on the training signal, the task, and the feature contrast learning to complete the construction of the self-supervised learning framework.

[0094] In this embodiment, the self-supervised learning module adopted is based on the following objective function: ; wherein, represents expectation, is the objective function, is the data distribution, is the data pair (such as the original data and its transformed version), is the model prediction function, is a contrast loss function for pulling the representations of the same data points in different views closer.

[0095] In one embodiment, the multi-modal data generalization learning method based on causal invariant transformation further includes a process of predicting the label of the node pair, and the specific process includes: determining a data point in the multi-modal data, obtaining a fused feature representation corresponding to the data point; designing a label prediction network according to the data point and the fused feature representation corresponding to the data point; and predicting the label of the node pair using the label prediction network.

[0096] A key task in self-supervised learning is to predict the labels of node pairs, which can be achieved by designing a label prediction network. The label prediction network can be represented as: ; wherein, is a label prediction network, and are the fusion feature representations of data points O and , respectively, is a label prediction function.

[0097] In one embodiment, the multi-modal data generalization learning method based on causal invariant transformation provided can further include a feature contrast learning process, and the specific process includes: obtaining a sample pair according to a training sample and a fusion feature representation, dividing the sample pair into a positive sample pair and a negative sample pair; calculating a distance difference between the positive sample pair and the negative sample pair, and minimizing the distance difference to learn the feature representation.

[0098] In order to improve the learning efficiency, the feature contrast learning method is used in the embodiment to learn the feature representation by minimizing the distance difference between the positive sample pair and the negative sample pair: ; wherein, is a learned feature representation, is a distance function between feature representations, and K is the number of negative samples.

[0099] Self-supervised learning enables the multi-modal data generalization learning method in the embodiment to effectively learn without or with only a small amount of labeled data, which is particularly important in application scenarios where the labeling cost is high or it is difficult to obtain sufficient labeled data.

[0100] In the embodiment, by constructing and analyzing a causal graph to identify the causal relationship between different modalities, and then designing a causal invariant transformation to simulate the causal relationship and generate new training samples, the model can learn a feature representation consistent with the causal structure, thereby maintaining stable performance on data of different distributions; by fusing information of different modalities, a comprehensive feature representation that can capture common and complementary information in multi-modal data can be extracted; by constructing a self-supervised learning framework, without additional labeled information, by predicting the labels of node pairs as a pre-training task, the representation and causal structure of the data are learned, which can effectively improve the generalization ability of the model and enhance the interpretability and stability of the model.

[0101] And the multi-modal data generalization learning method based on causal invariant transformation provided in this embodiment allows more processing in the local environment of data generation, such as edge devices, reducing the computing and storage requirements of the central server, which not only reduces the communication overhead, but also improves the efficiency of the overall system; It has good adaptability and scalability and can be easily applied to different multi-modal data sets and analysis tasks, whether in social media analysis, recommendation systems or sentiment analysis, etc. It can provide effective solutions, and it is easy to adjust and optimize according to specific application scenarios; and the causal graph construction and analysis provide a clear representation of causal relationships, which not only helps to improve the generalization ability of the model, but also enhances the interpretability of the model, making the decision-making process of the model more transparent.

[0102] It should be understood that although each step in the above flowchart is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0103] In one embodiment, as shown in Figure 3 A multi-modal data generalization learning system based on causal invariant transformation is provided, comprising: a causal graph construction and analysis module 310, a causal invariant transformation design module 320, a multi-modal feature fusion module 330 and a self-supervised learning module 340, wherein:

[0104] The causal graph construction and analysis module 310 is configured to obtain multi-modal data, construct a causal graph based on the multi-modal data using causal inference technology, and identify the causal relationships between different modalities in the multi-modal data based on the causal graph;

[0105] The causal invariant transformation design module 320 is configured to design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, keep the causal features in the causal relationship unchanged, change the non-causal features, and generate new training samples;

[0106] The multi-modal feature fusion module 330 is configured to fuse the information of different modalities based on the training samples using a multi-modal feature fusion technology to obtain a fused feature representation;

[0107] The self-supervised learning module 340 is configured to construct a self-supervised learning framework, and learn the feature representation and causal structure of the multi-modal data based on the self-supervised learning framework, according to the training sample and the fusion feature representation to predict the label of the node pair as a pre-training task, and complete the multi-modal data generalization learning.

[0108] In one embodiment, the causal graph construction and analysis module 310 is further configured to obtain the multi-modal data, determine respective random variables corresponding to the multi-modal data, and perform a conditional independence test on the respective random variables to determine whether there is a direct causal relationship between the respective random variables; take each random variable having a direct causal relationship as a node, and take the causal relationship between the respective random variables as an edge, to obtain a node set and an edge set; and perform orientation on the edges according to the node set, the edge set, and the judgment result of whether there is a direct causal relationship, to form a directed acyclic graph as the causal graph.

[0109] In one embodiment, the causal graph construction and analysis module 310 is further configured to introduce a causal effect function to describe the mutual influence between the respective nodes based on the causal graph; perform causal relationship identification according to the mutual influence by using Granger causality test, instantaneous causality analysis, intervention, and counterfactual analysis; quantize the identified causal relationship by using a structural equation model, and update the causal graph.

[0110] The structure of the causal graph construction and analysis module is shown in Figure 4 The structure of the causal graph construction and analysis module includes the following main parts: variable selection, responsible for selecting relevant variables from the multi-modal data to provide a basis for causal relationship identification; conditional independence test, responsible for performing a conditional independence test on the variable pairs to determine whether there is a direct causal relationship between the variables; graph orientation, according to the result of the conditional independence test, to orient the edges in the causal graph to form a directed acyclic graph; causal graph optimization, responsible for searching and scoring all possible DAG structures to find the optimal causal graph that best reflects the causal relationships in the data; and dynamic update mechanism, responsible for dynamically updating the causal graph according to new data or environmental changes to ensure that it always reflects the latest data distribution.

[0111] In Figure 4In this diagram, arrows represent data flow and information transmission. The output of variable selection points to the conditional independence test via an arrow, indicating the transmission of variable selection information; the output of the conditional independence test points to the graph orientation via an arrow, indicating the transmission of the conditional independence test results; the output of the graph orientation points to the causal graph optimization via an arrow, indicating the transmission of the causal graph structure; after optimizing the causal graph, its output points to the dynamic update mechanism via an arrow, indicating the transmission of the optimal causal graph; the dynamic update mechanism, after dynamically updating the causal graph, feeds back the update results to the conditional independence test and the graph orientation, indicating the transmission of dynamic update information.

[0112] In one embodiment, the causal invariant transformation design module 320 is further configured to define transformation operations and causal features, and design a causal invariant transformation based on the transformation operations and causal features; maintain the causal features unchanged when performing transformation operations based on the causal invariant transformation; optimize the causal invariant transformation, and complete the causal invariant transformation design.

[0113] The structural diagram of the causal invariant transformation design module is as follows: Figure 5 The diagram illustrates the design process of causal invariant transformation (CIT) for multimodal data generalization learning. Specifically, the CIT design module comprises the following main parts: Causal Feature Preservation: This ensures that the transformation operation does not alter the causal characteristics of the data, a core principle of CIT design; Diversity Generation: This generates diverse CITs to cover different data distributions, enhancing the model's generalization ability; Reversibility Verification: This verifies the reversibility of the transformation, ensuring that the original data can be recovered from the transformed data through the inverse transformation; CIT Generation Method: This generates specific CITs based on the principles of causal feature preservation, diversity, and reversibility; CIT Optimization Objective: This defines and optimizes the objective function of the CIT to minimize the distributional differences of the data before and after the transformation.

[0114] exist Figure 5 In this diagram, arrows represent data flow and information transmission. The output of causal feature preservation points to diversity generation via an arrow, signifying the transmission of the causal feature preservation principle. The output of diversity generation points to reversibility verification via an arrow, signifying the transmission of diversity generation information. Reversibility verification verifies the reversibility of CIT, and its result points to the CIT generation method via an arrow, signifying the transmission of the reversibility verification result. The CIT generation method generates CIT based on these principles, and its output points to the CIT optimization objective via an arrow, signifying the transmission of the CIT generation method. The CIT optimization objective defines the objective function for optimizing CIT, and its result is fed back to causal feature preservation via an arrow, signifying the transmission of the optimization objective.

[0115] In one embodiment, the multi-modal feature fusion module 330 is further configured to perform feature extraction on each modality in the training sample using a feature extraction network to obtain extracted features; perform feature alignment processing on the extracted features, and perform feature fusion on the aligned features using a corresponding fusion strategy to obtain a fused feature representation.

[0116] In one embodiment, the multi-modal feature fusion module 330 is further configured to design a feature alignment function, and perform feature alignment processing on the extracted features using the feature alignment function through feature mapping or dimension reduction techniques.

[0117] The structure of the multi-modal feature fusion module is shown in Figure 6 , which shows the multi-modal feature fusion process of multi-modal data generalization learning. Specifically, the structure of the multi-modal feature fusion module includes the following main parts: feature extraction, responsible for extracting features from the original data of each modality, providing a basis for subsequent feature fusion; feature alignment, responsible for aligning features of different modalities to ensure that they have the same feature space and scale; feature fusion strategy, responsible for selecting and designing appropriate feature fusion strategies according to task requirements and modality characteristics; fused feature optimization, responsible for optimizing the fused feature representation to improve the performance and generalization ability of the model.

[0118] In Figure 6 , each part is represented by an arrow indicating data flow and information transmission. The output of feature extraction is directed to feature alignment by an arrow, indicating the transmission of features extracted from the original data; after feature alignment, its output is directed to the feature fusion strategy by an arrow, indicating the transmission of aligned features; the feature fusion strategy generates fused features according to the fusion strategy, and its output is directed to the fused feature optimization by an arrow, indicating the transmission of fused features; the fused feature optimization optimizes the fused features to further improve the performance of the model.

[0119] In one embodiment, the self-supervised learning module 340 is further configured to define a target function of self-supervised learning, perform target training based on the training sample and the fused feature representation to obtain a training signal; determine a node pair and predict the label of the node pair as a task; obtain a sample pair based on the training sample and the fused feature representation, and perform feature contrast learning; perform joint learning coordination based on the training signal, the task, and the feature contrast learning to complete the construction of the self-supervised learning framework.

[0120] In one embodiment, the self-supervised learning module 340 is further configured to determine a data point in the multi-modal data, obtain a fused feature representation corresponding to the data point; design a label prediction network according to the data point and the fused feature representation corresponding to the data point; and predict the label of the node pair using the label prediction network.

[0121] In one embodiment, the self-supervised learning module 340 is further configured to divide the sample pairs into positive sample pairs and negative sample pairs according to the training samples and the fused feature representations, calculate the distance difference between the positive sample pairs and the negative sample pairs, and minimize the distance difference to learn the feature representations.

[0122] The structure of the self-supervised learning module is shown in Figure 7 The self-supervised learning process for unsupervised pre-training is shown. Specifically, the structure of the self-supervised learning module includes the following main parts: objective function definition, responsible for defining the objective function of self-supervised learning, providing training signals for the model; node pair label prediction, responsible for predicting the label of a given node pair as the key task of self-supervised learning; feature contrast learning, responsible for learning discriminative feature representations by contrast learning to pull positive sample pairs and push negative sample pairs; joint learning coordination, responsible for integrating information from different views or tasks under the joint learning framework to improve the effect of feature learning.

[0123] In Figure 7 , each part is represented by an arrow to represent data flow and information transmission. The objective function defined by the objective function definition is pointed to the node pair label prediction by the arrow, indicating the transmission of the training target; the output of the node pair label prediction is pointed to the feature contrast learning by the arrow, indicating the transmission of the node pair label prediction result; the feature contrast learning result of the feature contrast learning is pointed to the joint learning coordination by the arrow, indicating the transmission of the contrast learning feature; the joint learning coordination integrates and feeds back the information from multiple parties to the objective function definition as a joint learning coordinator to achieve more effective feature learning.

[0124] In one embodiment, as shown in Figure 8 , a structure diagram of a multi-modal data generalization learning method and system based on causal invariant transformation is provided, that is Figure 8 a multi-modal data generalization learning framework based on causal invariant transformation is shown, which mainly includes the following main parts: causal graph construction and analysis module, responsible for constructing and analyzing the causal graph of the data, providing theoretical basis for the subsequent design of causal invariant transformation; causal invariant transformation design module, designing a series of CIT to simulate the causal relationship in the data and generate new training samples; multi-modal feature fusion module, integrating information of different modalities, extracting comprehensive feature representations, and improving the generalization ability of the model; self-supervised learning module, learning the representation and causal structure of the data by predicting the label of the node pair as the pre-training task without additional annotation information.

[0125] In Figure 8In the specific implementation process, each part is represented by an arrow indicating data flow and information transmission. The output of the causal graph construction and analysis module is directed by an arrow to the causal invariant transformation design module, indicating the transmission of the construction and analysis results of the causal graph; the output of the causal invariant transformation design module is transmitted to the multi-modal feature fusion module through an arrow, indicating the transmission of the new samples generated by CIT; the multi-modal feature fusion module transmits the fused feature representation to the self-supervised learning module, indicating the transmission of the feature fusion results; the data representation obtained by the self-supervised learning module is fed back to the causal graph construction and analysis module and the causal invariant transformation design module, indicating the transmission of the learning results.

[0126] In one embodiment, the causal invariant transformation-based multi-modal data generalization learning method and system can be applied to a medical image diagnosis system, aiming to improve the accuracy and generalization ability of disease diagnosis by combining multi-modal data of medical images such as CT, MRI, and pathological images. As shown in the following figure, the specific implementation process is as follows: Figure 9

[0127] Multi-modal data collection: Collect various medical image data of patients, including CT images, MRI images, and pathological section images, etc.

[0128] Causal graph construction and analysis: Use the causal graph construction and analysis module to model and analyze the causal relationship between different modal data;

[0129] Feature extraction and fusion: Extract and integrate key features in different medical images through the multi-modal feature fusion module;

[0130] Causal invariant transformation generation: Apply causal invariant transformation (CIT) to generate new training samples to enhance the model's generalization ability for data under different imaging conditions;

[0131] Self-supervised learning pre-training: Use the self-supervised learning framework to pre-train the model to learn the representation and causal structure of the data;

[0132] Disease diagnosis model training: Combine the feature representation obtained by pre-training to train the final disease diagnosis model;

[0133] Model evaluation and optimization: Evaluate the model performance on the standard test set and optimize the model according to the evaluation results;

[0134] Diagnosis result explanation: Use the causal graph analysis module to provide decision support and enhance the interpretability of the model.

[0135] ​In one embodiment, a multi-modal data generalization learning method and system based on causal invariant transformation is provided, which can be applied in a multi-modal sentiment analysis system on a social media platform, aiming to improve the accuracy and generalization ability of sentiment analysis by combining text, image and audio data. As shown in Figure 10 the specific implementation process is as follows:

[0136] Multi-modal data collection: Collect user posts from social media platforms, including text content, attached pictures and audio information;

[0137] Causal graph construction and analysis: Use the causal graph construction and analysis module to model and analyze the causal relationship between text, image and audio data;

[0138] Feature extraction and fusion: Through the multi-modal feature fusion module, extract and integrate key emotional features in different data modalities;

[0139] Causal invariant transformation generation: Apply causal invariant transformation (CIT) to generate new training samples, enhance the generalization ability of the model to different emotional expression modes;

[0140] Self-supervised learning pre-training: Use the self-supervised learning framework to pre-train the model to learn the representation and causal structure of the data;

[0141] Sentiment analysis model training: Combine the feature representation obtained by pre-training to train the final multi-modal sentiment analysis model;

[0142] Model evaluation and optimization: Evaluate the model performance on the standard test set, and optimize the model according to the evaluation results;

[0143] Application of sentiment analysis results: Apply the trained model to real-time social media content analysis to provide emotional tendency feedback to users.

[0144] In one embodiment, a multi-modal data generalization learning method and system based on causal invariant transformation is provided, which can be applied in an intelligent traffic monitoring system, aiming to improve the accuracy and response speed of abnormal event monitoring by combining video monitoring, traffic flow data and environmental sensor data. As shown in Figure 11 the specific implementation process is as follows:

[0145] Multi-modal data collection: Collect various data in the traffic monitoring system, including video stream, traffic flow data and environmental sensor readings;

[0146] Causal graph construction and analysis: Use the causal graph construction and analysis module to model and analyze the causal relationship between video, traffic flow and environmental data;

[0147] Feature extraction and fusion: Through a multi-modal feature fusion module, key features in different data sources are extracted and integrated for feature representation of abnormal events;

[0148] Causal invariant transformation generation: causal invariant transformation (CIT) is applied to generate new training samples, enhancing the model's generalization ability for abnormal events under different traffic conditions;

[0149] Self-supervised learning pre-training: a self-supervised learning framework is used to pre-train the model to learn the representation and causal structure of the data;

[0150] Abnormal event monitoring model training: combining the feature representation obtained by pre-training, the final abnormal event monitoring model is trained;

[0151] Model evaluation and optimization: evaluate the model performance on the historical traffic event dataset, and optimize the model according to the evaluation results;

[0152] Real-time abnormal event response: apply the trained model to real-time traffic monitoring to achieve rapid detection and automatic alarm of abnormal events.

[0153] In an embodiment, a computer device, which can be a terminal, can have an internal structure diagram as shown in Figure 12 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a multi-modal data generalization learning method based on causal invariant transformation. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad provided on the computer device housing. It can also be an external keyboard, touchpad, or mouse, etc.

[0154] Those skilled in the art can understand that Figure 12 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of the method for multi-modal data generalization learning based on causal invariant transformation when executing the computer program.

[0156] In one embodiment, a computer readable storage medium is provided, storing a computer program, the computer program implementing the steps of the method for multi-modal data generalization learning based on causal invariant transformation when executed by a processor.

[0157] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium and can include the processes of the above-mentioned embodiments when executed. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM) and the like.

[0158] Any combination of the technical features of the above embodiments can be made, and in order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0159] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-modal data generalization learning method based on causal invariant transformation, characterized in that, The method comprises: acquiring multi-modal data, and constructing a causal graph based on the multi-modal data using a causal inference technique, and identifying a causal relationship between each modality in the multi-modal data based on the causal graph; designing a causal invariant transformation, including: defining a transformation operation, a causal feature, and designing a causal invariant transformation based on the transformation operation and the causal feature; when the transformation operation is performed based on the causal invariant transformation, the causal feature is kept unchanged; the causal invariant transformation is optimized to complete the design of the causal invariant transformation; the causal relationship is simulated through the causal invariant transformation, the causal feature in the causal relationship is kept unchanged, the non-causal feature is changed, and a new training sample is generated; wherein, the causal invariant transformation is generated in the following ways: data enhancement: a new training sample is generated by applying a known causal invariant transformation to the original data; or, a generation model: a generation model is used to learn the latent representation of the data and generate new samples with the same causal feature; or, domain adaptation: through domain adaptation technology, the data is converted from one domain to another while keeping the causal feature unchanged; using a multi-modal feature fusion technology to fuse information of different modalities based on the training sample to obtain a fused feature representation; constructing a self-supervised learning framework, and learning a feature representation and a causal structure of the multi-modal data based on the training sample and the fused feature representation as a pre-training task to predict a label of a node pair based on the self-supervised learning framework, complete multi-modal data generalization learning, including: defining an objective function of self-supervised learning, performing target training based on the training sample and the fused feature representation to obtain a training signal; determining a node pair and predicting a label of the node pair as a task; obtaining a sample pair based on the training sample and the fused feature representation, and performing feature contrast learning; performing joint learning coordination based on the training signal, the task, and the feature contrast learning to complete the construction of the self-supervised learning framework. 2.The multi-modal data generalization learning method based on causal invariant transformation of claim 1, wherein, acquiring multi-modal data, and constructing a causal graph based on the multi-modal data using a causal inference technique, including: acquiring multi-modal data, determining each random variable corresponding to the multi-modal data, and performing conditional independence test on each random variable to determine whether there is a direct causal relationship between each random variable; each random variable having a direct causal relationship is taken as a node, and a causal relationship between each random variable is taken as an edge to obtain a node set and an edge set; directing the edges based on the node set, the edge set, and the judgment result of whether there is a direct causal relationship to form a directed acyclic graph as a causal graph. 3.The multi-modal data generalization learning method based on causal invariant transformation of claim 2, wherein, based on the causal graph, introducing a causal effect function to describe the mutual influence between each node; using Granger causality test, instantaneous causality analysis, intervention, and counterfactual analysis to identify the causal relationship based on the mutual influence; using a structural equation model to quantify the identified causal relationship, and updating the causal graph. ​ 4.The multi-modal data generalization learning method based on causal invariant transformation of claim 1, wherein, The multi-modal feature fusion technology is used to fuse information of different modalities based on the training samples to obtain a fused feature representation, including: The feature extraction network is used to extract features of each modality in the training samples to obtain extracted features; The extracted features are subjected to feature alignment processing, and the aligned features are subjected to feature fusion using a corresponding fusion strategy to obtain a fused feature representation.

5. The multi-modal data generalization learning method based on the causally invariant transformation according to claim 4, characterized in that, The extracted features are subjected to feature alignment processing, including: A feature alignment function is designed, and the extracted features are subjected to feature alignment processing through feature mapping or dimension reduction technology using the feature alignment function.

6. The causal invariance transformation based multi-modal data generalization learning method according to claim 1, characterized in that, The label of the node pair is predicted, including: A data point in the multi-modal data is determined, and a fused feature representation corresponding to the data point is obtained; A label prediction network is designed according to the data point and the fused feature representation corresponding to the data point; The label prediction network is used to predict the label of the node pair.

7. The causal invariance transformation based multi-modal data generalization learning method according to claim 1, characterized in that, The sample pair is obtained according to the training sample and the fused feature representation, and feature contrast learning is performed, including: The sample pair is obtained according to the training sample and the fused feature representation, and the sample pair is divided into positive sample pairs and negative sample pairs; The distance difference between the positive sample pairs and the negative sample pairs is calculated, and the distance difference is minimized to learn the feature representation.

8. A multi-modal data generalization learning system based on causally invariant transformations, characterized by, The system includes: A causal graph construction and analysis module is configured to obtain multi-modal data, construct a causal graph based on the multi-modal data using causal inference technology, and identify causal relationships between modalities in the multi-modal data based on the causal graph; A causal invariant transformation design module is configured to design a causal invariant transformation, including: defining a transformation operation and a causal feature, and designing a causal invariant transformation based on the transformation operation and the causal feature; when the transformation operation is performed based on the causal invariant transformation, the causal feature remains unchanged; the causal invariant transformation is optimized to complete the design of the causal invariant transformation; the causal relationship is simulated through the causal invariant transformation, the causal feature in the causal relationship remains unchanged, the non-causal feature is changed, and a new training sample is generated; wherein, the causal invariant transformation generation mode is: data enhancement: a new training sample is generated by applying a known causal invariant transformation to the original data; or, generation model: a generation model is used to learn the latent representation of the data and generate a new sample with the same causal feature; or, domain adaptation: through domain adaptation technology, the data is converted from one domain to another domain while keeping the causal feature unchanged; A multi-modal feature fusion module is configured to fuse information of different modalities based on the training samples using a multi-modal feature fusion technology to obtain a fused feature representation. The self-supervised learning module is configured to construct a self-supervised learning framework, learn the feature representation and causal structure of the multi-modal data based on the self-supervised learning framework, complete the generalization learning of the multi-modal data, and includes the following steps: defining a target function of self-supervised learning, performing target training based on the training sample and the fusion feature representation to obtain a training signal; determining a node pair and predicting the label of the node pair as a task; obtaining a sample pair based on the training sample and the fusion feature representation and performing feature contrast learning; and performing joint learning coordination based on the training signal, the task, and the feature contrast learning to complete the construction of the self-supervised learning framework.

Images