Multi-modal data generalization learning method and system based on causal invariant transformation

Abstract

The invention relates to a multi-modal data generalization learning method and system based on causal invariant transformation. The method comprises the following steps: acquiring multi-modal data to construct a causal graph; designing causal invariant transformation to simulate a causal relationship, keeping causal features in the causal relationship unchanged, changing non-causal features, and generating a new training sample; fusing information of different modes; and constructing a self-supervised learning framework, predicting labels of the node pairs as pre-training tasks, and learning feature representation and causal structures of the multi-modal data. A causal graph is constructed and analyzed, causal invariant transformation is designed to simulate a causal relationship and generate a new training sample, and the model can learn feature representation consistent with a causal structure, so that stable performance is kept on different distributed data; according to the method, a self-supervised learning framework is constructed, extra labeling information is not needed, data representation and causal structures are learned by predicting labels of node pairs as pre-training tasks, and the generalization ability of the model can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a multimodal data generalization learning method and system based on causal invariant transformation. Background Art

[0002] With the development of artificial intelligence, machine learning and deep learning technologies have made continuous breakthroughs, making multimodal data generalization learning a hot topic of research. The core goal of multimodal data generalization learning is to give the model the ability to process and accurately predict out-of-distribution (OOD) multimodal data. This ability has significant application value in many fields such as natural language processing, computer vision, and recommendation systems.

[0003] When processing data, existing multimodal learning models often focus on the feature extraction and fusion of a single modality, ignoring the causal relationship between modalities. This single-modal processing strategy cannot fully utilize the complementary information between different modalities, resulting in limited generalization ability of the model in the face of OOD data; for example, some models may over-rely on keywords in text data and ignore emotional cues in image or audio modalities, which limits the model's comprehensive understanding and application of multimodal information. In addition, existing technologies often use simple data enhancement or feature fusion strategies in multimodal data processing, lacking in-depth analysis of data generation mechanisms, which results in the features learned by the model containing a large amount of noise that is irrelevant to the task, affecting the model's prediction performance on OOD data; at the same time, existing models often ignore the interpretability and stability of the model when designing, which is crucial for building user trust and understanding model prediction results. The black-box characteristics of existing models make it difficult to explain the prediction process of integrating information from different modalities.

[0004] Therefore, traditional multimodal learning models often have the problem of insufficient model generalization ability when processing data. Summary of the invention

[0005] Based on this, in order to solve the above technical problems, a multimodal data generalization learning method and system based on causal invariant transformation is provided, which can improve the generalization ability of the model and enable the model to extract richer feature representations.

[0006] A multimodal data generalization learning method based on causal invariant transformation, the method comprising: Acquire multimodal data, and construct a causal graph based on the multimodal data using causal inference technology, and identify the causal relationship between each mode in the multimodal data based on the causal graph; Designing a causal invariant transformation, simulating the causal relationship through the causal invariant transformation, keeping the causal features in the causal relationship unchanged, changing the non-causal features, and generating new training samples; Using multimodal feature fusion technology to fuse information of different modalities based on the training samples to obtain a fused feature representation; A self-supervised learning framework is constructed, and based on the self-supervised learning framework, labels of node pairs are predicted according to the training samples and fused feature representations as pre-training tasks to learn the feature representation and causal structure of the multimodal data and complete the generalization learning of multimodal data.

[0007] In one embodiment, obtaining multimodal data and constructing a causal graph based on the multimodal data using a causal inference technique includes: Acquire multimodal data, determine each random variable corresponding to the multimodal data, and perform a conditional independence test on each random variable to determine whether there is a direct causal relationship between each random variable; Taking random variables with direct causal relationships as nodes and the causal relationships between the random variables as edges, a node set and an edge set are obtained; According to the judgment result of whether the node set, the edge set, and the direct causal relationship exist, the edges are directed to form a directed acyclic graph as a causal graph.

[0008] In one embodiment, identifying the causal relationship between each modality in the multimodal data based on the causal graph includes: Based on the causal graph, a causal effect function is introduced to describe the mutual influence between each of the nodes; Granger causality test, instantaneous causal analysis, intervention and counterfactual analysis were used to identify causal relationships based on the mutual influences; The identified causal relationships were quantified using structural equation modeling, and the causal diagram was optimized and updated.

[0009] In one embodiment, a causal invariant transformation is designed, including: defining transformation operations and causal characteristics, and designing causal invariant transformations based on the transformation operations and causal characteristics; When performing the transformation operation based on the causal invariant transformation, the causal feature is kept unchanged; The causal invariant transformation is optimized to complete the causal invariant transformation design.

[0010] In one embodiment, a multimodal feature fusion technology is used to fuse information of different modalities based on the training sample to obtain a fused feature representation, including: Using a feature extraction network to extract features from each modality in the training sample to obtain extracted features; The extracted features are subjected to feature alignment processing, and the aligned features are subjected to feature fusion using a corresponding fusion strategy to obtain a fused feature representation.

[0011] In one embodiment, performing feature alignment processing on the extracted features includes: A feature alignment function is designed, and the feature alignment function is used to perform feature alignment processing on the extracted features through feature mapping or dimensionality reduction technology.

[0012] In one embodiment, a self-supervised learning framework is constructed, and labels of node pairs are predicted based on the training samples and fused feature representations as pre-training tasks to learn feature representations and causal structures of the multimodal data, including: Defining an objective function of self-supervised learning, performing target training based on the training samples and fusion feature representation, and obtaining a training signal; Determine a node pair, predict a label of the node pair, and use the predicted label of the node pair as a task; Obtaining sample pairs according to the training samples and fused feature representations, and performing feature comparison learning; Based on the training signals, tasks, and feature contrast learning, joint learning coordination is performed to complete the construction of a self-supervised learning framework.

[0013] In one embodiment, predicting the label of the node pair includes: Determine a data point in the multimodal data, and obtain a fused feature representation corresponding to the data point; Designing a label prediction network according to the data points and the fusion feature representations corresponding to the data points; The label prediction network is used to predict labels for the node pairs.

[0014] In one embodiment, obtaining sample pairs based on the training samples and fused feature representations, and performing feature comparison learning, includes: According to the training samples and the fused feature representation, sample pairs are obtained, and the sample pairs are divided into positive sample pairs and negative sample pairs; The distance difference between the positive sample pair and the negative sample pair is calculated, and the feature representation is learned by minimizing the distance difference.

[0015] A multimodal data generalization learning system based on causal invariant transformation, the system comprising: A causal graph construction and analysis module, which is used to obtain multimodal data, and use causal inference technology to construct a causal graph based on the multimodal data, and identify the causal relationship between each mode in the multimodal data based on the causal graph; A causal invariant transformation design module, used to design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, keep the causal features in the causal relationship unchanged, change the non-causal features, and generate new training samples; A multimodal feature fusion module, used to fuse information of different modalities based on the training samples using a multimodal feature fusion technology to obtain a fused feature representation; The self-supervised learning module is used to construct a self-supervised learning framework. Based on the self-supervised learning framework, the labels of the predicted node pairs are used as pre-training tasks according to the training samples and the fused feature representations to learn the feature representations and causal structures of the multimodal data and complete the generalization learning of the multimodal data.

[0016] The above-mentioned multimodal data generalization learning method and system based on causal invariant transformation, by constructing and analyzing causal graphs to identify causal relationships between different modalities, and then designing causal invariant transformations to simulate causal relationships and generate new training samples, can enable the model to learn feature representations consistent with the causal structure, thereby maintaining stable performance on data with different distributions; by fusing information from different modalities, a comprehensive feature representation that can capture common and complementary information in multimodal data can be extracted; a self-supervised learning framework is constructed without the need for additional annotation information, and the representation and causal structure of data are learned by predicting the labels of node pairs as a pre-training task, which can effectively improve the generalization ability of the model and enhance the interpretability and stability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an application environment diagram of a multimodal data generalization learning method based on causal invariant transformation in one embodiment; Figure 2 1 is a flow chart of a multimodal data generalization learning method based on causal invariant transformation in one embodiment; Figure 3 is a structural block diagram of a multimodal data generalization learning system based on causal invariant transformation in one embodiment; Figure 4 A schematic diagram of the structure of a cause-effect graph construction and analysis module in one embodiment; Figure 5 A schematic diagram of the structure of a causal invariant transformation design module in one embodiment; Figure 6 Schematic diagram of the structure of a multimodal feature fusion module in one embodiment; Figure 7is a schematic diagram of the structure of a self-supervised learning module in one embodiment; Figure 8 Schematic diagram of a multimodal data generalization learning framework based on causal invariant transformation in one embodiment; Fig. 9 is a schematic diagram of a flow chart applied to a medical image diagnosis system in one embodiment; Fig.10 is a schematic diagram of a process of a multimodal sentiment analysis system applied to a social media platform in one embodiment; Fig.11 A schematic diagram of a flow chart applied to an intelligent traffic monitoring system in one embodiment; Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] The multimodal data generalization learning method based on causal invariant transformation provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment includes a computer device 110. The computer device 110 can obtain multimodal data, and use causal inference technology to construct a causal graph based on the multimodal data, and identify the causal relationship between each modality in the multimodal data based on the causal graph; the computer device 110 can design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, and keep the causal features in the causal relationship unchanged, change the non-causal features, and generate new training samples; the computer device 110 can use multimodal feature fusion technology to fuse information of different modalities based on training samples to obtain fused feature representation; the computer device 110 can construct a self-supervised learning framework, and based on the self-supervised learning framework, predict the label of the node pair according to the training samples and the fused feature representation as a pre-training task, learn the feature representation and causal structure of the multimodal data, and complete the generalization learning of multimodal data. Among them, the computer device 110 can be, but not limited to, various personal computers, laptops, smart phones, robots, unmanned aerial vehicles and other devices.

[0020] In one embodiment, Figure 2 As shown, a multimodal data generalization learning method based on causal invariant transformation is provided, comprising the following steps: Step 202, obtaining multimodal data, and using causal inference technology to construct a causal graph based on the multimodal data, and identifying the causal relationship between each mode in the multimodal data based on the causal graph.

[0021] Constructing and analyzing the causal graph of multimodal data is the basis for generalization learning of multimodal data. In generalization learning of multimodal data, understanding the causal relationship between the modes in the data is crucial to improving the generalization ability of the model. In this embodiment, the construction of the causal graph adopts advanced causal inference technology, which can accurately reflect the potential causal structure in multimodal data. And by identifying the causal relationship between the modes in multimodal data, it can provide a theoretical basis for the subsequent design of causal invariant transformation.

[0022] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation is provided, which may also include a process of constructing a causal graph, and the specific process includes: obtaining multimodal data, determining each random variable corresponding to the multimodal data, and performing a conditional independence test on each random variable to determine whether there is a direct causal relationship between each random variable; taking each random variable with a direct causal relationship as a node, and the causal relationship between each random variable as an edge, to obtain a node set and an edge set; and orienting the edges according to the node set, the edge set, and the judgment result of whether a direct causal relationship exists, to form a directed acyclic graph as a causal graph.

[0023] Among them, the causal graph is a directed acyclic graph (DAG), where nodes represent random variables and edges represent causal relationships between variables. The purpose of constructing a causal graph is to reveal the direct causal effects between variables, thereby providing a structural basis for causal inference.

[0024] In this embodiment, the random variable set is assumed to be ,in, Represents each node, d is the dimension of the variable, and a causal graph G=(V, E) is constructed, where V is a set of nodes and E is a set of edges. The implementation steps of causal graph construction may include: variable selection: determine all variables that need to be included in the causal graph; conditional independence test: test the variables for A conditional independence test is performed to determine whether a direct causal relationship exists; Graph orientation: Based on the results of the conditional independence test, the edges are oriented to form a directed acyclic graph.

[0025] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation is provided, which may also include a process of identifying causal relationships, the specific process including: based on the causal graph, introducing a causal effect function to describe the mutual influence between each node; using Granger causality test, instantaneous causal analysis, intervention and counterfactual analysis to identify causal relationships based on mutual influence; using a structural equation model to quantify the identified causal relationships, and optimizing and updating the causal graph.

[0026] In a causal graph, the node right The direct impact can be seen through the side In order to formalize this relationship, the causal effect function f is introduced. The causal effect function describes How it affects ;in, represents other unobserved variables or noise.

[0027] In this embodiment, the mutual influence between nodes can be identified through Granger causality test, instantaneous causal analysis, intervention and counterfactual analysis. Granger causality test is used to test the causal relationship between variables in time series data; instantaneous causal analysis is used to analyze the instantaneous causal effect between variables; intervention and counterfactual analysis is used to evaluate the causal effect under intervention or counterfactual conditions.

[0028] In order to further quantify the causal relationship in the causal diagram, the structural equation model (SEM) can be used. Can be expressed as a function of its parent node: ;in, yes The parent node set of is from arrive The causal effect coefficient, is the error term.

[0029] In this embodiment, the learning and optimization process of the causal graph may also be included. Specifically, the process of constructing a causal graph can be regarded as an optimization problem. The goal is to find the causal structure that best explains the data, which can be completed through two steps: searching and scoring. Among them, the search is to search for the graph structure that best fits the data distribution among all possible directed acyclic graphs DAG; scoring is to use a scoring function to evaluate the fit of each DAG, such as the Bayesian information criterion or the minimum description length. The optimization problem can be expressed as: ;in, is the optimal causal diagram, is the scoring function, X is a set of random variables, and G is a causal graph. The constructed causal graph not only provides a theoretical basis for the design of causal invariant transformations, but can also be used to guide model training and generalization performance optimization. Through the causal graph, the variables that have the greatest impact on model prediction can be identified, so that more attention can be paid to these variables during model design.

[0030] In this embodiment, since the causal relationship of data may change over time or due to other factors, the causal graph construction and analysis also includes a dynamic update mechanism to adapt to changes in data distribution. The mathematical expression of dynamic update can be expressed as: ;in, is the causal graph at time t, is the causal graph at time t+1, is the set of random variables at time t+1, It is an update operation.

[0031] Dynamic updates can be achieved through online learning or incremental learning. Online learning refers to gradually updating the structure and parameters of the causal graph when new data arrives, rather than rebuilding the entire graph; incremental learning refers to making local adjustments to the causal graph to reflect the new data distribution when new data is significantly different from existing data.

[0032] The constructed causal graph will directly guide model training. During model training, features can be selected, network structure can be designed, or regularization can be applied based on the structure of the causal graph. If the causal graph shows that a variable is the cause of another variable, corresponding weights or constraints can be given in the model. The mathematical expression of integrating causal graphs and model training can be expressed as: ;in, represents the optimal model, h is the model, L is the loss function, is the true label of the i-th sample, R is the regularization term, is the regularization coefficient and G is the causal graph.

[0033] By constructing and analyzing the causal graph, the causal relationship in the data can be accurately identified and modeled. Compared with the traditional multimodal learning method, the generalization ability of the model can be significantly improved when processing OOD data. And the experimental results show that in the multimodal sentiment analysis task, the multimodal data generalization learning method in this embodiment improves the average accuracy by about 4% and 6% on the MNIST and C-MNIST data sets compared with the existing method DANN, proving the improvement of the model's generalization ability.

[0034] Step 204, designing a causal invariant transformation, simulating the causal relationship through the causal invariant transformation, keeping the causal features in the causal relationship unchanged, changing the non-causal features, and generating new training samples.

[0035] After clarifying the causal structure of the data, the next step is to design a series of causal invariant transformations (CITs) to simulate these causal relationships. Causal invariant transformations aim to keep the causal features of the data unchanged while changing the non-causal features to generate new training samples, thereby helping the model learn feature representations consistent with the causal structure. Among them, the design of CIT is based on a core concept: although the observed data may change due to various factors, the causal structure of the data should remain unchanged.

[0036] Specifically, the design of CIT follows the following principles: preservation of causal characteristics: the transformation should keep the causal characteristics unchanged; diversity: the transformation should cover a variety of situations to simulate different data distributions; reversibility: the transformation should be reversible to facilitate analysis and experimental verification.

[0037] Among them, CIT can be generated through one of the following three methods: data augmentation: generating new training samples by applying known causal invariant transformations (such as rotation, flipping, etc.) to the original data; generative model: using generative models (such as GANs or VAEs) to learn the latent representation of the data and generate new samples with the same causal characteristics; domain adaptation: through domain adaptation technology, data is converted from one domain to another while keeping the causal characteristics unchanged.

[0038] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation is provided, which may also include a process of designing a causal invariant transformation, the specific process including: defining transformation operations and causal features, and designing a causal invariant transformation based on the transformation operations and causal features; when performing transformation operations based on the causal invariant transformation, keeping the causal features unchanged; optimizing the causal invariant transformation to complete the design of the causal invariant transformation.

[0039] Here, let T be the transformation operation, g(O) be the causal feature, and T(g(O))=g(O) indicates that the transformation T is causally invariant, which means that no matter how the transformation acts on the data O, the causal feature g(O) will not change.

[0040] In this embodiment, the optimization goal of CIT is to minimize the distribution difference of the data before and after the transformation while maintaining the consistency of the causal characteristics. This can be formalized through the optimization problem: ;in, represents expectation, D is the distribution difference measure, P is the probability distribution, T is the transformation operation, L is the causal feature consistency loss, is the regularization coefficient, Y represents the label. For example, O is the first six columns of data, and Y represents the predicted label of the seventh column. The goal of the formula is to find a transformed data T (O), while maintaining the same causal characteristics as the original data O, and its transformed distribution is as close to the distribution of the original data as possible, while maintaining the consistency of the causal characteristics; by minimizing the sum of these two objective functions, new training samples can be generated. These samples change the non-causal characteristics while keeping the causal structure unchanged, thereby helping the model learn the feature representation consistent with the causal structure and improving the model's generalization ability for out-of-distribution OOD data. The causal invariant transformation design can not only generate new training samples that conform to the causal structure, but also effectively improve the model's generalization ability for OOD data by combining with the causal graph. In this embodiment, the causal invariant transformation design can not only generate new training samples that conform to the causal structure, but also effectively improve the model's generalization ability for OOD data by combining with the causal graph.

[0041] Through the causal invariant transformation design, new training samples can be generated. These samples change the non-causal features while keeping the causal structure unchanged. This data augmentation strategy makes the model more robust when facing OOD data.

[0042] Step 206: Multimodal feature fusion technology is used to fuse information of different modalities based on training samples to obtain a fused feature representation.

[0043] In the generalization learning of multimodal data, fusing features from different modalities is the key to improving the generalization ability of the model. Based on the new training samples generated by CIT, multimodal feature fusion technology can be further used to integrate information from different modalities, thereby extracting a comprehensive feature representation that can capture the common and complementary information in multimodal data, thereby improving the generalization ability of the model.

[0044] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation is provided, which may also include an information fusion process, and the specific process includes: using a feature extraction network to extract features for each modality in the training sample to obtain extracted features; performing feature alignment processing on the extracted features, and fusing the aligned features using corresponding fusion strategies to obtain fused feature representations.

[0045] First, for each modality, a specific feature extraction network can be used to extract features. For example, for the image modality, a convolutional neural network is used to extract visual features; for the text modality, a Transformer is used to extract text features. The feature extraction process can be expressed as: ;in, is the characteristic representation of the i-th mode, is the input data of the ith mode, is the feature extraction function of the i-th mode, are the parameters of the function.

[0046] In one embodiment, when performing feature alignment processing, it specifically includes: designing a feature alignment function, and using the feature alignment function to perform feature alignment processing on the extracted features through feature mapping or dimensionality reduction technology.

[0047] Specifically, since features of different modalities may have different distributions and dimensions, feature alignment is required to ensure that they can be effectively fused. Feature alignment can be achieved through techniques such as feature mapping or dimensionality reduction. The feature alignment process can be expressed as: ;in, is the aligned feature representation, is the feature alignment function, is the characteristic representation of the i-th mode, is the parameter of the feature alignment function.

[0048] Since feature fusion is the core step in multimodal learning, a variety of fusion strategies can be used to integrate features of different modalities, including but not limited to weighted fusion, series fusion, cascade fusion, etc. The choice of fusion strategy depends on the features of different modalities and task requirements. The feature fusion process can be expressed as: ; Among them, F is the fused feature representation, is the feature fusion function, are the parameters of the function, are the aligned feature representations that need to be fused, and m is the number of modal types.

[0049] The multimodal feature fusion module integrates information from different modalities to extract richer and more robust feature representations. On the MVSA dataset, by fusing text and image modalities, it demonstrates higher classification accuracy and robustness compared to methods that only use a single modality, thereby improving the quality of data representation.

[0050] Step 208, construct a self-supervised learning framework, based on the self-supervised learning framework, predict the labels of node pairs according to training samples and fused feature representation as pre-training tasks, learn the feature representation and causal structure of multimodal data, and complete the generalization learning of multimodal data.

[0051] In order to effectively utilize the generated new training samples and fused feature representations, a self-supervised learning framework can be constructed. The framework does not require additional annotation information, but learns the representation and causal structure of the data by predicting the labels of node pairs as a pre-training task. In multimodal datasets, the acquisition of labeled data is often costly and time-consuming. Self-supervised learning can effectively reduce the dependence on labeled data by using the structural information of the data itself as a supervisory signal, while improving the generalization ability of the model.

[0052] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation is provided, which may also include a process of constructing a self-supervised learning framework. The specific process includes: defining the objective function of self-supervised learning, performing target training based on training samples and fused feature representations to obtain training signals; determining node pairs, and predicting labels of node pairs, and using the predicted labels of node pairs as tasks; obtaining sample pairs based on training samples and fused feature representations, and performing feature contrast learning; and performing joint learning coordination based on training signals, tasks, and feature contrast learning to complete the construction of the self-supervised learning framework.

[0053] In this embodiment, the self-supervised learning module used is based on the following objective function: ;in, Express expectations, is the objective function, is the data distribution, are data pairs (e.g., original data and its transformed version), is the model prediction function, It is a contrastive loss function that is used to bring the representation of the same data point closer in different views.

[0054] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation is provided, which may also include a process of predicting the labels of node pairs, and the specific process includes: determining data points in the multimodal data, and obtaining fused feature representations corresponding to the data points; designing a label prediction network based on the data points and the fused feature representations corresponding to the data points; and using the label prediction network to predict the labels of the node pairs.

[0055] The key task in self-supervised learning is to predict the labels of node pairs, which can be achieved by designing a label prediction network. The label prediction network can be expressed as: ;in, is the label prediction network, and are data points O and The fusion feature representation is is the label prediction function.

[0056] In one embodiment, a multimodal data generalization learning method based on causal invariant transformation is provided, which may also include a feature contrast learning process, wherein the specific process includes: obtaining sample pairs based on training samples and fused feature representations, and dividing the sample pairs into positive sample pairs and negative sample pairs; calculating the distance difference between the positive sample pairs and the negative sample pairs, and minimizing the distance difference to learn the feature representation.

[0057] In order to improve learning efficiency, this embodiment adopts a feature contrast learning method to learn feature representation by minimizing the distance difference between positive sample pairs and negative sample pairs: ;in, is the learned feature representation, is the distance function between feature representations, and K is the number of negative samples.

[0058] Self-supervised learning enables the multimodal data generalization learning method in this embodiment to perform effective learning with no or only a small amount of labeled data, which is particularly important in application scenarios where labeling costs are high or it is difficult to obtain sufficient labeled data.

[0059] In this embodiment, by constructing and analyzing a causal graph to identify the causal relationship between different modalities, and then designing a causal invariant transformation to simulate the causal relationship and generate new training samples, the model can learn a feature representation that is consistent with the causal structure, thereby maintaining stable performance on data with different distributions; by fusing information from different modalities, a comprehensive feature representation that can capture common and complementary information in multimodal data can be extracted; a self-supervised learning framework is constructed without the need for additional annotation information, and the representation and causal structure of the data are learned by predicting the labels of node pairs as a pre-training task, which can effectively improve the generalization ability of the model and enhance the interpretability and stability of the model.

[0060] In addition, a multimodal data generalization learning method based on causal invariant transformation provided in this embodiment allows more processing to be performed in the local environment where data is generated (such as edge devices), reducing the computing and storage requirements of the central server, which not only reduces communication overhead, but also improves the efficiency of the overall system; it has good adaptability and scalability, and can be easily applied to different multimodal data sets and analysis tasks, whether in the fields of social media analysis, recommendation systems or sentiment analysis, it can provide effective solutions and is easy to adjust and optimize according to specific application scenarios; and the construction and analysis of the causal graph provides a clear representation of causal relationships, which not only helps to improve the generalization ability of the model, but also enhances the interpretability of the model, making the decision-making process of the model more transparent.

[0061] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0062] In one embodiment, Figure 3 As shown, a multimodal data generalization learning system based on causal invariant transformation is provided, including: a causal graph construction and analysis module 310, a causal invariant transformation design module 320, a multimodal feature fusion module 330 and a self-supervised learning module 340, wherein: A causal graph construction and analysis module 310 is used to obtain multimodal data, and use causal inference technology to construct a causal graph based on the multimodal data, and identify the causal relationship between each mode in the multimodal data based on the causal graph; A causal invariant transformation design module 320 is used to design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, keep the causal features in the causal relationship unchanged, change the non-causal features, and generate new training samples; A multimodal feature fusion module 330, used to fuse information of different modalities based on training samples using a multimodal feature fusion technology to obtain a fused feature representation; The self-supervised learning module 340 is used to build a self-supervised learning framework. Based on the self-supervised learning framework, the labels of the predicted node pairs are used as pre-training tasks according to the training samples and the fused feature representation, so as to learn the feature representation and causal structure of the multimodal data and complete the generalization learning of the multimodal data.

[0063] In one embodiment, the causal graph construction and analysis module 310 is also used to obtain multimodal data, determine each random variable corresponding to the multimodal data, and perform a conditional independence test on each random variable to determine whether there is a direct causal relationship between each random variable; use each random variable with a direct causal relationship as a node, and the causal relationship between each random variable as an edge to obtain a node set and an edge set; and orient the edges based on the node set, the edge set, and the judgment result of whether there is a direct causal relationship to form a directed acyclic graph as a causal graph.

[0064] In one embodiment, the causal graph construction and analysis module 310 is also used to introduce a causal effect function based on the causal graph to describe the mutual influence between each node; use Granger causality test, instantaneous causal analysis, intervention and counterfactual analysis to identify causal relationships based on mutual influence; use a structural equation model to quantify the identified causal relationships, and optimize and update the causal graph.

[0065] The structure of the cause-effect diagram construction and analysis module is as follows Figure 4 As shown in the figure, the causal graph construction and analysis process of multimodal data generalization learning is demonstrated. Specifically, the structure of the causal graph construction and analysis module includes the following main parts: variable selection: responsible for selecting relevant variables from multimodal data to provide a basis for causal relationship identification; conditional independence test: responsible for conditional independence test of variable pairs to determine whether there is a direct causal relationship between variables; graph orientation: according to the results of the conditional independence test, the edges in the causal graph are oriented to form a directed acyclic graph; causal graph optimization: responsible for searching and scoring all possible DAG structures to find the optimal causal graph that best reflects the causal relationship in the data; dynamic update mechanism: responsible for dynamically updating the causal graph according to new data or environmental changes to ensure that it always reflects the latest data distribution.

[0066] exist Figure 4 In the figure, arrows are used to represent data flow and information transmission in each part. The output of variable selection points to the conditional independence test through arrows, indicating the transmission of variable selection information; the output of conditional independence test points to the orientation of the graph through arrows, indicating the transmission of conditional independence test results; the output of the orientation of the graph points to the causal graph optimization through arrows, indicating the transmission of the causal graph structure; after the causal graph optimization optimizes the causal graph, its output points to the dynamic update mechanism through arrows, indicating the transmission of the optimal causal graph; after the dynamic update mechanism dynamically updates the causal graph, it feeds back the update results to the conditional independence test and the orientation of the graph, indicating the transmission of dynamic update information.

[0067] In one embodiment, the causal invariant transformation design module 320 is also used to define transformation operations and causal characteristics, and design a causal invariant transformation based on the transformation operations and causal characteristics; when performing transformation operations based on the causal invariant transformation, the causal characteristics are kept unchanged; and the causal invariant transformation is optimized to complete the causal invariant transformation design.

[0068] The structure of the causal invariant transformation design module is shown below: Figure 5As shown in the figure, the causal invariant transformation design process for generalization learning of multimodal data is presented. Specifically, the structure of the causal invariant transformation design module includes the following main parts: causal feature preservation: responsible for ensuring that the transformation operation does not change the causal characteristics of the data, which is the core principle of CIT design; diversity generation: responsible for generating diverse CITs to cover different data distributions and enhance the generalization ability of the model; reversibility verification: responsible for verifying the reversibility of the transformation to ensure that the original data can be restored from the transformed data through the inverse transformation; CIT generation method: responsible for generating specific CITs based on the principles of causal feature preservation, diversity and reversibility; CIT optimization goal: responsible for defining and optimizing the objective function of CIT to minimize the distribution difference of data before and after the transformation.

[0069] exist Figure 5 In the figure, arrows are used to represent data flow and information transfer in each part. The output of causal feature preservation points to diversity generation through arrows, indicating the transfer of causal feature preservation principles; the output of diversity generation points to reversibility verification through arrows, indicating the transfer of diversity generation information; reversibility verification verifies the reversibility of CIT, and its results point to the CIT generation method through arrows, indicating the transfer of reversibility verification results; the CIT generation method generates CIT according to these principles, and its output points to the CIT optimization target through arrows, indicating the transfer of the CIT generation method. The CIT optimization target defines and optimizes the objective function of CIT, and its results are fed back to causal feature preservation through arrows, indicating the transfer of optimization targets.

[0070] In one embodiment, the multimodal feature fusion module 330 is also used to use a feature extraction network to extract features from each modality in the training sample to obtain extracted features; perform feature alignment on the extracted features, and fuse the aligned features using a corresponding fusion strategy to obtain a fused feature representation.

[0071] In one embodiment, the multimodal feature fusion module 330 is further used to design a feature alignment function, and use the feature alignment function to perform feature alignment processing on the extracted features through feature mapping or dimensionality reduction technology.

[0072] The structure of the multimodal feature fusion module is shown in the figure below: Figure 6As shown in the figure, the multimodal feature fusion process of multimodal data generalization learning is demonstrated. Specifically, the structure of the multimodal feature fusion module includes the following main parts: Feature extraction: responsible for extracting features from the original data of each modality to provide a basis for subsequent feature fusion; Feature alignment: responsible for aligning the features of different modalities to ensure that they have the same feature space and scale; Feature fusion strategy: responsible for selecting and designing appropriate feature fusion strategies based on task requirements and modal characteristics; Fusion feature optimization: responsible for optimizing the fused feature representation to improve the performance and generalization ability of the model.

[0073] exist Figure 6 In the figure, arrows are used to represent data flow and information transfer in each part. The output of feature extraction points to feature alignment through arrows, indicating the transfer of features extracted from the original data; after feature alignment aligns the features, its output points to feature fusion strategy through arrows, indicating the transfer of aligned features; feature fusion strategy generates fusion features according to the fusion strategy, and its output points to fusion feature optimization through arrows, indicating the transfer of fusion features; fusion feature optimization optimizes the fusion features to further improve model performance.

[0074] In one embodiment, the self-supervised learning module 340 is also used to define the objective function of self-supervised learning, perform target training based on training samples and fused feature representations to obtain training signals; determine node pairs, and predict labels of node pairs, and use the predicted labels of node pairs as tasks; obtain sample pairs based on training samples and fused feature representations, and perform feature contrast learning; perform joint learning coordination based on training signals, tasks, and feature contrast learning to complete the construction of a self-supervised learning framework.

[0075] In one embodiment, the self-supervised learning module 340 is also used to determine data points in the multimodal data, obtain fused feature representations corresponding to the data points; design a label prediction network based on the data points and the fused feature representations corresponding to the data points; and use the label prediction network to predict labels for node pairs.

[0076] In one embodiment, the self-supervised learning module 340 is also used to obtain sample pairs based on training samples and fused feature representations, and divide the sample pairs into positive sample pairs and negative sample pairs; calculate the distance difference between the positive sample pairs and the negative sample pairs, and minimize the distance difference to learn feature representations.

[0077] The structure of the self-supervised learning module is shown in the figure below. Figure 7As shown in the figure, the self-supervised learning process for unsupervised pre-training is demonstrated. Specifically, the structure of the self-supervised learning module includes the following main parts: objective function definition: responsible for defining the objective function of self-supervised learning and providing training signals for the model; node pair label prediction: responsible for predicting the label of a given node pair, as a key task of self-supervised learning; feature contrast learning: responsible for bringing positive sample pairs closer and pushing negative sample pairs away through contrast learning to learn discriminative feature representations; joint learning coordination: responsible for integrating information from different views or tasks under the joint learning framework to improve the effect of feature learning.

[0078] exist Figure 7 In the figure, arrows are used to represent data flow and information transfer in each part. The objective function defined by the objective function points to the node for label prediction through arrows, indicating the transfer of training objectives; the output of the node for label prediction points to the feature contrast learning through arrows, indicating the transfer of the node for label prediction results; the feature contrast learning results of feature contrast learning point to the joint learning coordination through arrows, indicating the transfer of contrast learning features; the joint learning coordination, as a joint learning coordinator, integrates information from multiple parties and feeds it back to the objective function definition to achieve more effective feature learning.

[0079] In one embodiment, Figure 8 As shown, a schematic diagram of the structure of a multimodal data generalization learning method and system based on causal invariant transformation is provided, namely, Figure 8 A multimodal data generalization learning framework based on causal invariant transformation is demonstrated, which mainly includes the following main parts: causal graph construction and analysis module: responsible for constructing and analyzing the causal graph of the data, providing a theoretical basis for the subsequent causal invariant transformation design; causal invariant transformation design module: designing a series of CITs to simulate the causal relationship in the data and generate new training samples; multimodal feature fusion module: integrating information from different modalities, extracting comprehensive feature representations, and improving the generalization ability of the model; self-supervised learning module: learning the representation and causal structure of the data by predicting the labels of node pairs as a pre-training task without the need for additional annotation information.

[0080] exist Figure 8 In the figure, arrows are used to represent data flow and information transmission in each part. The output of the causal graph construction and analysis module points to the causal invariant transformation design module through arrows, indicating the transmission of the causal graph construction and analysis results; the output of the causal invariant transformation design module is passed to the multimodal feature fusion module through arrows, indicating the transmission of new samples generated by CIT; the multimodal feature fusion module passes the fused feature representation to the self-supervised learning module, indicating the transmission of feature fusion results; the self-supervised learning module feeds back the data representation obtained through self-supervised learning to the causal graph construction and analysis module and the causal invariant transformation design module, indicating the transmission of learning results.

[0081] In one embodiment, a multimodal data generalization learning method and system based on causal invariant transformation is provided, which can be applied to medical image diagnosis system, aiming to improve the accuracy and generalization ability of disease diagnosis by combining multimodal data of medical images (such as CT, MRI, pathological images, etc.). Fig. 9 The specific implementation process is as follows: Multimodal data collection: Collect various medical image data of patients, including CT images, MRI images, and pathological slice images; Causal graph construction and analysis: Use the causal graph construction and analysis module to model and analyze the causal relationship between different modal data; Feature extraction and fusion: extract and integrate key features from different medical images through multimodal feature fusion module; Causal Invariant Transform Generation: Applying Causal Invariant Transform (CIT) to generate new training samples enhances the model’s generalization ability to data under different imaging conditions. Self-supervised learning pre-training: Use a self-supervised learning framework to pre-train the model and learn the representation and causal structure of the data; Disease diagnosis model training: Combine the feature representation obtained from pre-training to train the final disease diagnosis model; Model evaluation and optimization: Evaluate model performance on a standard test set and optimize the model based on the evaluation results; Interpretation of diagnostic results: Use the causal graph analysis module to provide decision support and enhance the interpretability of the model.

[0082] In one embodiment, a multimodal data generalization learning method and system based on causal invariant transformation is provided, which can be applied to a multimodal sentiment analysis system on a social media platform, aiming to improve the accuracy and generalization ability of sentiment analysis by combining text, image and audio data. Fig.10 The specific implementation process is as follows: Multimodal data collection: Collecting user posts from social media platforms, including text content, attached images, and audio information; Causal graph construction and analysis: Use the causal graph construction and analysis module to model and analyze the causal relationship between text, image, and audio data; Feature extraction and fusion: Through the multimodal feature fusion module, key sentiment features in different data modalities are extracted and integrated; Causal Invariant Transform Generation: Applying Causal Invariant Transform (CIT) to generate new training samples enhances the model’s generalization ability to different emotional expressions; Self-supervised learning pre-training: Use a self-supervised learning framework to pre-train the model and learn the representation and causal structure of the data; Sentiment analysis model training: Combine the feature representation obtained by pre-training to train the final multimodal sentiment analysis model; Model evaluation and optimization: Evaluate model performance on a standard test set and optimize the model based on the evaluation results; Application of sentiment analysis results: Apply the trained model to real-time social media content analysis to provide users with sentiment tendency feedback.

[0083] In one embodiment, a multimodal data generalization learning method and system based on causal invariant transformation is provided, which can be applied to intelligent traffic monitoring systems, aiming to improve the accuracy and response speed of abnormal event monitoring by combining video monitoring, traffic flow data and environmental sensor data. Fig.11 The specific implementation process is as follows: Multimodal data collection: Collect multiple types of data in traffic monitoring systems, including video streams, traffic flow data, and environmental sensor readings; Causal graph construction and analysis: Use the causal graph construction and analysis module to model and analyze the causal relationship between video, traffic flow and environmental data; Feature extraction and fusion: Through the multimodal feature fusion module, key features from different data sources are extracted and integrated for feature representation of abnormal events; Causal Invariant Transform Generation: Applying Causal Invariant Transform (CIT) to generate new training samples, enhancing the model’s generalization ability to abnormal events under different traffic conditions; Self-supervised learning pre-training: Use a self-supervised learning framework to pre-train the model and learn the representation and causal structure of the data; Abnormal event monitoring model training: Combine the feature representation obtained by pre-training to train the final abnormal event monitoring model; Model evaluation and optimization: Evaluate model performance on historical traffic event datasets and optimize the model based on the evaluation results; Real-time abnormal event response: Apply the trained model to real-time traffic monitoring to achieve rapid detection and automatic alarm of abnormal events.

[0084] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.12As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal data generalization learning method based on causal invariant transformation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0085] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0086] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of a multimodal data generalization learning method based on causal invariant transformation are implemented.

[0087] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a multimodal data generalization learning method based on causal invariant transformation are implemented.

[0088] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multimodal data generalization learning method based on causal invariant transformation, characterized in that: The method comprises: Acquire multimodal data, and construct a causal graph based on the multimodal data using causal inference technology, and identify the causal relationship between each mode in the multimodal data based on the causal graph; Designing a causal invariant transformation, simulating the causal relationship through the causal invariant transformation, keeping the causal features in the causal relationship unchanged, changing the non-causal features, and generating new training samples; Using multimodal feature fusion technology to fuse information of different modalities based on the training samples to obtain a fused feature representation; A self-supervised learning framework is constructed, and based on the self-supervised learning framework, labels of node pairs are predicted according to the training samples and fused feature representations as pre-training tasks to learn the feature representation and causal structure of the multimodal data and complete the generalization learning of multimodal data.

2. The multimodal data generalization learning method based on causal invariant transformation according to claim 1 is characterized in that: Acquiring multimodal data and constructing a causal graph based on the multimodal data using causal inference techniques, including: Acquire multimodal data, determine each random variable corresponding to the multimodal data, and perform a conditional independence test on each random variable to determine whether there is a direct causal relationship between each random variable; Taking random variables with direct causal relationships as nodes and the causal relationships between the random variables as edges, a node set and an edge set are obtained; According to the judgment result of whether the node set, the edge set, and the direct causal relationship exist, the edges are directed to form a directed acyclic graph as a causal graph.

3. The multimodal data generalization learning method based on causal invariant transformation according to claim 2 is characterized in that: Identifying the causal relationship between each modality in the multimodal data based on the causal graph includes: Based on the causal graph, a causal effect function is introduced to describe the mutual influence between each of the nodes; Granger causality test, instantaneous causal analysis, intervention and counterfactual analysis were used to identify causal relationships based on the mutual influences; The identified causal relationships were quantified using structural equation modeling, and the causal diagram was optimized and updated.

4. The multimodal data generalization learning method based on causal invariant transformation according to claim 1 is characterized in that: Design causally invariant transformations, including: defining transformation operations and causal characteristics, and designing causal invariant transformations based on the transformation operations and causal characteristics; When performing the transformation operation based on the causal invariant transformation, the causal feature is kept unchanged; The causal invariant transformation is optimized to complete the causal invariant transformation design.

5. The multimodal data generalization learning method based on causal invariant transformation according to claim 1 is characterized in that: The multimodal feature fusion technology is used to fuse the information of different modalities based on the training samples to obtain a fused feature representation, including: Using a feature extraction network to extract features from each modality in the training sample to obtain extracted features; The extracted features are subjected to feature alignment processing, and the aligned features are subjected to feature fusion using a corresponding fusion strategy to obtain a fused feature representation.

6. The multimodal data generalization learning method based on causal invariant transformation according to claim 5 is characterized in that: Performing feature alignment processing on the extracted features includes: A feature alignment function is designed, and the feature alignment function is used to perform feature alignment processing on the extracted features through feature mapping or dimensionality reduction technology.

7. The multimodal data generalization learning method based on causal invariant transformation according to claim 1 is characterized in that: A self-supervised learning framework is constructed, and labels of node pairs are predicted based on the training samples and the fused feature representation as pre-training tasks to learn the feature representation and causal structure of the multimodal data, including: Defining an objective function of self-supervised learning, performing target training based on the training samples and fusion feature representation, and obtaining a training signal; Determine a node pair, predict a label of the node pair, and use the predicted label of the node pair as a task; Obtaining sample pairs according to the training samples and fused feature representations, and performing feature comparison learning; Based on the training signals, tasks, and feature contrast learning, joint learning coordination is performed to complete the construction of a self-supervised learning framework.

8. The multimodal data generalization learning method based on causal invariant transformation according to claim 7 is characterized in that: Predicting the label of the node pair includes: Determine a data point in the multimodal data, and obtain a fused feature representation corresponding to the data point; Designing a label prediction network according to the data points and the fusion feature representations corresponding to the data points; The label prediction network is used to predict labels for the node pairs.

9. The multimodal data generalization learning method based on causal invariant transformation according to claim 7, characterized in that: According to the training samples and the fused feature representation, sample pairs are obtained, and feature comparison learning is performed, including: According to the training samples and the fused feature representation, sample pairs are obtained, and the sample pairs are divided into positive sample pairs and negative sample pairs; The distance difference between the positive sample pair and the negative sample pair is calculated, and the feature representation is learned by minimizing the distance difference.

10. A multimodal data generalization learning system based on causal invariant transformation, characterized in that: The system comprises: A causal graph construction and analysis module, which is used to obtain multimodal data, and use causal inference technology to construct a causal graph based on the multimodal data, and identify the causal relationship between each mode in the multimodal data based on the causal graph; A causal invariant transformation design module, used to design a causal invariant transformation, simulate the causal relationship through the causal invariant transformation, keep the causal features in the causal relationship unchanged, change the non-causal features, and generate new training samples; A multimodal feature fusion module, used to fuse information of different modalities based on the training samples using a multimodal feature fusion technology to obtain a fused feature representation; The self-supervised learning module is used to construct a self-supervised learning framework. Based on the self-supervised learning framework, the labels of the predicted node pairs are used as pre-training tasks according to the training samples and the fused feature representations to learn the feature representations and causal structures of the multimodal data and complete the generalization learning of the multimodal data.

Images