A method and apparatus for causal decoupling based on multi-scale noise and adversarial supervision

Abstract

The application discloses a kind of based on multi-scale noise and the causality decoupling method and device of confrontation supervision, the method includes: by simulating the causal relationship of variable in causal graph to generate observation image data, and according to observation image data and its corresponding causal label information and causal graph, training set and test set are constructed;Causal decoupling model is constructed, and is trained under multi-scale noise using training set in confrontation supervision;Using test set and the causal graph matrix after training, using the trained observation data coding module and observation data decoding module obtains counterfactual intervention data.The present application fully combines self-encoder with causal acyclic constraint, discriminant module is trained under multi-scale noise, high-quality confrontation supervision is carried out, so as to improve the representation learning ability of model, enhance the representation understanding of model to data with causal relationship, improve the accuracy of implicit causal network prediction, improve the accuracy of causal decoupling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image representation learning and causal disentanglement, and particularly relates to a causal disentanglement method and device based on multi-scale noise and adversarial supervision. BACKGROUND

[0002] Image representation learning is an important direction in the fields of computer science, artificial intelligence and computer vision, and its core goal is to use deep learning technology to extract high-dimensional information from images and learn meaningful semantic features, so as to realize the abstract understanding of the real world by the machine. Image representation learning research is extensive and covers a variety of tasks, including but not limited to self-supervised learning, image generation, transfer learning, cross-modal learning, etc. The research on image representation learning can make the model learn the features and even the generation mechanism behind the data, so that the model can adapt to new tasks in the case of few samples or even no samples. Through the learning of image representation, the specific working mechanism and decision-making process of the black box model can be revealed, and the explainability of the model can be enhanced.

[0003] Causal disentanglement is a key direction in the field of machine learning, and its core goal is to separate independent causal variables from complex data and learn representations with clear causal semantics, so as to realize the understanding of the operation mechanism behind the data and make the model maintain robustness and generalization ability in dynamic environment, distribution change or few sample scene. Causal disentanglement identifies and separates potential causal variables in data, so that the representation can independently encode these factors, increasing the credibility of the model. Through modeling of causal relationships, the model can generate samples that are more consistent with the laws and logical constraints of the real world, laying a solid foundation for the application of artificial intelligence in social, economic and scientific fields. SUMMARY

[0004] The present application aims at the deficiencies of the prior art, and provides a causal disentanglement method and device based on multi-scale noise and adversarial supervision.

[0005] The purpose of the present application is achieved by the following technical solutions: the first aspect of the embodiment of the present application provides a causal disentanglement method based on multi-scale noise and adversarial supervision, comprising the following steps:

[0006] (1) using prior causal mechanism to simulate the causal relationship of variables in the causal graph, generating observed image data, and constructing training set and test set according to the observed image data, its corresponding causal label information and causal graph;

[0007] (2) constructing a causal decoupling model, and using the training set to perform adversarial supervised training under multi-scale noise to obtain the trained causal decoupling model; wherein the causal decoupling model comprises an observation data encoding module, an observation data decoding module and a multi-scale noise discrimination module;

[0008] (3) sending the observation image data in the test set, the causal label information and the trained causal graph matrix into the trained observation data encoding module to obtain causal hidden vector features, selecting a causal variable to be intervened, calculating an intervention value of the causal variable, modifying the corresponding causal variable in the causal hidden vector features, obtaining an intervened causal hidden vector feature, and sending the intervened causal hidden vector feature into the trained observation data decoding module to obtain counterfactual intervention data.

[0009] Further, the observation image data is specifically obtained by the following method:

[0010] According to the plurality of potential causal variables, a directed acyclic graph is constructed as a real causal graph; exogenous noise and variable values are input into the causal graph to construct causal label information in a simulation environment, and a causal function model is used to simulate and generate observation values of variables in the causal graph; based on the causal graph, each node is sampled following the relationship on the causal graph to obtain causal graph observation data, and image data generated based on the causal graph observation data is the simulation generated observation image data.

[0011] Further, the observation data encoding module comprises an encoding module convolution layer, an encoding module causal embedding layer and an encoding module data compression layer; the observation data decoding module comprises a decoding module data decoding layer, a decoding module causal embedding layer and a decoding module inverse convolution layer; the multi-scale noise discrimination module comprises an adaptive multi-scale noise adding layer, a discrimination module convolution layer, a discrimination module causal embedding layer, a discrimination module data compression layer and a classification layer;

[0012] The observed image data, the causal label information and the causal graph matrix are input into the causal decoupling model, the observed image data is obtained through convolution representation of an encoding module convolution layer, the convolution layer representation, the causal label information and the causal graph matrix are input into an encoding module causal embedding layer for calculation, and then causal hidden vector features are obtained after passing through an encoding module data compression layer; the causal hidden vector features are first input into a decoding module data decoding layer to obtain decoding representation, the decoding representation, the causal label information and the causal graph matrix are input into a decoding module causal embedding layer for calculation, and then reconstructed data is obtained after passing through a decoding module inverse convolution layer; the reconstructed data and the observed image data are respectively input into an adaptive multi-scale noise adding layer to obtain noise-added reconstructed data and noise-added observed image data, and the noise-added reconstructed data and the noise-added observed image data are simultaneously input into a discrimination module convolution layer to obtain convolution features, the convolution features, the causal label information and the causal graph matrix are input into a discrimination module causal embedding layer for calculation, and then a true or false discrimination probability is output after passing through a discrimination module data compression layer and a classification layer in sequence.

[0013] Further, the adaptive multi-scale noise adding layer specifically comprises: initializing a noise adding step number and generating a time embedding, and then using a forward noise adding mode of a diffusion model to add noise to the reconstructed data and the observed image data to obtain noise-added reconstructed data and noise-added observed image data.

[0014] Further, the initializing the noise adding step number and generating the time embedding specifically comprises:

[0015] Initializing a maximum noise scale and a time scale embedding dimension d, using a triangular embedding to encode the time scale, calculating a time scale embedding basis through a frequency function, processing the noise adding step number sequence T using the time scale embedding basis, and then performing sine and cosine transformations, and performing a splicing operation on the sine and cosine transformed embeddings to obtain a time scale embedding vector with a length of d.

[0016] Further, the using the forward noise adding mode of the diffusion model to add noise to the reconstructed data and the observed image data specifically comprises:

[0017] Based on the normal distribution noise and the noise adding step number, the reconstructed data and the observed image data are added noise through a cumulative noise factor to obtain noise-added reconstructed data and noise-added observed image data; wherein the cumulative noise factor is obtained according to the noise variance on the corresponding time scale and the time scale embedding vector.

[0018] Further, the using the training set to perform adversarial supervision training under multi-scale noise to obtain the trained causal decoupling model specifically comprises:

[0019] The observation image data, the causal label information and the causal graph matrix in the training set are input into the causal decoupling model to obtain causal hidden vector features, reconstructed data and authenticity discrimination probability;

[0020] The autoencoder loss is calculated based on the observation image data, the causal graph matrix, the causal hidden vector features, the reconstructed data and the authenticity discrimination probability, and the discriminant adversarial loss is calculated based on the authenticity discrimination probability corresponding to the observation image data and the reconstructed data;

[0021] The autoencoder loss and the discriminant adversarial loss are alternately optimized during the training process. When the autoencoder loss is optimized, only the parameters of the observation data encoding module and the observation data decoding module are updated. When the discriminant adversarial loss is optimized, only the parameters of the observation data decoding module and the multi-scale noise discrimination module are updated. Until the preset training round is reached or the autoencoder loss and the discriminant adversarial loss converge, the trained causal decoupling model is obtained.

[0022] Further, the autoencoder loss is specifically obtained by the following method:

[0023] The root mean square error between the observation image data and the reconstructed data is calculated as the reconstruction loss. The KL divergence loss is calculated based on the causal hidden vector features and their prior distribution. The generation loss is calculated according to the authenticity discrimination probability. The total variation loss and the entropy loss are calculated according to the reconstructed data. The adversarial generation loss is calculated according to the generation loss, the total variation loss and the entropy loss. The and directed acyclic loss is calculated according to the causal graph matrix. The autoencoder loss is calculated according to the reconstruction loss, the KL divergence loss, the adversarial generation loss and the directed acyclic loss.

[0024] Further, the step (3) specifically comprises:

[0025] The observation image data, the causal label information and the trained causal graph matrix in the test set are input into the trained observation data encoding module. The observation image data is obtained through the convolution layer of the encoding module to obtain the convolution representation. The convolution layer representation, the causal label information and the trained final causal graph matrix are input into the encoding module causal embedding layer for calculation, and then the causal hidden vector features are obtained after the data compression layer of the encoding module.

[0026] The causal variable that needs to be intervened is selected, the intervention value is calculated, and the corresponding causal variable in the causal hidden vector features is modified according to the intervention value to obtain the intervened causal hidden vector features.

[0027] The intervened causal hidden vector feature, the causal label information and the trained causal graph matrix are sent into the trained observation data decoding module, the intervened causal hidden vector feature is obtained through data decoding of the decoding module, the decoding feature, the causal label information and the trained causal graph matrix are input into the causal embedding layer of the decoding module for calculation, and counterfactual intervention data is obtained after inverse convolution of the decoding module.

[0028] The second aspect of the embodiment of the application provides a causal decoupling device based on multi-scale noise and adversarial supervision, which is used for realizing the causal decoupling method based on multi-scale noise and adversarial supervision.

[0029] The causal data synthesis module is used for simulating the causal relationship of variables in the causal graph by using the prior causal mechanism, generating observation image data, and constructing a training set and a test set according to the observation image data, the corresponding causal label information and the causal graph;

[0030] The model construction and training module is used for constructing a causal decoupling model and performing adversarial supervised training under multi-scale noise by using the training set, so as to obtain a trained causal decoupling model; wherein the causal decoupling model comprises an observation data encoding module, an observation data decoding module and a multi-scale noise discrimination module.

[0031] The causal counterfactual intervention module is used for sending the observation image data in the test set, the causal label information and the trained causal graph matrix into the trained observation data encoding module to obtain a causal hidden vector feature, selecting a causal variable to be intervened, calculating an intervention value of the causal variable, modifying the corresponding causal variable in the causal hidden vector feature, obtaining an intervened causal hidden vector feature, and sending the intervened causal hidden vector feature into the trained observation data decoding module to obtain counterfactual intervention data.

[0032] Compared with the prior art, the application has the following beneficial effects:

[0033] (1) The application is a causal decoupling method based on multi-scale noise and adversarial supervision, which can perform adversarial training through a discrimination module under multi-scale noise, thereby improving the accuracy of causal decoupling.

[0034] (2) The observation data encoding module and the observation data decoding module involved in the application are used for explicitly modeling the causal relationship through a directed acyclic graph, and then embedding the causal relationship into a model through a neural network for training, so that the model main loss and the directed acyclic loss are simultaneously optimized.

[0035] (3) The multi-scale noise discrimination module involved in the application is trained under multi-scale noise after adaptive noise adding, and the causal embedding information output from the autoencoder is input into the multi-scale noise discrimination module, and the observation data encoding module, the observation data decoding module and the multi-scale noise discrimination module are alternately modeled and trained.

[0036] (4) The application trains a strong discrimination module using noise data under multi-scale, trains the observation data encoding module and the observation data decoding module in an adversarial supervision manner, and combines causal embedding and directed acyclic constraints during the training process, so that the model can learn beneficial causal representation; the application uses a multi-objective balanced loss function to guide the training of the model, and then uses an adaptive noise scale adversarial supervision method to learn causal representation, solving the problems of low accuracy of causal decoupling and insufficient model generalization ability in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of the causal decoupling method based on multi-scale noise and adversarial supervision of the application;

[0038] Figure 2 is a workflow diagram of the causal data synthesis module of the application;

[0039] Figure 3 is an architecture flowchart of the causal decoupling model of the application;

[0040] Figure 4 is a workflow diagram of the observation data encoding module of the application;

[0041] Figure 5 is a workflow diagram of the observation data decoding module of the application;

[0042] Figure 6 is a workflow diagram of the multi-scale noise discrimination module of the application;

[0043] Figure 7 is a framework diagram of the multi-scale noise and adversarial supervision of the application;

[0044] Figure 8 is a workflow diagram of the causal counterfactual intervention module of the application;

[0045] Figure 9 is a structural diagram of the causal decoupling device based on multi-scale noise and adversarial supervision of the application. DETAILED DESCRIPTION

[0046] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, like numbers in the attached drawings refer to the same or similar elements. Obviously, the drawings shown herein are only a few embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0047] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0048] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] The present application will be described in detail below with reference to the attached drawings. The features of the embodiments and implementations described below can be combined with each other as long as there is no conflict.

[0050] Referring to Figure 1 The causal decoupling method based on multi-scale noise and adversarial supervision of the present application specifically includes the following steps:

[0051] (1) Simulate the causal relationship of variables in the causal graph using the prior causal mechanism, generate observation image data containing the causal relationship, and construct a training set and a test set according to the observation image data, the corresponding causal label information and the causal graph, for subsequent training and testing of the causal decoupling model.

[0052] In this embodiment, as Figure 2As shown, the observation image data is specifically obtained by the following method: by making assumptions about the simulation environment in accordance with physical logic, variables with causal relationships are represented by a causal graph, since the causal graph is used to describe the causal relationship between different variable characteristics, it must satisfy the acyclicity, so first, a directed acyclic graph is constructed according to the plurality of potential causal variables as a real causal graph, and the causal graph is stored in the form of an adjacency matrix, that is, the causal graph is mapped into a causal graph matrix by a causal vector mapping, and the causal graph is directly represented by the causal graph matrix subsequently. Then, exogenous noise and variable values are input into the causal graph to construct causal label information in the simulation environment. Specifically, the variables in the causal graph are topologically sorted, and the variable values are given in the order of topological sorting. The causal variables The exogenous noise is independently and identically distributed wherein Z represents a set of causal variables, represents a causal variable i, represents a set of exogenous noise, represents the exogenous noise assigned to the causal variable i, the exogenous noise is a homoscedastic Gaussian distribution, which is used to describe the disturbance factor in the system; and the observation values of the variables in the causal graph are generated by simulation using the causal function model. Finally, the causal graph observation data obtained after a single input (i.e., the exogenous noise and variable values input into the causal graph) to the causal graph is regarded as a sampling of the causal mechanism, and a set constructed by a plurality of such samplings is the causal graph observation data. The set of image data generated on the basis of the causal graph observation data is causal observation data, which is taken as the simulation generated observation image data. Specifically, in the causal graph, a variable that has no other variable directly pointing to it is a root cause variable, and a variable that has one or more variables pointing to it is a non-root cause variable. Based on the causal graph, each node is sampled to obtain causal graph observation data in accordance with the relationship on the causal graph. The image data generated on the basis of the causal graph observation data is the simulation generated observation image data. Wherein, for the root cause variable, its observation value is directly determined by the exogenous noise; for the non-root cause variable, its observation value is determined by the exogenous noise, the causal function and the parent node observation value.

[0053] It should be understood that the observation image data can be generated in a simulation manner, and can also be directly obtained from a real scene to represent the behavior of the causal system under synthetic conditions or real conditions. In order to obtain a biased estimate of the observation image data, the causal label information is also input into the causal decoupling model for training without intervention on the data.

[0054] It should be noted that the causal label information reflects the supervision signal of the semantic of the potential causal variable, and is used to construct weak prior information and then learn the causal network. Specifically, the causal label information herein refers to the causal graph observation data in the generation process of each observation image data sample. The causal label information needs to be consistent with the observation image data, that is, the causal label information and the observation image data need to be spliced into a vector and then input into the causal decoupling model. Although both the causal label information and the observation image data need to be input into the causal decoupling model, the input positions are different. The causal label information, as the quantitative information of the causal variable, is input in the causal embedding layer; and the observation image data is directly input into the convolution layer and then input into the causal embedding layer.

[0055] Further, the causal function model of the nonlinear structure can be expressed as:

[0056]

[0057] In the formula, Xi represents the observation value of the causal variable i, Xi represents the observation value of the parent node of the causal variable i in the causal graph, Xi represents the exogenous noise allocated to the causal variable i; Xi represents the nonlinear transformation corresponding to the causal variable i, which is used to apply nonlinear transformation to Xi directly applies nonlinear transformation; Xi represents the nonlinear transformation corresponding to the causal variable i and the corresponding exogenous noise, which is used to apply nonlinear transformation to Xi and Xi and

[0058] (2) Construct a causal decoupling model and use the training set to perform adversarial supervision training under multi-scale noise to obtain a trained causal decoupling model. The causal decoupling model includes an observation data encoding module, an observation data decoding module and a multi-scale noise discrimination module, as shown in Figure 3 .

[0059] In this embodiment, the observation data encoding module includes an encoding module convolution layer, an encoding module causal embedding layer and an encoding module data compression layer, as shown in Figure 4 ; the observation data decoding module includes a decoding module data decoding layer, a decoding module causal embedding layer and a decoding module inverse convolution layer, as shown in Figure 5 ; and the multi-scale noise discrimination module includes an adaptive multi-scale noise adding layer, a discrimination module convolution layer, a discrimination module causal embedding layer, a discrimination module data compression layer and a classification layer, as shown in Figure 6 .

[0060] Specifically, as shown in Figure 3As shown, the observed image data, causal label information and causal graph matrix are input into the causal decoupling model, the observed image data is obtained by convolution representation through the encoding module convolution layer, the convolution layer representation, the causal label information and the causal graph matrix are input into the encoding module causal embedding layer for calculation, and then the causal hidden vector feature is obtained after the encoding module data compression layer. The causal hidden vector feature is first obtained by decoding representation through the decoding module data decoding layer, the decoding representation, the causal label information and the causal graph matrix are input into the decoding module causal embedding layer for calculation, and then the reconstructed data is obtained after the decoding module inverse convolution layer. The reconstructed data and the observed image data are sent into the adaptive multi-scale noise layer respectively to obtain the noise-added reconstructed data and the observed image data. The noise-added reconstructed data and the observed image data are simultaneously subjected to the discriminant module convolution layer to obtain the convolution feature, the convolution feature, the causal label information and the causal graph matrix are input into the discriminant module causal embedding layer for calculation, and then the true and false discrimination probability is output after the discriminant module data compression layer and the classification layer, that is, the probability that the reconstructed data is the real observed image data. The true and false discrimination probability can be used for subsequent adversarial supervision training under multi-scale noise.

[0061] The specific working process of the observed data encoding module is as shown in Figure 4 The observed data encoding module is used for modeling the distribution of observed image data, extracting and compressing data features, and outputting high-dimensional hidden vector representation containing causal information embedding, that is, the causal hidden vector feature. Therefore, the causal hidden vector feature of the observed image data can be extracted by using the observed data encoding module. The observed image data is input into the observed data encoding module, and the causal graph (i.e. the causal graph matrix) and the causal label information in the optimization are put into the observed data encoding module through the encoding module causal embedding layer. Finally, the causal hidden vector feature is generated after the encoding module data compression layer. The causal hidden vector feature can be transmitted into the observed data decoding module.

[0062] The specific working process of the observed data decoding module is as shown in Figure 5 The observed data decoding module is used for decoding data distribution and generating reconstructed data, and its input is high-dimensional causal hidden vector representation containing causal representation, that is, the causal hidden vector feature. The causal graph (i.e. the causal graph matrix) and the causal label information in the optimization are put into the observed data decoding module through the decoding module causal embedding layer. The data is restored from the high-dimensional causal hidden vector representation to generate reconstructed observed image data, that is, the reconstructed data, through the decoding module inverse convolution layer. Therefore, the causal hidden vector feature can be decoded based on the causal graph matrix and the causal label information by using the observed data decoding module to obtain the reconstructed data, which is used for calculating the loss and optimizing the causal decoupling model.

[0063] The specific working process of the multi-scale noise discrimination module is as shown in Figure 6As shown, reconstructed data and observed image data are input as discrimination data into the multi-scale noise discrimination module. The module uses an adaptive multi-scale noise-adding layer to apply adaptive multi-scale noise to both the reconstructed and observed image data. In this layer, the number of noise-adding steps is initialized, and a temporal embedding is generated and input into the multi-scale noise discrimination module. Then, a forward noise-adding method using a diffusion model is used to add noise to both the real observed image data and the reconstructed data. The noise-adding reconstructed and observed image data are then passed through the convolutional layer of the discrimination module to obtain convolutional features. These features, along with causal label information and the causal graph matrix, are input into the causal embedding layer of the discrimination module for calculation. Finally, the data passes through the data compression layer and classification layer of the discrimination module, outputting the true / false discrimination probability. Therefore, the multi-scale noise discrimination module can be used to discriminate the reconstructed data, determining the probability that it is real observed image data, i.e., the true / false discrimination probability.

[0064] Furthermore, the adaptive multi-scale noise layer specifically includes: first, initializing the number of noise addition steps and generating temporal embeddings, and then using the forward noise addition method of the diffusion model to add noise to the reconstructed data and observed image data, so as to obtain the noise-added reconstructed data and observed image data.

[0065] Furthermore, the number of noise-adding steps is initialized and the temporal embedding is generated, specifically including: initializing the maximum noise-adding scale. Given a timescale embedding dimension d, triangular embedding is used to encode the timescale. Specifically, the embedding of the noisy timescale sequence consists of a frequency function, angle calculation, and sine and cosine transforms. The frequency function and the corresponding timescale embedding basis are expressed as follows:

[0066]

[0067]

[0068] In the formula, This represents the frequency of the k-th dimension in the time-scale embedding basis, where the value of k ranges from the first dimension to half of the total time-scale embedding dimension d. Indicates length is The time-scale embedding basis is calculated using the above formula to obtain a length of [missing information]. The temporal-scale embedding basis is then used to refine the noisy step sequence. After processing, sine and cosine transformations are performed respectively, where This indicates the first noise level. This indicates the second noise level. Indicates the third noise-adding scale, the maximum noise-adding scale. is the upper limit value of the noise adding step sequence T, and the embedding after the sine and cosine transformation is spliced to obtain a time scale embedding vector with a length of d, denoted as

[0069]

[0070] In the formula, denotes the embedding vector of the time scale t, denotes the splicing operation, denotes the sine transformation, denotes the cosine transformation. Here, the embedding after the sine and cosine transformation is spliced, and the output dimension is a time scale embedding vector of d. Subsequently, the forward noise adding mode of the diffusion model can be used to add noise to the reconstructed data and observed image data based on the obtained time scale embedding vector.

[0071] Further, the forward noise adding mode of the diffusion model is used to add noise to the reconstructed data and observed image data, specifically including: generating normally distributed noise , initializing the variance sequence through linear scheduling , the variance sequence is mainly used to control the intensity of noise at different scales, and the construction process is represented as:

[0072]

[0073] In the formula, denotes the noise variance at the tth scale, denotes the maximum length of all noise adding step sequences T, and denote the noise variance at the first scale and the last scale in the variance sequence respectively. The actual added noise is controlled by a set of cumulative noise factors , and the noise needs to pass through a variance control unit when added, and the calculation formula of the cumulative noise factor is:

[0074]

[0075] In the formula, denotes the cumulative noise factor at the tth scale, denotes a noise variance adjustment network, and the input is the original noise variance and the time embedding corresponding to the noise step at this time, denotes the embedding vector of the time scale s, denotes the noise variance at the sth scale, and s denotes the smallest noise adding unit at a specific noise scale, which takes a value range of , represents the gradual noise adding process from the original data (i.e. the original image observation data or the reconstructed data) to the tth scale, and the initialized noise adding step is The reconstructed data and the observed image data can be quickly added with noise by accumulating the noise factors, and the added noise formulas are respectively:

[0076]

[0077]

[0078] wherein, denotes the added noise reconstructed data, denotes the added noise observed image data, denotes the reconstructed data, and X denotes the observed image data, denotes the noise obeys a normal distribution with a mean of 0 and a covariance matrix of a unit matrix . .

[0079] It should be understood that the number of added noise steps determines the number of time embeddings, for example, if the initialization added noise step is 500, there will be 500 time scale embedding vectors, each with a length of d, that is, there will be a 500*d time embedding set, and then the cumulative noise factor is calculated to add noise to the reconstructed data and the observed image data.

[0080] In this embodiment, the training set is used for adversarial supervision training under multi-scale noise to obtain a trained causal decoupling model, as shown in Figure 3 and Figure 7 , specifically including: first, input the observed image data , causal label information and causal graph matrix A in the training set into the causal decoupling model to obtain causal hidden vector features b, reconstructed data and authenticity discrimination probability, wherein the causal graph matrix A is initialized as an upper triangular matrix when input into the causal decoupling model. Then, the autoencoder loss is calculated based on the observed image data, the causal graph matrix, the causal hidden vector feature, the reconstructed data and the authenticity discrimination probability, and the discrimination adversarial loss is calculated based on the authenticity discrimination probability corresponding to the observed image data and the reconstructed data. Subsequently, the autoencoder loss and the discrimination adversarial loss are back propagated to obtain a gradient, the model is optimized using AdamW, and the autoencoder loss and the discrimination adversarial loss are alternately optimized during the training process. When optimizing the autoencoder loss, only the parameters of the observed data encoding module and the observed data decoding module are updated; when optimizing the discrimination adversarial loss, only the parameters of the observed data decoding module and the multi-scale noise discrimination module are updated; until the preset training round or the autoencoder loss and the discrimination adversarial loss converge, the trained observed data encoding module, observed data decoding module and multi-scale noise discrimination module are obtained, that is, the trained causal decoupling model is obtained.

[0081] Furthermore, the autoencoder loss is calculated based on the observed image data, the causal graph matrix, the causal latent vector features, the reconstructed data, and the true / false discrimination probability. Specifically, this includes: calculating the root mean square error between the observed image data and the reconstructed data as the reconstruction loss; calculating the KL divergence loss based on the causal latent vector features and their prior distribution; calculating the generation loss based on the true / false discrimination probability; calculating the total variational loss and entropy loss based on the reconstructed data; calculating the adversarial generation loss based on the generation loss, total variational loss, and entropy loss; calculating the directed acyclic loss based on the causal graph matrix; and calculating the autoencoder loss based on the reconstruction loss, KL divergence loss, adversarial generation loss, and directed acyclic loss. The calculation formula is as follows:

[0082]

[0083] In the formula, This represents the loss of an autoencoder with anti-supervision capabilities; This represents the root mean square error, and also the reconstruction loss; Represents observed image data, This represents the observation data of variable i in the observed image data, and n represents the total number of variables; This represents the reconstructed data corresponding to the observed image data X. This represents the reconstructed data of variable i in the reconstructed data; Indicates the KL divergence loss; Indicates parameters The distribution of causal latent vector features b in the causal decoupling model under the input condition of observed image data X; This represents the prior distribution of the causal latent vector feature b; Indicates adversarial generation loss; Let be the directed acyclic constraint on the causal graph matrix A, and let represent the directed acyclic loss of the causal graph matrix A. , and They are in balance , and Hyperparameters.

[0084] The formula for calculating the adversarial generation loss is as follows:

[0085]

[0086] In the formula, Indicates the generation loss. This represents the total variational loss. Represents entropy loss, , and They are in balance , and hyperparameters. Wherein the calculation formula of generation loss, total variation loss and entropy loss are respectively:

[0087]

[0088]

[0089]

[0090] In the formula, denotes the reconstructed data after adding noise when the noise scale is t; denotes the probability that the multi-scale noise discrimination module judges the reconstructed data after adding noise to be true, which is obtained according to the true and false discrimination probability output by the multi-scale noise discrimination module; denotes the expectation of the reconstructed data after adding noise ; denotes the causal decoupling model containing parameters ; the distribution of the reconstructed data after adding noise under the input condition of causal hidden vector feature b; denotes the expectation of the data block at position coordinates ; denotes the reconstructed data block at position coordinates ; denotes the difference value of adjacent reconstructed data in the height direction, denotes the difference value of adjacent reconstructed data in the width direction; denotes noise; V, C, K are batch size, channel number and corresponding soft histogram bucket number of input data respectively; denotes the data of the i-th batch and the j-th channel on the i-th bucket of the soft histogram, v c k

[0091]

[0092] In the formula, is the data normalization value calculated based on V and C, denotes the value of the reconstructed data at the i-th batch, the j-th channel and the g-th position, and G denotes the sampling point number for calculating the soft histogram distribution, v c denotes the mean value of the i-th bucket of the soft histogram, k denotes the standard deviation of the soft histogram, which is used to measure the smoothness of the histogram.

[0093] ​​​​​​Directed acyclic loss of causal graph matrix A The calculation formula is:

[0094]

[0095] In the formula, This represents the trace taken after calculating the matrix exponent of the Hadamard product of the causal graph matrix A, where n is the dimension of the causal graph; For the directed acyclic loss of the causal graph matrix A, when When the value approaches 0, it indicates that the causal graph is a directed acyclic graph.

[0096] Furthermore, the formula for calculating the adversarial loss is as follows:

[0097]

[0098] In the formula, This represents the discriminative adversarial loss under multi-scale noise. This represents the reconstructed data after adding noise, where the noise scale is t. This represents the noise-added observed image data at a noise scale of t; This indicates the multi-scale noise discrimination module's handling of the noisy reconstructed data. The probability of being judged as true is obtained based on the true / false judgment probability output by the multi-scale noise discrimination module; This indicates the multi-scale noise discrimination module's handling of the noisy observed image data. The probability of being judged as true is obtained based on the true / false judgment probability output by the multi-scale noise discrimination module; This represents the observed image data after adding noise. Calculate the expectation; Represents the observed image data after adding noise. The true distribution.

[0099] Furthermore, in the adaptive multi-scale noise layer, the number of noise addition steps (i.e., the noise addition scale) is adaptively generated, specifically including: setting a noise scale update threshold. thr Obtain the maximum noise scale In optimizing loss , Reconstructed data output by the observation data decoding module in real time ,Will Input the multi-scale noise discrimination module to obtain the true / false discrimination probability. ,like Greater than thr This indicates that the generation quality of the observation data decoding module exceeds the threshold range, requiring an increase in the training difficulty of the multi-scale noise discrimination module, and a reduction in the noise scale. t Increase CStep. Discrimination is performed for each training until t increased to T max End of increment.

[0100] It should be understood that by conducting adversarial supervision training on the causal decoupling model under multi-scale noise, a strong discrimination module with discrimination ability under high disturbance conditions can be trained. Combined with causal constraints, the observation data encoding module, the observation data decoding module, and the multi-scale noise discrimination module are trained under multi-scale noise, the causal representation is learned, and the final robust trained causal decoupling model is obtained.

[0101] In the present embodiment, the architecture of the causal decoupling model based on multi-scale noise adversarial supervision is as shown in Figure 3 The observation image data is fused with the causal label and the causal graph feature to obtain a high-dimensional hidden vector with a causal representation, i.e., a causal hidden vector feature, through the observation data encoding module, and then the corresponding reconstructed data is obtained through the observation data decoding module. Under multi-scale noise, the observation data encoding module, the observation data decoding module, and the multi-scale noise discrimination module are trained adversarially. In order to enhance the discrimination ability of the multi-scale noise discrimination module, self-adaptive multi-scale noise fusion is used to add noise to the observation image data and the reconstructed data, and the noise adding mode follows the forward noise adding formula. Through adversarial supervision training, the multi-scale noise discrimination module can discriminate the quality of the reconstructed data, so that the observation data decoding module generates data conforming to the true distribution, and the decoupling ability of the causal decoupling model for the causal hidden vector feature is enhanced.

[0102] (3) The observation image data in the test set, the causal label information, and the trained causal graph matrix are input into the trained observation data encoding module to obtain the causal hidden vector feature, the causal variable to be intervened is selected, and its intervention value is calculated to modify the corresponding causal variable in the causal hidden vector feature, to obtain the intervened causal hidden vector feature, which is input into the trained observation data decoding module to obtain the counterfactual intervention data.

[0103] In the present embodiment, based on the trained causal decoupling model, the trained observation data encoding module, the trained observation data decoding module, and the trained causal graph matrix can be obtained. Since the causal graph matrix is trained after being input into the causal decoupling model, when the training is completed, the causal graph matrix is also trained. At this time, the trained causal graph matrix can be directly used for subsequent testing and online application. In short, the causal graph matrix can be regarded as a trainable parameter in the causal decoupling model, which can be directly used after training.

[0104] Specifically, as shown in Figure 8As shown, the observation image data in the test set, the causal label information and the trained final causal graph matrix are input into the trained observation data encoding module. The observation image data is obtained through the convolution layer of the encoding module to obtain the convolution representation. The convolution layer representation, the causal label information and the trained final causal graph matrix are input into the causal embedding layer of the encoding module for calculation, and then the causal hidden vector features are obtained after the data compression layer of the encoding module. The causal variable needing intervention is selected, the intervention value is calculated, and the corresponding causal variable in the causal hidden vector feature is modified according to the intervention value to obtain the causal hidden vector feature after intervention. The causal hidden vector feature after intervention, the causal label information and the trained final causal graph matrix are input into the trained observation data decoding module. The causal hidden vector feature after intervention is input into the data decoding layer of the decoding module to obtain the decoding representation. The decoding representation, the causal label information and the trained final causal graph matrix are input into the causal embedding layer of the decoding module for calculation, and then the counterfactual intervention data is obtained after the inverse convolution layer of the decoding module.

[0105] It should be noted that the trained observation data encoding module and the trained observation data decoding module can output reconstructed data, or perform counterfactual intervention on existing data to generate counterfactual intervention data that conforms to the causal relationship but does not exist in the test set.

[0106] Further, the counterfactual intervention data can verify whether the model has truly learned the hidden mechanism behind the data, test whether different latent variables control different semantics, and evaluate the degree of good causal representation learning, and test the learning ability of the model to causal representation and causal decoupling. The proportion of data conforming to the causal relationship in the counterfactual intervention data is calculated to evaluate the causal decoupling ability of the model. Specifically, the intervention classification accuracy can be used to evaluate the generation quality of the counterfactual intervention data, and the representation learning ability of the model is evaluated by calculating the number of samples with correct semantic transformation in the causal counterfactual intervention data divided by the total number of samples.

[0107] In summary, the causal decoupling method based on multi-scale noise and adversarial supervision aims to automatically learn variables with causal semantic features, establish constraints using causal embedding, and determine the causal mechanism of these variables in the real physical world. It can provide extensive support for downstream tasks, including transfer generalization, interpretable learning, multi-modal learning and representation alignment, consistency controllable generation, etc. Most of the current generation or representation models focus on fitting the surface distribution of observed data, ignoring the underlying generation mechanism of the data, which has bottlenecks in learning the real physical mechanism. Although such methods have made significant progress in image and text generation tasks, they still have insufficient generalization for out-of-distribution data. The model may have "hallucinations" and poor consistency modeling ability in complex situations, and cannot be applied in high-precision scenarios. However, the causal decoupling method based on multi-scale noise and adversarial supervision can separate background noise data unrelated to the underlying causal relationship through adversarial supervision training of the multi-scale noise discrimination module, observation data encoding module and observation data decoding module, and enhance the learning ability of the representation to the causal structure. Specifically, in the robot complex scene object grasping and decision planning task, the image data received by the visual sensor contains light changes, obstacle occlusions, environmental background noise and other disturbances. Traditional representation learning methods rely on statistical correlation and are prone to overfitting to these disturbances, leading to task failure. By using the causal decoupling method based on multi-scale noise and adversarial supervision, the robust representation can be output to the downstream task after learning, which can separate the factors directly related to the grasping decision task from the irrelevant disturbances. Even in scenes with light changes and partial occlusions, the grasping position in the image can be accurately calculated. Further, since the representation depends on the causal structure unrelated to the background features, even when migrating to a new task scenario, the dependence on the old task scenario can be reduced, improving the accuracy of downstream task execution and the robustness of the model.

[0108] It is worth mentioning that the embodiment of the present application also provides a causal decoupling device based on multi-scale noise and adversarial supervision, which is used to realize the causal decoupling method based on multi-scale noise and adversarial supervision in the above embodiment. As shown in the Figure 9 causal data synthesis module, a model construction and training module, and a causal counterfactual intervention module.

[0109] In this embodiment, the causal data synthesis module is used to simulate the causal relationship of variables in the causal graph using the prior causal mechanism, to generate observation image data containing causal relationship, as shown in Figure 2 and construct a training set and a test set according to the observation image data and its corresponding causal label information and the causal graph.

[0110] In this embodiment, the model construction and training module is configured to construct a causal decoupling model, and perform adversarial supervised training under multi-scale noise using a training set to obtain a trained causal decoupling model, as shown in Figure 7

[0111] In this embodiment, the causal counterfactual intervention module is configured to input the observation image data, the causal label information and the trained causal graph matrix in the test set into the trained observation data encoding module to obtain causal latent vector features, select a causal variable to be intervened, calculate an intervention value of the causal variable, modify the corresponding causal variable in the causal latent vector features, obtain intervened causal latent vector features, and input the intervened causal latent vector features into the trained observation data decoding module to obtain counterfactual intervention data, as shown in Figure 8

[0112] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.​​

Claims

1. A method for causal disentanglement based on multi-scale noise and adversarial supervision, characterized in that, The method comprises the following steps: (1) Simulate the causal relationship of variables in a causal graph using a prior causal mechanism, generate observed image data, and construct a training set and a test set according to the observed image data, the corresponding causal label information and the causal graph; (2) Construct a causal decoupling model and perform adversarial supervised training under multi-scale noise using the training set to obtain a trained causal decoupling model; wherein the causal decoupling model comprises an observed data encoding module, an observed data decoding module and a multi-scale noise discrimination module; the observed data encoding module comprises an encoding module convolution layer, an encoding module causal embedding layer and an encoding module data compression layer; the observed data decoding module comprises a decoding module data decoding layer, a decoding module causal embedding layer and a decoding module inverse convolution layer; the multi-scale noise discrimination module comprises an adaptive multi-scale noise adding layer, a discrimination module convolution layer, a discrimination module causal embedding layer, a discrimination module data compression layer and a classification layer; The observed image data, the causal label information and the causal graph matrix are input into the causal decoupling model, the observed image data is obtained through the encoding module convolution layer to obtain a convolution representation, the convolution representation, the causal label information and the causal graph matrix are input into the encoding module causal embedding layer for calculation, and then the causal hidden vector feature is obtained after passing through the encoding module data compression layer; the causal hidden vector feature is first obtained through the decoding module data decoding layer to obtain a decoding representation, the decoding representation, the causal label information and the causal graph matrix are input into the decoding module causal embedding layer for calculation, and then the reconstructed data is obtained after passing through the decoding module inverse convolution layer; the reconstructed data and the observed image data are input into the adaptive multi-scale noise adding layer to obtain the noise-added reconstructed data and the observed image data, the noise-added reconstructed data and the observed image data are simultaneously input into the discrimination module convolution layer to obtain a convolution feature, the convolution feature, the causal label information and the causal graph matrix are input into the discrimination module causal embedding layer for calculation, and then the true-false discrimination probability is output after passing through the discrimination module data compression layer and the classification layer in sequence; (3) The observed image data, the causal label information and the trained causal graph matrix in the test set are input into the trained observed data encoding module to obtain the causal hidden vector feature, the causal variable to be intervened is selected, the intervention value thereof is calculated, the corresponding causal variable in the causal hidden vector feature is modified to obtain the intervened causal hidden vector feature, and the intervened causal hidden vector feature is input into the trained observed data decoding module to obtain the counterfactual intervention data.

2. The method of claim 1, wherein, The observed image data is specifically obtained by the following method: A directed acyclic graph is constructed as a real causal graph according to a plurality of potential causal variables; exogenous noise and variable values are input into the causal graph to construct causal label information in a simulated environment, and the observed values of variables in the causal graph are simulated and generated using a causal function model; Based on the causal graph, each node is sampled to obtain causal graph observation data according to the relationship on the causal graph, and the image data generated based on the causal graph observation data is the simulated observed image data.

3. The method of claim 1, wherein, The adaptive multi-scale noise layer specifically comprises: initializing a noise step number and generating a time embedding, and then using a forward noise mode of a diffusion model to add noise to the reconstructed data and the observed image data to obtain the noise-added reconstructed data and the noise-added observed image data.

4. The method of claim 3, wherein, The initializing of the noise step number and the generating of the time embedding specifically comprise: Initializing a maximum noise scale And the time scale embedding dimension d, the time scale is encoded by using the triangular embedding, the time scale embedding base is calculated by the frequency function, the noise step sequence T is processed by using the time scale embedding base, and then the sine and cosine transformations are performed respectively, and the embedding after the sine and cosine transformations is spliced to obtain a time scale embedding vector with a length of d.

5. The method of claim 3, wherein, The using of the forward noise mode of the diffusion model to add noise to the reconstructed data and the observed image data specifically comprises: Based on the normal distribution noise and the noise step number, the reconstructed data and the observed image data are added noise through a cumulative noise factor to obtain the noise-added reconstructed data and the noise-added observed image data; wherein the cumulative noise factor is obtained according to the noise variance on the corresponding time scale and the time scale embedding vector.

6. The method of claim 1, wherein, The using of the training set to perform adversarial supervision training under the multi-scale noise to obtain the trained causal decoupling model specifically comprises: The observed image data, the causal label information and the causal graph matrix in the training set are input into the causal decoupling model to obtain the causal hidden vector feature, the reconstructed data and the authenticity discrimination probability; The autoencoder loss is calculated based on the observed image data, the causal graph matrix, the causal hidden vector feature, the reconstructed data and the authenticity discrimination probability, and the discrimination adversarial loss is calculated based on the authenticity discrimination probability corresponding to the observed image data and the reconstructed data; In the training process, the autoencoder loss and the discrimination adversarial loss are alternately optimized, when the autoencoder loss is optimized, only the parameters of the observed data encoding module and the observed data decoding module are updated; when the discrimination adversarial loss is optimized, only the parameters of the observed data decoding module and the multi-scale noise discrimination module are updated; until a preset training round or the autoencoder loss and the discrimination adversarial loss converge, the trained causal decoupling model is obtained.

7. The method of claim 6, wherein, The autoencoder loss is specifically obtained by the following method: The root mean square error between the observed image data and the reconstructed data is calculated as the reconstruction loss; the KL divergence loss is calculated based on the causal hidden vector feature and its prior distribution; the generation loss is calculated according to the authenticity discrimination probability, the total variation loss and the entropy loss are calculated according to the reconstructed data, the adversarial generation loss is calculated according to the generation loss, the total variation loss and the entropy loss; the acyclic graph loss is calculated according to the causal graph matrix; and the autoencoder loss is calculated according to the reconstruction loss, the KL divergence loss, the adversarial generation loss and the acyclic graph loss.

8. The method of claim 1, wherein, The step (3) specifically comprises: The observed image data, the causal label information and the trained causal graph matrix in the test set are input into the trained observed data encoding module, the observed image data is obtained through the encoding module convolution layer to obtain the convolution representation, the convolution layer representation, the causal label information and the trained final causal graph matrix are input into the encoding module causal embedding layer for calculation, and then the causal hidden vector feature is obtained after the encoding module data compression layer. The causal variable to be intervened is selected, the intervention value thereof is calculated, and the corresponding causal variable in the causal hidden vector feature is modified according to the intervention value to obtain the intervened causal hidden vector feature. The intervened causal hidden vector feature, the causal label information and the trained causal graph matrix are input into the trained observation data decoding module, the intervened causal hidden vector feature is first subjected to data decoding of the decoding module to obtain a decoding representation, the decoding representation, the causal label information and the trained causal graph matrix are input into a causal embedding layer of the decoding module for calculation, and counterfactual intervention data is obtained after inverse convolution of the decoding module.

9. An apparatus for causally decoupling based on multi-scale noise and adversarial supervision, configured to implement the method for causally decoupling based on multi-scale noise and adversarial supervision according to any one of claims 1-8, characterized in that, The device comprises: The causal data synthesis module is configured to simulate the causal relationship of variables in the causal graph by using a prior causal mechanism, generate observation image data, and construct a training set and a test set according to the observation image data, the corresponding causal label information and the causal graph; The model construction and training module is configured to construct a causal decoupling model, and perform adversarial supervised training under multi-scale noise by using the training set to obtain the trained causal decoupling model; wherein the causal decoupling model comprises an observation data encoding module, an observation data decoding module and a multi-scale noise discrimination module; The causal counterfactual intervention module is configured to input the observation image data in the test set, the causal label information and the trained causal graph matrix into the trained observation data encoding module to obtain a causal hidden vector feature, select a causal variable to be intervened, calculate an intervention value of the causal variable, modify the corresponding causal variable in the causal hidden vector feature, obtain an intervened causal hidden vector feature, and input the intervened causal hidden vector feature into the trained observation data decoding module to obtain counterfactual intervention data.

Images