Method, device, equipment and medium for extracting aggregatable causal information from medical images

Abstract

The present application provides a method, device, equipment and medium for extracting aggregable causal information from medical images. It relates to the technical fields of representation learning, causal inference and deep generative models. The method includes: constructing a causal representation learning framework; the causal representation learning framework includes an encoder, a structural causal model, a decoder and a discriminator, the encoder is used to encode the input medical image into a low-dimensional exogenous variable, the structural causal model takes the low-dimensional exogenous variable as input, and generates a causal representation based on the low-dimensional exogenous variable, the decoder is used to intervene and reconstruct the causal representation, and the discriminator is used for adversarial training; a model training loss function is established, and the causal representation learning framework is trained based on the model training loss function, and the trained causal representation learning framework is used to realize the extraction of aggregable causal information in medical images. The causal graph identified by the present application can clearly describe the causal relationship between different features, providing more transparent decision support for clinicians.

Description

Technical Field

[0001] The present application relates to the technical fields of representation learning, causal inference and deep generative models, and in particular to a method, apparatus, device and medium for extracting aggregatable causal information from medical images. Background Art

[0002] With the rapid development of medical imaging technologies (such as X-rays, CT scans, MRI scans, and ultrasound), medical images are playing an increasingly important role in disease diagnosis, treatment planning, and efficacy evaluation. Traditional deep learning models require large amounts of labeled data for training. However, labeling medical image data is expensive, scarce, and inconsistent. Furthermore, deep learning models are often "black box" models, making it difficult to provide explanatory information of practical value to medical professionals. Causal representation learning offers a new paradigm for addressing these issues. It explicitly models the causal relationships between variables and extracts causally meaningful representations, thereby enhancing the model's generalization and interpretability.

[0003] By learning causal representations from medical images, we can model causal relationships in the data and effectively remove confounding factors introduced by data noise and bias, thereby more accurately identifying disease characteristics. This causal-driven analysis approach helps reduce misdiagnoses and missed diagnoses caused by subjective factors or equipment differences, and improves diagnostic reliability and consistency. Furthermore, the medical field places high demands on model interpretability. Causal graphs can clearly reveal the causal relationships between variables, helping clinicians understand the rationale behind model outputs and thereby increasing trust in AI-powered decision-making.

[0004] Current causal representation learning algorithms are mainly divided into two categories: VAE-based algorithms and GAN-based algorithms. In 2018, Kocaoglu et al. proposed the CausalGAN model, which is a causal implicit generative model for learning a given causal graph. They used the causal graph to represent the dependency structure between binary labels. During the generation process, binary labels can be used to control relevant attributes in facial images, such as smiling, squinting, and other attributes. The limitation of CausalGAN is that the real causal information is directly input into the generator as a conditional attribute. The model does not adjust the dimensional alignment between the latent variables and the latent generation factors. Therefore, CausalGAN does not perform representation learning in the decoupling stage. In 2021, the CausalVAE model proposed by Yang et al. introduced the structural causal model (SCM) and binary label information of image attributes in the decoupling stage, input the encoded representation into the SCM for re-modulation, and output a representation vector with causal information. During the generation process, the latent vector is nonlinearly processed and the scene is reconstructed, realizing causally controllable decoupled representation learning.

[0005] Metric learning is an important branch of machine learning that aims to learn effective distance or similarity metrics between data. Deep metric learning can perform nonlinear mapping on input features and has been widely used in the field of computer vision. Graph Attention Networks (GATs) are novel convolutional neural networks that incorporate masked self-attention layers and can process graph-structured data. The graph attention layer used by GATs is computationally efficient (no expensive matrix operations are required and all nodes in the graph can be processed in parallel), allowing different importance to be (implicitly) assigned to different nodes within a neighborhood, while also processing neighborhoods of different sizes.

[0006] Against this background, how to capture the complex nonlinear causal relationships in medical image data, effectively remove the confounding factors introduced by data noise and bias, increase the independence of the extracted representations, and thus more accurately identify disease characteristics and provide precise support for personalized diagnosis and treatment, is a technical problem that urgently needs to be solved. Summary of the Invention

[0007] The present application provides a method, apparatus, device and medium for extracting aggregatable causal information from medical images. Taking into account the complex nonlinear distribution in medical images, metric learning and graph attention mechanism are introduced into the traditional causal representation learning framework to construct a GAT-based SCM, thereby improving the quality of causal representation and inferring causal relationships in images.

[0008] In a first aspect, the present application provides a method for extracting aggregatable causal information from medical images, comprising:

[0009] Constructing a causal representation learning framework; wherein the causal representation learning framework includes an encoder, a structural causal model, a decoder, and a discriminator, wherein the encoder is used to encode the input medical image into a low-dimensional exogenous variable, the structural causal model takes the low-dimensional exogenous variable as input and generates a causal representation based on the low-dimensional exogenous variable, the decoder is used to intervene and reconstruct the causal representation, and the discriminator is used for adversarial training;

[0010] A model training loss function is established, and the causal representation learning framework is trained based on the model training loss function, so as to realize the extraction of aggregatable causal information in medical images using the trained causal representation learning framework.

[0011] In one possible design, the structural causal model is expressed as:

[0012]

[0013] Where z represents the causal representation, T represents the matrix transpose, f and h represent nonlinear functions, and g represents the parameterized graph attention network. represents low-dimensional exogenous variables, represents the domain of real numbers, d represents the number of causal attributes, A represents the prior causal graph, Represents the causal structure matrix;

[0014] Gradient-based continuous update strategy optimizes graph attention network to obtain causal structure matrix

[0015] The causal structure matrix Input into the structural causal model to generate a causal representation.

[0016] In one possible design, a gradient-based continuous update strategy optimizes the graph attention network to obtain the causal structure matrix include:

[0017] Take i and j as the causal structure matrix For any two nodes in , we learn the causal edge between i and j by multiplying the connection feature with the shared attention mechanism a(·) to generate an attention score e ij ; The attention score is expressed as:

[0018] e ij =a(W∈ i ,W∈ j )

[0019] Where, represents a shared weight matrix;

[0020] For node i, only node j∈Pa is calculated i , where Pa i Represents the parent nodes and uses the softmax function to normalize them. The calculation process is as follows:

[0021]

[0022] Where, represents the normalized attention coefficient generated by the h-th attention head, and H represents the number of attention heads;

[0023] Constructed by the following formula

[0024]

[0025] Where, represents the causal strength of row i and column j in the causal structure matrix, τ>0, is the temperature parameter that controls the outcome to be close to 0 or 1; g1 and g0 represent samples drawn independently from the Gumbel(0,1) distribution, and σ(·) represents the logistic sigmoid function;

[0026] Iteratively train the graph attention network to obtain the causal structure matrix

[0027] In one possible design, the causal structure matrix Input into the structural causal model to generate a causal representation, including:

[0028] The encoder based on metric learning is used to generate the ideal causal representation of triple samples. The triple network fits the feature distribution in the latent space by measuring the distance between the anchor point and the positive / negative samples, thereby obtaining a causal representation that is not affected by distribution shift. The triple loss is expressed as:

[0029]

[0030] In the formula, (z a ,z p ,z n ) represents a triplet, including an anchor sample z a , a positive sample z p and a negative sample z n , max represents the maximum value function, D(z a ,z n ) represents the distance between the anchor sample and the negative sample, D(z a ,z p ) represents the distance between the anchor sample and the positive sample, D(z i ,z j )=||f(z i )-f(z j )||2 represents the Euclidean distance between two vectors, and m represents the margin.

[0031] In one possible design, the causal representation is intervened and reconstructed through the following formula:

[0032]

[0033] Where, f -1 represents the inverse mapping of f after intervention, represents a set of intervention operations, represents the causal representation after the intervention, Represents the causal structure matrix after the intervention.

[0034] In one possible design, the model training loss function is established as follows;

[0035] By learning an approximate variational distribution q of the true distribution p of the causal representation z, we can fit the actual posterior distribution;

[0036] Get the training dataset and joint distribution Maximize the evidence lower bound, which is expressed as:

[0037]

[0038] Where ELBO represents the lower bound of evidence; Indicates the data distribution Take the expectation of all samples to ensure that the optimization of the model's objective function can reflect the overall performance of the dataset; represents the approximate posterior distribution q of the latent variable z φ The expected value calculated under (z|x,u); p θ (x|z) represents the distribution of observed data generated from the latent variable, parameterized by θ; q φ (∈|x,u) represents the probability distribution of ∈ given the observation data x and the conditional variable u; p ∈ (∈) represents the prior distribution of the noise variable ∈; q φ (z|x,u) represents the approximate posterior distribution of the latent variable z; p θ (z|u) represents the prior distribution of the latent variable z;

[0039] and represents the KL divergence term, which measures the difference between the two distributions separated by ||; x represents high-dimensional medical image data; u represents the corresponding label of the image;

[0040] The model learns the causal structure by setting a prior on the representation generated by the structural causal model and computing the KL divergence, using the following loss function:

[0041]

[0042] in

[0043]

[0044] Where α and β are regularization hyperparameters, represents the expected log-likelihood of reconstruction ability, Indicates that the distribution of latent variables is constrained by prior knowledge; Represents the distance between features in the latent space by optimizing; Represents the overall loss function of the VAE framework; D KL (q φ (∈|x,u)‖p ∈ (∈)) represents q φ (∈|x,u) and q φ KL divergence of (∈|x,u), where q φ(∈|x,u) is a variational approximation that models the posterior distribution of the latent variable ∈, p ∈ (∈);D KL (q φ (z|x,u)||p θ (z|u)) represents the distribution q φ (z|x,u) and p θ KL divergence between (z|u), where q φ (z|x,u) is the approximate posterior distribution learned by the encoder network in VAE, which represents the probability distribution of the latent variable z given the observation data x and condition u; x a represents the anchor point sample; x p represents samples that are semantically close to the anchor samples, i.e., positive samples; x n It represents the samples that are semantically different from the anchor samples, that is, negative samples; m represents the margin; D(x a ,x p ) represents the distance between the anchor sample and the positive sample in the feature space, which is generally calculated using the Euclidean distance; D(x a ,x n ) represents the distance between the anchor sample and the negative sample in the feature space;

[0045] The discriminator optimizes a binary classifier using the following loss function:

[0046]

[0047] Where, Represents the discriminator loss; Dis(x) represents the output of the discriminator, and its value range is 0≤Dis(x)≤1. Represents the data generated by the decoder;

[0048] The model training loss function Expressed as:

[0049]

[0050] In a second aspect, the present application provides a device for extracting aggregatable causal information from medical images, the device comprising:

[0051] a framework construction module configured to construct a causal representation learning framework; wherein the causal representation learning framework includes an encoder, a structural causal model, a decoder, and a discriminator, wherein the encoder is used to encode an input medical image into a low-dimensional exogenous variable, the structural causal model takes the low-dimensional exogenous variable as input and generates a causal representation based on the low-dimensional exogenous variable, the decoder is used to intervene in and reconstruct the causal representation, and the discriminator is used for adversarial training;

[0052] The framework training module is configured to establish a model training loss function, train the causal representation learning framework based on the model training loss function, and realize the extraction of aggregatable causal information in medical images using the trained causal representation learning framework.

[0053] In a third aspect, an embodiment of the present application provides an electronic device comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the aggregatable causal information extraction method for medical images as described in the first aspect and various possible designs of the first aspect.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method for extracting aggregatable causal information from medical images as described in the first aspect and various possible designs of the first aspect.

[0055] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for extracting aggregatable causal information from medical images as described in the first aspect and various possible designs of the first aspect.

[0056] The method, apparatus, device, and medium for extracting aggregatable causal information from medical images provided in this application effectively quantify detection uncertainty, significantly improve the reliability and adaptability of the model in different scenarios, and have greater robustness. The main advantages are summarized as follows:

[0057] 1. Based on the encoder-decoder framework, this application introduces a decoupling method of metric learning on the encoder side to separate representations, increase the distance between negative samples and anchor points, and reduce the distance between positive samples and anchor points. A structural causal model is introduced between the encoder and decoder, using a graph attention network (GAT) to identify causal relationships between variables, and a causal layer to transform exogenous variables into endogenous variables. This can capture the complex nonlinear causal relationships in medical image data, effectively remove confounding factors introduced by data noise and bias, increase the independence of the extracted representations, and thus more accurately identify disease characteristics, providing precise support for personalized diagnosis and treatment.

[0058] 2. In order to learn causal representations from medical image data, remove confounding factors introduced by data noise and bias, thereby more accurately identifying disease features and improving the interpretability of the model, a causal representation learning model based on a triplet network and a graph attention mechanism is proposed. The graph attention network (GAT) is injected into the structural causal model (SCM), and the causal structure matrix is obtained by aggregating the causal information of context nodes and continuously updating the GAT using a gradient-based strategy. The triplet loss is used to reduce the distance between similar samples in the latent space distribution to obtain a more effective causal representation. This application can learn causal relationships in medical images with high accuracy; this application uses a triplet network to generate more effective causal representations by reducing the distance between similar samples in the latent space distribution; the causal graph identified by this application can clearly describe the causal relationship between different features, providing more transparent decision support for clinicians. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] Figure 1 A schematic diagram of the structure of the causal representation learning framework provided in an embodiment of the present application;

[0061] Figure 2 A flowchart of a method for extracting aggregatable causal information from medical images provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the structure of an apparatus for extracting aggregatable causal information from medical images provided in an embodiment of the present application.

[0063] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0065] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0066] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0067] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] An embodiment of the present application provides a method for extracting aggregatable causal information from medical images. The method comprises: constructing a causal representation learning framework; establishing a model training loss function, training the causal representation learning framework based on the model training loss function, and extracting aggregatable causal information from medical images using the trained causal representation learning framework. The aggregatable causal information extracted in this embodiment is a causal graph, which can provide more transparent decision support for clinicians.

[0069] Figure 1 A schematic diagram of the structure of a causal representation learning framework provided in an embodiment of the present application, wherein the causal representation learning framework includes an encoder, a decoder, a discriminator, and a structural causal model SCM.

[0070] Encoder, used to encode nonlinear feature distribution images into low-dimensional exogenous variables.

[0071] SCM, used to define the causal structure in the data, consists of a graph attention network (GAT) and a set of structural equations containing nonlinear transformations to achieve the conversion from exogenous variables to endogenous variables.

[0072] The decoder is used to reconstruct the original image or the intervened image from the low-dimensional representation, helping to verify the validity of the underlying causal representation.

[0073] The discriminator is used for adversarial training, which enables the discriminator to measure samples and thus guide the decoder to generate more realistic intervention samples.

[0074] like Figure 1 As shown, the encoder accepts a triple (x a ,x p ,x n) as input, which contains anchor samples, positive samples with similar features to the anchor samples, and negative samples with dissimilar features to the anchor samples. The encoder encodes the inputs as independent exogenous variables, and their prior distribution is assumed to be a standard Gaussian distribution. Then, the SCM converts them into causal representations (z a ,z p ,z n ), and calculate the triplet loss to update the model. The discriminator calculates the discriminator loss by inputting the generated image and the real image, which is used to update the discriminator for adversarial training.

[0075] To learn highly generalizable causal representations from medical images, we first incorporate a Graph Attention Network (GAT) into the SCM to aggregate causal information from contextual nodes and encode this information into the GAT parameters. Secondly, we use a triplet loss to reduce the distance between similar samples, adapting to the true distribution and obtaining an effective causal representation. Furthermore, the triplet loss encourages the decoder to generate more realistic intervention samples, which are used to evaluate the generated causal representations. This is achieved by jointly training a variational autoencoder (VAE) and a generative adversarial network (GAN).

[0076] In a specific embodiment, Figure 2 This is a flow chart of a method for extracting aggregatable causal information from medical images provided in an embodiment of the present application. Figure 2 As shown, the method for extracting aggregatable causal information from medical images includes the following steps S10 to S40, which are described in detail below.

[0077] Step S10: construct a causal representation learning framework based on GAT and metric learning.

[0078] For the latent factors of data x If the encoder E learns a decoupled representation of z (i.e., a low-dimensional exogenous variable), that is, for each i=1,...,m, there exists a corresponding function [E(x)] i =z i .

[0079] Contrary to the fact, most previous methods use independent priors to describe z. Assume a more general case where there are causal relationships between latent factors. A structural causal model (SCM) represents a tuple Where z represents a set of endogenous variables represents a set of exogenous variables Structural equations is a function that determines z, where z i =f i (Pa i ,∈ i ), Pa iRepresents z in the causal structure matrix i The parent node set of . SCM uses the deterministic function f and exogenous variables ∈ to build a causal model, and uses the encoder E to generate independent exogenous variables from x.

[0080] Let X = {x n |1≤n≤N} is the data input set containing d potential factors, U={u n |1≤n≤N} is the corresponding attribute label set. By generating a triplet dataset, we can obtain the triplet (x a ,x p ,x n ), representing anchor points, positive samples, and negative samples. For an image x, first use the encoder network E (such as ResNet) to generate exogenous variables This embodiment uses SCM and GAT to learn semantic association information from exogenous variables to obtain the matrix So, set the SCM to:

[0081]

[0082] where g(·) represents the parameterized GAT, f i (·) and h(·) represent nonlinear functions, A represents the prior causal structure, Represents the causal structure matrix.

[0083] Optimize the multi-layer GAT via a gradient-based continuous update strategy to obtain the causal structure matrix Then Input into the structural equation to generate the causal representation z.

[0084] The task of the causal representation learning framework consists of three parts:

[0085] 1) Learn the causal structure matrix based on the supervision label u(u∈U)

[0086] 2) Using the triplet dataset (x a ,x p ,x n ) Generate causal representations;

[0087] 3) Intervene in the causal representation z, use an independent causal mechanism to intervene and reconstruct the causal representation, and generate the intervened image through the decoder, that is, the aggregatable causal information extracted from the medical image.

[0088] Step S20, learning the causal structure matrix

[0089] This embodiment uses GAT based on aggregated causal information to learn the causal structure matrix. Considering the permutation invariance and message passing mechanism of GAT, it can process nodes without losing performance, effectively transmit causal effects, and obtain the dynamic causal strength on the edge. Let i and j be For any two nodes in , we learn the causal edge between i and j by multiplying the connection feature with the shared attention mechanism a(·) to generate an attention score e ij ;e ij reflects the probability of causal edges. By stacking multiple layers of networks, e ij Can be propagated from parent node to child node:

[0090] e ij =a(W∈ i ,W∈ j )

[0091] in, represents a shared weight matrix. The prior causal structure is then injected into the causal attention mechanism by performing masked attention. For node i, only node j∈Pa i , where Pa i Represents the parent node. In order to balance the influence of different parent nodes, the softmax function is used to normalize them; the process is as follows:

[0092]

[0093] Denotes the normalized attention coefficient generated by the hth attention head, H represents the number of attention heads; by using Gumbel-Sigmoid to identify causal edges, this can approximate the sample (threshold = 0.5) and ensure that the parameters are suitable for gradient optimization. It can be expressed as follows:

[0094]

[0095] in, represents the causal strength of row i and column j in the causal structure matrix, τ>0 is the temperature parameter that controls the result to be close to 0 or 1; here, '1' indicates the existence of a causal edge, g1 and g0 represent samples drawn independently from the Gumbel(0,1) distribution, and σ(·) represents the logistic sigmoid function. The above formula is used to iteratively train the GAT to obtain the causal structure matrix

[0096] Step S30: Generate causal representation z.

[0097] We use a metric learning-based VAE to generate ideal causal representations for triplet samples. The triplet network fits the feature distribution in the latent space by measuring the distance between the anchor point and the positive / negative samples, thereby obtaining a causal representation that is not affected by distribution shift. For an anchor sample, we select samples with causal properties close to it as its positive sample, and otherwise as negative samples. Triplet samples share encoder weights, and the triplet loss is defined as follows:

[0098]

[0099] According to the above formula, the triple (z a ,z p ,z n ) is generated by SCM, which contains an anchor sample z a , a positive sample z p and a negative sample z n The goal of the above formula is to ensure that the distance D(z a ,z n ) exceeds the distance D(z) between the anchor sample and the positive sample a ,z p ), and the difference between the two is at least a margin m; where D(z i ,z j )=||f(z i )-f(z j )||2 represents the Euclidean distance between two vectors; this is done to effectively utilize the triplet loss to enhance the performance of VAE in learning causal representations.

[0100] Step S40: Reconstructing the causal representation of the intervention

[0101] According to the causal ladder theory, an “intervention” involves a change in the distribution of a variable, and the intervention affects only those nodes that belong to the same module as the intervened node.

[0102] By using “interventions” to manipulate and reconstruct variables and using SCM to simulate the propagation of causal effects, the causal structure matrix Changes in can indicate causal relationships after intervention;

[0103]

[0104] Among them, f -1 represents the inverse function of f, that is, the inverse mapping of f after intervention. The above equation is obtained by f -1 Simplify the complexity of nonlinear causal relationships into linear form. represents a set of intervention operations, represents the causal representation after the intervention, represents the causal structure matrix after intervention. The intervened causal node will change the effect node, otherwise it will not change. This is because in SCM, causal information is only transferred from causal nodes to effect nodes.

[0105] Step S50: define the model training loss function.

[0106] By learning the approximate variational distribution q of the true distribution p of the latent variable z, to fit the actual posterior distribution; given a training dataset and joint distribution Try to maximize the Evidence Lower Bound (ELBO), which is a variational lower bound of:

[0107]

[0108] Among them, θ and φ are learnable parameters; considering the correlation between ∈ and z, the equation can be written as follows:

[0109]

[0110] Where ELBO represents the lower bound of evidence; Indicates the data distribution Take the expectation of all samples to ensure that the optimization of the model's objective function can reflect the overall performance of the dataset; represents the approximate posterior distribution q of the latent variable z φ The expected value calculated under (z|x,u); p θ (x|z) represents the distribution of observed data generated from the latent variable, parameterized by θ; q φ (∈|x,u) represents the probability distribution of ∈ given the observation data x and the conditional variable u; p ∈ (∈) represents the prior distribution of the noise variable ∈; q φ (z|x,u) represents the approximate posterior distribution of the latent variable z; p θ (z|u) represents the prior distribution of the latent variable z;

[0111] and represents the KL divergence term, which measures the difference between the two distributions separated by ||; x represents high-dimensional medical image data; u represents the corresponding label of the image.

[0112] The model learns the causal structure by setting a prior on the representation generated by the structural causal model and computing the Kullback-Leibler (KL) divergence, using the following loss function:

[0113]

[0114] in

[0115]

[0116]

[0117] Among them, α and β are regularization hyperparameters, represents the expected log-likelihood of reconstruction ability, Indicates that the distribution of latent variables is constrained by prior knowledge; Represents the distance between features in the latent space by optimizing; Represents the overall loss function of the VAE framework; D KL (q φ (∈|x,u)‖p ∈ (∈)) represents q φ (∈|x,u) and q φ KL divergence of (∈|x,u), where q φ (∈|x,u) is a variational approximation that models the posterior distribution of the latent variable ∈, p ∈ (∈);D KL (q φ (z|x,u)||p θ (z|u)) represents the distribution q φ (z|x,u) and p θ KL divergence between (z|u), where q φ (z|x,u) is the approximate posterior distribution learned by the encoder network in VAE, which represents the probability distribution of the latent variable z given the observation data x and condition u; x a represents the anchor point sample; x p represents samples that are semantically close to the anchor samples, i.e., positive samples; x n It represents the samples that are semantically different from the anchor samples, that is, negative samples; m represents the margin; D(x a ,x p ) represents the distance between the anchor sample and the positive sample in the feature space, which is generally calculated using the Euclidean distance; D(x a ,x n ) represents the distance between the anchor sample and the negative sample in the feature space.

[0118] The discriminator aims to optimize a binary classifier using the following loss function to better distinguish between real and generated images while encouraging the generator to fit the true distribution;

[0119]

[0120] Where Dis(x) represents the output of the discriminator, and its value range is 0≤Dis(x)≤1. Represents data generated by the decoder.

[0121] The model training loss function consists of the following equation:

[0122]

[0123] Where, Train the loss function for the model.

[0124] The embodiment of the present application also provides a device for extracting causal information from medical images. Figure 3 As shown, the device for extracting aggregatable causal information from a medical image includes:

[0125] A framework construction module 301 is configured to construct a causal representation learning framework; wherein the causal representation learning framework includes an encoder, a structural causal model, a decoder, and a discriminator, wherein the encoder is used to encode an input medical image into a low-dimensional exogenous variable, the structural causal model takes the low-dimensional exogenous variable as input and generates a causal representation based on the low-dimensional exogenous variable, the decoder is used to intervene in and reconstruct the causal representation, and the discriminator is used for adversarial training;

[0126] The framework training module 302 is configured to establish a model training loss function, train the causal representation learning framework based on the model training loss function, and implement the extraction of aggregatable causal information in medical images using the trained causal representation learning framework.

[0127] In some embodiments, the framework building module is further configured to represent the structural causal model as:

[0128]

[0129] Where z represents the causal representation, T represents the matrix transpose, f and h represent nonlinear functions, and g represents the parameterized graph attention network. represents low-dimensional exogenous variables, represents the real number domain, d represents the dimension of causal representation, A represents the prior causal structure, Represents the causal structure matrix;

[0130] Gradient-based continuous update strategy optimizes graph attention network to obtain causal structure matrix

[0131] The causal structure matrix Input into the structural causal model to generate a causal representation.

[0132] In some embodiments, the framework building module is further configured to:

[0133] Take i and j as the causal structure matrix For any two nodes in , we learn the causal edge between i and j by multiplying the connection feature with the shared attention mechanism a(·) to generate an attention score e ij ; The attention score is expressed as:

[0134] e ij =a(W∈ i ,W∈ j )

[0135] Where, represents a shared weight matrix;

[0136] For node i, only node j∈Pa is calculated i , where Pa i Represents the parent nodes and uses the softmax function to normalize them. The calculation process is as follows:

[0137]

[0138] Where, represents the normalized attention coefficient generated by the h-th attention head, and H represents the number of attention heads;

[0139] Constructed by the following formula

[0140]

[0141] Where, represents the causal strength of row i and column j in the causal structure matrix, τ>0, is the temperature parameter that controls the outcome to be close to 0 or 1; g1 and g0 represent samples drawn independently from the Gumbel(0,1) distribution, and σ(·) represents the logistic sigmoid function;

[0142] Iteratively train the graph attention network to obtain the causal structure matrix

[0143] In some embodiments, the framework building module is further configured to:

[0144] The encoder based on metric learning is used to generate the ideal causal representation of triple samples. The triple network fits the feature distribution in the latent space by measuring the distance between the anchor point and the positive / negative samples, thereby obtaining a causal representation that is not affected by distribution shift. The triple loss is expressed as:

[0145]

[0146] In the formula, (z a ,z p ,z n ) represents a triplet, including an anchor sample z a, a positive sample z p and a negative sample z n , max represents the maximum value function, D(z a ,z n ) represents the distance between the anchor sample and the negative sample, D(z a ,z p ) represents the distance between the anchor sample and the positive sample, D(z i ,z j )=||f(z i )-f(z j )||2 represents the Euclidean distance between two vectors, and m represents the margin.

[0147] In some embodiments, the framework building module is further configured to intervene and reconstruct the causal representation through the following formula:

[0148]

[0149] Where, f -1 represents the inverse mapping of f after intervention, represents a set of intervention operations, represents the causal representation after the intervention, Represents the causal structure matrix after the intervention.

[0150] In some embodiments, the framework training module is further configured to establish a model training loss function by the following method;

[0151] By learning an approximate variational distribution q of the true distribution p of the causal representation z, we can fit the actual posterior distribution;

[0152] Get the training dataset and joint distribution Maximize the evidence lower bound, which is expressed as:

[0153]

[0154] Where ELBO represents the lower bound of evidence; Indicates the data distribution Take the expectation of all samples to ensure that the optimization of the model's objective function can reflect the overall performance of the dataset; represents the approximate posterior distribution q of the latent variable z φ The expected value calculated under (z|x,u); p θ (x|z) represents the distribution of observed data generated from the latent variable, parameterized by θ; q φ (∈|x,u) represents the probability distribution of ∈ given the observation data x and the conditional variable u; p ∈ () represents the prior distribution of the noise variable ∈; qφ (z|x,u) represents the approximate posterior distribution of the latent variable z; p θ (z|u) represents the prior distribution of the latent variable z;

[0155] and represents the KL divergence term, which measures the difference between the two distributions separated by ||; x represents high-dimensional medical image data; u represents the corresponding label of the image;

[0156] The model learns the causal structure by setting a prior on the representation generated by the structural causal model and computing the KL divergence, using the following loss function:

[0157]

[0158] in

[0159]

[0160] Among them, α and β are regularization hyperparameters, represents the expected log-likelihood of reconstruction ability, Indicates that the distribution of latent variables is constrained by prior knowledge; Represents the distance between features in the latent space by optimizing; Represents the overall loss function of the VAE framework; D KL (q φ (∈|x,u)‖p ∈ (∈)) represents q φ (∈|x,u) and q φ KL divergence of (∈|x,u), where q φ (∈|x,u) is a variational approximation that models the posterior distribution of the latent variable ∈, p ∈ (∈);D KL (q φ (z|x,u)||p θ (z|u)) represents the distribution q φ (z|x,u) and p θ KL divergence between (z|u), where q φ (z|x,u) is the approximate posterior distribution learned by the encoder network in VAE, which represents the probability distribution of the latent variable z given the observation data x and condition u; x a represents the anchor point sample; x p represents samples that are semantically close to the anchor samples, i.e., positive samples; x n It represents the samples that are semantically different from the anchor samples, that is, negative samples; m represents the margin; D(x a ,x p) represents the distance between the anchor sample and the positive sample in the feature space, which is generally calculated using the Euclidean distance; D(x a ,x n ) represents the distance between the anchor sample and the negative sample in the feature space;

[0161] The discriminator optimizes a binary classifier using the following loss function:

[0162]

[0163] Where, Represents the discriminator loss; Dis(x) represents the output of the discriminator, and its value range is 0≤Dis(x)≤1. Represents the data generated by the decoder;

[0164] The model training loss function Expressed as:

[0165]

[0166] An embodiment of the present application provides an electronic device, which may include a processor and a memory, wherein the processor and the memory can communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0167] The processor executes the computer-executable instructions stored in the memory, so that the processor implements the solutions in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0168] The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, among others. System buses can be divided into address buses, data buses, and control buses. For ease of illustration, the figure shows only one thick line, but this does not imply that there is only one bus or type of bus. Transceivers are used to enable communication between the database access device and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0169] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0170] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the technical solution of the method for extracting aggregable causal information from medical images in the above embodiment.

[0171] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the method for extracting aggregable causal information from medical images in the above embodiment.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0173] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected to implement the solution of this embodiment based on actual needs.

[0174] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more modules may be integrated into a single unit. The above-mentioned modules may be implemented in the form of hardware or hardware plus software functional units.

[0175] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0176] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0177] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0178] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0179] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0180] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0181] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for extracting aggregatable causal information from medical images, characterized in that: The method comprises: Constructing a causal representation learning framework; wherein the causal representation learning framework includes an encoder, a structural causal model, a decoder, and a discriminator, wherein the encoder is used to encode the input medical image into a low-dimensional exogenous variable, the structural causal model takes the low-dimensional exogenous variable as input and generates a causal representation based on the low-dimensional exogenous variable, the decoder is used to intervene and reconstruct the causal representation, and the discriminator is used for adversarial training; Establishing a model training loss function, training the causal representation learning framework based on the model training loss function, and extracting aggregatable causal information from medical images using the trained causal representation learning framework; The structural causal model is expressed as: Where z represents the causal representation, T represents the matrix transpose, f and h represent nonlinear functions, and g represents the parameterized graph attention network. represents low-dimensional exogenous variables, represents the domain of real numbers, d represents the number of causal attributes, A represents the prior causal graph, Represents the causal structure matrix; Gradient-based continuous update strategy optimizes graph attention network to obtain causal structure matrix The causal structure matrix Input into the structural causal model to generate a causal representation.

2. The method for extracting aggregatable causal information from medical images according to claim 1, characterized in that: Gradient-based continuous update strategy optimizes graph attention network to obtain causal structure matrix include: Take i and j as the causal structure matrix For any two nodes in , we learn the causal edge between i and j by multiplying the connection feature with the shared attention mechanism a(·) to generate an attention score e ij ; The attention score is expressed as: have been ij =a(W i ∈,W j ) Where, represents a shared weight matrix; For node i, only node j∈Pa is calculated i , where Pa i Represents the parent nodes and uses the softmax function to normalize them. The calculation process is as follows: Where, represents the normalized attention coefficient generated by the h-th attention head, and H represents the number of attention heads; Constructed by the following formula Where, represents the causal strength of row i and column j in the causal structure matrix, τ>0, is the temperature parameter that controls the outcome to be close to 0 or 1; g1 and g0 represent samples drawn independently from the Gumbel(0,1) distribution, and σ(·) represents the logistic sigmoid function; Iteratively train the graph attention network to obtain the causal structure matrix 3. The method for extracting aggregatable causal information from medical images according to claim 1, characterized in that: The causal structure matrix Input into the structural causal model to generate a causal representation, including: The encoder based on metric learning is used to generate the ideal causal representation of triple samples. The triple network fits the feature distribution in the latent space by measuring the distance between the anchor point and the positive / negative samples, thereby obtaining a causal representation that is not affected by distribution shift. The triple loss is expressed as: In the formula, (z a ,z p ,z n ) represents a triplet, including an anchor sample z a , a positive sample z p and a negative sample z n , max represents the maximum value function, D(z a ,z n ) represents the distance between the anchor sample and the negative sample, D(z a ,z p ) represents the distance between the anchor sample and the positive sample, D(z i ,z j )=||f(z i )-f(z j )||2 represents the Euclidean distance between two vectors, and m represents the margin.

4. The method for extracting aggregatable causal information from medical images according to claim 1, characterized in that: Intervene and reconstruct the causal representation through the following formula: Where, f -1 represents the inverse mapping of f after intervention, represents a set of intervention operations, represents the causal representation after the intervention, Represents the causal structure matrix after the intervention.

5. The method for extracting aggregatable causal information from medical images according to claim 1, characterized in that: The model training loss function is established by the following method; By learning an approximate variational distribution q of the true distribution p of the causal representation z, we can fit the actual posterior distribution; Get the training dataset and joint distribution Maximize the evidence lower bound, which is expressed as: Where ELBO represents the lower bound of evidence; Indicates the data distribution Take the expectation of all samples to ensure that the optimization of the model's objective function can reflect the overall performance of the dataset; represents the approximate posterior distribution q of the latent variable z φ The expected value calculated under (z|x,u); p θ (x|z) represents the distribution of observed data generated from the latent variable, parameterized by θ; q φ (∈|x,u) represents the probability distribution of ∈ given the observation data x and the conditional variable u; p ∈ (∈) represents the prior distribution of the noise variable ∈; q φ (z|x,u) represents the approximate posterior distribution of the latent variable z; p θ (z|u) represents the prior distribution of the latent variable z; and represents the KL divergence term, which measures the difference between the two distributions separated by ||; x represents high-dimensional medical image data; u represents the corresponding label of the image; The model learns the causal structure by setting a prior on the representation generated by the structural causal model and computing the KL divergence, using the following loss function: in Where α and β are regularization hyperparameters, represents the expected log-likelihood of reconstruction ability, Indicates that the distribution of latent variables is constrained by prior knowledge; Represents the distance between features in the latent space by optimizing; Represents the overall loss function of the VAE framework; D KL (q φ (∈|x,u)‖p ∈ (∈)) represents q φ (∈|x,u) and q φ KL divergence of (∈|x,u), where q φ (∈|x,u) is a variational approximation that models the posterior distribution of the latent variable ∈, p ∈ (∈) is the prior distribution of the exogenous variable ∈; D KL (q φ (z|x,u)||p θ (z|u)) represents the distribution q φ (z|x,u) and p θ KL divergence between (z|u), where q φ (z|x,u) is the approximate posterior distribution learned by the encoder network in VAE, which represents the probability distribution of the latent variable z given the observation data x and condition u; x a represents the anchor point sample; x p represents samples that are semantically close to the anchor samples, i.e., positive samples; x n It represents the samples that are semantically different from the anchor samples, that is, negative samples; m represents the margin; D(x a ,x p ) represents the distance between the anchor sample and the positive sample in the feature space, which is calculated using the Euclidean distance; D(x a ,x n ) represents the distance between the anchor sample and the negative sample in the feature space; The discriminator optimizes a binary classifier using the following loss function: Where, Represents the discriminator loss; Dis(x) represents the output of the discriminator, and its value range is 0≤Dis(x)≤1. Represents the data generated by the decoder; The model training loss function Expressed as:

6. A device for extracting aggregatable causal information from medical images, characterized in that: The device comprises: a framework construction module configured to construct a causal representation learning framework; wherein the causal representation learning framework includes an encoder, a structural causal model, a decoder, and a discriminator, wherein the encoder is used to encode an input medical image into a low-dimensional exogenous variable, the structural causal model takes the low-dimensional exogenous variable as input and generates a causal representation based on the low-dimensional exogenous variable, the decoder is used to intervene in and reconstruct the causal representation, and the discriminator is used for adversarial training; a framework training module configured to establish a model training loss function, train the causal representation learning framework based on the model training loss function, and extract aggregatable causal information from medical images using the trained causal representation learning framework; The structural causal model is expressed as: Where z represents the causal representation, T represents the matrix transpose, f and h represent nonlinear functions, and g represents the parameterized graph attention network. represents low-dimensional exogenous variables, represents the domain of real numbers, d represents the number of causal attributes, A represents the prior causal graph, Represents the causal structure matrix; Gradient-based continuous update strategy optimizes graph attention network to obtain causal structure matrix The causal structure matrix Input into the structural causal model to generate a causal representation.

7. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for extracting aggregatable causal information from medical images according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for extracting aggregatable causal information from medical images according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for extracting aggregatable causal information from a medical image according to any one of claims 1 to 5.

Images