Time sequence brain image calibration method based on multi-task adversarial learning

Abstract

A time sequence brain image calibration method based on multi-task adversarial learning belongs to the technical field of medical images, and comprises the following steps: pre-processing a pre-acquired time sequence brain image to remove irrelevant information; a brain image phenotype calibration network is divided into two sub-tasks (local detail reconstruction and global structure transformation), learning is carried out in different modes, and a discriminator in an adversarial learning strategy is adopted to restrain and generate a better calibration brain image. The method is very valuable for improving the brain image and gene association performance, and the state of the human brain can be effectively observed and understood.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical images, and particularly relates to a time sequence brain image calibration method based on multi-task adversarial learning. BACKGROUND

[0002] As an important research content of complex brain state change analysis, gene and brain image association analysis is a typical multi-disciplinary cross subject combining the fields of machine learning, neuroimaging and gene sequencing technology, and needs the participation of researchers in different fields, including mathematics, computers and biology, etc. The gene and brain image association analysis can mine the correlation and evolution law between neuroimaging and gene data and the brain state, find the biomarkers related to the brain state, and thus provide data-dependent explanation for the development mechanism of the complex brain state, and provide basic support for realizing how genes affect the neural structure and function of the brain. In the clinic, the time sequence brain image data can reflect and reveal the changes of the brain structure and function over time, including the changes of the whole brain volume, the gray matter volume, the ventricle size, the frontal and temporal lobe cortex thickness and surface area, the hippocampus and amygdala volume, etc. The time sequence brain image and gene association analysis effectively reveals the influence mechanism of genes on the changes of brain structure or function in the disease process by establishing the correlation between brain images at different times and genes, and can more comprehensively understand the nature and individual differences of brain diseases.

[0003] Considering that the patients receive imaging scans from different devices or doctors at different time points, the time sequence brain image data is prone to present heterogeneity in time sequence, i.e. heterogeneity between phenotypes. This heterogeneity not only comes from the inherent differences of the imaging devices themselves in image acquisition, processing and reconstruction technology, but is further intensified due to the specific parameters selected by the personnel at the time of scanning, the maintenance and calibration state of the device, and possible operation differences. The time sequence heterogeneity of the time sequence brain image will bring many challenges to the training and prediction of the subsequent association analysis model, such as the decrease of model stability and training difficulty. Therefore, the application proposes a time sequence brain image calibration method based on multi-task adversarial learning, which divides the brain image phenotype calibration network into two sub-tasks (one is local detail reconstruction, and the other is global structure transformation), learns them in different ways, and uses the discriminator in the adversarial learning strategy to constrain the generation of better calibrated brain images. The application calibrates the brain images at different times to the same structure space, reduces the heterogeneity of the time sequence brain image data, which is very valuable for improving the performance of brain image and gene association, and can effectively observe and understand the state of the human brain.

[0004] The differences compared with the prior art are as follows:

[0005] Technical comparison with patent CN112308833A “one-shot brain image segmentation method based on cyclic consistent correlation”

[0006] I. Patent CN112308833A aims to improve the segmentation efficiency of brain image by using the LT-NET network model, and improving the segmentation efficiency through forward mapping and backward mapping with supervised loss. However, the present invention focuses on the calibration task of time-series brain images, and divides the brain image phenotype calibration network into two sub-tasks: local detail reconstruction and global structure transformation. It uses an adversarial learning strategy to generate more optimal calibration images to improve the performance of brain image and gene association.

[0007] II. Patent CN112308833A focuses on improving the performance of one-way correlation learning to optimize image segmentation. However, the present invention optimizes local and global features through multi-task learning, and focuses more on the calibration of time-series brain images, which can more effectively observe and understand the state of the human brain.

[0008] Comparison with patent CN112348786B "one-shot brain image segmentation method based on bidirectional correlation"

[0009] I. Patent CN112348786B aims to segment brain images and construct an image transformation model to learn bidirectional mapping. It constrains forward mapping through backward mapping to improve the accuracy of forward mapping. However, the present invention focuses on the calibration of time-series brain images and uses multi-task adversarial learning to learn local detail reconstruction and global structure transformation respectively. It uses a discriminator to generate more optimal calibration images.

[0010] II. Patent CN112348786B optimizes segmentation results through bidirectional mapping constraints. However, the present invention enhances the capture of dynamic features of time-series brain images through multi-task division, which is more conducive to improving the association performance of brain images and genes, thereby deepening the understanding of the state of the human brain. SUMMARY

[0011] To solve the above technical problems, the present invention is a time-series brain image calibration method based on multi-task adversarial learning, which can calibrate brain images at different time points to the same structural space, reducing the heterogeneity of time-series brain image data. This is very valuable for improving the performance of brain image and gene association, and can effectively observe and understand the state of the human brain.

[0012] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0013] A time-series brain image calibration method based on multi-task adversarial learning, the specific process is as follows:

[0014] (1) Pre-process the pre-acquired time-series brain images to remove irrelevant information;

[0015] (2) From the pixel level, constrain the consistency between the generated calibration brain image and the template brain image, and construct a reconstruction loss term ;

[0016] (3) Regression loss from the brain image phenotype feature at time n to the template brain image phenotype feature, to construct a feature transformation loss term ;

[0017] (4) Restrict the network to generate more realistic calibration brain image phenotype, to construct an adversarial loss term ;

[0018] (5) The brain image phenotype calibration network is divided into two sub-tasks, two sub-tasks are local detail reconstruction and global structure transformation respectively, different ways are used to learn, and the discriminator in the adversarial learning strategy is used to constrain to generate better calibration brain image.

[0019] As a further improvement of the application, the reconstruction loss term in step (2) is :

[0020] ;

[0021] Wherein, assuming and respectively represent the calibration brain image and time brain image, the size is , here , respectively represent the length and width of the brain image phenotype; It is an identity network and consists of an encoder And the decoder , the input and output part is the same template brain image phenotype, which prompts the network to focus on learning the encoding / decoding of detail information in the training process, without learning the structure transformation.

[0022] As a further improvement of the application, the feature transformation loss term in step (3) is :

[0023] ;

[0024] Wherein, and respectively represent the length and width of the feature brain image; Indicates the feature transformer.

[0025] As a further improvement of the application, the adversarial loss term in step (4) is :

[0026] ;

[0027] Wherein, is a transformation network and consists of an encoder , a feature transformer and a decoder , the input and output are respectively the brain image phenotype at time t and the corresponding calibrated brain image phenotype, denotes the discriminator in the adversarial network.

[0028] As a further improvement of the application, the step (5) is implemented as follows:

[0029] (51) According to steps (2), (3) and (4), assuming and represent the calibrated brain image and the brain image at time t respectively, and represent the generator and the discriminator in the adversarial network architecture respectively, in the proposed calibration method, the generator includes three sub-modules, an encoder , a feature transformer and a decoder , wherein the encoder is responsible for encoding the detailed information of the input brain image phenotype, the feature encoding is responsible for mapping the feature expression of the brain image phenotype at time t in the hidden space to the feature space of the corresponding calibrated brain image phenotype, and the decoder is responsible for reconstructing the calibrated brain image according to the feature expression; The purpose of this method is to learn a generator

[0030] that can unify any time series brain image phenotype to the space where the calibrated brain image is located, that is . The essence of the application is to divide the brain image phenotype calibration network into two sub-tasks (one is local detail reconstruction, and the other is global structure transformation), learn them in different ways, and use the discriminator in the adversarial learning strategy to constrain the generation of better calibrated brain images. In summary, the loss function of the proposed missing feature completion method contains multiple loss terms, that is:

[0031]

[0032] ;

[0033] wherein, and are weight parameters balancing the importance of each term;

[0034] (52) Train the two generators and simultaneously in a multi-task manner, wherein is an identity network and consists of an encoder​ and decoder consists of the same template brain image phenotype, which prompts the network to focus on learning the encoding / decoding of detailed information in the training process without learning the structural transformation, is a transformation network and is composed of an encoder , a feature transformer and a decoder , and the input and output are respectively the brain image phenotype at the moment and the corresponding calibration brain image phenotype, and the encoder and the decoder inshare parameters so as to share the ability to encode / decode detailed information, and the network constrains the feature transformer to learn the mapping from the feature space of the brain image phenotype at the moment to the feature space of the template brain image phenotype without considering how to encode detailed information, and a discriminator is introduced to distinguish the calibration brain image phenotype generated by the generator from the template brain image phenotype, so as to constrain the generator to generate better calibration brain image. Advantages: compared with the prior art,

[0035] The present application divides the brain image phenotype calibration network into two sub-tasks (one is local detail reconstruction, and the other is global structure transformation), learns them in different ways respectively, and uses the discriminator in the adversarial learning strategy to constrain the generation of better calibration brain images. The present application calibrates brain image phenotypes at different moments to the same structural space, reduces the heterogeneity of the time-series brain image data, which is very valuable for improving the performance of brain image and gene association, and can effectively observe and understand the state of the human brain.

[0036] BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of the method of the present application. DETAILED DESCRIPTION

[0038] The technical solutions of the application will be further described in detail below with reference to the drawings. The described embodiments are only a part of the embodiments involved in the patent. All non-innovative embodiments of other researchers in the field on the embodiments belong to the protection scope of the patent.

[0039] The present application proposes a time-series brain image calibration method based on multi-task adversarial learning, which divides the brain image phenotype calibration network into two sub-tasks (one is local detail reconstruction, and the other is global structure transformation), learns them in different ways respectively, and uses the discriminator in the adversarial learning strategy to constrain the generation of better calibration brain images. For example,Figure 1 As shown, specifically includes the following steps:

[0040] Step 1: Pre-process the pre-acquired time-series brain images to remove irrelevant information.

[0041] Step 2: From the pixel level, constrain the generated calibration brain image to maintain consistency with the template brain image, and construct the reconstruction loss term :

[0042]

[0043] where, assuming and represent the calibration brain image and time brain image, both of which have a size of Here, , represent the length and width of the brain image phenotype, respectively; (consisting of an encoder and a decoder ) is an identity network, that is, its input and output parts are the same template brain image phenotype, which prompts the network to focus on learning the encoding / decoding of detailed information during the training process, without learning the structural transformation.

[0044] Step 3: Regression loss from the n-time brain image phenotype feature to the template brain image phenotype feature, construct feature transformation loss term :

[0045]

[0046] where, and represent the length and width of the feature brain image, respectively; represents the feature transformer.

[0047] Step 4: Constrain the network to generate more realistic calibration brain image phenotypes, construct the adversarial loss term ;

[0048]

[0049] where, (by encoder , feature transformer , and decoder ) is a transformation network, and the input and output are time brain image phenotype and the corresponding calibration brain image phenotype. represents the discriminator in the adversarial network.

[0050] Step 5: Divide the brain image phenotypic calibration network into two sub-tasks (local detail reconstruction and global structural transformation), learn them using different methods, and employ a discriminator from an adversarial learning strategy to constrain the generation of better calibrated brain images. Specifically, this includes the following steps:

[0051] 1) Based on steps (2), (3), and (4), assume and Representing the calibration of brain images and Momentary brain images and These represent the generator and discriminator in an adversarial network architecture, respectively. In the proposed calibration method, the generator comprises three sub-modules: an encoder (…). ), feature transformer ( ) and decoder ( The encoder is responsible for encoding the detailed information of the phenotypic characteristics of the input brain image, while the feature encoder is responsible for... The method maps the feature representations of the brain image phenotype in the latent space to the feature space of the corresponding calibration brain image phenotype, and the decoder is responsible for reconstructing the calibration brain image based on the feature representations. The goal of this method is to learn a generator. It can convert brain image phenotypes at any time sequence. Unified to calibrated brain images In the space where it is located, that is This invention essentially divides the brain image phenotypic calibration network into two sub-tasks (local detail reconstruction and global structural transformation), learning them in different ways, and employing a discriminator from an adversarial learning strategy to constrain the generation of better calibrated brain images. In summary, the loss function of the proposed missing feature completion method includes multiple loss terms, namely:

[0052]

[0053] in, and It is a weighting parameter that balances the importance of each component.

[0054] 2) Train two generators simultaneously using a multi-task approach. and .in, (by encoder) and decoder The network is an identity network, meaning its input and output are based on the same template brain image phenotype. This allows the network to focus on learning the encoding / decoding of detailed information during training, without needing to learn structural transformations. (by encoder) Feature Transformer and decoder (The structure) is a transform network, with inputs and outputs being... Time-based brain image phenotypes and corresponding calibrated brain image phenotypes. and encoder in and decoder It shares parameters to enable the sharing of encoding / decoding details. Furthermore, the network constrains the feature transformer through a feature regression loss term. Learning from The model maps the phenotypic feature space of the brain image at any given time to the phenotypic feature space of the template brain image without considering how to encode detailed information. Furthermore, a discriminator is introduced to distinguish between the calibrated brain image phenotypes generated by the generator and the template brain image phenotypes, thereby constraining the generator to produce better calibrated brain image phenotypes.

[0055] This invention uses Python software to test the above method on a time-series brain image dataset. To verify the effectiveness of the method, this embodiment evaluates its performance on the real ADNI dataset. Time-series brain images, including 114 samples, were used. This invention is valuable for improving the correlation between brain images and genes, enabling effective observation and understanding of the human brain's state.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A method for calibrating time-series brain images based on multi-task adversarial learning, characterized in that: The specific process is as follows: (1) preprocessing the pre-acquired time sequence brain image, removing irrelevant information; (2) From the pixel level constraint generated calibration brain image and template brain image to keep consistency, construct reconstruction loss term ; (3) Regression loss from the brain image phenotype features at time n to the template brain image phenotype features, to construct a feature transformation loss term ; (4) Constrained network generates more realistic calibrated brain image phenotypes, builds adversarial loss term ; (5) the brain image phenotype calibration network is divided into two subtasks, two subtasks are local detail reconstruction and global structure transformation respectively, different ways are used to learn respectively, and a discriminator in an adversarial learning strategy is used to constrain to generate a better calibrated brain image. 2.The multi-task adversarial learning based time series brain image alignment method of claim 1, wherein: The step (2) the reconstruction loss term is: ； where the assumption and denote the calibrated brain image and the brain image at time , respectively, both of which have the size , represent the length and width of the brain image phenotype, respectively, is used to denote the Frobenius norm; is an identity network and consists of an encoder and a decoder , the input and output parts are the same template brain image phenotype. 3.The multi-task adversarial learning based method for calibrating time-series brain images according to claim 2, characterized in that: The step (3) the feature transformation loss term is: ； wherein, and denote the length and width of the characteristic brain image, respectively; denotes a characteristic transformer. 4.The method of claim 3, wherein: The step (4) the confrontation loss term is: ； wherein, is a transformation network and consists of an encoder , a feature transformer , and a decoder with input and output respectively, denotes a discriminator in the adversarial network.

5. The multi-task adversarial learning-based timing brain image calibration method according to claim 4, characterized in that: The step (5) realizes the process as follows: (51) Based on steps (2), (3) and (4), assume and Representing the calibration of brain images and Momentary brain images and These represent the generator and discriminator in an adversarial network architecture, respectively. In the proposed calibration method, the generator comprises three sub-modules: an encoder, an encoder, and a discriminator. Feature Transformer and decoder In this process, the encoder is responsible for encoding the detailed information of the phenotypic characteristics of the input brain image, while the feature encoder is responsible for... The feature representations of brain image phenotypes at each time step are mapped from the latent space to the feature space of the corresponding calibrated brain image phenotype. The decoder is responsible for reconstructing the calibrated brain image based on the feature representations. In summary, the loss function of the proposed missing feature imputation method contains multiple loss terms, namely: ； wherein, and are weight parameters balancing the importance of each term; (52) Train two generators simultaneously in a multi-task manner and ,in It is an identity network and is encoded by an encoder. and decoder The composition, input, and output parts are the same template brain image phenotype, while It is a transform network and is composed of an encoder Feature Transformer and decoder The components, inputs, and outputs are respectively: Time-based brain image phenotypes and corresponding calibrated brain image phenotypes, and encoder in and decoder It shares parameters to enable the sharing of encoding / decoding details; the network constrains the feature transformer through a feature regression loss term. Learning from The mapping from the time-bound brain image phenotypic feature space to the template brain image phenotypic feature space is implemented without considering how to encode detailed information. A discriminator is also introduced to distinguish between the calibrated brain image phenotypic generated by the generator and the template brain image phenotypic, thereby constraining the generator to produce a better calibrated brain image phenotypic.

Images