A multi-task adversarial learning-based temporal brain image calibration method

Abstract

A temporal brain image calibration method based on multi-task adversarial learning, belonging to the field of medical imaging technology, includes the following steps: preprocessing pre-acquired temporal brain images to remove irrelevant information; dividing the brain image phenotypic calibration network into two sub-tasks (local detail reconstruction and global structural transformation), learning them in different ways, and employing a discriminator in an adversarial learning strategy to constrain the generation of better calibrated brain images. This invention is highly valuable for improving the correlation performance between brain images and genes, enabling effective observation and understanding of the human brain state.

Description

Technical Field

[0001] This invention belongs to the field of medical image technology, specifically a temporal brain image calibration method based on multi-task adversarial learning. Background Technology

[0002] As an important research area for analyzing changes in complex brain states, gene-brain image association analysis is a typical multidisciplinary project combining machine learning, neuroimaging, and gene sequencing technologies. It requires the participation of researchers from various fields, including mathematics, computer science, and biology. Gene-brain image association analysis can uncover the correlations and evolutionary patterns between neuroimaging and genetic data and brain states, identify biomarkers related to brain states, and thus provide data-dependent explanations for the developmental mechanisms of complex brain states. It also provides fundamental support for understanding how genes influence the neural structure and function of the brain. Clinically, time-series brain imaging data can reflect and reveal changes in brain structure and function over time, including changes in whole-brain volume, gray matter volume, ventricular size, frontotemporal cortical thickness and surface area, and hippocampal and amygdala volume. By establishing the association between brain images and genes at different times, time-series brain image-gene association analysis effectively reveals the mechanisms by which genes influence changes in brain structure or function during disease progression, enabling a more comprehensive understanding of the nature of brain diseases and individual differences.

[0003] Given that patients receive imaging scans from different devices or doctors at different time points, temporal brain image data is prone to heterogeneity, i.e., heterogeneity between different phenotypes. This heterogeneity stems not only from inherent differences in image acquisition, processing, and reconstruction techniques of the imaging equipment itself, but is further exacerbated by the specific parameters selected by the personnel during scanning, the maintenance and calibration status of the equipment, and potential operational differences. Temporal heterogeneity in brain images poses numerous challenges to the training and prediction of subsequent association analysis models, such as decreased model stability and training difficulties. Therefore, this invention proposes a temporal brain image calibration method based on multi-task adversarial learning. The brain image phenotype calibration network is divided into two sub-tasks (local detail reconstruction and global structural transformation), each learned using different methods. A discriminator from the adversarial learning strategy is used to constrain the generation of better calibrated brain images. This invention calibrates brain image phenotypes at different times into the same structural space, reducing the heterogeneity of temporal brain image data. This is highly valuable for improving the correlation performance between brain images and genes, enabling effective observation and understanding of the human brain's state.

[0004] The differences compared to existing technologies are as follows:

[0005] Technical Comparison with Patent CN112308833A "A One-Shot Brain Image Segmentation Method Based on Cyclic Consistency Correlation"

[0006] I. Patent CN112308833A addresses brain image segmentation tasks by employing the LT-NET network model, which improves segmentation efficiency through a combination of forward and backward mapping and supervised loss. This invention, however, targets temporal brain image calibration tasks. It divides the brain image phenotypic calibration network into two sub-tasks: local detail reconstruction and global structural transformation. An adversarial learning strategy is used to generate better calibration images, thereby improving the correlation between brain images and genes.

[0007] Second, patent CN112308833A focuses on improving the performance of unidirectional correlation learning to optimize image segmentation. This invention, however, optimizes local and global features through multi-task learning, placing greater emphasis on the calibration of temporal brain images, thus enabling more effective observation and understanding of the human brain's state.

[0008] Technical Comparison with Patent CN112348786B "A One-Shot Brain Image Segmentation Method Based on Bidirectional Correlation"

[0009] I. Patent CN112348786B addresses brain image segmentation by constructing an image transformation model to learn bidirectional mappings. It uses backward mapping to constrain forward mapping to improve the accuracy of forward mapping. In contrast, this invention addresses temporal brain image calibration by employing multi-task adversarial learning. It learns both local detail reconstruction and global structural transformations separately, and uses a discriminator to constrain and generate a better calibration image.

[0010] II. Patent CN112348786B optimizes segmentation results through bidirectional mapping and mutual constraints. This invention, however, enhances the capture of dynamic features in temporal brain images through multi-task partitioning, which is more conducive to improving the correlation between brain images and genes, thereby providing a deeper understanding of the human brain state. Summary of the Invention

[0011] To address the aforementioned technical problems, this invention provides a temporal brain image calibration method based on multi-task adversarial learning. This method can calibrate brain image phenotypes at different times into the same structural space, reducing the heterogeneity of temporal brain image data. This is highly valuable for improving the correlation between brain images and genes, and can effectively observe and understand the state of the human brain.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] A temporal brain image calibration method based on multi-task adversarial learning is described below:

[0014] (1) Preprocess the pre-acquired temporal brain images to remove irrelevant information;

[0015] (2) Constrain the consistency between the generated calibrated brain image and the template brain image at the pixel level, and construct a reconstruction loss term. ;

[0016] (3) Construct a feature transformation loss term by regressing brain image phenotypic features from time n to template brain image phenotypic features. ;

[0017] (4) Constrain the network to generate more realistic calibrated brain image phenotypes and construct adversarial loss terms. ;

[0018] (5) The brain image phenotypic calibration network is divided into two sub-tasks: local detail reconstruction and global structural transformation. The two sub-tasks are learned in different ways, and the discriminator in the adversarial learning strategy is used to constrain the generation of better calibrated brain images.

[0019] As a further improvement of the present invention, the reconstruction loss term in step (2) for:

[0020] ;

[0021] Where, assuming and Representing the calibration of brain images and Time-lapse brain images, all of which are [size missing]. , here , These represent the length and width of the brain image phenotype, respectively; It is an identity network and is encoded by an encoder. and decoder The input and output components are based on the same template brain image phenotype, which prompts the network to focus on learning the encoding / decoding of detailed information during training, without having to learn structural transformations.

[0022] As a further improvement of the present invention, the feature transformation loss term in step (3) for:

[0023] ;

[0024] in, and These represent the length and width of the feature brain image, respectively; This represents a feature transformer.

[0025] As a further improvement of the present invention, the adversarial loss term in step (4) for:

[0026] ;

[0027] in, It is a transform network and is composed of an encoder Feature Transformer and decoder The components, inputs, and outputs are respectively: Time-based brain image phenotypes and corresponding calibrated brain image phenotypes, This represents the discriminator in an adversarial network.

[0028] As a further improvement of the present invention, step (5) is implemented as follows:

[0029] (51) Based on steps (2), (3) and (4), assume and Representing the calibration of brain images and Momentary brain images and These represent the generator and discriminator in an adversarial network architecture, respectively. In the proposed calibration method, the generator comprises three sub-modules: an encoder, an encoder, and a discriminator. Feature Transformer and decoder In this process, the encoder is responsible for encoding the detailed information of the phenotypic characteristics of the input brain image, while the feature encoder is responsible for... The feature representation of the brain image phenotype at any given time in the latent space is mapped to the feature space of the corresponding calibrated brain image phenotype, and the decoder is responsible for reconstructing the calibrated brain image based on the feature representation.

[0030] The goal of this method is to learn a generator. It can convert brain image phenotypes at any time sequence. Unified to calibrated brain images In the space where it is located, that is The essence of this invention is to divide the brain image phenotypic calibration network into two sub-tasks (one is local detail reconstruction, and the other is global structural transformation), learn them in different ways, and use the discriminator in the adversarial learning strategy to constrain the generation of better calibrated brain images.

[0031] In summary, the loss function of the proposed missing feature completion method includes multiple loss terms, namely:

[0032] ;

[0033] in, and It is a weighting parameter that balances the importance of each component;

[0034] (52) Train two generators simultaneously in a multi-task manner and ,in It is an identity network and is encoded by an encoder. and decoder The input and output components share the same template brain image phenotype, which encourages the network to focus on learning the encoding / decoding of detailed information during training, without needing to learn structural transformations. It is a transform network and is composed of an encoder Feature Transformer and decoder The components, inputs, and outputs are respectively: Time-based brain image phenotypes and corresponding calibrated brain image phenotypes, and encoder in and decoder It shares parameters to enable the sharing of encoding / decoding details; the network constrains the feature transformer through a feature regression loss term. Learning from The mapping from the time-bound brain image phenotypic feature space to the template brain image phenotypic feature space is implemented without considering how to encode detailed information. A discriminator is also introduced to distinguish between the calibrated brain image phenotypic generated by the generator and the template brain image phenotypic, thereby constraining the generator to produce a better calibrated brain image phenotypic.

[0035] Beneficial effects: Compared with existing technologies,

[0036] This invention divides the brain image phenotypic calibration network into two sub-tasks (local detail reconstruction and global structural transformation), learning them in different ways and employing a discriminator from an adversarial learning strategy to constrain the generation of better calibrated brain images. This invention calibrates brain image phenotypes from different time points into the same structural space, reducing the heterogeneity of temporal brain image data. This is highly valuable for improving the correlation between brain images and genes, enabling effective observation and understanding of the human brain's state. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0038] The technical solution of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this patent. All non-innovative embodiments based on this embodiment by other researchers in the art are within the protection scope of this patent.

[0039] This invention proposes a temporal brain image calibration method based on multi-task adversarial learning. The brain image phenotypic calibration network is divided into two sub-tasks (local detail reconstruction and global structural transformation), which are learned using different methods. A discriminator from the adversarial learning strategy is employed to constrain the generation of better calibrated brain images. Figure 1 As shown, the specific steps include the following:

[0040] Step 1: Preprocess the pre-acquired temporal brain images to remove irrelevant information.

[0041] Step 2: Constrain the consistency between the generated calibrated brain image and the template brain image at the pixel level, and construct the reconstruction loss term. :

[0042]

[0043] Where, assuming and Representing the calibration of brain images and Time-lapse brain images, all of which are [size missing]. , here , These represent the length and width of the brain image phenotype, respectively; (by encoder) and decoder The network is an identity network, meaning that its input and output parts are the same template brain image phenotype. This prompts the network to focus on learning the encoding / decoding of detailed information during training, without having to learn structural transformations.

[0044] Step 3: Construct a feature transformation loss term by calculating the regression loss from brain image phenotypic features at time n to template brain image phenotypic features. :

[0045]

[0046] in, and These represent the length and width of the feature brain image, respectively; This represents a feature transformer.

[0047] Step 4: Constrain the network to generate more realistic calibrated brain image phenotypes and construct adversarial loss terms. ;

[0048]

[0049] in, (by encoder) Feature Transformer and decoder (The structure) is a transform network, with inputs and outputs being... Time-based brain image phenotypes and corresponding calibrated brain image phenotypes. This represents the discriminator in an adversarial network.

[0050] Step 5: Divide the brain image phenotypic calibration network into two sub-tasks (local detail reconstruction and global structural transformation), learn them using different methods, and employ a discriminator from an adversarial learning strategy to constrain the generation of better calibrated brain images. Specifically, this includes the following steps:

[0051] 1) Based on steps (2), (3), and (4), assume that and Representing the calibration of brain images and Momentary brain images and These represent the generator and discriminator in an adversarial network architecture, respectively. In the proposed calibration method, the generator comprises three sub-modules: an encoder (…). ), feature transformer ( ) and decoder ( The encoder is responsible for encoding the detailed information of the phenotypic characteristics of the input brain image, while the feature encoder is responsible for... The method maps the feature representations of the brain image phenotype in the latent space to the feature space of the corresponding calibration brain image phenotype, and the decoder is responsible for reconstructing the calibration brain image based on the feature representations. The goal of this method is to learn a generator. It can convert brain image phenotypes at any time sequence. Unified to calibrated brain images In the space where it is located, that is This invention essentially divides the brain image phenotypic calibration network into two sub-tasks (local detail reconstruction and global structural transformation), learning them in different ways, and employing a discriminator from an adversarial learning strategy to constrain the generation of better calibrated brain images. In summary, the loss function of the proposed missing feature completion method includes multiple loss terms, namely:

[0052]

[0053] in, and It is a weighting parameter that balances the importance of each component.

[0054] 2) Train two generators simultaneously using a multi-task approach. and .in, (by encoder) and decoder The network is an identity network, meaning its input and output are based on the same template brain image phenotype. This allows the network to focus on learning the encoding / decoding of detailed information during training, without needing to learn structural transformations. (by encoder) Feature Transformer and decoder (The structure) is a transform network, with inputs and outputs being... Time-based brain image phenotypes and corresponding calibrated brain image phenotypes. and encoder in and decoder It shares parameters to enable the sharing of encoding / decoding details. Furthermore, the network constrains the feature transformer through a feature regression loss term. Learning from The model maps the phenotypic feature space of the brain image at any given time to the phenotypic feature space of the template brain image without considering how to encode detailed information. Furthermore, a discriminator is introduced to distinguish between the calibrated brain image phenotypes generated by the generator and the template brain image phenotypes, thereby constraining the generator to produce better calibrated brain image phenotypes.

[0055] This invention uses Python software to test the above method on a time-series brain image dataset. To verify the effectiveness of the method, this embodiment evaluates its performance on the real ADNI dataset. Time-series brain images, including 114 samples, were used. This invention is valuable for improving the correlation between brain images and genes, enabling effective observation and understanding of the human brain's state.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A temporal brain image calibration method based on multi-task adversarial learning, characterized in that: The specific process is as follows: (1) Preprocess the pre-acquired temporal brain images to remove irrelevant information; (2) Constrain the consistency between the generated calibrated brain image and the template brain image at the pixel level, and construct a reconstruction loss term. ; The reconstruction loss term in step (2) for: ； Where, assuming and These represent the calibration of brain images and Time-lapse brain images, all of which are [size missing]. , here , These represent the length and width of the brain image phenotype, respectively; It is an identity network and is encoded by an encoder. and decoder The components, input and output parts, are the same template brain image phenotype; (3) Construct a feature transformation loss term by regressing brain image phenotypic features from time n to template brain image phenotypic features. ; The feature transformation loss term in step (3) for: ； in, and These represent the length and width of the feature brain image, respectively; Represents a feature transformer; (4) Constrain the network to generate more realistic calibrated brain image phenotypes and construct adversarial loss terms. ; The anti-loss term in step (4) for: ； in, It is a transform network and is composed of an encoder Feature Transformer and decoder The components, inputs, and outputs are respectively: Time-based brain image phenotypes and corresponding calibrated brain image phenotypes, This represents a discriminator in an adversarial network. (5) The brain image phenotypic calibration network is divided into two sub-tasks: local detail reconstruction and global structural transformation. The two sub-tasks are learned in different ways, and the discriminator in the adversarial learning strategy is used to constrain the generation of better calibrated brain images. The implementation process of step (5) is as follows: (51) Based on steps (2), (3) and (4), assume and These represent the calibration of brain images and Momentary brain images and These represent the generator and discriminator in an adversarial network architecture, respectively. In the proposed calibration method, the generator comprises three sub-modules: an encoder, an encoder, and a discriminator. Feature Transformer and decoder In this process, the encoder is responsible for encoding the detailed information of the phenotypic characteristics of the input brain image, while the feature encoder is responsible for... The feature representations of brain image phenotypes at each time step are mapped from the latent space to the feature space of the corresponding calibrated brain image phenotype. The decoder is responsible for reconstructing the calibrated brain image based on the feature representations. In summary, the loss function of the proposed missing feature imputation method contains multiple loss terms, namely: ； in, and It is a weighting parameter that balances the importance of each item; (52) Train two generators simultaneously in a multi-task manner and ,in It is an identity network and is encoded by an encoder. and decoder The composition, input, and output parts are the same template brain image phenotype, while It is a transform network and is composed of an encoder Feature Transformer and decoder The components, inputs, and outputs are respectively: Time-based brain image phenotypes and corresponding calibrated brain image phenotypes, and encoder in and decoder It shares parameters to enable the sharing of encoding / decoding details; the network constrains the feature transformer through a feature regression loss term. Learning from The mapping from the time-bound brain image phenotypic feature space to the template brain image phenotypic feature space is implemented without considering how to encode detailed information. A discriminator is also introduced to distinguish between the calibrated brain image phenotypic generated by the generator and the template brain image phenotypic, thereby constraining the generator to produce a better calibrated brain image phenotypic.

Images