Brain image generation method and system based on multi-level structure constraint, and medium

Abstract

The invention relates to a brain image generation method and system based on multilevel structure constraint, and a medium. The generation method comprises the following steps: preprocessing brain MRI image data; generating a high-fidelity cross-modal brain image through decoupling content features and attribute features by using a generative adversarial network framework of decoupling and comparative learning; pixel-level granularity discrimination: discriminating pixel-level features by using a brain memory library to enhance discrimination ability and optimize a memory space; and structure-level granularity discrimination: constraining different brain structures by dynamically adjusting a weight strategy so as to improve the learning effect of samples difficult to recognize. Compared with the prior art, the hierarchical granularity discrimination method provided by the invention aims to solve the problem of neglecting spatial coherence of structural hierarchical representation and anatomical consistency of content in a model training process by utilizing multi-level semantic information in a medical image.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and medical imaging, and in particular to a method, system and medium for generating brain images based on multi-level structural constraints. Background Art

[0002] As a non-invasive imaging technique, magnetic resonance imaging (MRI) has the advantages of high spatial resolution, good soft tissue contrast, and multiple imaging modalities. It is an important means of brain science research and clinical research on brain diseases. Among them, multimodal MRI data is very important for clinical diagnosis and decision-making. Multimodal MRI data provides comprehensive brain structure and function information by comprehensively utilizing different imaging sequences, such as T1-weighted, T2-weighted, diffusion-weighted imaging (DWI), and functional MRI (fMRI). This diverse data enables doctors to more accurately assess the nature, extent, and progression of brain lesions, thereby formulating more precise treatment plans. Given that obtaining multimodal MRI data is usually expensive, and in some situations, it becomes impractical to obtain data due to difficulties in patient cooperation or the requirements of medical data privacy protection, synthetic MRI data has become a practical alternative, providing valuable auxiliary information for clinical research and diagnosis.

[0003] The challenges faced by existing generative techniques are mainly: 1. Maintaining the integrity and consistency of anatomical structures. In medical image generation tasks, it is crucial to maintain the integrity and consistency of image structures. These tasks are of great significance to disease diagnosis, treatment planning, and medical research. The model needs to accurately reflect the rich semantic information in the image and ensure that the segmentation results can retain subtle anatomical structures, especially in the boundary areas. The diagnosis of many diseases relies on the observation of these tiny structures, such as the edge features of tumors, which may affect the judgment of malignancy. Therefore, the generative model must have a high degree of restoration to avoid losing detailed information that is critical to diagnosis. 2. Processing and separating coupled features in multimodal images. The challenge is that the model not only needs to extract meaningful information from complex high-dimensional data, but also needs to be able to identify and deconstruct the interdependent and entangled parts of this information. This process requires the model to have sophisticated representation learning capabilities and a deep insight into the complex interactions between different attributes in the input data. 3. Generating high-resolution and high-quality images is an important challenge. The details in medical images are very important, and any small distortion or artifacts may affect the diagnosis results. Therefore, the generative model must be able to avoid noise and artifacts while maintaining details and resolution. 4. Data scarcity and imbalance. Medical image data is usually scarce and unbalanced, especially for some rare diseases. This data scarcity and imbalance will affect the training effect of the model, resulting in poor performance of the model in practical applications. To this end, the model needs to be able to effectively utilize limited data and improve its generalization ability and robustness through new technologies.

[0004] In summary, medical image generation technology needs to comprehensively consider challenges such as structural integrity, multimodal feature separation, image quality, data scarcity, etc. Therefore, how to introduce more sophisticated representation learning mechanisms and multi-level structural constraints into the model to achieve more accurate and robust generation effects in different modalities and complex data environments has become an important technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a brain image generation method, system, and medium based on multi-level structural constraints in order to overcome the defects of the above-mentioned prior art. The present invention develops a medical image generation model that can maintain structural consistency and proposes a hierarchical granularity discrimination method, which aims to utilize the multi-level semantic information in medical images. A three-level discrimination granularity is adopted: first, pixel-level discrimination, which is realized through the brain memory library; second, structural-level discrimination, which constrains different brain structures through a strategy of dynamically adjusting weights to better learn difficult-to-recognize samples; and finally, global-level discrimination to ensure anatomical consistency throughout the conversion process.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A first aspect of the present invention provides a method for generating a brain image based on multi-level structural constraints, comprising the following steps:

[0008] S1: preprocessing brain MRI image data to improve data quality, wherein the preprocessing includes denoising, standardization, registration, and resampling;

[0009] S2: Using a generative adversarial network framework of decoupling and contrastive learning to generate high-fidelity cross-modal brain images by decoupling content features and attribute features, the generative adversarial network framework includes a content encoder, an attribute encoder, a generator, and a modality discriminator;

[0010] S3: Pixel-level granularity discrimination: Use the brain memory bank to discriminate pixel-level features to enhance discrimination ability and optimize memory space;

[0011] S4: Structural-level granularity discrimination: Constrain different brain structures by dynamically adjusting weight strategies to improve the learning effect of difficult-to-recognize samples;

[0012] S5: Ensure global anatomical consistency during the conversion process through a global-level granularity discrimination module.

[0013] Furthermore, in S2, by using the generative adversarial network framework of decoupling and contrastive learning, the specific process of generating high-fidelity cross-modal brain images includes:

[0014] A decoupled learning process is used to separate the intrinsic content features and variable attribute features in brain images;

[0015] Acting on the input brain image through a content encoder and an attribute encoder respectively, wherein the content encoder focuses on extracting stable content in the image to ensure that the stable content is preserved in the generated image, and the attribute encoder captures the mutable attributes of the modality;

[0016] The decoupled features are fed into the generator, which uses the decoupled features to generate high-fidelity cross-modal brain images.

[0017] The quality of the generated cross-modal brain images is controlled and optimized through a modality discriminator.

[0018] Furthermore, S3 specifically includes the following processes:

[0019] Using the brain memory library to extract features from each pixel of the cross-modal brain image generated in step S2, the features include anatomical structure and texture detail information in the image;

[0020] The generative adversarial network framework model queries the keys in the brain memory bank by forming query items through content features, so as to obtain enhanced features through the brain memory bank;

[0021] After extraction and query, the generative adversarial network framework model fuses the content features with the enhanced attribute features retrieved from the brain memory bank;

[0022] Operate on each pixel, calculate the relevance of each pixel with respect to different categories, and generate an affine score;

[0023] Calculate a weight for each update based on the similarity between the query term and the items stored in the brain memory bank;

[0024] Use the weights calculated above to update the keys and values ​​in the brain memory bank.

[0025] Furthermore, S4 specifically includes the following processes:

[0026] Integrate the features obtained from pixel-level discrimination in S3, initialize the features of different brain structures, and assign initial weights;

[0027] Use the annotation information in the training data to constrain the model to identify and process samples that are difficult to identify;

[0028] By comparing the real-time performance during the learning process, the weights are adjusted to optimize the model's sensitivity to structural changes and recognition accuracy;

[0029] According to the adjusted weights, the structural level discriminant loss is calculated, and the results of the loss function are used to update the model parameters. Through the iterative optimization process, the model's ability to recognize and generate complex structures in medical images is improved, ensuring the structural consistency and accuracy of the generated images.

[0030] Furthermore, S5 specifically includes the following processes:

[0031] Integrate the features obtained from the structural level in S4 to further extract global features, which include the overall shape of the image, the location of anatomical parts and their relationship information;

[0032] The model introduces consistency constraints by comparing the global features of the original image and the target image to ensure that the converted image is consistent with the original image in terms of overall structure;

[0033] Use a loss function to quantify and minimize the difference between global features;

[0034] The generator produces images with characteristics of the target modality, while the discriminator evaluates the authenticity of the generated images. Through adversarial training, the generator learns how to better simulate the global anatomical structure of the target modality, while the discriminator improves its ability to distinguish between real and generated images.

[0035] Furthermore, in S3, during the pixel-level granularity discrimination process, pixel-level features are stored and retrieved through the brain memory bank to improve the detail fidelity of the generated image. The retrieval process is expressed by the following formula:

[0036]

[0037] Among them, the sim(·,·) function is used to calculate the cosine similarity between two vectors. It is a content feature The query vector for the nth pixel in , is the enhanced representation for each pixel, and k t are the value and key stored in the tth slot in the brain memory bank respectively.

[0038] Furthermore, the process of updating the keys and values ​​in the brain memory bank is represented by the following formula:

[0039]

[0040] Among them, α p is a hyperparameter that controls the update rate, and k t are the value and key of slot t in the BMB, respectively; To update the weights;

[0041] The pixel-level contrast loss is:

[0042]

[0043] in, is a positive sample pair, which is a pixel from different modalities of the same individual. τ1 is the temperature scaling parameter.

[0044] Furthermore, the structure-level granularity determination process is as follows:

[0045] By imposing structural-level discrimination constraints on different brain structures, the weights of each structure are dynamically adjusted to improve the model's ability to recognize complex brain structures. The formula is as follows:

[0046]

[0047] in, For a specific structure i Content characteristics, Represents different structures, is the structure label, ⊙ represents the Hadamard product, (x, y) is the position coordinate, is a binary indicator function, is the positive sample set,

[0048] The hyperparameter α s Controls the degree of smoothness in the exponential function, and rank(·) is responsible for sorting the elements in the collection;

[0049] Ω' c =Ω c \{m i}, τ2 is the temperature scaling parameter.

[0050] A second aspect of the present invention provides a brain image generation system based on multi-level structural constraints, comprising:

[0051] MRI data preprocessing module: preprocess the brain MRI image data to improve the data quality. The preprocessing includes denoising, standardization, registration, and resampling. The generative adversarial network framework of decoupling and contrastive learning is used to decouple content features and attribute features to generate high-fidelity cross-modal brain images. The generative adversarial network framework includes a content encoder, an attribute encoder, a generator, and a modality discriminator.

[0052] Pixel-level granularity discrimination module: Pixel-level granularity discrimination: Use the brain memory bank to discriminate pixel-level features to enhance discrimination ability and optimize memory space;

[0053] Structural-level granularity discrimination module: constrains different brain structures by dynamically adjusting weight strategies to improve the learning effect of difficult-to-recognize samples;

[0054] A global-level granularity discrimination module ensures global anatomical consistency during the conversion process.

[0055] A third aspect of the present invention provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, is used to execute the above-mentioned brain image generation method based on multi-level structural constraints.

[0056] Compared with the prior art, the present invention has the following technical advantages:

[0057] 1) Improve structural fidelity: Unlike most existing methods, the method of the present invention solves the problem of ignoring the spatial coherence of structural hierarchical representation and the anatomical consistency of content during model training by exploring a new direction of generating high-fidelity images using hierarchical granularity discrimination. This solution to the structural protection challenge helps generate more accurate and realistic medical images.

[0058] 2) Completion of missing modalities: The method of the present invention can complete the missing modalities in the medical imaging data set while maintaining the structure, solving clinical challenges such as patients unable to fully cooperate, emergency situations requiring rapid decision-making, and incomplete historical data. This allows clinicians to obtain a comprehensive view of the patient's condition without the need for additional scans, which helps to improve future analysis and research.

[0059] 3) Responding to urgent and time-sensitive situations: In emergency situations or when surgical intervention is urgently needed, the method of the present invention can quickly generate synthetic equivalents of missing imaging modalities, which may be the key to saving lives. Such a rapid response can speed up the speed at which experts make informed decisions and increase the chances of successful treatment.

[0060] 4) Enhanced historical databases: By completing missing modalities in historical medical databases, the method of the present invention enriches these resources, making them more valuable for research and training. This enhanced dataset facilitates retrospective research and improves the training of machine learning models, providing a more comprehensive dataset.

[0061] 5) As a data augmentation strategy: The method of the present invention is also a powerful form of data augmentation, especially suitable for unbalanced or skewed clinical datasets. By generating high-quality, multimodal disease cases from diverse source data, the model can enrich the dataset. This enhancement supports the development of more general and robust diagnostic algorithms, especially in machine learning applications, where model performance is often impaired by uneven distribution of clinical center data. By providing synthetically generated, realistic examples, these problems can be alleviated, by providing a more balanced representation of rare and common conditions, thereby improving the accuracy and reliability of predictive models in different clinical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a structural framework diagram of a method for generating brain magnetic resonance images based on multi-level structural constraints (HiFi-Syn) according to an embodiment of the present invention;

[0063] Figure 2 It is a flowchart of implementing BMB for assisting pixel-level discrimination in a method for generating brain magnetic resonance images based on multi-level structural constraints according to an embodiment of the present invention;

[0064] Figure 3 is a schematic diagram of the HiFi-Syn structure of an embodiment of the present invention;

[0065] Figure 4 This is a diagram of the generation result of HiFi-Syn according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Component models, material names, connection structures, control methods, algorithms and other features not clearly described in this technical solution are all considered to be common technical features disclosed in the prior art.

[0067] Example 1

[0068] The conversion model in this embodiment includes two content encoders Two attribute encoders Two generators And two modality discriminators For the input image m i ,m j In the first round of modal conversion, content features are extracted through content encoder and attribute encoder respectively. and attribute characteristics Exchange their attribute features to generate the transformed image pair m i→j ,m j→i In the second round of style transfer, features are extracted from the synthesized image Exchange the attribute features again and get the reconstructed image

[0069]

[0070] This embodiment introduces a pixel-level granularity discrimination (PGD) mechanism to maintain structural consistency during image conversion. This method is different from the traditional instance-level discrimination method, which treats the entire image as a single entity and is limited in capturing local features. In contrast, the pixel-level granularity method adopted by this method is to convert the content encoder E c Each pixel in the encoded feature is regarded as an independent entity, thus achieving a more refined and semantically distinguishable feature expression. In addition, this method constructs a BMB (Brain Memory Bank), such as Figure 2 As shown in Figure 2, the BMB contains T slots to track these feature expressions. Each slot stores a C-dimensional vector to record the attribute representation of a specific domain, and the key is used as an index to retrieve the related memory item.

[0071] In the training phase, the model first uses content features Construct query terms to query the key of BMB to obtain enhanced features through BMB Then, the model transforms the content features With enhanced attribute features Perform fusion to prepare for subsequent domain conversion.

[0072] In particular, unlike those memory techniques that only rely on the content features of the original image, our method enhances the performance of the model by generating structural consistency, integrating the content features of the original image and the transformed image. This method effectively maintains the consistency of pixel-level structure in image conversion, ensuring the structural accuracy and reliability of the converted image.

[0073] In dealing with content features When , the dimension of its feature map is H×W×C, where H, W, and C represent the height, width, and number of channels, respectively. In order to perform detailed pixel-level feature operations, this method defines a set of pixel-level query items Where N = H × W, each query item is a 1×1×C vector representing a single pixel in the content feature map. In order to calculate the relevance of each pixel relative to different categories, this method introduces a cosine similarity-based calculation method, which is the query item of the nth pixel and each key k. t Generate affine scores

[0074]

[0075] Among them, the sim(·,·) function is used to calculate the cosine similarity between two vectors, and Content Features The query vector of the nth pixel in . Through this method, the method can assign an affine score to each pixel for each category, thus providing a solid foundation for the model in the subsequent pixel-level transformation and feature enhancement steps. This pixel-level processing method enables the model to maintain and convey structural information more meticulously when performing image transformation, especially when processing medical images with complex anatomical structures, ensuring a high degree of structural consistency and image authenticity.

[0076] To obtain pixel-level enhanced representation The storage module will look for the value b stored for the key close to t , and weighted. The process can be defined as follows:

[0077]

[0078] After operating on each pixel, the final enhanced representation at the feature map level can be obtained

[0079] To update the items in the memory, this method adopts a strategy to update each item in the memory. and k t Calculate update weights Next, update the key-value pairs in the BMB using the following method:

[0080]

[0081] Among them, α p It is used to control the update rate and plays a role in dynamically balancing the fusion of historical information and new information. Especially in the task of medical image generation, momentum smoothing is particularly important for maintaining the stability of the memory library during the update process.

[0082] Similarly, by calculating It is possible to update attribute store items from other domains. In this process, the above formula is responsible for updating the keys and values ​​in the memory. Update weights Based on the similarity between the query and the stored items, the content association between the original image and the converted image is considered. This content-based update mechanism maintains the consistency of structural information during the conversion process and ensures the dynamic and adaptable nature of the BMB. In this way, the model can effectively manage and update key features, thereby improving the effect and accuracy of the conversion. The dynamic update mechanism ensures that the memory library can continuously adapt to changes in the input data, thereby more accurately capturing and retaining key structural features and attributes during the conversion process. Therefore, the model proposed in this method can process complex medical image data more efficiently and maintain structural consistency and accuracy in image-to-image conversion.

[0083] In order to promote the learning of more refined feature disentangled representation, this method designs a comparison mechanism for each individual. i and its transformed image m i→j sharing the same anatomical structure, the two should remain close in the latent content space (forming a positive pair, ), and images of different individuals (even if they belong to the same modality) should be far away from each other. Based on this assumption, this method adopts a pixel-level contrast loss, which is formulated as follows:

[0084]

[0085] in τ1 is the temperature scaling parameter.

[0086] This method further introduces the structure-level granularity discrimination (SGD) mechanism, with the aim of better calibrating and maintaining the consistency of structural semantics during the image conversion process. By utilizing labeled brain structure training data, SGD learns decoupled features at the structural level to capture the key changes between different anatomical structures of the brain. The experimental results of this method show that compared with the existing technology, SGD, as one of the core components, can bring more efficient and stable learning results at the pixel level and the global level. This method uses a contrastive learning mechanism in SGD, which aims to narrow the semantic feature distance within the same structural region while maintaining the distinction between different structural features. To achieve this goal, this method reuses decoupled content features. and the corresponding structure tags The labels are downsampled to fit the size of H and W. Assume that the total number of structures is L, and use Represents different structures, according to The mode M can be calculated as follows: i Specific structures in i Content Features

[0087]

[0088] Among them, ⊙ represents the Hadamard product (element-wise product), and (x, y) is the position coordinate. is a binary indicator function used to indicate whether each pixel belongs to structure s i Similarly, the converted Belongs to mode M i→j Although the appearance determined by different attribute encodings may be different, the same individual The consistency of the anatomical content should be maintained to form a positive sample set Negative sample set Contains other structures and content features of different individuals. In addition, in order to encourage the network to focus more on samples that are difficult to identify, this method designs a weighting scheme based on content similarity to automatically adjust the weight of each sample. Specifically, for a set of structures s of an individual i , deformation diagram under different modes The weighted calculation adopts a ranking-based method by analyzing the deformation graph between each pair of positive samples. The deformation map quantifies the difference in the shape of the structure before and after the image is generated, which helps to identify samples with significant structural changes. The specific weighting steps are as follows: 1. Calculate the deformation map: For each structure s i , calculate the deformation map between it and other positive samples Capture the degree of structural shape distortion between different samples. 2. Ranking based on deformation graph: Rank each positive sample according to the degree of deformation. Sample pairs with larger deformation are considered as difficult examples and need special attention. 3. Weighted calculation: Assign weights to each pair of positive samples according to the ranking. Sample pairs with higher weights account for a larger proportion in the total loss function, prompting the model to learn these difficult examples first.

[0089] Through the above steps, the goal of SGD is to enhance the model's sensitivity to structural changes, thereby improving its performance in processing complex structures in medical images. SGD promotes the model's accurate recognition of subtle differences and unique changes between structures by focusing on difficult examples, which is extremely critical for maintaining structural consistency during image conversion. This process can be described by the following formula:

[0090]

[0091] Among them, the hyperparameter α s controls the degree of smoothing in the exponential function, while rank(·) is responsible for sorting the elements in the collection. The value range is from 0 to 1. Sample pairs with larger deformation are assigned higher weights to guide the network to correct the mismatch caused by image generation, thereby maintaining the integrity of the brain structure during modality conversion. Therefore, the overall expression of the structure-level granularity discrimination loss can be defined as follows:

[0092]

[0093] in, Ω' c =Ω c \{m i}. τ2 is the temperature scaling parameter.

[0094] By assigning higher weights to pairs of samples with greater deformation, this method forces the model to pay more attention to the structural distortions that may be encountered during the image conversion process. This approach not only improves the learning efficiency of the model, but also enhances the structural consistency of the converted image, which is extremely important for ensuring the diagnostic value of medical image conversion.

[0095] Similar to structural granularity discrimination, the global anatomical content of the same individual should remain consistent even if the image modalities are different. To achieve this goal, this method introduces a global-level granularity discrimination (GGD) mechanism. And use a contrastive learning loss similar to SGD, the difference is that the weights in the formula It is uniformly set to 1. To avoid redundancy, the specific definition of the formula is not given here.

[0096] By introducing GGD, the model obtains a more comprehensive feature representation capability, which can effectively capture and maintain the anatomical structure of the image at the global level, which is particularly important for applications that require image-to-image conversion in the absence of detailed annotation information during testing. Combining pixel-level, structural-level, and global-level discrimination mechanisms, the model can more deeply understand and retain key features in the image at multiple levels, thereby improving the overall quality of the converted image.

[0097] In summary, global granularity discrimination is a key supplement to the model. It effectively improves the performance of the model in the absence of structural annotations and plays a key role in maintaining the global consistency of the image. This mechanism not only enhances the adaptability of the model, but also provides a more comprehensive constraint for handling complex medical image conversion tasks.

[0098] The bidirectional reconstruction loss promotes decoupling by collaboratively updating the gradient propagation between modules. We represent the cross-loop reconstruction and self-reconstruction as and The present invention defines these losses as follows:

[0099]

[0100] and For adversarial learning, the former ensures that the image modality cannot be inferred from the content code alone, which helps to decouple attribute information from content features. The latter ensures that the synthesized image has the same modality-specific features as the real image. These losses can be calculated as follows:

[0101]

[0102] The overall loss function of this model framework is as follows:

[0103]

[0104] Where λ1 to λ7 are hyperparameters.

[0105] This embodiment also proposes a brain image generation system based on multi-level structural constraints, including:

[0106] MRI data preprocessing module: preprocess the brain MRI image data to improve the data quality. The preprocessing includes denoising, standardization, registration, and resampling. The generative adversarial network framework of decoupling and contrastive learning is used to decouple content features and attribute features to generate high-fidelity cross-modal brain images. The generative adversarial network framework includes a content encoder, an attribute encoder, a generator, and a modality discriminator.

[0107] Pixel-level granularity discrimination module: Pixel-level granularity discrimination: Use the brain memory bank to discriminate pixel-level features to enhance discrimination ability and optimize memory space;

[0108] Structural-level granularity discrimination module: constrains different brain structures by dynamically adjusting weight strategies to improve the learning effect of difficult-to-recognize samples;

[0109] A global-level granularity discrimination module ensures global anatomical consistency during the conversion process.

[0110] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional units.

[0112] The present embodiment also proposes a computer-readable storage medium, wherein the storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned brain image generation method based on multi-level structural constraints. The storage medium can be an electronic medium, a magnetic medium, an optical medium, an electromagnetic medium, an infrared medium or a semiconductor system or a propagation medium. The storage medium can also include a semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a hard disk and an optical disk. The optical disk can include a compact disk-read-only memory (CD-ROM), a compact disk-read-write (CD-RW) and a DVD.

[0113] Application Example 1

[0114] like Figure 1As shown, the brain magnetic resonance imaging generation method based on multi-level structural constraints provided by the present invention includes a data enhancement module, a generation-contrast learning framework encoder, a pixel-level granularity discrimination module, a structure-level granularity discrimination module, a global-level granularity discrimination module and a generation-contrast learning framework decoder;

[0115] The overall framework is as follows Figure 3 As shown in Figure 2, the encoder of the generative-contrastive learning framework consists of an attribute encoder and a content encoder, and the decoder of the generative-contrastive learning framework consists of a generator. a In the example, data propagation starts from an initial input m j Initially, the image is processed through a 7x7 convolution kernel with 64 output channels, which is mainly used to capture basic visual features. Subsequently, the input data passes through three consecutive 4x4 convolution layers with the number of channels expanded to 128 and 256 respectively. These layers gradually extract more complex features while increasing the depth and complexity of the features. The last 4x4 convolution layer keeps the number of channels at 256 to ensure the consistency of deep features. After these convolution layers, the features are downsampled through an average pooling layer, which is usually used to reduce the feature dimension and improve the generalization ability of the network. The entire process effectively extracts and compresses highly complex features from the input through gradually deepening convolution and pooling layers, providing a powerful feature representation for subsequent analysis or classification tasks.

[0116] Figure 3 The content encoder E shown in c is used to process the input image m i The complex neural network architecture consists of multiple convolutional layers. The initial input is processed through a 7x7 convolutional layer with 64 channels, which is mainly used to capture the basic visual features of the image. Subsequently, the features are further refined through four consecutive 3x3 convolutional layers, where the number of channels is gradually increased to 128 and finally 256. The purpose of this series of convolutional layers is to deeply explore the content features in the image and increase the model's ability to capture details. The last two 3x3 convolutional layers continue to deepen the dimension of the features, keeping the number of channels at 256, further enriching the model's expression of the image content. This content encoder E c The design is sophisticated and focuses on extracting detailed content features from the input image, providing a solid feature foundation for subsequent image processing tasks.

[0117] In the proposed architecture, the discriminator D mi It is designed to process the input through a series of convolutional layers to assess its authenticity. mi m iAs input, it is first processed through a convolutional layer with a 3x3 convolution kernel and 64 output channels. The output then passes through multiple consecutive layers with 3x3 convolution kernels, with the number of output channels increasing from 128 to 256, 512, and 1024. This is followed by two layers of convolution operations with 3x3 convolution kernels and 1024 output channels. Finally, a 1x1 convolutional layer generates a single output for binary classification, indicating whether the input is real or generated.

[0118] The generator module shown in the figure is used to convert the input features into images of the target modality. The generator receives the content encoder E c and attribute encoder E a The features of are combined together through the concatenation operation as the input of the generator. The generator contains multiple 3x3 convolutional layers, each with 256 output channels, and the continuity and consistency of the information flow are maintained through residual blocks and jump connections. After processing multiple convolutional layers, the generator converts the high-dimensional feature map back to the image space through a 3x3 convolutional transpose layer to achieve modal conversion and generate the target modal image m i→j The structural design of the generator ensures that the input features can retain important structural information during the conversion process while generating high-quality target images.

[0119] Content Identifier D c Receive z c As input, it first passes through a convolutional layer using three 7x7 convolutional kernels and 256 output channels, and then passes through a 4x4 convolutional kernel layer. The final layer is a 1x1 convolutional layer that produces a single output for binary classification. c It aims to distinguish the extracted content representations between the two modalities and promote disentangled representation learning.

[0120] The brain memory bank (BMB) is divided into two different parts. The items belonging to the pixel-level granularity discrimination (PGD) module only focus on knowledge at the structural level, while the remaining items contain global-level information, attributed to the global-level granularity discrimination (GGD). According to experience, the two parts are symmetrical in size, and each part contains the same number of items. According to this symmetrical configuration, the number of memory items in the present invention is set to 38. Taking the BraTS dataset as an example, the 38 memory items are divided into 2 for background, 2 for CSF, 3 for GM, 3 for WM, 9 for three brain lesions, and 19 for global structure.

[0121] Compared with MGUIT, the present invention adopts structure-level annotations instead of bounding boxes to locate pixel positions, thereby providing a more accurate structural description. In addition, the present invention introduces items that record global values ​​in the brain memory bank (BMB), recognizing that when no annotations are available during the test phase, the network is expected to learn contextual differences from the input rather than relying solely on labels, thereby ensuring the robustness and generalization ability of the network.

[0122] Hyperparameter λ 1-4 The settings are the same as DRIT++. According to experience, set α p =0.01,α s =0.5, λ5, λ7=1, and λ6=2.

[0123] This image-to-image translation network was trained in an unpaired manner, and their performance was evaluated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). By focusing on the segmented structures of interest, a more reliable and comprehensive measure of structural fidelity can be achieved during image synthesis.

[0124] In a specific case where the system of this embodiment is applied, Figure 4 As shown, the proposed framework is compared with other unsupervised image-to-image translation methods, including CycleGAN, AttentionGAN, MUNIT, DRIT++, UVCGAN, StarGAN-v2, and the recent state-of-the-art instance-level I2I methods MGUIT and InstaFormer. In addition, four translation methods with medical structure constraints are introduced to show the effectiveness of the proposed method. These methods include GC-CycleGAN, SC-CycleGAN, COSMOS, and RegGAN.

[0125] HiFi-Syn shows a significant lead over all methods. This performance improvement may be attributed to the complex semantic information contained in pathological tumor images, which requires a multi-level learning strategy, which may make it difficult for other methods to identify different structures. Instance-level I2I networks such as InstaFormer and MGUIT achieved suboptimal results. Unlike those methods that focus on global content, these methods capture more detailed local information by considering instance-level representations. However, the limitations of instance fineness of instance-level learning methods pose a challenge, making it difficult for these methods to efficiently handle complex anatomical structures. This limitation becomes particularly evident when a deep understanding of complex structures and the details therein is required, resulting in these methods being in a suboptimal position in the evaluation results. Through visual inspection of the generated results, it was found that InstaFormer and other baseline methods introduced different degrees of distortion in the tumor region, which may affect their application in downstream tasks.

[0126] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A brain image generation method based on multi-level structural constraints, characterized in that: The following steps are involved: S1: preprocessing brain MRI image data to improve data quality, wherein the preprocessing includes denoising, standardization, registration, and resampling; S2: Using a generative adversarial network framework of decoupling and contrastive learning to generate high-fidelity cross-modal brain images by decoupling content features and attribute features, the generative adversarial network framework includes a content encoder, an attribute encoder, a generator, and a modality discriminator; S3: Pixel-level granularity discrimination: Use the brain memory bank to discriminate pixel-level features to enhance discrimination ability and optimize memory space; S4: Structural-level granularity discrimination: Constrain different brain structures by dynamically adjusting weight strategies to improve the learning effect of difficult-to-recognize samples; S5: Ensure global anatomical consistency during the conversion process through a global-level granularity discrimination module.

2. A brain image generation method based on multi-level structural constraints according to claim 1, characterized in that: In S2, the generation of high-fidelity cross-modal brain images is achieved by using a generative adversarial network framework of decoupling and contrastive learning. The specific process includes: A decoupled learning process is used to separate the intrinsic content features and variable attribute features in brain images; Acting on the input brain image through a content encoder and an attribute encoder respectively, wherein the content encoder focuses on extracting stable content in the image to ensure that the stable content is preserved in the generated image, and the attribute encoder captures the mutable attributes of the modality; The decoupled features are fed into the generator, which uses the decoupled features to generate high-fidelity cross-modal brain images. The quality of the generated cross-modal brain images is controlled and optimized through a modality discriminator.

3. The method for generating brain images based on multi-level structural constraints according to claim 1, characterized in that: In S3, the specific process includes the following: Using the brain memory library to extract features from each pixel of the cross-modal brain image generated in step S2, the features include anatomical structure and texture detail information in the image; The generative adversarial network framework model queries the keys in the brain memory bank by forming query items through content features, so as to obtain enhanced features through the brain memory bank; After extraction and query, the generative adversarial network framework model fuses the content features with the enhanced attribute features retrieved from the brain memory bank; Operate on each pixel, calculate the relevance of each pixel with respect to different categories, and generate an affine score; Calculate a weight for each update based on the similarity between the query term and the items stored in the brain memory bank; Use the weights calculated above to update the keys and values ​​in the brain memory bank.

4. The method for generating brain images based on multi-level structural constraints according to claim 1, characterized in that: S4, specific The process includes: Integrate the features obtained from pixel-level discrimination in S3, initialize the features of different brain structures, and assign initial weights; Use the annotation information in the training data to constrain the model to identify and process samples that are difficult to identify; By comparing the real-time performance during the learning process, the weights are adjusted to optimize the model's sensitivity to structural changes and recognition accuracy; According to the adjusted weights, the structural level discriminant loss is calculated, and the results of the loss function are used to update the model parameters. Through the iterative optimization process, the model's ability to recognize and generate complex structures in medical images is improved, ensuring the structural consistency and accuracy of the generated images.

5. The method for generating brain images based on multi-level structural constraints according to claim 1, characterized in that: S5 specifically includes the following processes: Integrate the features obtained from the structural level in S4 to further extract global features, which include the overall shape of the image, the location of anatomical parts and their relationship information; The model introduces consistency constraints by comparing the global features of the original image and the target image to ensure that the converted image is consistent with the original image in terms of overall structure; Use a loss function to quantify and minimize the difference between global features; The generator produces images with characteristics of the target modality, while the discriminator evaluates the authenticity of the generated images. Through adversarial training, the generator learns how to better simulate the global anatomical structure of the target modality, while the discriminator improves its ability to distinguish between real and generated images.

6. The method for generating brain images based on multi-level structural constraints according to claim 1, characterized in that: In S3, during the pixel-level granularity discrimination process, pixel-level features are stored and retrieved through the brain memory bank to improve the detail fidelity of the generated image. The retrieval process is expressed by the following formula: Among them, the sim(·,·) function is used to calculate the cosine similarity between two vectors. It is a content feature The query vector for the nth pixel in , is the enhanced representation for each pixel, and k t are the value and key stored in the tth slot in the brain memory bank respectively.

7. The method for generating brain images based on multi-level structural constraints according to claim 2, characterized in that: The process of updating keys and values ​​in the brain memory bank is represented by the following formula: Among them, α p is a hyperparameter that controls the update rate, and k t are the value and key of slot t in the BMB, respectively; To update the weights; The pixel-level contrast loss is: in, is a positive sample pair, which is a pixel from different modalities of the same individual. τ1 is the temperature scaling parameter.

8. The method for generating brain images based on multi-level structural constraints according to claim 2, characterized in that: The structure-level particle size determination process is as follows: By imposing structural-level discrimination constraints on different brain structures, the weights of each structure are dynamically adjusted to improve the model's ability to recognize complex brain structures. The formula is as follows: in, For a specific structure i Content characteristics, Represents different structures, is the structure label, ⊙ represents the Hadamard product, (x, y) is the position coordinate, is a binary indicator function, is the positive sample set, The hyperparameter α s Controls the degree of smoothness in the exponential function, and rank(·) is responsible for sorting the elements in the collection; Ω' c =Ω c \{m i }, τ2 is the temperature scaling parameter.

9. A brain image generation system based on multi-level structural constraints, characterized in that: include: MRI data preprocessing module: preprocess the brain MRI image data to improve the data quality. The preprocessing includes denoising, standardization, registration, and resampling. The generative adversarial network framework of decoupling and contrastive learning is used to decouple content features and attribute features to generate high-fidelity cross-modal brain images. The generative adversarial network framework includes a content encoder, an attribute encoder, a generator, and a modality discriminator. Pixel-level granularity discrimination module: Pixel-level granularity discrimination: Use the brain memory bank to discriminate pixel-level features to enhance discrimination ability and optimize memory space; Structural-level granularity discrimination module: constrains different brain structures by dynamically adjusting weight strategies to improve the learning effect of difficult-to-recognize samples; A global-level granularity discrimination module ensures global anatomical consistency during the conversion process.

10. A storage medium containing computer executable instructions, characterized in that: The storage medium of the computer executable instructions is used to execute the brain image generation method based on multi-level structural constraints as described in any one of claims 1 to 8 when executed by a computer processor.

Images