A medical image cross-domain retrieval method and system based on a generative adversarial network

Abstract

The application belongs to the technical field of medical image cross-domain retrieval, and provides a medical image cross-domain retrieval method and system based on a generative adversarial network, which comprises the following steps: acquiring MRI images and CT images; performing unsupervised cross-domain mapping of the acquired images based on a bidirectional generative adversarial network, and calculating an adversarial loss function of the cross-domain mapping; calculating a gradient consistency loss function and a feature contrast loss function based on a multilevel cyclic consistency constraint mechanism; adaptively and dynamically adjusting the obtained loss functions to obtain a medical image cross-domain retrieval loss function, training a medical image cross-domain retrieval model, performing similarity retrieval based on the trained medical image cross-domain retrieval model, and completing medical image cross-domain retrieval based on the generative adversarial network. The application is free from the dependence on paired MRI / CT image samples, realizes high-quality image domain conversion through set-level supervision, and realizes feasible cross-domain retrieval in a real non-paired medical image environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image cross-domain retrieval, and particularly relates to a medical image cross-domain retrieval method and system based on a generative adversarial network. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the rapid development of medical imaging technology, accurate medical image analysis and diagnosis have become key elements for improving treatment effectiveness and patient care. Medical image retrieval (MIR) includes image processing, management and integration, and identifies clinically relevant image data from a large data set according to specific query requirements.

[0004] Medical cross-domain image retrieval (MCDMIR) is a generalization of traditional image retrieval, aiming to input a certain medical image (such as computed tomography CT) and retrieve relevant image data of other domains (such as magnetic resonance imaging MRI) from a medical data set. MCDMIR plays an indispensable role in multiple medical application fields: in terms of diagnostic support, it provides similar historical cases for clinicians, thereby significantly improving diagnostic accuracy; for medical students and residents, MCDMIR provides visually similar cases, serving as an important education and training tool. Medical image retrieval is considered a key means to effectively utilize massive medical data and promote the development of medical and research fields. The core obstacle faced is the "visual domain gap", which manifests as significant differences in color, texture, brightness, gradient and gray level features of images from different imaging carriers, spectrums, mechanisms or lighting conditions; especially in the medical imaging field, due to the huge differences in contrast, texture and noise level between different imaging fields (such as CT and magnetic resonance imaging MRI), the challenge of cross-domain adaptation is particularly prominent.

[0005] Currently, in the process of medical image cross-domain retrieval, low-level features (such as texture, shape and color) of the image can be extracted and retrieved, that is, cross-domain matching is performed using a manually designed feature descriptor such as SIFT and HOG, or domain conversion is realized through gray histogram alignment; however, there is an inherent difference between the automatically extracted low-level visual features and the high-level human semantic understanding or clinical concept, and the uniqueness and complexity of medical images make the manual feature method not effective when directly applied to medical images. Or, relying on paired cross-domain data (such as MRI-CT image pairs of the same patient), shared features are extracted through convolutional neural network (CNN) or twin network; more and more high-level features are learned directly from the input data to realize high-level understanding of objects, scenes and semantics; however, such methods also have certain limitations, and the main challenge of deep learning models, especially supervised learning methods, is the dependence on large-scale labeled data sets, which is particularly scarce in the field of medical images; the labeling process of medical images is extremely time-consuming and tedious.

[0006] Therefore, the prior art has the following problems in medical image cross-domain retrieval (such as MRI to CT): manual features cannot capture the deep semantic association between different domains (such as the difference between the soft tissue contrast of MRI and the density signal of CT), which results in a certain gap between the semantic features, and further results in low retrieval accuracy; supervised learning methods require a large number of strictly paired cross-domain data, and the paired data are strongly dependent, but in the clinic, it is often difficult to obtain or align multi-domain images of the same patient; traditional domain conversion methods (such as gray mapping) will destroy the anatomical structure (such as the blurring of the tumor edge), distort the local structure, and affect the reliability of diagnosis.

[0007] The essential reason for the above problems is:

[0008] (1) Nonlinear mapping between domains, that is, the imaging principles of different medical image domains (such as MRI / CT) are very different, and simple linear transformation (such as gray normalization) cannot model the complex relationship.

[0009] (2) Training dilemma without paired data, that is, traditional deep learning requires pixel-level supervision signals, while hospitals usually only have independent single-domain image libraries (such as MRI library and CT library), and lack of paired labeling.

[0010] (3) Conflict between structure preservation and style conversion, that is, existing methods are difficult to simultaneously realize domain style conversion (such as MRI→CT appearance adaptation) and key anatomical structure preservation (such as lesion position invariance).

[0011] (4) Imbalance and gradient conflict problems easily occurred in multi-loss collaborative training. SUMMARY

[0012] To solve the above problems, the application provides a medical image cross-domain retrieval method and system based on a generative adversarial network, which breaks away from the dependence on paired MRI / CT image samples and realizes high-quality image domain conversion through set level supervision; the MRI and CT domains are bidirectionally modeled through a double generator-discriminator structure, and the generator is driven to approach the target domain distribution in combination with an adversarial loss, so that the image semantic migration is successfully completed under the condition of no paired data, the threshold for clinical data preparation is greatly reduced, and feasible cross-domain retrieval in a real non-paired medical image environment is realized.

[0013] According to some embodiments, the first aspect of the application provides a medical image cross-domain retrieval method based on a generative adversarial network, which adopts the following technical scheme:

[0014] A medical image cross-domain retrieval method based on a generative adversarial network comprises:

[0015] MRI images and CT images are acquired;

[0016] The acquired MRI images and CT images are unsupervisedly cross-domain mapped based on a bidirectional generative adversarial network, and an adversarial loss function of the cross-domain mapping is calculated;

[0017] In the unsupervised cross-domain mapping process, the cross-domain mapping is optimized in two dimensions of structural precision and semantic consistency based on a multi-level cyclic consistency constraint mechanism, and a gradient consistency loss function and a feature contrast loss function are calculated;

[0018] The weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function and feature contrast loss function are adaptively and dynamically adjusted to obtain a medical image cross-domain retrieval loss function;

[0019] The training of a medical image cross-domain retrieval model is completed according to the obtained medical image cross-domain retrieval loss function, the similarity retrieval is performed based on the trained medical image cross-domain retrieval model, and the medical image cross-domain retrieval based on the generative adversarial network is completed.

[0020] As a further technical limitation, the bidirectional generative adversarial network is used to realize cross-domain image conversion and true or false image discrimination, and comprises a first generator, a first discriminator, a second generator and a second discriminator; the image distribution of the target domain is obtained based on the adversarial training of the bidirectional generative adversarial network without relying on paired image samples.

[0021] Further, the first generator receives the acquired MRI images to generate pseudo CT images, the second discriminator distinguishes the acquired CT images from the generated pseudo CT images, and the adversarial loss function in the direction of the MRI images to CT images is ; wherein, represents the image samples of the CT image domain. denotes a CT image; denotes a second discriminator for judging whether an input image is a real CT image; denotes an image sample in an MRI image domain; denotes an expected value, i.e., a mean value of a sample distribution; denotes a first generator from an MRI image to a CT image; denotes a predicted log probability of a discriminator for an input image being "real"; denotes a predicted log probability of a discriminator for a generated image being "fake".

[0022] Further, the second generator receives the acquired CT image, generates a pseudo MRI image, the first discriminator distinguishes the acquired MRI image and the generated pseudo MRI image, and the adversarial loss function in the CT image to MRI image direction is ; wherein, denotes a second generator from a CT image to an MRI image; denotes an MRI image; denotes a first discriminator for judging whether an input image is a real MRI image; the adversarial loss function of the cross-domain mapping includes an adversarial loss function in the MRI image to CT image direction and an adversarial loss function in the CT image to MRI image direction; that is .

[0023] As a further technical limitation, the multi-level cyclic consistency constraint includes a gradient consistency constraint, which adopts a structural gradient preservation strategy of a multi-directional Sobel kernel to extract gradient information of an input image in different directions, strengthens high-frequency structures of image edge contours, normalizes the extracted gradient information in different directions, and obtains a gradient consistency loss function of structural preservation according to the normalized gradient information .

[0024] As a further technical limitation, the multi-level cyclic consistency constraint also includes a semantic contrast constraint, which adopts a shared Transformer encoder, extracts semantic features of MRI images and CT images through a multi-layer attention mechanism, aggregates the extracted semantic features, and obtains a feature contrast loss function .

[0025] As a further technical limitation, the medical image cross-domain retrieval loss function is a weighted sum of the cross-domain mapping adversarial loss function, the gradient consistency loss function, and the feature contrast loss function, and the weight adopts a dynamic weight.

[0026] Further, the dynamic weight is a product of a training stage scheduling factor and a loss dynamic response factor; the training stage scheduling factor is used for controlling a change trend of the loss weight with training progress; and the loss dynamic response factor is a weight dynamically adjusted according to stability of the loss.

[0027] As a further technical limitation, after the training of the medical image cross-domain retrieval model is completed, the features of the MRI image and the CT image are extracted by using the shared encoder, the query image is input into the shared encoder to obtain a semantic embedding vector, the similarity retrieval of the query image is performed in the target field image features, and the medical image cross-domain retrieval is completed.

[0028] A medical image cross-domain retrieval system based on a generative adversarial network comprises:

[0029] An acquisition module configured to acquire MRI images and CT images;

[0030] A mapping module configured to perform unsupervised cross-domain mapping on the acquired MRI images and CT images based on a bidirectional generative adversarial network, and calculate an adversarial loss function of the cross-domain mapping;

[0031] An optimization module configured to, in the unsupervised cross-domain mapping process, based on a multi-level cyclic consistency constraint mechanism, simultaneously optimize the cross-domain mapping in two dimensions of structural accuracy and semantic consistency, and calculate a gradient consistency loss function and a feature contrast loss function;

[0032] An adjustment module configured to adaptively and dynamically adjust weights of the obtained adversarial loss function of the cross-domain mapping, the gradient consistency loss function and the feature contrast loss function, and obtain a medical image cross-domain retrieval loss function;

[0033] A retrieval module configured to complete training of a medical image cross-domain retrieval model according to the obtained medical image cross-domain retrieval loss function, perform similarity retrieval based on the trained medical image cross-domain retrieval model, and complete medical image cross-domain retrieval based on the generative adversarial network.

[0034] Compared with the prior art, the present application has the following beneficial effects:

[0035] The present application gets rid of the dependence on paired MRI / CT image samples, and realizes high-quality image domain conversion through set level supervision; the present application successfully completes image semantic migration under the condition of no paired data by bidirectional modeling of the MRI and CT domains through a double generator-discriminator structure, combining the adversarial loss to drive the generator to approximate the target domain distribution, greatly reducing the threshold of clinical data preparation, and realizing feasible cross-domain retrieval in a real non-paired medical image environment. BRIEF DESCRIPTION OF DRAWINGS

[0036] The drawings constituting a part of the specification of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments and descriptions of this embodiment are used to explain this embodiment and do not constitute an improper limitation on this embodiment.

[0037] Figure 1 Flowchart of the cross-domain retrieval method for medical images based on generative adversarial networks in Example 1 of the present invention;

[0038] Figure 2 This is an architecture diagram of the medical image cross-domain retrieval method based on generative adversarial network in Example 1 of the present invention;

[0039] Figure 3 This is a flowchart of cross-domain retrieval in Example 1 of the present invention;

[0040] Figure 4 This is a structural block diagram of the medical image cross-domain retrieval system based on generative adversarial networks in Example 2 of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0044] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0045] Example 1

[0046] Embodiment 1 of the present invention introduces a cross-domain retrieval method for medical images based on a generative adversarial network.

[0047] like Figure 1 A cross-domain medical image retrieval method based on a generative adversarial network is shown, comprising:

[0048] Acquire MRI images and CT images;

[0049] The acquired MRI image and CT image are unsupervised cross-domain mapped based on a bidirectional generative adversarial network, and an adversarial loss function of cross-domain mapping is calculated.

[0050] In the unsupervised cross-domain mapping process, based on a multi-level cyclic consistency constraint mechanism, the cross-domain mapping is optimized in two dimensions of structural accuracy and semantic consistency, and a gradient consistency loss function and a feature contrast loss function are calculated.

[0051] The weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function and feature contrast loss function are adaptively and dynamically adjusted to obtain a medical image cross-domain retrieval loss function.

[0052] According to the obtained medical image cross-domain retrieval loss function, the training of the medical image cross-domain retrieval model is completed, the similarity retrieval is performed based on the trained medical image cross-domain retrieval model, and the medical image cross-domain retrieval based on the generative adversarial network is completed.

[0053] As shown in Figure 2 The medical image cross-domain retrieval introduced in the embodiment includes an unsupervised cross-domain mapping framework based on a generative adversarial network, a multi-level cyclic consistency constraint mechanism and a dynamic weight adjustment strategy of multi-loss cooperative optimization.

[0054] In the following, the embodiment is expanded and described in detail:

[0055] (1) Unsupervised cross-domain mapping framework based on generative adversarial network

[0056] To overcome the problem that the traditional method needs to rely on strictly paired cross-domain data, the embodiment adopts an unsupervised cross-domain image mapping generative adversarial framework, performs unsupervised cross-domain mapping of images based on a generative adversarial network (GAN), and for the first time, gets rid of the dependence on paired MRI / CT image samples, and only realizes high-quality image domain conversion through set-level supervision; through a double-generator-discriminator structure, the MRI and CT domains are bidirectionally modeled, the generator is driven to approximate the target domain distribution combined with the adversarial loss, the image semantic migration is successfully completed under the condition of no paired data, the clinical data preparation threshold is greatly reduced, and feasible cross-domain retrieval under a real non-paired medical image environment is realized.

[0057] Generative adversarial networks (GANs) are a powerful class of artificial intelligence algorithms composed of two neural networks: a generator (G) and a discriminator (D). These two networks are trained simultaneously through an adversarial competition. The generator's role is to generate synthetic medical images from random noise, striving to replicate the characteristics of real data. Meanwhile, the discriminator evaluates these images, distinguishing between real and generated images. Through iterative feedback from the discriminator, the generator continually optimizes its output to create increasingly realistic synthetic data. A key advantage of GANs lies in their ability to generate highly realistic images and effectively handle domain shifts. Additionally, GANs can generate high-quality data with minimal or no labeled data, which is a significant advantage in data-scarce domains such as medical imaging.

[0058] Cross-domain supervised mapping requires converting images from one domain to another domain representation (e.g., converting MRI images to CT images). In traditional supervised learning environments, contrastive learning methods are often used to reduce the distance metric between features from different domains but with the same semantics in feature space (e.g., reducing the Euclidean distance between MRI and CT images of the same patient). However, MRI and CT images of the same patient are often difficult to obtain, so a method is needed to achieve cross-domain mapping without paired input of multi-domain images. The goal of introducing a generative adversarial network is to use images lacking paired samples to supervise at the set level.

[0059] To achieve cross-domain mapping between the MRI domain (A) and the CT domain (B), the present embodiment uses a bidirectional generative adversarial network structure, which includes a generator 、 and a discriminator 、 to achieve cross-domain image conversion and real or fake image discrimination between the MRI domain and the CT domain.

[0060] Without relying on paired image samples, the present embodiment uses adversarial training to enable the generator to learn the image distribution of the target domain. Taking MRI → CT as an example, the generator receives MRI images and generates pseudo-CT images , and the discriminator attempts to distinguish between real CT images and generated pseudo images ; its adversarial loss function is:

[0061] ;

[0062] wherein represents image samples from the MRI image domain (source domain A); represents image samples from the CT image domain (target domain B). denotes an image generator from MRI to CT; denotes an image generator from CT to MRI; denotes a discriminator for judging whether an input image is a real CT image; denotes a discriminator for judging whether an input image is a real MRI image; denotes a pseudo CT image generated by an MRI image; denotes a pseudo MRI image generated by a CT image; denotes an expected value, i.e., a mean value of a sample distribution; denotes a predicted log probability of a discriminator for an input image being "real"; denotes a predicted log probability of a discriminator for a generated image being "fake".

[0063] Similarly, a definition of an adversarial loss function in the CT→MRI direction can be obtained as follows: ; therefore, an adversarial loss function for cross-domain mapping is ; which lays a foundation for cross-domain feature alignment and subsequent unified retrieval representation learning.

[0064] (2) Multi-level cyclic consistency constraint mechanism

[0065] To solve the problem that simple linear changes cannot model complex relationships and achieve style conversion and key structure preservation, the multi-level cyclic consistency constraint is adopted in this embodiment, a gradient preservation loss and a feature space contrast loss are introduced into the traditional CycleGAN architecture for collaborative optimization, and the fidelity of low-level structures (such as tumor edges) and high-level semantics (field-independent representations) of images is considered; while maintaining the style migration ability, the cross-domain mapping is optimized in two dimensions of structural accuracy and semantic consistency for the first time, which significantly improves the clinical diagnostic reliability and retrievability of generated images, and the structural similarity index (SSIM) is improved by more than 30% compared with traditional methods.

[0066] The generative adversarial network can realize cross-domain mapping by training the generator, and in an ideal state, the generator can convert the image information of the A domain to , i.e., map the A domain to a domain with the same distribution as the B domain; the multi-layer semantic information of the image itself is easily ignored, i.e., although obeys the domain , it can present completely different semantic characteristics from .

[0067] The embodiment adopts multi-layer cycle consistency constraints, implements constraints at the pixel level and high-level feature space, introduces gradient constraint differences at the instance level, and forces the original image to retain the original anatomical structure (such as tumor edge and blood vessel direction) after experiencing A->B mapping and B->A mapping; at the feature level, the shared encoder E is used to extract domain-independent high-level semantic information of cross-domain images, and the contrast loss discriminative between learning samples, so that in the public subspace, the positive sample pair is close to each other, and the negative sample pair is far away from each other. Through multi-layer cycle consistency constraints, it is ensured that the generator and the encoder retain the pixel-level features and high-level semantic features of the original image while performing cross-domain conversion.

[0068] To improve the structural fidelity and semantic consistency of cross-domain mapping, the embodiment introduces multi-level structure constraints on the basis of the pixel-level cycle consistency of the traditional CycleGAN, including gradient consistency and semantic contrast constraints:

[0069] 1) Gradient consistency loss of structure preservation

[0070] To more accurately preserve the structural details in cross-domain image conversion, the embodiment introduces a structure gradient preservation strategy based on a multi-directional Sobel kernel on the basis of the traditional gradient consistency constraint to strengthen the ability to preserve edges, contours and other high-frequency structures.

[0071] Specifically, the embodiment uses a multi-directional Sobel filter set to extract gradient information of the input image in different directions (such as horizontal direction, vertical direction, diagonal direction, etc.). The image gradient in each direction is represented as ; wherein, represents any image, represents a convolution operation, and the direction kernel includes The Sobel convolution kernel corresponding to the direction is defined as follows:

[0072] Horizontal direction (0°): ;

[0073] Vertical direction (90°): ;

[0074] Positive diagonal direction (45°): ;

[0075] Negative diagonal direction (135°): .

[0076] To avoid the gradient scale difference between different directions from causing unstable effects on loss convergence, the gradients in each direction are further normalized; the normalization method uses min-max standardization for each image,

[0077] ;

[0078] wherein, is a stability constant.

[0079] Therefore, the gradient consistency loss function of structure preservation is defined as:

[0080]

[0081] The loss function measures the difference between the reconstructed image and the original image in the multi-directional edge structure, effectively strengthening the structural consistency and explainability of the generated image in key areas such as boundaries and organ contours, thereby improving the cross-domain structural alignment quality.

[0082] 2) Semantic-preserving feature consistency contrastive loss

[0083] To achieve higher-level cross-domain semantic alignment, this embodiment designs a shared Transformer encoder to extract domain-independent semantic feature representations of MRI and CT images . Unlike traditional CNNs that only focus on local texture information, Transformer has global modeling capabilities and can capture image structure and semantics at a more abstract level, making it suitable for alignment tasks between domains with significant appearance differences but consistent semantics.

[0084] In this embodiment, the encoder selects a visual Transformer model (DINOv2-ViT-S / 16) pre-trained on large-scale image data and fine-tuned in this task. Transformer models the long-range dependencies between different regions of an image through multiple layers of self-attention mechanisms, and its deep output represents a high semantic embedding vector of the image, which can effectively capture abstract semantic information such as organ structure and lesion category.

[0085] To guide the model to aggregate semantically similar images in the feature space, a feature consistency loss function based on contrastive learning is designed in this shared feature space. Specifically, for the input MRI image , it is obtained through the bidirectional generator and to obtain the reconstructed image . It is expected that they will maintain consistency in the semantic space, therefore, the feature contrastive loss function is:

[0086] ;

[0087] wherein, is the cosine similarity; exp is the power of e; Temperature parameter (settable to 0.1-0.2, control positive and negative sample difference sensitivity); Negative sample set, from other images in the same batch.

[0088] The loss encourages the model to aggregate the embeddings of the images with consistent semantics (i.e., the original image and the cyclically generated image) in the feature space, while distinguishing other images with inconsistent semantics. With the global representation capability of the Transformer, the model can complete the cross-domain alignment between the MRI and the CT at a high semantic level, providing support for the construction of a unified retrieval space.

[0089] (3) Dynamic weight adjustment strategy for multi-loss collaborative optimization

[0090] To solve the imbalance and gradient conflict problems that easily occur in multi-loss collaborative training, the embodiment adaptively and dynamically adjusts the weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function and feature comparison loss function to obtain a medical image cross-domain retrieval loss function; in combination with a training phase scheduling and loss fluctuation response mechanism, the weights are adaptively allocated according to the convergence speed and fluctuation degree of different loss terms in the training process, so as to ensure that the model focuses on the most challenging optimization target at different stages, thereby improving the training stability and final retrieval performance.

[0091] In view of the inconsistent convergence speed of loss terms and frequent gradient conflicts in multi-task training, the embodiment adopts a dynamic weight adjustment strategy based on training phase perception and loss fluctuation response, which monitors the change trend of each type of loss term in real time and adaptively adjusts the weight distribution, so that the model automatically focuses on the most challenging optimization target at each training stage, effectively suppressing the "strong loss dominant" in the early training stage or the "weak loss submerged" in the later stage; a new paradigm is provided for weight scheduling in multi-objective optimization, which realizes a more stable training process and better final performance without introducing additional supervision information, and the average mAP is improved by up to 5% in the cross-domain retrieval task, which is significantly better than the traditional static weighting method.

[0092] To cope with the differences in convergence speed and stability of each loss term in the training process and realize collaborative optimization of multi-task loss, the embodiment adopts a two-stage dynamic weight adjustment mechanism combining training phase progress and loss fluctuation feedback, so that the model can adaptively adjust the weight distribution of each loss according to the training state.

[0093] The medical image cross-domain retrieval loss function is the weighted sum of the cross-domain mapping adversarial loss function, the gradient consistency loss function and the feature comparison loss function, i.e. ; wherein, represents the i-th loss, including the adversarial loss , the gradient consistency loss and the feature comparison loss and feature contrast loss ; the dynamic weight of the first loss at the training progress.

[0094] the dynamic weight is ; wherein, is a training stage scheduling factor, controlling the trend of the loss weight with the training progress; is a loss dynamic response factor, dynamically adjusting the weight according to the stability of the loss.

[0095] In order to make the model focus on different task targets at different training stages, a scheduling function is designed; wherein, is the current training step, ; is the total training step; is the base weight of the first loss, controlling its initial importance; is an exponential hyperparameter that adjusts the growth rate of the weight, determining the growth curve shape of the loss weight with the training progress.

[0096] By adjusting , the model can focus on structure fidelity (such as gradient consistency) in the early training stage, and gradually strengthen semantic alignment (such as feature consistency) in the later stage.

[0097] In order to cope with the differences and fluctuations of different loss convergences, the embodiment designs a dynamic response mechanism based on loss fluctuation, reduces the weight of the loss with larger fluctuation, and avoids the influence of instability in the training process on the overall optimization; that is, ; wherein, is the standard deviation of the loss in the sliding window, reflecting the stability of the loss; is a smoothing constant (such as ), preventing division by zero operation.

[0098] In order to realize the unsupervised cross-domain image retrieval task between MRI images and CT images; after the medical image cross-domain retrieval model is trained, as shown in Figure 3 , the embodiment uses a shared encoder to extract the unified feature representation of the image, by inputting the query image (such as MRI image) into the encoder, the semantic embedding vector is obtained, and the similarity retrieval is performed in the target field (such as CT image) image feature database; the design of the embodiment based on the unified feature space makes the model not need to rely on paired samples, and can also perform cross-domain similar image matching without labeled information, providing a feasible solution for heterogeneous image assisted diagnosis in actual clinical practice.

[0099] ​This embodiment gets rid of the dependence on paired MRI / CT image samples and can achieve high-quality image domain conversion through set-level supervision; it uses a dual generator-discriminator structure to perform bidirectional modeling of MRI and CT domains, and combines adversarial loss to drive the generator to approximate the target domain distribution, successfully completing image semantic transfer without paired data, greatly reducing the threshold for clinical data preparation, and realizing feasible cross-domain retrieval in a real unpaired medical image environment.

[0100] Example 2

[0101] Embodiment 2 of the present invention introduces a medical image cross-domain retrieval system based on a generative adversarial network.

[0102] like Figure 4 A cross-domain medical image retrieval system based on a generative adversarial network is shown, comprising:

[0103] an acquisition module configured to acquire MRI images and CT images;

[0104] A mapping module is configured to perform unsupervised cross-domain mapping on the acquired MRI images and CT images based on a bidirectional generative adversarial network, and calculate an adversarial loss function for the cross-domain mapping;

[0105] An optimization module, which is configured to simultaneously optimize cross-domain mapping in terms of structural accuracy and semantic consistency based on a multi-level cycle consistency constraint mechanism during the unsupervised cross-domain mapping process, and calculate the gradient consistency loss function and the feature contrast loss function;

[0106] An adjustment module is configured to adaptively and dynamically adjust the weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function to obtain a medical image cross-domain retrieval loss function;

[0107] The retrieval module is configured to complete the training of the medical image cross-domain retrieval model according to the obtained medical image cross-domain retrieval loss function, perform similarity retrieval based on the trained medical image cross-domain retrieval model, and complete the medical image cross-domain retrieval based on the generative adversarial network.

[0108] The detailed steps are the same as those of the cross-domain retrieval method for medical images based on generative adversarial networks provided in Example 1 and will not be repeated here.

[0109] The above description is merely a preferred embodiment of this embodiment and is not intended to limit this embodiment. Those skilled in the art will readily appreciate that this embodiment may be modified and varied in various ways. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this embodiment shall be within the scope of protection of this embodiment.

Claims

1.A medical image cross-domain retrieval method based on a generative adversarial network, characterized in that, The method comprises the steps of: obtaining MRI images and CT images; performing unsupervised cross-domain mapping on the obtained MRI images and CT images based on a bidirectional generative adversarial network, and calculating an adversarial loss function of the cross-domain mapping; In the unsupervised cross-domain mapping process, based on the multi-level cyclic consistency constraint mechanism, the cross-domain mapping is optimized in two dimensions of structural accuracy and semantic consistency, and the gradient consistency loss function and the feature comparison loss function are calculated; the multi-level cyclic consistency constraint includes gradient consistency constraint, the gradient consistency constraint adopts the structural gradient retention strategy of multi-directional Sobel kernel to extract the gradient information of the input image in different directions, and the high frequency structure of the image edge contour is strengthened, wherein the image gradient in each direction is expressed as ; represents any image, represents a convolution operation, the directional kernel includes , the Sobel convolution kernel corresponding to the direction is defined as follows: horizontal direction, i.e. 0°: ; perpendicular direction, i.e. 90°: ; directly opposite the diagonal direction, i.e. 45°: ; The opposite direction, i.e. 135°: ; the gradient information in different directions is normalized, the normalization method adopts the min-max standardization of each picture, and the gradient consistency loss function of structure preservation is obtained according to the normalized gradient information ; the multi-level cycle consistency constraint further includes a semantic comparison constraint, the semantic comparison constraint adopts a shared Transformer encoder, extracts semantic features of the MRI image and the CT image through a multi-layer attention mechanism, aggregates the extracted semantic features, and obtains a feature comparison loss function ; performing adaptive dynamic adjustment on weights of the obtained adversarial loss function of the cross-domain mapping, gradient consistency loss function and feature contrast loss function to obtain a medical image cross-domain retrieval loss function; the medical image cross-domain retrieval loss function is a weighted sum of the adversarial loss function of the cross-domain mapping, the gradient consistency loss function and the feature contrast loss function, and the weights adopt dynamic weights; the dynamic weights are a product of a training stage scheduling factor and a loss dynamic response factor; the training stage scheduling factor is used to control the change trend of the loss weight with the training progress; and the loss dynamic response factor is a weight dynamically adjusted according to the stability of the loss; training a medical image cross-domain retrieval model according to the obtained medical image cross-domain retrieval loss function, performing similarity retrieval based on the trained medical image cross-domain retrieval model, and completing medical image cross-domain retrieval based on the generative adversarial network. 2.The medical image cross-domain retrieval method based on a generative adversarial network according to claim 1, wherein, The bidirectional generative adversarial network is used to realize cross-domain image conversion and true or false image discrimination, and comprises a first generator, a first discriminator, a second generator and a second discriminator; the image distribution of the target domain is obtained based on adversarial training of the bidirectional generative adversarial network without relying on paired image samples. 3.The medical image cross-domain retrieval method based on a generative adversarial network according to claim 2, characterized in that, The first generator receives the acquired MRI image, generates a pseudo CT image, the second discriminator distinguishes the acquired CT image and the generated pseudo CT image, and the adversarial loss function of the MRI image to the CT image direction is ; wherein, represents the image sample in the CT image domain; represents the MRI image; represents the CT image; represents the second discriminator for judging whether the input image is a real CT image; represents the image sample in the MRI image domain; represents the expected value, i.e. the mean value of the sample distribution; represents the first generator from the MRI image to the CT image; represents the predicted logarithmic probability of the discriminator that the input image is "real"; represents the predicted logarithmic probability of the discriminator that the generated image is "fake". 4.The medical image cross-domain retrieval method based on a generative adversarial network according to claim 3, characterized in that, The second generator receives the acquired CT image, generates a pseudo-MRI image, the first discriminator distinguishes the acquired MRI image and the generated pseudo-MRI image, and the adversarial loss function in the CT image to MRI image direction is ; wherein, represents a second generator from a CT image to an MRI image; represents an MRI image; represents a first discriminator for judging whether an input image is a real MRI image; the adversarial loss function of the cross-domain mapping includes an adversarial loss function in the MRI image to CT image direction and an adversarial loss function in the CT image to MRI image direction; that is . 5.The medical image cross-domain retrieval method based on a generative adversarial network according to claim 1, wherein, After the training of the medical image cross-domain retrieval model is completed, the features of the MRI images and the CT images are extracted by using a shared encoder, a semantic embedding vector is obtained by inputting a query image into the shared encoder, similarity retrieval of the query image is performed in the image features of the target field, and medical image cross-domain retrieval is completed. 6.A medical image cross-domain retrieval system based on a generative adversarial network, characterized in that, The method comprises the steps of: an acquisition module configured to obtain MRI images and CT images; a mapping module configured to perform unsupervised cross-domain mapping on the obtained MRI images and CT images based on a bidirectional generative adversarial network, and calculate an adversarial loss function of the cross-domain mapping; An optimization module is configured to, in an unsupervised cross-domain mapping process, based on a multi-level cyclic consistency constraint mechanism, simultaneously optimize the cross-domain mapping in two dimensions of structural precision and semantic consistency, and calculate a gradient consistency loss function and a feature contrast loss function; the multi-level cyclic consistency constraint includes a gradient consistency constraint, which extracts gradient information of an input image in different directions by using a multi-directional Sobel kernel structure gradient preservation strategy, and strengthens high-frequency structures of image edge contours, wherein the image gradient in each direction is expressed as ; represents an arbitrary image, represents a convolution operation, and the directional kernel includes The Sobel convolution kernel corresponding to the direction is defined as follows: horizontal direction, i.e. 0°: ; perpendicular direction, i.e. 90°: ; directly opposite the diagonal direction, i.e. 45°: ; The opposite direction, i.e. 135°: ; the gradient information in different directions is normalized, the normalization method adopts the min-max standardization of each picture, and the gradient consistency loss function of structure preservation is obtained according to the normalized gradient information ; the multi-level cycle consistency constraint further includes a semantic comparison constraint, the semantic comparison constraint adopts a shared Transformer encoder, extracts semantic features of the MRI image and the CT image through a multi-layer attention mechanism, aggregates the extracted semantic features, and obtains a feature comparison loss function ; an adjustment module configured to perform adaptive dynamic adjustment on weights of the obtained adversarial loss function of the cross-domain mapping, gradient consistency loss function and feature contrast loss function to obtain a medical image cross-domain retrieval loss function; the medical image cross-domain retrieval loss function is a weighted sum of the adversarial loss function of the cross-domain mapping, the gradient consistency loss function and the feature contrast loss function, and the weights adopt dynamic weights; the dynamic weights are a product of a training stage scheduling factor and a loss dynamic response factor; the training stage scheduling factor is used to control the change trend of the loss weight with the training progress; and the loss dynamic response factor is a weight dynamically adjusted according to the stability of the loss; a retrieval module configured to train a medical image cross-domain retrieval model according to the obtained medical image cross-domain retrieval loss function, perform similarity retrieval based on the trained medical image cross-domain retrieval model, and complete medical image cross-domain retrieval based on the generative adversarial network.

Images