Medical image cross-domain retrieval method and system based on generative adversarial network

Abstract

The invention belongs to the technical field of medical image cross-domain retrieval, and provides a medical image cross-domain retrieval method and system based on a generative adversarial network, and the method comprises the steps: obtaining an MRI image and a CT image; performing unsupervised cross-domain mapping of the acquired image based on the bidirectional generative adversarial network, and calculating an adversarial loss function of the cross-domain mapping; calculating a gradient consistency loss function and a feature comparison loss function based on a multi-level cyclic consistency constraint mechanism; carrying out adaptive dynamic adjustment on the obtained loss function to obtain a medical image cross-domain retrieval loss function, training a medical image cross-domain retrieval model, and carrying out similarity retrieval based on the trained medical image cross-domain retrieval model to complete medical image cross-domain retrieval based on the generative adversarial network. According to the method, the dependence on paired MRI / CT image samples is eliminated, high-quality image domain conversion is realized through integrated level supervision, and feasible cross-domain retrieval in a real non-paired medical image environment is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-domain retrieval of medical images, and specifically relates to a cross-domain retrieval method and system for medical images based on a generative adversarial network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of medical imaging technology, accurate medical image analysis and diagnosis have become key elements in improving treatment outcomes and patient care. Medical image retrieval (MIR) involves image processing, management, and integration, identifying clinically relevant imaging data from large datasets based on specific query requirements.

[0004] Medical Cross Domain Image Retrieval (MCDMIR) is a generalization of traditional image retrieval. It aims to input a specific medical image (e.g., computed tomography (CT)) and retrieve relevant images from other domains (e.g., magnetic resonance imaging (MRI)) from medical datasets. MCDMIR plays an indispensable role in multiple medical application areas: In diagnostic support, it provides clinicians with similar historical cases, significantly improving diagnostic accuracy; and for medical students and residents, MCDMIR serves as an important educational and training tool by providing visually similar cases. Medical image retrieval is considered a key means of effectively utilizing massive amounts of medical data and advancing both healthcare and research. However, a core obstacle is the "visual domain gap," characterized by significant differences in color, texture, brightness, gradient, and grayscale features between images derived from different imaging carriers, spectra, mechanisms, or illumination conditions. In the medical imaging field, cross-domain adaptation presents a particularly significant challenge due to the significant differences in contrast, texture, and noise levels between different imaging domains (e.g., CT and MRI).

[0005] Currently, cross-domain retrieval of medical images can be achieved by extracting and retrieving low-level features (such as texture, shape, and color) from images, using hand-crafted feature descriptors such as SIFT and HOG for cross-domain matching, or by achieving domain conversion through grayscale histogram alignment. However, there are inherent differences between automatically extracted low-level visual features and high-level human semantic understanding or clinical concepts. The uniqueness and complexity of medical images make manual feature methods ineffective when directly applied to medical images. Alternatively, shared features can be extracted using convolutional neural networks (CNNs) or twin networks, relying on paired cross-domain data (such as MRI-CT image pairs from the same patient). These methods directly learn increasingly higher-level features from the input data to achieve advanced understanding at the object, scene, and semantic levels. However, these methods also have certain limitations. The main challenge of deep learning models, especially supervised learning methods, is their reliance on large-scale annotated datasets, which are particularly scarce in the field of medical imaging. The annotation process for medical images is extremely time-consuming and cumbersome.

[0006] Therefore, existing technologies have the following problems in cross-domain retrieval of medical images (such as MRI to CT): manual features cannot capture the deep semantic associations between different domains (such as the difference in soft tissue contrast in MRI and density signals in CT), which leads to a certain gap between feature semantics and low retrieval accuracy; supervised learning methods require a large amount of strictly paired cross-domain data, and paired data is highly dependent, but multi-domain images of the same patient in clinical practice are often difficult to obtain or align; traditional domain conversion methods (such as grayscale mapping) will destroy anatomical structures (such as blurred tumor edges) and distort local structures, affecting diagnostic reliability.

[0007] The fundamental reasons for the above problems are: (1) Nonlinear mapping between fields. That is, the imaging principles of different fields of medical imaging (such as MRI / CT) are very different, and simple linear transformations (such as grayscale normalization) cannot model complex relationships.

[0008] (2) The training dilemma of unpaired data, that is, traditional deep learning requires pixel-level supervision signals, but hospitals usually only have independent single-domain image libraries (such as MRI libraries and CT libraries) and lack paired annotations.

[0009] (3) The contradiction between structure preservation and style transfer, that is, it is difficult for existing methods to simultaneously achieve domain style transfer (such as MRI→CT appearance adaptation) and key anatomical structure preservation (such as unchanged lesion location).

[0010] (4) Imbalance and gradient conflict problems that are prone to occur in multi-loss collaborative training. Summary of the Invention

[0011] To solve the above problems, the present invention proposes a cross-domain retrieval method and system for medical images based on a generative adversarial network, which gets rid of the dependence on paired MRI / CT image samples and can achieve high-quality image domain conversion through set-level supervision; bidirectional modeling of MRI and CT domains is performed through a dual generator-discriminator structure, and the adversarial loss is used to drive the generator to approximate the target domain distribution, successfully completing image semantic transfer without paired data, greatly reducing the threshold for clinical data preparation, and realizing feasible cross-domain retrieval in a real unpaired medical image environment.

[0012] According to some embodiments, a first solution of the present invention provides a cross-domain medical image retrieval method based on a generative adversarial network, which adopts the following technical solutions: A cross-domain medical image retrieval method based on a generative adversarial network, comprising: Acquire MRI images and CT images; Based on the bidirectional generative adversarial network, the acquired MRI images and CT images are unsupervisedly mapped across domains, and the adversarial loss function of the cross-domain mapping is calculated; In the unsupervised cross-domain mapping process, based on a multi-level cycle consistency constraint mechanism, the cross-domain mapping is optimized simultaneously in two dimensions: structural accuracy and semantic consistency, and the gradient consistency loss function and feature contrast loss function are calculated. The weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function are adaptively and dynamically adjusted to obtain the cross-domain retrieval loss function for medical images. The training of the medical image cross-domain retrieval model is completed according to the obtained medical image cross-domain retrieval loss function, and similarity retrieval is performed based on the trained medical image cross-domain retrieval model to complete the medical image cross-domain retrieval based on the generative adversarial network.

[0013] As a further technical limitation, the bidirectional generative adversarial network is used to achieve cross-domain image conversion and true and false image discrimination, including a first generator, a first discriminator, a second generator and a second discriminator; it does not rely on paired image samples, and obtains the image distribution of the image target domain based on adversarial training of the bidirectional generative adversarial network.

[0014] Furthermore, the first generator receives the acquired MRI image and generates a pseudo CT image, and the second discriminator distinguishes the acquired CT image from the generated pseudo CT image. The adversarial loss function from the MRI image to the CT image direction is: ;in, represents an image sample in the CT image domain; represents a CT image; represents a second discriminator for determining whether the input image is a real CT image; represents an image sample in the MRI image domain; represents the expected value, which is the mean of the sample distribution; represents a first generator from MRI images to CT images; represents the discriminator's predicted log probability that the input image is "real"; represents the discriminator’s predicted log probability that the generated image is “fake”.

[0015] Furthermore, the second generator receives the acquired CT image and generates a pseudo MRI image, the first discriminator distinguishes the acquired MRI image from the generated pseudo MRI image, and the adversarial loss function from the CT image to the MRI image direction is ;in, represents the second generator from CT image to MRI image; represents an MRI image; Represents a first discriminator for determining whether an input image is a real MRI image; the cross-domain mapping adversarial loss function of the MRI image to CT image direction includes an adversarial loss function of the MRI image to CT image direction and an adversarial loss function of the CT image to MRI image direction; that is .

[0016] As a further technical limitation, the multi-level cycle consistency constraint includes a gradient consistency constraint, which uses a structural gradient preservation strategy of a multi-directional Sobel kernel to extract the gradient information of the input image in different directions, strengthen the high-frequency structure of the image edge contour, and normalize the extracted gradient information in different directions. The structure-preserving gradient consistency loss function is obtained based on the normalized gradient information. .

[0017] As a further technical limitation, the multi-level cycle consistency constraint also includes a semantic contrast constraint, which uses a shared Transformer encoder to extract semantic features of MRI images and CT images through a multi-layer attention mechanism, aggregates the extracted semantic features, and obtains a feature contrast loss function .

[0018] As a further technical limitation, the medical image cross-domain retrieval loss function is a weighted sum of a cross-domain mapping adversarial loss function, a gradient consistency loss function, and a feature contrast loss function, and the weight adopts a dynamic weight.

[0019] Furthermore, the dynamic weight is the product of the training phase scheduling factor and the loss dynamic response factor; the training phase scheduling factor is used to control the changing trend of the loss weight with the training progress; the loss dynamic response factor is a weight dynamically adjusted according to the stability of the loss.

[0020] As a further technical limitation, after completing the training of the cross-domain retrieval model for medical images, a shared encoder is used to extract the features of MRI images and CT images, and the query image is input into the shared encoder to obtain a semantic embedding vector. The query image similarity is retrieved in the target domain image features to complete the cross-domain retrieval of medical images.

[0021] A cross-domain medical image retrieval system based on generative adversarial networks, including: an acquisition module configured to acquire MRI images and CT images; A mapping module is configured to perform unsupervised cross-domain mapping on the acquired MRI images and CT images based on a bidirectional generative adversarial network, and calculate an adversarial loss function for the cross-domain mapping; An optimization module, which is configured to simultaneously optimize cross-domain mapping in terms of structural accuracy and semantic consistency based on a multi-level cycle consistency constraint mechanism during the unsupervised cross-domain mapping process, and calculate the gradient consistency loss function and the feature contrast loss function; An adjustment module is configured to adaptively and dynamically adjust the weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function to obtain a medical image cross-domain retrieval loss function; The retrieval module is configured to complete the training of the medical image cross-domain retrieval model according to the obtained medical image cross-domain retrieval loss function, perform similarity retrieval based on the trained medical image cross-domain retrieval model, and complete the medical image cross-domain retrieval based on the generative adversarial network.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The present invention gets rid of the dependence on paired MRI / CT image samples and can achieve high-quality image domain conversion through set-level supervision; it uses a dual generator-discriminator structure to perform bidirectional modeling of MRI and CT domains, and combines adversarial loss to drive the generator to approximate the target domain distribution, successfully completing image semantic migration without paired data, greatly reducing the threshold for clinical data preparation, and realizing feasible cross-domain retrieval in a real unpaired medical image environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings constituting a part of the specification of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments and descriptions of this embodiment are used to explain this embodiment and do not constitute an improper limitation on this embodiment.

[0024] Figure 1 Flowchart of the cross-domain retrieval method for medical images based on generative adversarial networks in Example 1 of the present invention; Figure 2 This is an architecture diagram of the medical image cross-domain retrieval method based on generative adversarial network in Example 1 of the present invention; Figure 3 This is a flowchart of cross-domain retrieval in Example 1 of the present invention; Figure 4 This is a structural block diagram of the medical image cross-domain retrieval system based on generative adversarial networks in Example 2 of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0027] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0028] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0029] Example 1 Embodiment 1 of the present invention introduces a cross-domain retrieval method for medical images based on a generative adversarial network.

[0030] like Figure 1 A cross-domain medical image retrieval method based on a generative adversarial network is shown, comprising: Acquire MRI images and CT images; Based on the bidirectional generative adversarial network, the acquired MRI images and CT images are unsupervisedly mapped across domains, and the adversarial loss function of the cross-domain mapping is calculated; In the unsupervised cross-domain mapping process, based on a multi-level cycle consistency constraint mechanism, the cross-domain mapping is optimized simultaneously in two dimensions: structural accuracy and semantic consistency, and the gradient consistency loss function and feature contrast loss function are calculated. The weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function are adaptively and dynamically adjusted to obtain the cross-domain retrieval loss function for medical images. The training of the medical image cross-domain retrieval model is completed according to the obtained medical image cross-domain retrieval loss function, and similarity retrieval is performed based on the trained medical image cross-domain retrieval model to complete the medical image cross-domain retrieval based on the generative adversarial network.

[0031] like Figure 2 As shown, the cross-domain retrieval of medical images introduced in this embodiment includes an unsupervised cross-domain mapping framework based on a generative adversarial network, a multi-level cycle consistency constraint mechanism, and a dynamic weight adjustment strategy for multi-loss collaborative optimization.

[0032] The following describes this embodiment in detail: (1) Unsupervised cross-domain mapping framework based on generative adversarial networks To overcome the difficulty that traditional methods require to rely on strictly paired cross-domain data, this embodiment adopts a generative adversarial framework for unsupervised cross-domain image mapping without paired samples. It performs unsupervised cross-domain mapping of images based on a generative adversarial network (GAN), which for the first time gets rid of the dependence on paired MRI / CT image samples and achieves high-quality image domain conversion only through set-level supervision. It uses a dual generator-discriminator structure to perform bidirectional modeling of MRI and CT domains, and combines adversarial loss to drive the generator to approximate the target domain distribution. It successfully completes image semantic transfer without paired data, greatly reducing the threshold for clinical data preparation and realizing feasible cross-domain retrieval in a real unpaired medical image environment.

[0033] Generative Adversarial Networks (GANs) are a powerful class of artificial intelligence algorithms consisting of two neural networks: a generator (G) and a discriminator (D). These two networks are trained simultaneously through an adversarial competition. The generator generates synthetic medical images from random noise, striving to replicate the characteristics of real data. Simultaneously, the discriminator evaluates these images, distinguishing between real and generated images. Through iterative feedback from the discriminator, the generator continuously optimizes its output to create increasingly realistic synthetic data. A key advantage of GANs is their ability to generate highly realistic images and effectively handle domain drift. Furthermore, GANs can generate high-quality data with little or no labeled data, which is a significant advantage in data-scarce fields such as medical imaging.

[0034] Cross-domain supervised mapping requires converting images from one domain into image representations from another (e.g., converting MRI images to CT images). In traditional supervised learning environments, contrastive learning methods are often used to narrow the distance metrics of semantically identical features from different domains in feature space (e.g., narrowing the Euclidean distance between MRI and CT images of the same patient). However, MRI and CT images of the same patient are often difficult to obtain, so a method is needed to achieve cross-domain mapping without the need for paired input of multi-domain images. The goal of introducing generative adversarial networks is to use image outputs that lack paired samples for supervision at the set level.

[0035] To achieve cross-domain mapping between the MRI domain (A) and the CT domain (B), this embodiment adopts a bidirectional generative adversarial network structure, which includes a generator 、 and the discriminator 、 , realizing cross-domain image conversion and true and false image discrimination between MRI domain and CT domain.

[0036] Without relying on paired image samples, this embodiment uses adversarial training to enable the generator to learn the image distribution of the target domain. Taking MRI → CT as an example, the generator Receiving MRI images And generate pseudo CT images , discriminator Then try to distinguish the real CT image and the generated pseudo image ; Its adversarial loss function is: ; in, represents an image sample from the MRI image domain (source domain A); represents an image sample from the CT image domain (target domain B); represents the image generator from MRI to CT; represents the image generator from CT to MRI; represents the discriminator used to determine whether the input image is a real CT image; : A discriminator used to determine whether the input image is a real MRI image; represents a pseudo CT image generated from an MRI image; represents a pseudo MRI image generated from a CT image; represents the expected value, that is, the average value of the sample distribution; represents the discriminator's predicted log probability that the input image is "real"; represents the discriminator’s predicted log probability that the generated image is “fake”.

[0037] Similarly, the adversarial loss function in the CT→MRI direction can be defined as: ; Therefore, the adversarial loss function of cross-domain mapping is ; Lay the foundation for cross-domain feature alignment and subsequent unified retrieval representation learning.

[0038] (2) Multi-level cycle consistency constraint mechanism To address the problem that simple linear changes cannot model complex relationships and achieve style transfer and key structure preservation, this embodiment adopts multi-level cycle consistency constraints and introduces a collaborative optimization strategy of gradient preservation loss and feature space contrast loss on the traditional CycleGAN architecture, taking into account the fidelity of both the low-level structure of the image (such as tumor edges) and high-level semantics (domain-independent representation); while maintaining the style transfer capability, for the first time, cross-domain mapping is simultaneously optimized in two dimensions: structural accuracy and semantic consistency, significantly improving the clinical diagnostic credibility and retrievability of the generated images, with an improvement of more than 30% in the structural similarity index (SSIM) compared to traditional methods.

[0039] Generative adversarial networks can achieve cross-domain mapping by training the generator. Ideally, the generator Able to transmit image information of domain A Convert to , that is, mapping domain A to a domain with the same distribution as domain B It is easy to ignore the multi-layer semantic information of the image itself, that is, although Obey Domain , which may appear similar to Completely different semantic properties.

[0040] This embodiment adopts multi-layer cycle consistency constraints to implement constraints in pixel-level details and high-level feature space. At the instance level, gradient constraint difference is introduced to force the original image to retain the original anatomical structure (such as tumor edge and blood vessel direction) after A->B mapping and B->A mapping. At the feature level, a shared encoder E is used to extract domain-independent high-level semantic information of cross-domain images, and the contrast loss is used to extract domain-independent high-level semantic information of cross-domain images. The discriminability between samples is learned so that positive pairs are close to each other and negative pairs are far away in the common subspace. Through multi-layer cycle consistency constraints, the generator and encoder are ensured to preserve the pixel-level features and high-level semantic features of the original image while performing cross-domain conversion.

[0041] To improve the structural fidelity and semantic consistency of cross-domain mapping, this embodiment introduces multi-level structural constraints based on the pixel-level cycle consistency of traditional CycleGAN, including gradient consistency and semantic contrast constraints: 1) Structure-preserving gradient consistency loss To more accurately preserve structural details in cross-domain image conversion, this embodiment introduces a structural gradient preservation strategy based on a multi-directional Sobel kernel on the basis of traditional gradient consistency constraints to enhance the ability to retain high-frequency structures such as edges and contours.

[0042] Specifically, this embodiment adopts a multi-directional Sobel filter bank , which is used to extract the gradient information of the input image in different directions (such as horizontal, vertical, diagonal, etc.). The image gradient in each direction is expressed as ;in, represents any image, Represents the convolution operation, the directional kernel includes , the Sobel convolution kernels in the corresponding directions are defined as follows: Horizontal direction (0°): ; Vertical direction (90°): ; Diagonal direction (45°): ; Against the diagonal direction (135°): .

[0043] In order to avoid the unstable effect of gradient scale differences between different directions on loss convergence, the gradients in each direction are further normalized; the normalization method adopts min-max normalization for each image. ; in, is the stability constant.

[0044] Therefore, the structure-preserving gradient consistency loss function is defined as:

[0045] This loss function measures the difference between the reconstructed image and the original image in multi-directional edge structure, effectively enhancing the structural consistency and interpretability of the generated image in key areas such as boundaries and organ contours, thereby improving the quality of cross-domain structural alignment.

[0046] 2) Semantic-preserving feature consistency contrast loss To achieve higher-level cross-domain semantic alignment, this embodiment designs a shared Transformer encoder , for extracting domain-independent semantic feature representations from MRI and CT images Unlike traditional CNNs that only focus on local texture information, Transformer has global modeling capabilities and can capture image structure and semantics at a more abstract level. It is suitable for alignment tasks between domains with significant appearance differences but consistent semantics.

[0047] In this embodiment, the encoder A visual Transformer model (DINOv2-ViT-S / 16), pre-trained on large-scale image data, was selected and fine-tuned for this task. The Transformer uses a multi-layer self-attention mechanism to model long-range dependencies between different image regions. Its deep-layer output represents a highly semantically rich embedding vector of the image, effectively capturing abstract semantic information such as organ structure and lesion classification.

[0048] In order to guide the model to aggregate semantically similar images in the feature space, a feature consistency loss function based on contrastive learning is designed in the shared feature space. Specifically, for the input MRI image , which is generated by a bidirectional generator and Get the reconstructed image . They are expected to remain consistent in the semantic space, so the feature contrast loss function is: ; in, is the cosine similarity; exp is the power of e; is the temperature parameter (can be set to 0.1–0.2 to control the sensitivity of the difference between positive and negative samples); is a set of negative samples from other images in the same batch.

[0049] This loss encourages the model to aggregate semantically consistent embeddings (i.e., the original image and the recurrent generated image) in the feature space, while distinguishing between images from other domains or with inconsistent semantics. Combined with the global representation capabilities of the Transformer, the model can achieve cross-domain alignment between MRI and CT at a high semantic level, supporting the subsequent construction of a unified retrieval space.

[0050] (3) Dynamic weight adjustment strategy for multi-loss collaborative optimization To address the imbalance and gradient conflict issues that often occur in collaborative multi-loss training, this embodiment adaptively and dynamically adjusts the weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function to obtain a cross-domain retrieval loss function for medical images. Combining the training stage scheduling and loss fluctuation response mechanism, weights are adaptively allocated based on the convergence speed and fluctuation degree of different loss items during training, ensuring that the model focuses on the most challenging optimization goals at different stages, thereby improving training stability and final retrieval performance.

[0051] To address the problems of inconsistent convergence speed of loss terms and frequent gradient conflicts in multi-task training, this embodiment adopts a dynamic weight adjustment strategy based on training stage perception and loss fluctuation response, monitors the changing trends of various loss terms in real time, and adaptively adjusts the weight distribution, so that the model automatically focuses on the most challenging optimization goal at each stage of training, effectively suppressing the "strong loss dominance" in the early stage of training or "weak loss drowning" in the later stage; it provides a new paradigm for weight scheduling in multi-objective optimization, and achieves a more stable training process and better final performance without introducing additional supervision information. The average mAP in cross-domain retrieval tasks is improved by 5%, which is significantly better than the traditional static weighting method.

[0052] To address the differences in convergence speed and stability of various loss items during training and achieve collaborative optimization of multi-task losses, this embodiment adopts a two-stage dynamic weight adjustment mechanism that combines training stage progress and loss fluctuation feedback, so that the model can adaptively adjust the weight distribution of each loss according to the training status.

[0053] The cross-domain retrieval loss function of medical images is the weighted sum of the cross-domain mapping adversarial loss function, the gradient consistency loss function, and the feature contrast loss function, that is, ;in, Indicates the Losses, including counter-losses , gradient consistency loss and feature contrast loss ; For the Term loss in training progress Dynamic weight when .

[0054] The dynamic weight is ;in, It is the scheduling factor in the training phase, which controls the changing trend of loss weight as the training progresses; It is the loss dynamic response factor, which dynamically adjusts the weight according to the stability of the loss.

[0055] In order to make the model focus on different task objectives at different stages of training, a scheduling function is designed ;in, is the current training step number, ; is the total number of training steps; For the The basic weight of the loss item controls its initial importance; The exponential hyperparameter that adjusts the weight growth rate determines the shape of the growth curve of the loss weight as the training progresses.

[0056] By adjusting , the model can focus on structural fidelity (such as gradient consistency) in the early stage of training, and gradually strengthen semantic alignment (such as feature consistency) in the later stage.

[0057] In order to cope with the differences and fluctuations in the convergence of different losses, this embodiment designs a dynamic response mechanism based on loss fluctuations, reducing the weight of losses with large fluctuations to avoid the instability during training affecting the overall optimization; that is, ;in, For loss The standard deviation within the sliding window reflects the stability of the loss; is a smoothing constant (e.g. ), to prevent division by zero.

[0058] To achieve the unsupervised cross-domain image retrieval task between MRI images and CT images; after the training of the medical image cross-domain retrieval model is completed, Figure 3 As shown, this embodiment uses a shared encoder A unified feature representation of the image is extracted, and a semantic embedding vector is obtained by inputting the query image (such as an MRI image) into the encoder, and similarity retrieval is performed in the image feature database of the target domain (such as a CT image). This embodiment is based on the design of a unified feature space, so that the model can perform cross-domain similar image matching without relying on paired samples or labeling information, providing a feasible solution for heterogeneous image-assisted diagnosis in actual clinical practice.

[0059] This embodiment gets rid of the dependence on paired MRI / CT image samples and can achieve high-quality image domain conversion through set-level supervision; it uses a dual generator-discriminator structure to perform bidirectional modeling of MRI and CT domains, and combines adversarial loss to drive the generator to approximate the target domain distribution, successfully completing image semantic transfer without paired data, greatly reducing the threshold for clinical data preparation, and realizing feasible cross-domain retrieval in a real unpaired medical image environment.

[0060] Example 2 Embodiment 2 of the present invention introduces a medical image cross-domain retrieval system based on a generative adversarial network.

[0061] like Figure 4 A cross-domain medical image retrieval system based on a generative adversarial network is shown, comprising: an acquisition module configured to acquire MRI images and CT images; A mapping module is configured to perform unsupervised cross-domain mapping on the acquired MRI images and CT images based on a bidirectional generative adversarial network, and calculate an adversarial loss function for the cross-domain mapping; An optimization module, which is configured to simultaneously optimize cross-domain mapping in terms of structural accuracy and semantic consistency based on a multi-level cycle consistency constraint mechanism during the unsupervised cross-domain mapping process, and calculate the gradient consistency loss function and the feature contrast loss function; An adjustment module is configured to adaptively and dynamically adjust the weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function to obtain a medical image cross-domain retrieval loss function; The retrieval module is configured to complete the training of the medical image cross-domain retrieval model according to the obtained medical image cross-domain retrieval loss function, perform similarity retrieval based on the trained medical image cross-domain retrieval model, and complete the medical image cross-domain retrieval based on the generative adversarial network.

[0062] The detailed steps are the same as those of the cross-domain retrieval method for medical images based on generative adversarial networks provided in Example 1 and will not be repeated here.

[0063] The above description is merely a preferred embodiment of this embodiment and is not intended to limit this embodiment. Those skilled in the art will readily appreciate that this embodiment may be modified and varied in various ways. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this embodiment shall be within the scope of protection of this embodiment.

Claims

1. A cross-domain medical image retrieval method based on generative adversarial networks, characterized by: include: Acquire MRI images and CT images; Based on the bidirectional generative adversarial network, the acquired MRI images and CT images are unsupervisedly mapped across domains, and the adversarial loss function of the cross-domain mapping is calculated; In the unsupervised cross-domain mapping process, based on a multi-level cycle consistency constraint mechanism, the cross-domain mapping is optimized simultaneously in two dimensions: structural accuracy and semantic consistency, and the gradient consistency loss function and feature contrast loss function are calculated. The weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function are adaptively and dynamically adjusted to obtain the cross-domain retrieval loss function for medical images. The training of the medical image cross-domain retrieval model is completed according to the obtained medical image cross-domain retrieval loss function, and similarity retrieval is performed based on the trained medical image cross-domain retrieval model to complete the medical image cross-domain retrieval based on the generative adversarial network.

2. A cross-domain medical image retrieval method based on a generative adversarial network as claimed in claim 1, characterized in that: The bidirectional generative adversarial network is used to achieve cross-domain image conversion and true and false image discrimination, including a first generator, a first discriminator, a second generator and a second discriminator; it does not rely on paired image samples and obtains the image distribution of the image target domain based on adversarial training of the bidirectional generative adversarial network.

3. A cross-domain medical image retrieval method based on a generative adversarial network as described in claim 2, characterized in that: The first generator receives the acquired MRI image and generates a pseudo CT image. The second discriminator distinguishes the acquired CT image from the generated pseudo CT image. The adversarial loss function from the MRI image to the CT image is: ;in, represents an image sample in the CT image domain; represents a CT image; represents a second discriminator for determining whether the input image is a real CT image; represents an image sample in the MRI image domain; represents the expected value, which is the mean of the sample distribution; represents a first generator from MRI images to CT images; represents the discriminator's predicted log probability that the input image is "real"; represents the discriminator's predicted log probability that the generated image is "fake".

4. A cross-domain medical image retrieval method based on a generative adversarial network as claimed in claim 3, characterized in that: The second generator receives the acquired CT image and generates a pseudo MRI image. The first discriminator distinguishes the acquired MRI image from the generated pseudo MRI image. The adversarial loss function from the CT image to the MRI image is ;in, represents the second generator from CT image to MRI image; represents an MRI image; Represents a first discriminator for determining whether an input image is a real MRI image; the cross-domain mapping adversarial loss function of the MRI image to CT image direction includes an adversarial loss function of the MRI image to CT image direction and an adversarial loss function of the CT image to MRI image direction; that is .

5. The cross-domain medical image retrieval method based on generative adversarial networks as claimed in claim 1, characterized in that: The multi-level cycle consistency constraint includes a gradient consistency constraint, which uses a multi-directional Sobel kernel structure gradient preservation strategy to extract the gradient information of the input image in different directions, strengthen the high-frequency structure of the image edge contour, and normalize the extracted gradient information in different directions. The structure-preserving gradient consistency loss function is obtained based on the normalized gradient information. .

6. A cross-domain medical image retrieval method based on generative adversarial networks as claimed in claim 1, characterized in that: The multi-level cycle consistency constraint also includes a semantic contrast constraint, which uses a shared Transformer encoder to extract semantic features of MRI images and CT images through a multi-layer attention mechanism, aggregates the extracted semantic features, and obtains a feature contrast loss function .

7. The cross-domain medical image retrieval method based on generative adversarial networks as claimed in claim 1, characterized in that: The medical image cross-domain retrieval loss function is a weighted sum of a cross-domain mapping adversarial loss function, a gradient consistency loss function, and a feature contrast loss function, and the weight adopts a dynamic weight.

8. A cross-domain medical image retrieval method based on a generative adversarial network as claimed in claim 7, characterized in that: The dynamic weight is the product of the training phase scheduling factor and the loss dynamic response factor; the training phase scheduling factor is used to control the changing trend of the loss weight with the training progress; the loss dynamic response factor is a weight dynamically adjusted according to the stability of the loss.

9. The cross-domain medical image retrieval method based on generative adversarial networks as claimed in claim 1, characterized in that: After completing the training of the cross-domain retrieval model for medical images, a shared encoder is used to extract the features of MRI images and CT images. The query image is input into the shared encoder to obtain a semantic embedding vector. The query image similarity is retrieved in the target domain image features to complete the cross-domain retrieval of medical images.

10. A medical image cross-domain retrieval system based on generative adversarial networks, characterized by: include: an acquisition module configured to acquire MRI images and CT images; A mapping module is configured to perform unsupervised cross-domain mapping on the acquired MRI images and CT images based on a bidirectional generative adversarial network, and calculate an adversarial loss function for the cross-domain mapping; An optimization module, which is configured to simultaneously optimize cross-domain mapping in terms of structural accuracy and semantic consistency based on a multi-level cycle consistency constraint mechanism during the unsupervised cross-domain mapping process, and calculate the gradient consistency loss function and the feature contrast loss function; An adjustment module is configured to adaptively and dynamically adjust the weights of the obtained cross-domain mapping adversarial loss function, gradient consistency loss function, and feature contrast loss function to obtain a medical image cross-domain retrieval loss function; The retrieval module is configured to complete the training of the medical image cross-domain retrieval model according to the obtained medical image cross-domain retrieval loss function, perform similarity retrieval based on the trained medical image cross-domain retrieval model, and complete the medical image cross-domain retrieval based on the generative adversarial network.

Images