An ultrasound-to-magnetic resonance image synthesis method and system based on structure-guided cycle consistency adversarial network

Abstract

This invention relates to the field of intelligent medical image processing technology, and particularly to a method and system for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network. Addressing the problems of existing cross-modal translation methods that easily lead to anatomical distortion and fail to remove ultrasound speckle noise, this method acquires unpaired images to construct a training set; constructs a recurrent consistency adversarial network with a bidirectional shared modality-independent feature extractor; introduces modality-independent feature consistency loss in the recurrent reconstruction to force alignment of deep anatomical structures; uses a domain classifier combined with a gradient inversion layer for adversarial training to actively remove modality-specific physical noise; and iteratively optimizes the network using a multidimensional loss. This invention effectively overcomes the "tissue illusion" and geometric distortion caused by ultrasound artifacts, and uses a trained model to output anatomically accurate virtual magnetic resonance images with high soft tissue contrast in real time, significantly improving the accuracy of clinical biopsy and surgical navigation.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image processing technology, and in particular to a method and system for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network. Background Technology

[0002] In modern clinical medical diagnosis, early disease screening, and surgical navigation, medical imaging technology is an indispensable cornerstone. Among numerous imaging modalities, ultrasound (US) and magnetic resonance imaging (MRI) are the two most core and highly complementary non-ionizing radiation imaging technologies. Ultrasound imaging has advantages such as no ionizing radiation, portable equipment, and real-time dynamic imaging, and is widely used in intraoperative real-time navigation; however, the physical generation mechanism of ultrasound imaging determines that it has inherent physical defects that are difficult to overcome through conventional hardware upgrades. Ultrasound images are reconstructed from backscattered signals generated by the interaction between the internal tissue structures of the human body and the sound waves emitted by the probe. In this physical process, when there are tiny structures within the tissue smaller than the wavelength of the emitted sound waves, Rayleigh scattering occurs. These randomly scattered coherent echoes interfere constructively or destructively in space, thus forming characteristic "speckle noise" on the image. This inherent speckle noise not only severely reduces the signal-to-noise ratio and spatial resolution of images, but also masks small lesions or low-contrast diseased tissues (such as early micro-tumor infiltration boundaries or complex vascular networks), resulting in ultrasound images exhibiting strong textural heterogeneity and artifacts. Conversely, MRI offers extremely high soft tissue resolution, clearly distinguishing various types of lesions; however, its equipment is expensive, data acquisition is extremely slow, and it suffers from severe motion artifacts and stringent magnetic shielding conditions, making it difficult to widely adopt in real-time surgical navigation or primary healthcare. Therefore, directly synthesizing "low-quality ultrasound images acquired in real-time" into "virtual MRI images with high soft tissue contrast" has significant technical value.

[0003] In recent years, unsupervised cross-modal translation models based on deep learning (such as the standard CycleGAN) have been introduced into this field. In such unsupervised cross-modal translation tasks, existing techniques typically employ the standard CycleGAN and its derivatives (such as UNIT, MUNIT, DCLGAN, etc.). The core idea of ​​CycleGAN and its variants is to achieve image conversion between two different domains (such as ultrasound and MRI) by introducing cycle-consistency loss, without the need for costly collection of paired medical image datasets. However, existing cycle-consistency-based ultrasound-to-MRI methods suffer from the following technical drawbacks: First, existing technologies only perform cyclic reconstruction constraints at the "pixel-level" of the image. When mapping from the relatively information-scarce and noisy ultrasound domain to the extremely information-rich MRI domain, there is a risk of "one-to-many" ambiguity mapping. Due to the lack of strong structural prior constraints, the generator is prone to finding shortcuts to deceive the discriminator. This can lead to the appearance of non-existent tissue structures in the generated virtual MRI images (i.e., "tissue hallucinations"), or arbitrary alteration of the shape, position, size, and relative proximity of organs. In other words, current technology cannot actively remove the physical noise unique to ultrasound; instead, it often mistranslates strong ultrasound spots as anatomical lesions (such as tumors or hyperplasia) on the MRI image. Furthermore, during cross-modal mapping, it can easily cause severe geometric distortions of the original organ shape and lesion boundaries. Secondly, while the adversarial loss and pixel-level cyclic consistency loss used in current technology can ensure the coherence of the global image style and macroscopic structure to some extent, they are extremely insensitive to high-frequency textures and sharp boundaries. This often results in these high-frequency textures and sharp boundaries being overly smoothed during translation. The synthesized MRI images lack granularity and sharpness, failing to clearly display specific anatomical features. Finally, existing technologies employ more complex translation models, such as UNIT and MUNIT, to decouple images into "content encoding" and "style encoding," prioritizing uniformity of representation or diversity of style. However, speckle noise and acoustic artifacts in ultrasound images are caused by acoustic interference and physical obstruction, not anatomical content. When these existing models process ultrasound images, they cannot actively identify and discard these unique physical noise features. Instead, they often incorrectly retain strong ultrasound speckles or reverberation artifacts, rigidly translating them into anatomical features on the MRI image that resemble lesions or abnormal proliferations. This fails to effectively remove artifacts from medical images, affecting the final imaging effect and reducing imaging accuracy. Summary of the Invention

[0004] To address the technical problems of existing recurrent consistency adversarial network (RCA) techniques, such as geometric distortion caused by mapping ambiguity between ultrasound and MRI, lack of image granularity due to insensitivity to high-frequency textures, and inability of existing models to adapt to the complex artifacts and noise in medical images, this invention provides a method and system for ultrasound-to-MRI image synthesis based on structure-guided recurrent consistency adversarial networks (CRANs).

[0005] One method for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network includes the following steps: S1. Acquire unpaired ultrasound and magnetic resonance images; S2. Preprocess the unpaired ultrasound and magnetic resonance images, and construct ultrasound image training sets and magnetic resonance image training sets respectively based on the preprocessed ultrasound and magnetic resonance images. S3. Based on the cycle consistency loss constraint and combined with the modality-independent feature extractor, an adversarial generative network based on structure-guided cycle consistency is constructed. The adversarial generative network is trained according to the ultrasound image training set and the magnetic resonance image training set to obtain the trained adversarial generative network. S4. Obtain the real ultrasound image to be converted, and perform ultrasound-to-MRI conversion through a trained generative adversarial network to generate the final MRI image.

[0006] Furthermore, step S2 includes the following sub-steps: S201. Perform unified center cropping on unpaired ultrasound images and magnetic resonance images; S202. Scale the cropped ultrasound and magnetic resonance images to a preset fixed resolution matrix pixel size; S203. Based on a pre-set floating-point range, the grayscale pixel values ​​of the scaled ultrasound image and magnetic resonance image are linearly transformed and normalized until the floating-point range requirements are met; S204. Based on the ultrasound images and magnetic resonance images obtained in step S203, construct ultrasound image training sets and magnetic resonance image training sets.

[0007] Furthermore, step S3 includes the following sub-steps: S301. Based on the cycle consistency loss constraint and combined with a modality-independent feature extractor, an adversarial generative network based on structure-guided cycle consistency is constructed. The adversarial generative network includes a forward generator, a backward generator, a visual realism discriminator, a domain classifier, and a gradient inversion layer. The forward generator includes a modality-independent feature extractor and a magnetic resonance feature decoding mapping function, and the backward generator includes a modality-independent feature extractor and an ultrasound feature decoding mapping function. The forward generator and the backward generator share the same modality-independent feature extractor. The visual realism discriminator includes a magnetic resonance realism discriminator and an ultrasound realism discriminator. S302. Input the ultrasound image training set into the forward generator, extract the anatomical structure feature map of the ultrasound image through the modality-independent feature extractor, and map the anatomical structure feature map of the ultrasound image from the ultrasound domain to the magnetic resonance domain through the magnetic resonance feature decoding mapping function to obtain a virtual magnetic resonance image. Input the magnetic resonance image training set into the reverse generator, extract the anatomical structure feature map of the magnetic resonance image through the modality-independent feature extractor, and map the anatomical structure feature map of the magnetic resonance image from the magnetic resonance domain back to the ultrasound domain through the ultrasound feature decoding mapping function to obtain a virtual ultrasound image. S303. Input the virtual MRI image into the inverse generator, reconstruct the reconstructed ultrasound image through inverse cyclic data, and extract the anatomical structure feature map of the reconstructed ultrasound image through the modality-independent feature extractor. Input the virtual ultrasound image into the forward generator, reconstruct the reconstructed MRI image through inverse cyclic data, and extract the anatomical structure feature map of the reconstructed MRI image through the modality-independent feature extractor. Calculate the absolute error distance between the anatomical structure feature map of the ultrasound image and the anatomical structure feature map of the reconstructed ultrasound image, and the absolute error distance between the anatomical structure feature map of the MRI image and the anatomical structure feature map of the reconstructed MRI image, and construct the modality-independent feature consistency loss. S304. Obtain anatomical structure feature maps from ultrasound images and magnetic resonance images to construct an anatomical feature map set. Calculate the probability that the anatomical feature map set comes from the ultrasound image training set or the magnetic resonance image training set through a neighborhood classifier. Construct a neighborhood discrimination loss based on binary cross-entropy. After multiplying the neighborhood discrimination loss by the inversion coefficient through a gradient inversion layer to perform a negative inversion, input it into the modality-independent feature extractor. S305. Input the magnetic resonance image training set and the virtual magnetic resonance image obtained in step S202 into the magnetic resonance authenticity discriminator, and input the ultrasound image training set and the virtual ultrasound image obtained in step S202 into the ultrasound authenticity discriminator to construct magnetic resonance adversarial loss and ultrasound adversarial loss respectively. S306. Input the magnetic resonance image training set into the forward generator and construct the identity mapping loss; S307. Based on the cycle consistency loss, modality-independent feature consistency loss, domain discrimination loss, magnetic resonance adversarial loss, ultrasonic adversarial loss, and identity mapping loss, a weighted aggregation calculation is performed according to the preset weight hyperparameters to construct the total loss function. The network parameters of the adversarial generative network are iteratively updated with the goal of minimizing the total loss function until the total loss function converges, thus obtaining the optimal adversarial generative network.

[0008] Furthermore, the cycle consistency loss is expressed as: ;in, Indicates through the forward generator and reverse generator The calculated pixel-level cycle consistency loss function, This represents ultrasound image samples in the ultrasound image training set. This represents magnetic resonance image samples in the magnetic resonance image training set. This represents the training set of ultrasound images. This represents the training set of magnetic resonance images. Represents the mathematical expectation. This indicates that ultrasound image samples are extracted from the ultrasound image training set. This indicates that magnetic resonance image samples are extracted from the magnetic resonance image training set. Represents a virtual magnetic resonance image. Represents a virtual ultrasound image. This represents the reconstructed ultrasound image. Represents the reconstructed magnetic resonance image. Represents norm distance.

[0009] Furthermore, the modality-independent feature consistency loss is expressed as: ;in, This represents the modality-independent feature consistency loss. This represents a map showing the anatomical features of an ultrasound image. Map showing anatomical features in magnetic resonance imaging. This represents a map showing the anatomical features of a reconstructed magnetic resonance image. This represents a map showing the anatomical features of a reconstructed ultrasound image.

[0010] Furthermore, the probability is expressed as: ;in, This represents the image input to the modality-independent feature extractor. Atlas of anatomical features Represents the parameters of the weight matrix. Indicates the bias term parameter, This represents the transpose of the weight matrix parameters. This represents the Sigmoid activation function, used to map the result of linear computation to the (0,1) interval, forming probability values. This represents the current anatomical feature map under the influence of weight W and bias z. The probabilities derived from the ultrasound image training set or the magnetic resonance image training set; the neighborhood discrimination loss is expressed as: ;in, Indicates the domain for loss determination. Represents a domain classifier. Represents anatomical feature maps. This represents the total number of feature maps extracted in a single training epoch. Indicates the number of rounds in the current round. Image samples, Indicates the first One input image sample, A binary label indicating the true source of a sample modality, when the image originates from the ultrasound domain. When the image comes from the magnetic resonance domain , Represents the natural logarithm. Represents the current anatomical feature map The probability derived from the training set of ultrasound images or magnetic resonance images; The inversion coefficient is expressed as: ;in, Indicates the dynamic reversal coefficient. Represents the natural constant An exponential function with base 0. This represents a pre-set constant for controlling the dynamic scheduling rate. Indicates the current training progress, and .

[0011] Furthermore, the magnetic resonance countermeasure loss and the ultrasonic countermeasure loss are respectively expressed as: ; ; in, Indicates magnetic resonance resistance loss, Indicates ultrasound counteracting loss, This represents a magnetic resonance imaging (MRI) authenticity discriminator. This indicates an ultrasound authenticity discriminator. Indicates judging the input image This represents a probability assessment value for a real magnetic resonance image. Indicates the judgment of virtual magnetic resonance images This represents a probability assessment value for a real magnetic resonance image. Indicates judging the input image This represents a probability assessment value for a real ultrasound image. Indicates the judgment of virtual ultrasound images This is a probability assessment value for a real ultrasound image.

[0012] Furthermore, the identity mapping loss is expressed as: ;in, Denotes the identity mapping loss function. This indicates the actual magnetic resonance image. The source data is directly input into the forward generator. The output image, This indicates the use of real ultrasound images. The source data is directly input into the reverse generator. The image output later.

[0013] Furthermore, the total loss function is expressed as: ;in, Represents the total loss function. Indicates magnetic resonance resistance loss, Indicates ultrasound counteracting loss, This represents the loss of cycle consistency. This represents the modality-independent feature consistency loss. Indicates the domain for loss determination. This represents the loss from the identity mapping. This represents the first weighted hyperparameter, which is pre-set to control the strength of the cycle consistency loss and the mode-independent feature consistency loss in the total loss. This represents a pre-set second weight hyperparameter used to control the relative strength of the domain discrimination loss and identity mapping loss in the total loss.

[0014] This invention also provides an ultrasound-to-magnetic resonance (MRI) image synthesis system based on a structure-guided cyclic consistency adversarial network. This system is implemented based on any of the ultrasound-to-MRI image synthesis methods described above, and includes: a data acquisition and preprocessing module for acquiring unpaired ultrasound and MRI images, preprocessing the unpaired ultrasound and MRI images, and constructing ultrasound and MRI image training sets respectively based on the preprocessed ultrasound and MRI images; a model construction and training module for constructing a structure-guided cyclic consistency adversarial generative network (GGN) based on cyclic consistency constraints and a modality-independent feature extractor, and training the GGN based on the ultrasound and MRI image training sets to obtain a trained GGN; and an ultrasound-to-MRI module for acquiring the real ultrasound image to be converted, performing ultrasound-to-MRI processing through the trained GGN, and generating the final MRI image.

[0015] A computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform any of the above-mentioned methods for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network.

[0016] An electronic device includes: a memory for storing a computer program; and a processor for executing the computer program to implement an ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network as described above.

[0017] Compared with the prior art, the present invention has the following advantages: This invention constructs a model architecture where the forward and backward generators share a modality-independent feature extractor, and introduces modality-independent feature consistency loss. This enables forced comparison and reduction of the absolute feature distance between the original image and the cyclically reconstructed image in a deep, high-dimensional feature space. This strongly guides the cross-modal translation process to the underlying anatomical structure. The high-dimensional feature abstraction layer of the network intercepts and removes interference noise information unique to ultrasound, forcibly preserving and aligning only the anatomical structure of the organs, effectively improving the high-dimensional structural fidelity during ultrasound-to-MRI conversion. Secondly, addressing the problem of severe speckle noise in ultrasound being easily misinterpreted as MRI artifacts, this invention uses a domain classifier and gradient inversion layer to form an adversarial game mechanism, incorporating the error gradient into the backpropagation process. The negative inversion of the dynamics forces the feature extractor to confuse the modalities of the image, thereby actively filtering and stripping away the unique speckle noise information of ultrasound, retaining only the pure anatomical contours, and fundamentally eliminating the pathological false positive features in the synthetic MRI images. In addition, this invention, through this high-quality synthetic ultrasound and MRI data, combined with the hybrid pre-training mechanism of real ultrasound and MRI data, effectively overcomes the long-tail distribution and class imbalance defects caused by the scarcity of sample data in practical applications. This allows a lightweight neural network with extremely small parameters and extremely low computational cost to achieve or even surpass the sensitivity and specificity indicators of heavyweight models with tens of millions of parameters in disease feature recognition, greatly improving conversion efficiency and effectively reducing conversion costs in practical applications. Attached Figure Description

[0018] Figure 1 This is a flowchart of an ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network proposed in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the structure of an ultrasound-to-magnetic resonance image synthesis system based on a structure-guided recurrent consistency adversarial network proposed in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the terminal device structure of an ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network proposed in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of a computer-readable storage medium structure for an ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network proposed in an embodiment of the present invention.

[0022] In the diagram, 200 is the terminal device, 210 is the memory, 211 is the RAM, 212 is the cache memory, 213 is the ROM, 214 is the program / utility, 215 is the program module, 220 is the processor, 230 is the bus, 240 is the external device, 250 is the I / O interface, 260 is the network adapter, and 300 is the program product. Detailed Implementation

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, and not all of them. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention. It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0026] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or machine that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or machine. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or machine that includes said element.

[0027] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0028] Example 1: like Figure 1 As shown, this embodiment provides a method for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network, including the following steps: S1. Acquire unpaired ultrasound and magnetic resonance images; S2. Preprocess the unpaired ultrasound and magnetic resonance images, and construct ultrasound image training sets and magnetic resonance image training sets respectively based on the preprocessed ultrasound and magnetic resonance images. Furthermore, step S2 includes the following sub-steps: S201. Perform unified center cropping on unpaired ultrasound images and magnetic resonance images; S202. Scale the cropped ultrasound and magnetic resonance images to a preset fixed resolution matrix pixel size; S203. Based on a pre-set floating-point range, the grayscale pixel values ​​of the scaled ultrasound image and magnetic resonance image are linearly transformed and normalized until the floating-point range requirements are met; S204. Based on the ultrasound images and magnetic resonance images obtained in step S203, construct ultrasound image training sets and magnetic resonance image training sets.

[0029] Specifically, this step addresses the underlying data requirements for cross-modal image synthesis in medicine by independently acquiring ultrasound and MRI images from a multi-center database and performing rigorous standardization operations on spatial scale and pixel-level distribution. Existing techniques for cross-modal network training often rely on paired datasets with perfectly aligned anatomical locations. However, in clinical medicine, due to factors such as instrument bias and organ displacement during imaging, obtaining perfect paired data is extremely difficult, resulting in model training being limited by data scale and exhibiting registration errors. This method collects unpaired data and performs uniform cropping and floating-point normalization, constructing an unbiased, scale-uniform, and deep learning-compliant underlying multi-source state dataset for subsequent unsupervised generative adversarial algorithms.

[0030] Specifically, the implementation principle described above is as follows: First, ultrasound and MRI images of different batches of patients were independently acquired. Unnecessary black borders and instrument parameter characters at the edges of the ultrasound probe scanning sectors were removed using a unified center cropping method. Simultaneously, all images were uniformly scaled to a fixed resolution matrix using an interpolation algorithm. Finally, pixel intensity values, originally with different physical meanings and dimensions, were linearly mapped to the standard [-1,1] floating-point range. The acquired standardized data were packaged separately to construct a training set covering the basic distribution characteristics of the source and target domains. Furthermore, it should be noted that the feature extraction efficiency of neural network models is limited by the consistency of input data format and the stability of gradient flow. By using a standardized preprocessing network to achieve full-dimensional image format unification, the feature silos caused by different equipment manufacturers in traditional medical imaging are broken down. This allows the scattered physical quantities reflecting ultrasound acoustic reflection and MRI hydrogen proton relaxation to be aggregated into a high-fidelity, tensor-aligned underlying data pool, providing a complete data support foundation for subsequent global deep feature extraction and cross-domain mapping.

[0031] S3. Based on the cycle consistency loss constraint and combined with the modality-independent feature extractor, an adversarial generative network based on structure-guided cycle consistency is constructed. The adversarial generative network is trained according to the ultrasound image training set and the magnetic resonance image training set to obtain the trained adversarial generative network. Furthermore, step S3 includes the following sub-steps: S301. Based on the cycle consistency loss constraint and combined with a modality-independent feature extractor, an adversarial generative network based on structure-guided cycle consistency is constructed. The adversarial generative network includes a forward generator, a backward generator, a visual realism discriminator, a domain classifier, and a gradient inversion layer. The forward generator includes a modality-independent feature extractor and a magnetic resonance feature decoding mapping function, and the backward generator includes a modality-independent feature extractor and an ultrasound feature decoding mapping function. The forward generator and the backward generator share the same modality-independent feature extractor. The visual realism discriminator includes a magnetic resonance realism discriminator and an ultrasound realism discriminator. Specifically, the Modality Independent Feature Extractor (MAFE) is constructed by stacking deep residual convolutional blocks (ResNet Blocks) with instance normalization and ReLU activation functions. However, in scenarios involving large organs (such as whole-abdominal or whole-brain synthesis) where global spatial correlation is extremely important, the MAFE architecture can be completely replaced by a self-attention-based visual Transformer (the core mechanism of which is the Swin-Transformer or MobileViT). The Transformer, with its superior ability to capture long-range dependencies, can further improve the anatomical consistency of distant tissue structures.

[0032] Specifically, this step addresses the fundamental topology of the cross-modal network by deploying a shared modality-independent feature extractor at the encoding front end of the generative model, and deploying a domain classifier with a gradient inversion layer as a side branch after the extractor. Existing technologies for building CycleGAN architectures often rely on two completely independent generators (each with its own encoder and decoder). The data feature spaces of each encoder are isolated, inevitably resulting in strong speckle noise in the encoded ultrasound images. This method breaks down the encoding silos within the bidirectional generative network by constructing a shared modality-independent feature extractor in the bidirectional translation link, thus creating a unified, high-dimensional, and globally coherent network architecture for subsequent forced feature alignment and physical noise removal.

[0033] Specifically, the implementation principle described above is as follows: First, a shared modality-independent feature extractor deployed at the front end of the network architecture receives image data from the ultrasound and magnetic resonance domains in parallel throughout the entire timeframe, forcing these two distinct modalities to undergo dimensionality reduction and high-order feature encoding using the same set of convolutional kernel parameters. Furthermore, it should be noted that the translation efficiency of cross-modal medical images is limited by the difference in physical information between the source and target domains. By constructing a shared feature encoding network to achieve full-dimensional feature reduction and interception, the problem of modality information leakage caused by independent feature extraction in traditional dual encoders is overcome. Physical quantities reflecting the essence of medical anatomy, such as organ contours and lesion boundaries, are aggregated into a high-fidelity, highly coherent shared deep feature pool, providing a unified feature extraction model for subsequent decoding, rendering, and modality discrimination calculations.

[0034] S302. Input the ultrasound image training set into the forward generator, extract the anatomical structure feature map of the ultrasound image through the modality-independent feature extractor, and map the anatomical structure feature map of the ultrasound image from the ultrasound domain to the magnetic resonance domain through the magnetic resonance feature decoding mapping function to obtain a virtual magnetic resonance image. Input the magnetic resonance image training set into the reverse generator, extract the anatomical structure feature map of the magnetic resonance image through the modality-independent feature extractor, and map the anatomical structure feature map of the magnetic resonance image from the magnetic resonance domain back to the ultrasound domain through the ultrasound feature decoding mapping function to obtain a virtual ultrasound image. Furthermore, the cycle consistency loss is expressed as: ;in, Indicates through the forward generator and reverse generator The calculated pixel-level cycle consistency loss function, This represents ultrasound image samples in the ultrasound image training set. This represents magnetic resonance image samples in the magnetic resonance image training set. This represents the training set of ultrasound images. This represents the training set of magnetic resonance images. Represents the mathematical expectation. This indicates that ultrasound image samples are extracted from the ultrasound image training set. This indicates that magnetic resonance image samples are extracted from the magnetic resonance image training set. Represents a virtual magnetic resonance image. Represents a virtual ultrasound image. This represents the reconstructed ultrasound image. Represents the reconstructed magnetic resonance image. Represents norm distance.

[0035] Specifically, forward generator Responsible for processing the input low-contrast ultrasound images (source domain) Converted into high-quality virtual MRI images (target domain) To achieve the initial goal of structure-guided design, this invention will use a generator. The internal network architecture is innovatively decomposed into two cascaded sub-modules. The first part is the aforementioned "Modality-Agnostic Feature Extractor (MAFE)" module, whose purpose is to encode high-level anatomical features common to both ultrasound and MRI imaging modalities (such as the prostate capsule and the endometrial-myophyll junction of the uterus). The second part is the specific modality mapping function that follows the MAFE, which receives the pure anatomical features extracted by the MAFE and decodes and renders them to reflect the soft tissue grayscale contrast and high-frequency texture representation specific to the target MRI modality. Specifically, reverse generator Then, a reverse cross-modal translation task is performed, that is, the MRI image (target domain) is translated. Mapping back to ultrasound image (source domain) The key innovation of this invention lies in the reverse generator. With forward generator The same MAFE submodule is completely shared in terms of physical structure and parameters. This sharing mechanism ensures that both forward and reverse cross-modal translation must pass through a unified, standardized anatomical feature space. Following the shared MAFE, the reverse generator... A dedicated ultrasonic feature decoding mapping function is used to reconstruct structural features into images with ultrasonic physical properties; S303. Input the virtual MRI image into the inverse generator, reconstruct the reconstructed ultrasound image through inverse cyclic data, and extract the anatomical structure feature map of the reconstructed ultrasound image through the modality-independent feature extractor. Input the virtual ultrasound image into the forward generator, reconstruct the reconstructed MRI image through inverse cyclic data, and extract the anatomical structure feature map of the reconstructed MRI image through the modality-independent feature extractor. Calculate the absolute error distance between the anatomical structure feature map of the ultrasound image and the anatomical structure feature map of the reconstructed ultrasound image, and the absolute error distance between the anatomical structure feature map of the MRI image and the anatomical structure feature map of the reconstructed MRI image, and construct the modality-independent feature consistency loss. Furthermore, the modality-independent feature consistency loss is expressed as: ;in, This represents the modality-independent feature consistency loss. This represents a map showing the anatomical features of an ultrasound image. Map showing anatomical features in magnetic resonance imaging. This represents a map showing the anatomical features of a reconstructed magnetic resonance image. This represents a map showing the anatomical features of a reconstructed ultrasound image.

[0036] Specifically, this step addresses the structural distortion problem in medical cross-modal synthesis by directly implementing consistency constraints in a deep abstract feature space. Existing techniques for cyclic consistency constraints often rely on pixel-level reconstruction distance calculations. Pixel-level comparisons between ultrasound and MRI are highly susceptible to speckle noise and acoustic artifacts, resulting in a lack of structural constraints in higher-level calculations and a high risk of anatomical position shifts or "tissue illusions." This method, by bypassing shallow pixels and directly calculating errors at the level of the high-dimensional anatomical feature map output by the extractor, breaks the "visual deception trap" caused by traditional pixel-level constraints. This establishes a sub-millimeter-level structurally accurate and strongly topologically correlated feature alignment standard for subsequent medical diagnosis and surgical navigation.

[0037] Specifically, the implementation principle described above is as follows: First, by concatenating a forward generator and a backward generator, the ultrasound image is translated into a virtual magnetic resonance image and then reconstructed back into an ultrasound image. Simultaneously, both the original and reconstructed ultrasound images are fed back into a shared modality-independent feature extractor to extract their deep anatomical structural feature maps, and the absolute error distance between these two feature matrices is calculated. Furthermore, it should be noted that the structural fidelity of large medical organs is limited by feature drift during cross-domain transformations and real-time noise intervention. By using a consistent network in the feature space to achieve hard anchoring of deep topological information, this breaks the monitoring isolation of traditional CycleGANs that only focus on apparent pixel similarity. It aggregates deep physical features reflecting the true shape and positional relationships of pathological tissues into a high-fidelity, strongly constrained loss scalar, providing a robust mathematical foundation for forcibly preserving the patient's true anatomical appearance during subsequent network updates.

[0038] S304. Obtain anatomical structure feature maps from ultrasound images and magnetic resonance images to construct an anatomical feature map set. Calculate the probability that the anatomical feature map set comes from the ultrasound image training set or the magnetic resonance image training set through a neighborhood classifier. Construct a neighborhood discrimination loss based on binary cross-entropy. After multiplying the neighborhood discrimination loss by the inversion coefficient through a gradient inversion layer to perform a negative inversion, input it into the modality-independent feature extractor. Specifically, domain classifier (Domain Classifier) The domain classifier and gradient inversion layer (GRL) constitute the core driving force for achieving the "modality independent" characteristic of this invention. Connected separately to the shared MAFE module, its sole task is to receive the high-dimensional feature map extracted by MAFE and predict whether the feature map originally came from the ultrasound image (source domain). ) or MRI image (target domain) To eliminate modal differences in the features extracted by MAFE, this invention combines MAFE with a domain classifier. A gradient inversion layer (GRL) is embedded between them. During backpropagation, the GRL will adjust the classifier... The returned gradient is inverted (multiplied by a negative value) and then passed to MAFE. This means that the classifier... While striving to improve its accuracy in identifying modality sources, MAFE is forced to move towards "maximizing the confusion classifier". The parameters are updated according to the direction of the "judgment". Through this adversarial game, MAFE is ultimately forced to completely abandon the extraction of distinctive ultrasound speckle noise or MRI feature brightness, and can only extract core features that are insensitive to modality but highly sensitive to the underlying anatomical shape; Furthermore, the probability is expressed as: ;in, This represents the image input to the modality-independent feature extractor. Atlas of anatomical features Represents the parameters of the weight matrix. Indicates the bias term parameter, This represents the transpose of the weight matrix parameters. This represents the Sigmoid activation function, used to map the result of linear computation to the (0,1) interval, forming probability values. This indicates the current anatomical feature set under the influence of weight W and bias z. The probabilities derived from the ultrasound image training set or the magnetic resonance image training set; the neighborhood discrimination loss is expressed as: ;in, Indicates the domain for loss determination. Represents a domain classifier. Represents anatomical feature maps. This represents the total number of feature maps extracted in a single training epoch. Indicates the number of rounds in the current round. Image samples, Indicates the first One input image sample, A binary label indicating the true source of a sample modality, when the image originates from the ultrasound domain. When the image comes from the magnetic resonance domain , Represents the natural logarithm. Represents the current anatomical feature map The probability derived from the training set of ultrasound images or magnetic resonance images; Specifically, when backpropagating the error gradient through the Gradient Reversal Layer (GRL) to the MAFE, this invention does not use a fixed reversal coefficient, but proposes a nonlinear progressive dynamic scheduling strategy, wherein the reversal coefficient is expressed as: ;in, Indicates the dynamic reversal coefficient. Represents the natural constant An exponential function with base 0. This represents a pre-set constant for controlling the dynamic scheduling rate. This indicates the current training progress.

[0039] Specifically, It controls the rate constant of the scheduling. This represents the current training progress (i.e., the ratio of the current number of epochs to the total number of epochs). The function is a modified Sigmoid-type increasing curve, in the initial stage of training ( ), Approaching 0 allows the model to initially focus on learning basic image-to-image translation tasks and macroscopic contour features; as training steadily progresses ( (gradually increasing) The modal adversarial constraint is gradually introduced, increasing smoothly and non-linearly to near 1. This strategy effectively avoids network collapse caused by excessive modal stripping penalties in the early stages of training, ensuring absolute stability for training large-scale high-resolution medical images.

[0040] Specifically, this step addresses the persistent problem of difficult-to-eliminate physical noise specific to ultrasound images by deploying a dynamic gradient inversion game mechanism between the feature extractor and the domain classifier. Existing technologies, when performing cross-modal translation of images, largely rely on the network's spontaneous style transfer, failing to actively identify and discard physical interference noise. This results in ultrasound spots being directly and forcibly translated as false positives or artifacts on MRI images. This method, by introducing gradient-level adversarial inversion, eliminates noise interference during feature extraction, constructing denoised, high-frequency, and modally immune feature data for the subsequent generation of clean virtual MRI images.

[0041] Specifically, the implementation principle described above is as follows: First, a domain classifier deployed at the back end of the feature extractor assesses whether the current feature map originates from ultrasound or magnetic resonance imaging (MRI). Simultaneously, during backpropagation parameter updates, the gradient inversion layer intercepts the error gradient returned by the classifier, forcibly multiplies it by a negative coefficient, and continues propagating it. This forces the feature extractor to evolve towards maximizing the confusion of the classifier's judgment rather than conforming to its own during parameter updates. Furthermore, it should be noted that the conversion efficiency from ultrasound to MRI is limited by the strong interference of inherent speckle noise caused by Rayleigh scattering of ultrasound waves. The gradient inversion network achieves adversarial feature stripping across all dimensions, completely filtering out the modality-specific physical noise originally mixed in with the anatomical structure, thus obtaining a pure, high-fidelity bottom-level feature pool containing only anatomical shapes.

[0042] S305. Input the magnetic resonance image training set and the virtual magnetic resonance image obtained in step S202 into the magnetic resonance authenticity discriminator, and input the ultrasound image training set and the virtual ultrasound image obtained in step S202 into the ultrasound authenticity discriminator to construct magnetic resonance adversarial loss and ultrasound adversarial loss respectively. Specifically, visual realism discriminators include ultrasonic realism discriminators. Magnetic Resonance Authenticity Discriminator (Discriminators , It employs the advanced PatchGAN fully convolutional discriminator architecture. Discriminator The purpose is to examine the generator The generated virtual MRI images are distinguished from real clinical MRI images, thereby forcing the generator to work against real clinical MRI images through adversarial training. This synthesizes images that are visually indistinguishable from real MRI scans. Similarly, the discriminator... Used to distinguish between real ultrasound images and generator images. Synthesized ultrasound images ensure the physical coherence of the reverse translation link.

[0043] Furthermore, the magnetic resonance countermeasure loss and the ultrasonic countermeasure loss are respectively expressed as: ; ; in, Indicates magnetic resonance resistance loss, Indicates ultrasound counteracting loss, This represents a magnetic resonance imaging (MRI) authenticity discriminator. This indicates an ultrasound authenticity discriminator. Indicates judging the input image This represents a probability assessment value for a real magnetic resonance image. Indicates the judgment of virtual magnetic resonance images This represents a probability assessment value for a real magnetic resonance image. Indicates judging the input image This represents a probability assessment value for a real ultrasound image. Indicates the judgment of virtual ultrasound images The generator aims to minimize the loss to deceive the discriminator, which in turn aims to maximize the loss to identify fake images, given the probability assessment value of a real ultrasound image. Specifically, combating losses and The default is to use binary cross-entropy loss. If generator gradient vanishing or mode collapse issues are encountered when performing ultra-high-resolution MRI synthesis using higher-resolution ultrasound datasets (e.g., 512:512), the adversarial loss can be equivalently replaced with a bulldozer distance-based Wasserstein GAN loss with gradient penalty (WGAN-GP), or a least-squares adversarial loss (LSGAN). Both alternatives provide a smoother and more stable gradient flow for high-resolution synthesis.

[0044] S306. Input the magnetic resonance image training set into the forward generator and construct the identity mapping loss; Specifically, to further enhance the tonal realism of synthesized MRI images and suppress color distortions or spontaneous artifacts that do not conform to medical norms generated by the generator, this invention introduces an identity loss. That is, when the ultrasound is transferred to the MRI generator... When a real MRI image is directly input, the generator should not make any modifications and should output an image identical to the original. Figure 1 The image is consistent; Furthermore, the identity mapping loss is expressed as: ;in, Denotes the identity mapping loss function. This indicates the actual magnetic resonance image. The source data is directly input into the forward generator. The output image, This indicates the use of real ultrasound images. The source data is directly input into the reverse generator. The image output later.

[0045] S307. Based on the cycle consistency loss, modality-independent feature consistency loss, domain discrimination loss, magnetic resonance adversarial loss, ultrasonic adversarial loss, and identity mapping loss, a weighted aggregation calculation is performed according to the preset weight hyperparameters to construct the total loss function. The network parameters of the adversarial generative network are iteratively updated with the goal of minimizing the total loss function until the total loss function converges, thus obtaining the optimal adversarial generative network.

[0046] Furthermore, the total loss function is expressed as: ;in, Represents the total loss function. Indicates magnetic resonance resistance loss, Indicates ultrasound counteracting loss, This represents the loss of cycle consistency. This represents the modality-independent feature consistency loss. Indicates the domain for loss determination. This represents the loss from the identity mapping. This represents the first weighted hyperparameter, pre-set to control the strength of the cycle consistency loss and the mode-independent feature consistency loss in the total loss. This represents a pre-set second weighted hyperparameter used to control the relative strength of the domain discrimination loss and identity mapping loss in the total loss.

[0047] Specifically, the present invention sets The optimal value for is 10, because if The value is too low (e.g.) The network will be overly controlled in order to combat losses. and This method tends to generate extremely realistic MRI textures, but this often comes at the cost of sacrificing the original ultrasound organ morphology of the target image, easily causing lesion location deviation; if The value is too high (e.g.) If this happens, the network will fall into an excessive reconstruction self-preservation state. Due to the asymmetry in information content between ultrasound and MRI, the generator will refuse to perform sufficient cross-modal texture rendering, resulting in the synthesized virtual MRI still having the hazy feeling of ultrasound or noise residue. This invention sets The optimal value for is 5, because Used for balancing neighborhood classifiers In sync with the backbone network updates. If If the value is too large, the gradient reversal penalty passed to the classifier in the early stages of training will be too strong, causing the MAFE module to encounter gradient explosion or gradient vanishing, and failing to converge to learn the basic features; if If the size is too small, it will be unable to completely eliminate the speckle noise unique to ultrasound.

[0048] Specifically, this step targets the global optimization path of the adversarial generative network by deploying a multi-dimensional loss weighted aggregation optimization matrix based on dynamic hyperparameters during backpropagation. Existing techniques for training generative networks often rely on a single adversarial loss or a simple reconstruction loss. The various constraints are isolated and prone to gradient conflicts, leading to pattern collapse in upper-layer training or an inability to simultaneously maintain structural fidelity and color realism. This method, through the global physical interconnection and weighted game theory of multi-dimensional losses, constructs a smooth, stable parameter evolution path for subsequent model convergence, possessing the ability to find the global optimum.

[0049] Specifically, the implementation principle described above is as follows: First, by extracting the dynamic error scalars of cycle consistency loss, modality-independent feature consistency loss, neighborhood discrimination loss, magnetic resonance adversarial loss, ultrasonic adversarial loss, and identity mapping loss, these scalars are fused in multiple dimensions according to a preset weight system to construct a total loss scalar that covers the multi-task optimization features of the unit. Then, the gradient descent algorithm is used to update the weight parameters of each sub-network in parallel throughout the entire time period.

[0050] S4. Obtain the real ultrasound image to be converted, and perform ultrasound-to-MRI conversion through a trained generative adversarial network to generate the final MRI image.

[0051] The core design concept and innovation of this embodiment lies in introducing a specially designed shared "Modality-Agnostic Feature Extractor" (MAFE) into the traditional recurrent consistency generative adversarial network architecture. This is combined with a "Gradient Reversal Layer" (GRL) and a unique "Modality-Agnostic Feature Consistency Loss," forcing the neural network to actively strip away and discard the speckle noise specific to ultrasound and the grayscale distribution style specific to MRI during the intermediate stage of cross-modal translation. This forces precise alignment of pure anatomical structural content (such as organ contours, relative positions, lesion boundaries, etc.) in a deep, high-dimensional feature space. The overall task of this invention is to learn the mapping between two unpaired image domains: let the ultrasound image domain be... MRI image domain is The aim is to establish a method for translating real-time ultrasound images into anatomically accurate virtual MRI images with high quality.

[0052] Example 2

[0053] like Figure 2 As shown, as a preferred embodiment of the above embodiments, a system for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network is provided. This system is implemented based on any of the ultrasound-to-magnetic resonance image synthesis methods based on structure-guided recurrent consistency adversarial networks described above, and includes: The data acquisition and preprocessing module is used to acquire unpaired ultrasound and magnetic resonance images, preprocess the unpaired ultrasound and magnetic resonance images, and construct ultrasound image training sets and magnetic resonance image training sets based on the preprocessed ultrasound and magnetic resonance images, respectively. The model building and training module is used to build an adversarial generative network based on structure-guided cycle consistency constraints and modality-independent feature extractors, and to train the adversarial generative network based on ultrasound image training set and magnetic resonance image training set to obtain the trained adversarial generative network. The ultrasound-to-magnetic resonance module is used to acquire real ultrasound images to be converted, and then perform ultrasound-to-magnetic resonance processing through a trained generative adversarial network to generate the final magnetic resonance image.

[0054] Specifically, the implementation principle of the above embodiments is as follows: First, the data acquisition and preprocessing module deployed at the data front end acquires multi-source heterogeneous data of unpaired ultrasound and magnetic resonance images from a multi-center database in real time. To ensure the reliability of feature calculation and the security of tensor alignment of the underlying data in the deep neural network environment, the data acquisition and preprocessing module performs unified center cropping, resolution scaling and pixel value floating-point normalization on the image data through adaptive matrix transformation technology, and securely transmits it to the model building and training module. Then, after receiving the standardized preprocessed data, the model building and training module uses the shared modality-independent feature extractor (MAFE) based on the built-in forward and backward generator networks to perform cross-modal feature dimensionality reduction and extraction on the ultrasound and magnetic resonance data, and resolves the pure anatomical topological features that are independent of the interference of physical imaging equipment. Then, it uses the decoding mapping function of a specific modality to render virtual magnetic resonance images and virtual ultrasound images that reflect the original anatomical topological state. Next, the cyclic reconstruction and adversarial stripping module extracts the virtual image data output by the model building and training module. Utilizing the cyclic closed-loop reconstruction link it contains, it extracts high-dimensional features of the reconstructed image to calculate the modality-independent feature consistency distance. It dynamically generates constraint signals to force alignment of the underlying anatomical structures within the system and sends them down for execution. Simultaneously, based on the probability prediction results of the feature map modality source by the domain classifier, this module synchronously performs real-time adversarial loss calculation and backpropagates the penalty gradient after nonlinear negative inversion through the gradient inversion layer (GRL) upwards to the feature extraction node to forcibly strip modality-specific physical noise such as ultrasound spots. Finally, the loss joint and network optimization module receives various loss data generated by the above calculation process through the built-in hyperparameter weighting engine, intelligently analyzes the feature consistency loss, domain discrimination loss and adversarial and identity mapping loss generated by visual inspection, calls the gradient optimization interface of the underlying backpropagation, and outputs the optimal cross-modal generator model with dynamically converged weight parameters, distortion-free dissection and successful artifact removal to the application end through the optimization path of multi-dimensional loss aggregation. Furthermore, during the entire clinical application process of the system, the inference and computation logic of the ultrasound-to-MRI module is directly deployed to the edge workstation node of the operating room or ultrasound equipment. Utilizing the lightweight GPU acceleration chip built into the edge workstation, millisecond-level real-time ultrasound video stream cross-modal rendering instructions are processed locally. Based on the high-efficiency rendering interface deployed on the edge, visual data communication with the surgical display screen is maintained with extremely low latency. Thus, the physical architecture ensures the perfect synergy between the real-time response of highly complex intraoperative virtual MRI image synthesis and high-precision clinical lesion navigation.

[0055] Example 3

[0056] Building upon Examples 1 and 2, there exists a practical application scenario where the SG-CycleGAN ultrasound-to-MRI synthesis method and system of this invention are built and trained from scratch using a large, multi-center, unpaired medical image dataset. This scenario employs an ultrasound-to-MRI image synthesis method and system based on a structure-guided cyclic consistency adversarial network as described in the above examples, specifically including the following steps: Step 1: Data Acquisition and Standardization Preprocessing from Multiple Centers. To ensure that the anatomical features learned by the model are not affected by the bias of a specific model of ultrasound equipment in a single center, this embodiment extensively collects unpaired ultrasound and MRI datasets from multiple medical centers: 1. Source domain data (ultrasound) ): Two-dimensional transvaginal or transabdominal ultrasound (TVS / US) cross-sectional images of thousands of patients with confirmed diseases and healthy controls were extracted, totaling more than 36,000 ultrasound images. All images underwent a standardized image cropping procedure to remove non-diagnostic artifacts such as hospital logos and text parameter labels around the images.

[0057] 2. Target domain data (magnetic resonance) ): Preoperative multi-sequence MRI data of another group of patients (approximately 500, with patient IDs independent of the ultrasound group to meet unpaired training conditions) were independently and retrospectively collected, from which more than 34,000 high-contrast MRI slices, including axial T2-weighted (T2WI), were extracted.

[0058] Unified Normalization: To match the optimal tensor scale flow for the generative adversarial network, the two sets of images with different modalities are uniformly cropped and scaled to a fixed resolution matrix pixel size (e.g., 256:256), and their grayscale / intensity pixel values ​​are linearly transformed and normalized to the floating-point range of [-1,1] as the standard input format for the neural network.

[0059] Step 2: Specific Construction and Parameter Control of the Network Model Architecture To ensure that the system of this invention can be efficiently deployed and perform fast forward inference on the GPUs (such as NVIDIA RTX series graphics cards) of standard clinical imaging workstations in hospitals, strict parameter control was implemented when constructing the network topology. In this embodiment, a one-way generator (such as a generator for ultrasound to MRI) is used. The total number of parameters was precisely controlled at 213M (approximately 213 million).

[0060] The shared modality-independent feature extractor (MAFE) consists of a front-end extractor consisting of a large 7:7 receptive field convolutional layer (with a stride of 1 for initial feature aggregation), followed by two downsampling convolutional layers with a stride of 2 (to achieve spatial dimensionality reduction and receptive field expansion); then seven residual blocks containing instance normalization and non-linear ReLU activation are stacked to extract deep, high-order anatomical features.

[0061] Ultrasound feature decoding mapping function and magnetic resonance feature decoding mapping function: a shallow private branch design is adopted to maintain the equivalence and fairness of the number of parameters with the baseline model. The features extracted by MAFE are remapped to a synthetic image output with a resolution of 256:256 by using deconvolution (transposed convolution) or upsampling layer with subpixel convolution.

[0062] Domain classifier With discriminator Domain classifier Connected at the end of the MAFE residual block, it includes a global average pooling layer and two fully connected layers to achieve binary classification; the discriminator adopts a 5-layer PatchGAN convolutional architecture to evaluate the realism of local receptive field patches in the image.

[0063] Step 3: End-to-end closed-loop joint optimization training. The training process is executed on a high-performance GPU-accelerated workstation, following the minimax game alternating update strategy of Algorithm 1: Parameter initialization: Initialize the generator , Discriminator , and classifier Learnable weights. Setting GRL scheduler parameters. Let the training progress variable Initialize to 0. Set the weight of the total loss term. (Control loop and feature consistency) (Control adversarial domain discrimination and identity loss).

[0064] Batch data sampling: Set the batch size to 32. Data is sampled from the ultrasound dataset. and MRI set Micro-batch images are randomly selected from the data.

[0065] Discriminator update separately: Lock generator parameters. Calculate the virtual magnetic resonance image obtained from the generator's forward propagation. With virtual ultrasound images Using real and synthetic images, calculate and Adversarial loss is addressed by updating the discriminator using gradient ascent. and This maximizes its ability to detect forged images.

[0066] The core update for generator-feature consistency is: locking the discriminator. Perform a complete forward and cyclic backpropagation. Calculate the total generation loss. ;in We force a measurement and reduction of the absolute error distance between the MAFE depth features extracted from the ground truth map and the reconstructed map. The generator is updated using gradient descent via backpropagation. and The parameters are set so that the virtual image generated is sufficient to deceive the discriminator while maintaining the original anatomical appearance.

[0067] Domain classifier Adversarial updates against GRL, computational domain binary cross-entropy loss : The first step is to backpropagate and update the classifier in the conventional way. The weights are assigned to accurately identify the source of features.

[0068] The second step is to calculate the current GRL dynamic reversal coefficient: When the error gradient is determined by the classifier When the gradient propagates upward through the gradient inversion layer to the MAFE, it is forced to be multiplied by a negative value. This reverse-modified gradient is used to update the parameters of the shared MAFE, thereby achieving the game objective of forcing the extraction of "modality-independent" features.

[0069] Iteration and Convergence: The Adam stochastic gradient optimizer is used throughout the process (with an initial learning rate set). The training loop executes for approximately 200 epochs until the loss curves reach a dynamic convergence equilibrium point. Finally, a trained generator with millisecond-level forward inference capability is derived. The parameter model.

[0070] In order to objectively and scientifically demonstrate that the structure guidance mechanism of the present invention has achieved extremely significant technical progress compared with the prior art, and in response to industry bottlenecks such as "lack of anatomical consistency" and "smoothing of high-frequency details", this embodiment conducted a rigorous objective quantitative evaluation and comparison experiment on the quality of synthesized images under the same test conditions.

[0071] Comparative Experiment Design Description: A test set independently partitioned from the aforementioned multi-center dataset (completely unused during any model training phase) was used. To ensure absolute fairness in the performance comparison and avoid interference from differences in model size, the network layer depths of all current mainstream baseline models participating in the comparison were uniformly adjusted. This ensured that the neural network parameter counts of all generators, including standard CycleGAN, UNIT (Unsupervised Image-to-Image Translation), MUNIT (Multimodal Unsupervised Image-to-Image Translation), DCLGAN (Dual Contrastive Learning GAN), and the SG-CycleGAN of this invention, were strictly aligned to the same order of magnitude between 196M and 227M.

[0072] This test uses internationally recognized metrics for evaluating the image quality and distribution similarity of generative models: Fréchet Inception Distance (FID) and Kernel Inception Distance (KID). The lower the scores of these two metrics, the closer the artificial intelligence-generated virtual synthetic image is to the distribution of real clinical images in the target domain in terms of deep visual feature statistical distribution, and the more realistic the generated anatomical texture and physical texture.

[0073] Quantitative test result analysis: The ultrasound images of the test set were input into each converged generative model, and the quality comparison and evaluation data of the synthesized virtual images are shown in Table 1.

[0074] Table 1. Comparison of different cross-modal generative models in terms of image quality metrics.

[0075] The detailed data in Table 1 clearly demonstrates that existing cross-modal translation techniques exhibit severe limitations under the premise of strictly limiting parameter scale for fair comparison. The UNIT model has an FID score as high as 219.13, making it almost impossible to generate virtual images that conform to medical logic. The FID scores of the standard CycleGAN and MUNIT models hover between 133 and 165, proving that due to the lack of feature-level anatomical forced alignment constraints, these models generate images with a large amount of unfiltered ultrasound speckle noise residue and inevitably cause geometric distortion of local tissues. Even the DCLGAN model, which uses a relatively advanced bidirectional contrastive learning algorithm, can only achieve a minimum FID of 122.98. However, by introducing the original MAFE module and consistency feature loss, the FID distance of the SG-CycleGAN model in this invention drops dramatically to 73.25, and the KID drops to an extremely low level of 0.0636. This significant breakthrough proves from the underlying data logic that the gradient inversion mechanism of this invention can overcome the interference of ultrasound Rayleigh scattering speckle noise extremely efficiently, forcing the generated images to accurately fit the distribution of real high-contrast MRI images. Further feature space t-SNE clustering visualization analysis experiments show that the high-dimensional feature points of ultrasound and MRI extracted by the MAFE of this invention achieve a very high degree of overlap and fusion on the dimension-reduced two-dimensional plane, completely removing the modality-specific physical style imprint, confirming that this network successfully isolates and preserves the pure cross-modal general anatomical contour.

[0076] Example 4

[0077] like Figure 3 As shown, this embodiment proposes a terminal device for an ultrasound-to-magnetic resonance image synthesis method based on a structure-guided cyclic consistency adversarial network. The terminal device includes at least one memory, at least one processor, and a bus connecting different platform systems.

[0078] The memory may include readable media in the form of volatile memory, such as RAM 211 and / or cache memory, and may further include ROM 213.

[0079] The memory also stores a computer program that can be executed by a processor, causing the processor to perform any of the ultrasound-to-magnetic resonance image synthesis methods based on structure-guided recurrent consistency adversarial networks described in the embodiments of this application. The specific implementation and technical effects achieved are consistent with those described in the embodiments of the above methods, and some details will not be repeated here. The memory may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0080] Accordingly, the processor can execute the aforementioned computer programs, as well as executable programs / utilities.

[0081] A bus can represent one or more of several bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus structures.

[0082] The terminal device can also communicate with one or more external devices, such as keyboards, pointing devices, Bluetooth devices, etc., and with one or more devices capable of interacting with the terminal device, and / or with any device that enables the terminal device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed through I / O interfaces. Furthermore, the terminal device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter can communicate with other modules of the terminal device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the terminal device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0083] Example 5

[0084] like Figure 4 As shown, based on Embodiment 1, this embodiment proposes a computer-readable storage medium for an ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network. The computer-readable storage medium stores instructions that, when executed by a processor, implement any one of the aforementioned ultrasound-to-magnetic resonance image synthesis methods based on a structure-guided recurrent consistency adversarial network. Its specific implementation method and the achieved technical effects are consistent with those described in the embodiments of the above method, and some details will not be repeated.

[0085] This embodiment provides a program product for implementing the above-described method, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this embodiment, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0086] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to user computing devices via any type of network, including local area networks (LANs) or wide area networks (WANs), or they can be connected to external computing devices (e.g., via the Internet through an Internet service provider).

[0087] This invention is described from the perspectives of its intended use, effectiveness, progress, and novelty. Its practical and progressive features meet the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings are merely preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc., that are similar to or identical to those of this application, i.e., all equivalent substitutions or modifications made in accordance with the scope of this patent application, shall fall within the scope of protection of this patent application.

[0088] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network, characterized in that, Includes the following steps: S1. Acquire unpaired ultrasound and magnetic resonance images; S2. Preprocess the unpaired ultrasound and magnetic resonance images, and construct ultrasound image training sets and magnetic resonance image training sets respectively based on the preprocessed ultrasound and magnetic resonance images. S3. Based on the cycle consistency loss constraint and combined with the modality-independent feature extractor, an adversarial generative network based on structure-guided cycle consistency is constructed. The adversarial generative network is trained according to the ultrasound image training set and the magnetic resonance image training set to obtain the trained adversarial generative network. S4. Obtain the real ultrasound image to be converted, and perform ultrasound-to-MRI conversion through a trained generative adversarial network to generate the final MRI image.

2. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 1, characterized in that, Step S2 includes the following sub-steps: S201. Perform unified center cropping on unpaired ultrasound images and magnetic resonance images; S202. Scale the cropped ultrasound and magnetic resonance images to a preset fixed resolution matrix pixel size; S203. Based on a pre-set floating-point range, the grayscale pixel values ​​of the scaled ultrasound image and magnetic resonance image are linearly transformed and normalized until the floating-point range requirements are met; S204. Based on the ultrasound images and magnetic resonance images obtained in step S203, construct ultrasound image training sets and magnetic resonance image training sets.

3. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 1, characterized in that, Step S3 includes the following sub-steps: S301. Based on the cycle consistency loss constraint and combined with a modality-independent feature extractor, an adversarial generative network based on structure-guided cycle consistency is constructed. The adversarial generative network includes a forward generator, a backward generator, a visual realism discriminator, a domain classifier, and a gradient inversion layer. The forward generator includes a modality-independent feature extractor and a magnetic resonance feature decoding mapping function, and the backward generator includes a modality-independent feature extractor and an ultrasound feature decoding mapping function. The forward generator and the backward generator share the same modality-independent feature extractor. The visual realism discriminator includes a magnetic resonance realism discriminator and an ultrasound realism discriminator. S302. Input the ultrasound image training set into the forward generator, extract the anatomical structure feature map of the ultrasound image through the modality-independent feature extractor, and map the anatomical structure feature map of the ultrasound image from the ultrasound domain to the magnetic resonance domain through the magnetic resonance feature decoding mapping function to obtain a virtual magnetic resonance image. Input the magnetic resonance image training set into the reverse generator, extract the anatomical structure feature map of the magnetic resonance image through the modality-independent feature extractor, and map the anatomical structure feature map of the magnetic resonance image from the magnetic resonance domain back to the ultrasound domain through the ultrasound feature decoding mapping function to obtain a virtual ultrasound image. S303. Input the virtual MRI image into the inverse generator, reconstruct the reconstructed ultrasound image through inverse cyclic data, and extract the anatomical structure feature map of the reconstructed ultrasound image through the modality-independent feature extractor. Input the virtual ultrasound image into the forward generator, reconstruct the reconstructed MRI image through inverse cyclic data, and extract the anatomical structure feature map of the reconstructed MRI image through the modality-independent feature extractor. Calculate the absolute error distance between the anatomical structure feature map of the ultrasound image and the anatomical structure feature map of the reconstructed ultrasound image, and the absolute error distance between the anatomical structure feature map of the MRI image and the anatomical structure feature map of the reconstructed MRI image, and construct the modality-independent feature consistency loss. S304. Obtain anatomical structure feature maps from ultrasound images and magnetic resonance images to construct an anatomical feature map set. Calculate the probability that the anatomical feature map set comes from the ultrasound image training set or the magnetic resonance image training set through a neighborhood classifier. Construct a neighborhood discrimination loss based on binary cross-entropy. After multiplying the neighborhood discrimination loss by the inversion coefficient through a gradient inversion layer to perform a negative inversion, input it into the modality-independent feature extractor. S305. Input the magnetic resonance image training set and the virtual magnetic resonance image obtained in step S202 into the magnetic resonance authenticity discriminator, and input the ultrasound image training set and the virtual ultrasound image obtained in step S202 into the ultrasound authenticity discriminator to construct magnetic resonance adversarial loss and ultrasound adversarial loss respectively. S306. Input the magnetic resonance image training set into the forward generator and construct the identity mapping loss; S307. Based on the cycle consistency loss, modality-independent feature consistency loss, domain discrimination loss, magnetic resonance adversarial loss, ultrasonic adversarial loss, and identity mapping loss, a weighted aggregation calculation is performed according to the preset weight hyperparameters to construct the total loss function. The network parameters of the adversarial generative network are iteratively updated with the goal of minimizing the total loss function until the total loss function converges, thus obtaining the optimal adversarial generative network.

4. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 3, characterized in that, The cycle consistency loss is expressed as: ； in, Indicates through the forward generator and reverse generator The calculated pixel-level cycle consistency loss function, This represents ultrasound image samples in the ultrasound image training set. This represents magnetic resonance image samples in the magnetic resonance image training set. This represents the training set of ultrasound images. This represents the training set of magnetic resonance images. Represents the mathematical expectation. This indicates that ultrasound image samples are extracted from the ultrasound image training set. This indicates that magnetic resonance image samples are extracted from the magnetic resonance image training set. Represents a virtual magnetic resonance image. Represents a virtual ultrasound image. This represents the reconstructed ultrasound image. Represents the reconstructed magnetic resonance image. Represents norm distance.

5. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 4, characterized in that, The modality-independent feature consistency loss is expressed as: ； in, This represents the modality-independent feature consistency loss. This represents a map showing the anatomical features of an ultrasound image. Map showing anatomical features in magnetic resonance imaging. This represents a map showing the anatomical features of a reconstructed magnetic resonance image. This represents a map showing the anatomical features of a reconstructed ultrasound image.

6. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 4, characterized in that, The probability is expressed as: ； in, This represents the image input to the modality-independent feature extractor. Atlas of anatomical features Represents the parameters of the weight matrix. Indicates the bias term parameter, This represents the transpose of the weight matrix parameters. This represents the Sigmoid activation function, used to map the result of linear computation to the (0,1) interval, forming probability values. This indicates the current anatomical feature set under the influence of weight W and bias z. The probability derived from the training set of ultrasound images or magnetic resonance images; The domain discrimination loss is expressed as: ； in, Indicates the domain for loss determination. Represents a domain classifier. Represents anatomical feature maps. This represents the total number of feature maps extracted in a single training epoch. Indicates the number of rounds in the current round. Image samples, Indicates the first One input image sample, A binary label indicating the true source of a sample modality, when the image originates from the ultrasound domain. When the image comes from the magnetic resonance domain , Represents the natural logarithm. Indicates the current anatomical feature map The probability derived from the training set of ultrasound images or magnetic resonance images; The inversion coefficient is expressed as: ； in, Indicates the dynamic reversal coefficient. Represents the natural constant An exponential function with base 0. This represents a pre-set constant for controlling the dynamic scheduling rate. This indicates the current training progress.

7. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 4, characterized in that, The magnetic resonance countermeasure loss and the ultrasonic countermeasure loss are respectively expressed as: ； ； in, Indicates magnetic resonance resistance loss, Indicates ultrasound counteracting loss, This represents a magnetic resonance imaging (MRI) authenticity discriminator. This indicates an ultrasound authenticity discriminator. Indicates judging the input image This represents a probability assessment value for a real magnetic resonance image. Indicates the judgment of virtual magnetic resonance images This represents a probability assessment value for a real magnetic resonance image. Indicates judging the input image This represents a probability assessment value for a real ultrasound image. Indicates the judgment of virtual ultrasound images This is a probability assessment value for a real ultrasound image.

8. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 4, characterized in that, The identity mapping loss is expressed as: ； in, This represents the identity mapping loss function. This indicates the actual magnetic resonance image. The source data is directly input into the forward generator. The output image, This indicates the use of real ultrasound images. The source data is directly input into the reverse generator. The image output later.

9. The ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network according to claim 4, characterized in that, The total loss function is expressed as: ； in, Represents the total loss function. Indicates magnetic resonance resistance loss, Indicates ultrasound counteracting loss, This represents the loss of cycle consistency. This represents the modality-independent feature consistency loss. Indicates the domain for loss determination. This represents the loss from the identity mapping. This represents the first weighted hyperparameter, pre-set to control the strength of the cycle consistency loss and the mode-independent feature consistency loss in the total loss. This represents a pre-set second weighted hyperparameter used to control the relative strength of the domain discrimination loss and identity mapping loss in the total loss.

10. A system for synthesizing ultrasound-to-magnetic resonance images based on a structure-guided recurrent consistency adversarial network, the system being implemented based on the ultrasound-to-magnetic resonance image synthesis method based on a structure-guided recurrent consistency adversarial network as described in any one of claims 1-9, characterized in that, include: The data acquisition and preprocessing module is used to acquire unpaired ultrasound images and magnetic resonance images, and to preprocess the unpaired ultrasound images and magnetic resonance images. Based on the preprocessed ultrasound images and magnetic resonance images, ultrasound image training sets and magnetic resonance image training sets are constructed respectively. The model building and training module is used to build an adversarial generative network based on structure-guided cycle consistency constraints and modality-independent feature extractors, and to train the adversarial generative network based on ultrasound image training set and magnetic resonance image training set to obtain the trained adversarial generative network. The ultrasound-to-magnetic resonance module is used to acquire real ultrasound images to be converted, and then perform ultrasound-to-magnetic resonance processing through a trained generative adversarial network to generate the final magnetic resonance image.

Images