Ultrasound image enhancement method and system based on multi-modal image migration, and medium
By employing multimodal image transfer technology and deep learning methods, precise registration and style transfer between ultrasound images and CT/MRI images are achieved, solving the problems of insufficient tissue contrast and spatial resolution in ultrasound images, and improving the quality and diagnostic accuracy of ultrasound images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- THE THIRD XIANGYA HOSPITAL OF CENT SOUTH UNIV
- Filing Date
- 2025-07-24
- Publication Date
- 2026-07-31
AI Technical Summary
In current medical imaging technologies, ultrasound imaging has shortcomings in terms of tissue contrast and spatial resolution, and cannot be accurately registered and style-transferred with CT/MRI images, which affects the accuracy of ultrasound-guided surgery.
By constructing a joint representation space from CT/MRI to ultrasound, and employing a multimodal image transfer method combined with deep learning-based cross-modal image enhancement technology, precise registration and style transfer between ultrasound and CT/MRI images are achieved, thereby improving the spatial resolution and tissue contrast of ultrasound images.
It achieves precise registration and style consistency conversion between ultrasound images and CT/MRI images, significantly improving the clarity of ultrasound images and the integrity of diagnostic information, and supporting real-time intraoperative navigation and clinical diagnosis.
Smart Images
Figure CN120852200B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the intersection of medical image computing and artificial intelligence, and more specifically, to a method, system, and medium for ultrasound image enhancement based on multimodal image transfer. Background Technology
[0002] Among current medical imaging technologies, CT and MRI can provide high spatial resolution anatomical information, but they cannot achieve real-time imaging due to limitations in their imaging principles. While ultrasound imaging has the advantage of real-time performance, it has significant shortcomings in terms of tissue contrast and spatial resolution.
[0003] Achieving precise registration and style transfer between ultrasound and CT / MRI images to obtain high-quality fused images is a pressing technical challenge in the field of medical image fusion. Solving this challenge is crucial for improving the precision of ultrasound-guided surgery. Summary of the Invention
[0004] The purpose of this invention is to address the problem of how to achieve accurate registration between ultrasound images and CT / MRI images. It proposes an ultrasound image enhancement method, system, and medium based on multimodal image transfer. The cross-modal image enhancement technology, which integrates physical imaging models and deep learning, constructs a joint representation space from CT / MRI to ultrasound, and transfers high-resolution anatomical information to the ultrasound imaging system, significantly improving the spatial resolution and tissue contrast of ultrasound images. It is suitable for scenarios such as intraoperative real-time navigation, interventional treatment, and clinical diagnosis.
[0005] The technical solution of this invention is: This invention provides an ultrasound image enhancement method based on multimodal image transfer, comprising: S1. Acquire multimodal imaging data, including ultrasound imaging data and CT / MRI imaging data; S2. Spatial alignment of ultrasound image data and CT / MRI image data is performed using the registration module to obtain registered CT / MRI image data; S3. The registered CT / MRI image data is converted into pseudo-ultrasound image data with an ultrasound style through the pseudo-ultrasound generation module. S4. The acquired ultrasound image data is enhanced using a trained dual-path anatomy-texture fusion enhancement network. The anatomical structure enhancement pathway extracts and reconstructs the anatomical structure map in the ultrasound image data, and the texture enhancement pathway generates a guided texture map that conforms to the anatomical structure constraints. Finally, the enhanced ultrasound image data is generated by fusion. The anatomical structure enhancement pathway is trained with pseudo ultrasound image data as the supervision signal, and the texture enhancement pathway is trained with the original ultrasound image data as the supervision signal.
[0006] Furthermore, S2 includes: S21. Perform affine transformation registration on ultrasound image data and CT / MRI image data, and output the affine transformation matrix through affine transformation to correct spatial differences. S22. An elastic registration method based on coordinate convolution is adopted. The dense displacement field is predicted by the trained elastic deformation network. The affine-corrected CT / MRI image data is deformed according to the dense displacement field to obtain the registered CT / MRI image data that is pixel-level aligned with the ultrasound image data.
[0007] Furthermore, when constructing the elastic deformation network, coordinate information is introduced to generate a dense displacement field; the affine-corrected CT / MRI image data is then deformed based on the dense displacement field to obtain deformed CT / MRI image data. The similarity between deformed CT / MRI and ultrasound images was evaluated using a loss function to optimize the elastic deformation network for image registration.
[0008] Furthermore, the elastic deformation network includes multiple coordinate convolutional layers for processing the spatial location information of voxels, including; Before processing each coordinate convolutional layer, the three-dimensional spatial coordinates (x, y, z) of each voxel in the input feature map are obtained, and the three-dimensional spatial coordinates (x, y, z) are normalized and scaled to the interval [-1, 1]. The normalized three-dimensional spatial coordinate data is used as an additional channel and concatenated with the original channel of the input feature map to generate a concatenated feature tensor. The concatenated feature tensors are convolved using coordinate convolution layers to generate a dense displacement field that is related to the voxel spatial location. ; .
[0009] Furthermore, the loss function includes a normalized mutual information term, a structural similarity index term, and a gradient difference constraint term; The overall content similarity between deformed CT / MRI image data and ultrasound image data is calculated by normalizing the mutual information term. The degree of local structural matching between deformed CT / MRI and ultrasound imaging data is assessed using a structural similarity index. The edge differences between the deformed CT / MRI and ultrasound images in each direction are calculated using gradient difference constraints to force alignment of the edge structures.
[0010] Furthermore, S3 includes: A recurrent generative adversarial network architecture is used to generate pseudo-ultrasound image data with an ultrasound image style by using registered CT / MRI image data as the source domain input. The pseudo-ultrasound image data is converted back to the source domain image data by inverse transformation, and the cycle consistency loss is calculated to preserve the anatomical structure map. The generator and discriminator are trained using adversarial loss and cycle consistency loss, and the final pseudo-ultrasound image data is output.
[0011] Furthermore, the recurrent generative adversarial network architecture includes: The first generator is used to convert registered CT / MRI image data into pseudo ultrasound image data; The first discriminator is used to determine the similarity between the generated pseudo ultrasound image data and the real ultrasound image data; The second generator is used to convert pseudo-ultrasound image data back to the CT / MRI image data domain; The second discriminator is used to determine the similarity between the converted CT / MRI image data and the registered CT / MRI image data.
[0012] Furthermore, S4 includes: S41. Input the ultrasound image data into the dual-path anatomical-texture fusion enhancement network, wherein the dual-path anatomical-texture fusion enhancement network architecture includes an anatomical structure enhancement pathway and a texture enhancement pathway; S42. The generation process of texture enhancement pathways is constrained through cross-pathway guidance mechanisms, and texture features are guided and constrained based on anatomical structure enhancement pathways. S43. The anatomical structure map output by the anatomical structure enhancement pathway and the guided texture map output by the texture enhancement pathway are weighted and fused by the adaptive fusion module to generate enhanced ultrasound image data.
[0013] Furthermore, in S4, the anatomical structures extracted and reconstructed from ultrasound imaging data via the anatomical enhancement pathway include: A deep encoder-decoder structure is used to extract features from the raw ultrasound image data; Deep semantic features are extracted from raw ultrasound image data during the encoding stage; In the decoding stage, deep semantic features are upsampled to restore spatial structure information. Based on the spatial structure information, an anatomical structure map is generated. At the same time, multi-scale features at different levels are extracted as guide feature maps to be output to the texture enhancement layer path of the corresponding scale.
[0014] Furthermore, in S4, the generation of guided post-texture maps that conform to anatomical constraints through the texture enhancement pathway includes; During the decoding stage of the texture enhancement pathway, guided feature maps from the corresponding layers of the anatomical structure enhancement pathway are received; At the corresponding layer of the texture enhancement path, modulation parameters of the same size as the guiding feature map, including scaling factor γ and offset β, are predicted by a convolutional network based on the guiding feature map. The feature map of the texture enhancement path is subjected to pixel-by-pixel numerical transformation using modulation parameters to generate a guided texture map.
[0015] Furthermore, the enhanced ultrasound image data generated by fusing in S4 includes: The structure map and the guided texture map are input into the adaptive fusion module; a spatial fusion weight map is learned through a convolutional network, and the spatial fusion weight map assigns a fusion weight to each pixel position; the structure map and the texture map are weighted and averaged according to the fusion weight to generate the enhanced ultrasound image data.
[0016] Furthermore, the dual-path anatomy-texture fusion enhancement network described in S4 is jointly trained in an end-to-end manner, and the loss function of the dual-path anatomy-texture fusion enhancement network consists of three weighted parts, including; Lstruct, a structural pathway loss algorithm, is trained using pseudo-ultrasound image data as supervised labels to minimize the difference between the anatomical structure map output by the anatomical structure enhancement pathway and the supervised labels. The texture path loss Ltexture is trained using the original ultrasound image data as supervised labels, minimizing the difference between the guided texture map output by the texture path and the supervised labels. The fusion output loss Ltotal is trained using pseudo-ultrasound image data as supervised labels to minimize the difference between the final output enhanced ultrasound image data and the supervised labels.
[0017] A system used in an ultrasound image enhancement method based on multimodal image transfer includes: The data acquisition module is used to acquire multimodal image data, including ultrasound image data and CT / MRI image data; The registration module is used to spatially align ultrasound image data and CT / MRI image data to obtain registered CT / MRI image data. The pseudo-ultrasound generation module is used to convert registered CT / MRI image data into pseudo-ultrasound image data with an ultrasound style. A dual-path anatomy-texture fusion enhancement network is used to enhance acquired ultrasound image data. The anatomical structure enhancement pathway extracts and reconstructs anatomical structures from the ultrasound image data, while the texture enhancement pathway generates guided texture maps that conform to anatomical constraints. Finally, the enhanced ultrasound image data is fused together. The anatomical structure enhancement pathway is trained using pseudo-ultrasound image data as the supervision signal, while the texture enhancement pathway is trained using the original ultrasound image data as the supervision signal.
[0018] A computer-readable storage medium having a computer program thereon, which, when executed, implements the ultrasound image enhancement method based on multimodal image transfer.
[0019] The beneficial effects of this invention are: This invention discloses an ultrasound image enhancement method, system, and medium based on multimodal image transfer. Addressing the challenges of spatial registration, style transfer, and image quality enhancement between ultrasound and CT / MRI images in various business scenarios, it proposes an integrated solution. This solution enables precise registration, style consistency transfer, and anatomical and textural enhancement of ultrasound and CT / MRI images, achieving cross-modal consistency and high-quality presentation of image data. This provides efficient support for precision medical diagnosis and effectively improves diagnostic accuracy.
[0020] This invention first achieves global spatial registration through affine transformation to generate a transformation matrix; then, it uses an elastic registration method based on coordinate convolution to predict dense displacement fields and complete pixel-level alignment; next, it utilizes a recurrent generative adversarial network to achieve style conversion from CT / MRI images to ultrasound images, and optimizes the generator and discriminator through adversarial loss and cycle consistency loss to ensure the authenticity of style conversion; finally, it extracts anatomical structure maps and multi-scale features through a dual-path anatomy-texture fusion enhancement network, generates guided texture maps based on multi-scale feature constraints, and adaptively fuses the anatomical structure maps and guided texture maps to generate enhanced ultrasound image data, significantly improving the clarity of ultrasound images and the completeness of diagnostic information.
[0021] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description
[0022] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.
[0023] Figure 1 A flowchart of an ultrasound image enhancement method based on multimodal image transfer according to the present invention is shown.
[0024] Figure 2 The flowchart illustrating the multi-step registration process performed by the registration module in this invention is shown.
[0025] Figure 3 The flowchart of generating pseudo ultrasound images based on a recurrent generative adversarial network in this invention is shown.
[0026] Figure 4 A schematic diagram of the training process of the dual-path anatomy-texture fusion enhancement network in this invention is shown.
[0027] Figure 5 This diagram illustrates a comparison of the results after ultrasound image data enhancement processing in this invention. Detailed Implementation
[0028] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0029] Example 1: Figure 1 A flowchart of an ultrasound image enhancement method based on multimodal image transfer according to the present invention is shown.
[0030] like Figure 1 As shown, this invention provides an ultrasound image enhancement method based on multimodal image transfer, which includes the following steps: S1. Acquire multimodal imaging data, including ultrasound imaging data and CT / MRI imaging data; S2. Spatial alignment of ultrasound image data and CT / MRI image data is performed using the registration module to obtain registered CT / MRI image data; S3. The registered CT / MRI image data is converted into pseudo-ultrasound image data with an ultrasound style through the pseudo-ultrasound generation module. S4. The acquired ultrasound image data is enhanced using a trained dual-path anatomy-texture fusion enhancement network. The anatomical structure enhancement pathway extracts and reconstructs the anatomical structure map in the ultrasound image data, and the texture enhancement pathway generates a guided texture map that conforms to the anatomical structure constraints. Finally, the enhanced ultrasound image data is generated by fusion. The anatomical structure enhancement pathway is trained with pseudo ultrasound image data as the supervision signal, and the texture enhancement pathway is trained with the original ultrasound image data as the supervision signal.
[0031] In this embodiment, spatial alignment is achieved through affine transformation and elastic registration to establish data pairing. A conditional generative adversarial network is used to generate pseudo-ultrasound image data with anatomical structural information. A supervised learning framework is then used to enhance the ultrasound images, ultimately achieving content enhancement and optimization of ultrasound image quality. Figure 5 This diagram illustrates a comparison of the results after ultrasound image data enhancement processing in this invention.
[0032] Figure 2 The flowchart illustrating the multi-step registration process performed by the registration module in this invention is shown.
[0033] In one example, such as Figure 2 As shown, S2 includes: S21. Perform affine transformation registration on ultrasound image data and CT / MRI image data, and output an affine transformation matrix to correct spatial differences, including position, rotation and scaling differences. S22. An elastic registration method based on coordinate convolution is adopted. The dense displacement field is predicted by the trained elastic deformation network. The affine-corrected CT / MRI image data is deformed according to the dense displacement field to obtain the registered CT / MRI image data that is pixel-level aligned with the ultrasound image data.
[0034] When constructing the elastic deformation network, coordinate information is introduced to generate a dense displacement field; the affine-corrected CT / MRI image data is deformed according to the dense displacement field to obtain deformed CT / MRI image data; the similarity between the deformed CT / MRI image data and the ultrasound image data is evaluated by a loss function to optimize the elastic deformation network to achieve image registration.
[0035] The elastic deformation network contains multiple coordinate convolutional layers for processing the spatial location information of voxels, and the operation process includes: Before processing each coordinate convolutional layer, the three-dimensional spatial coordinates (x, y, z) of each voxel in the input feature map are obtained, and the three-dimensional spatial coordinates (x, y, z) are normalized and scaled to the interval [-1, 1]. The normalized three-dimensional spatial coordinate data is used as an additional channel and concatenated with the original channel of the input feature map to generate a concatenated feature tensor. The concatenated feature tensor contains the original feature information and spatial location information. The concatenated feature tensors are convolved using coordinate convolution layers to generate a dense displacement field that is related to the voxel spatial location. , .
[0036] The loss function of the elastic deformation network includes a normalized mutual information term, a structural similarity index term, and a gradient difference constraint term. The overall content similarity between deformed CT / MRI image data and ultrasound image data is calculated by normalizing the mutual information term; it is applicable to the registration of different modalities such as CT and US images, and does not require the brightness values of the images to correspond directly, but rather measures their statistical dependence. The structural similarity index is used to assess the degree of local structural matching between deformed CT / MRI and ultrasound images. This term evaluates the alignment of the two images from the perspective of structural similarity and focuses on the matching of local structures compared to NMI. The gradient difference constraint calculates the edge differences between deformed CT / MRI and ultrasound images in each direction, forcing alignment of edge structures. The gradient difference constraint calculates and minimizes the difference in gradients (i.e., edges) between the two images in the x, y, and z directions to force alignment of edge structures in the images. This is very helpful in overcoming the common speckle noise interference in ultrasound images and achieving accurate matching of sharp edges.
[0037] In this embodiment, standard convolution possesses "translation invariance," meaning that the calculation method and result of convolution are the same regardless of the location of a feature in the image. However, this is precisely a weakness for the task of predicting displacement fields. Because in registration, the specific amount and direction of displacement of a voxel are strongly correlated with its absolute position within the anatomical structure.
[0038] Therefore, the core technical advantages brought by introducing coordinate convolution in this embodiment include: 1. Breaking translation invariance and enhancing spatial perception: By using coordinate information as explicit input, the network can learn the absolute spatial location of features. It can perceive the absolute spatial location of each feature point, even if it cannot process local texture information of the image. This greatly enhances the network's ability to model spatial geometric relationships, enabling it to learn position-dependent deformation patterns. This allows it to predict accurate and smooth displacement fields even in regions lacking texture information, significantly improving the accuracy and robustness of local elastic registration.
[0039] 2. Improved registration accuracy: Especially in anatomical regions lacking significant texture features or exhibiting symmetry (such as the liver or the interior of the prostate), standard convolutional networks are prone to producing blurry or unreasonable displacement predictions. Coordinate information provides the network with powerful additional cues, enabling the generation of a smoother, more anatomically realistic, denser displacement field D, thereby achieving more accurate pixel-level alignment.
[0040] Figure 3 The flowchart of generating pseudo ultrasound images based on a recurrent generative adversarial network in this invention is shown.
[0041] In one example, such as Figure 3 As shown, S3 includes: A recurrent generative adversarial network architecture is used to generate pseudo-ultrasound image data with an ultrasound image style by using registered CT / MRI image data as the source domain input. The pseudo-ultrasound image data is converted back to the source domain image data by inverse transformation, and the cycle consistency loss is calculated to preserve the anatomical structure map. The generator and discriminator are trained using adversarial loss and cycle consistency loss, and the final pseudo-ultrasound image data is output.
[0042] The recurrent generative adversarial network architecture includes: The first generator is used to convert the registered CT / MRI image data into pseudo ultrasound image data; the first discriminator is used to determine the similarity between the generated pseudo ultrasound image data and the real ultrasound image data; the second generator is used to convert the pseudo ultrasound image data back into the CT / MRI image data domain; the second discriminator is used to determine the similarity between the converted CT / MRI image data and the registered CT / MRI image data.
[0043] In this embodiment, a forward transformation (CT / MR → US) is performed by a first generator to generate a synthetic pseudo-ultrasound US from the input CT / MR image. A first discriminator determines whether it resembles a real US image. Then, a reverse transformation (US → CT / MR) is performed by a second generator to generate a synthetic CT / MR from the input pseudo-ultrasound US image. The discriminator determines whether it resembles a real CT / MR image. Finally, cyclic consistency training is performed. The two sets of images (ultrasound domain, MR domain, and CT domain) fed into CycleGAN do not need an absolutely precise correspondence. CycleGAN limits the generator to retain the features of the original domain images by adding "cyclic generation" and optimizing the consistency loss. This ensures that the anatomical information of the original CT / MRI is preserved while converting the image style.
[0044] This embodiment utilizes a cycle consistency loss model. First, the generator is fixed, and the discriminator is trained. Training is done using real ultrasound (US) and generated pseudo-ultrasound US; and using real CT / MR and generated CT / MR. Then, the discriminator is fixed, and the generator is trained, optimizing the first generator G and the second generator F to ensure that the generated pseudo-ultrasound US images can fool the discriminator while minimizing cycle consistency error. This ensures that even when data is not paired, G(x) retains the anatomical information of x, obtaining a pseudo-ultrasound image (Pseudo-US) that is highly consistent with the input CT / MR in anatomical structure but presents a stylistic representation with certain tissue structures.
[0045] In one example, S4 includes: S41. Input the ultrasound image data into the dual-path anatomical-texture fusion enhancement network, wherein the dual-path anatomical-texture fusion enhancement network architecture includes an anatomical structure enhancement pathway and a texture enhancement pathway; S42. The generation process of texture enhancement pathways is constrained through cross-pathway guidance mechanisms, and texture features are guided and constrained based on anatomical structure enhancement pathways. S43. The anatomical structure map output by the anatomical structure enhancement pathway and the guided texture map output by the texture enhancement pathway are weighted and fused by the adaptive fusion module to generate enhanced ultrasound image data.
[0046] Figure 4 The end-to-end training process of the dual-path anatomy-texture fusion enhancement network of the present invention is shown. The network architecture, data processing flow and the functional implementation of each module are highly consistent with the actual application scenario, ensuring that the trained model can be directly deployed and used.
[0047] In the dual-path anatomy-texture fusion enhancement network of this invention, the extraction and reconstruction of anatomical structure maps from ultrasound image data through the anatomical structure enhancement pathway includes: extracting features from the original ultrasound image data using a deep encoder-decoder structure; extracting deep semantic features from the original ultrasound image data during the encoding stage; upsampling the deep semantic features during the decoding stage to restore spatial structural information; generating an anatomical structure map based on the spatial structural information; and simultaneously extracting multi-scale features at different levels as guiding feature maps for output to the corresponding scale texture enhancement level pathway. The training of this pathway is supervised by pseudo-ultrasound image data generated by S3, minimizing the structural loss between the anatomical structure map and the supervision label to optimize the network and ensure the accuracy of the output anatomical structure.
[0048] In the dual-path anatomy-texture fusion enhancement network of the present invention, generating a guided post-texture map conforming to anatomical constraints through a texture enhancement pathway includes: in the decoding stage of the texture enhancement pathway, receiving a guided feature map from the corresponding layer of the anatomical enhancement pathway; at the corresponding layer of the texture enhancement pathway, predicting modulation parameters of the same size as the guided feature map, including a scaling factor γ and an offset β, using a convolutional network based on the guided feature map; and performing pixel-by-pixel numerical transformation processing on the feature map of the texture enhancement pathway using the modulation parameters to generate the guided post-texture map. The training of the guided post-texture map output by this pathway uses the original input ultrasound image US_low itself as a supervision label, and optimizes by minimizing the texture / style loss between the two to ensure that the synthesized texture conforms to the statistical characteristics of real ultrasound.
[0049] In the dual-path anatomy-texture fusion enhancement network of the present invention, the process of fusing and generating enhanced ultrasound image data includes: inputting the structural map and the guided texture map into an adaptive fusion module; learning a spatial fusion weight map through a convolutional network, wherein the spatial fusion weight map assigns a fusion weight to each pixel position; and performing a weighted average processing on the structural map and the texture map according to the fusion weight to generate the enhanced ultrasound image data.
[0050] The dual-path anatomy-texture fusion enhancement network of this invention is jointly trained in an end-to-end manner. The loss function of the dual-path anatomy-texture fusion enhancement network consists of three weighted parts: structural pathway loss Lstruct, which is trained using pseudo-ultrasound image data as a supervisory label to minimize the difference between the anatomical structure map output by the anatomical structure enhancement pathway and the supervisory label; texture pathway loss Ltexture, which is trained using the original ultrasound image data as a supervisory label to minimize the difference between the guided texture map output by the texture pathway and the supervisory label; and fusion output loss Ltotal, which is trained using pseudo-ultrasound image data as a supervisory label to minimize the difference between the final output enhanced ultrasound image data and the supervisory label.
[0051] In this embodiment, to address the challenges of traditional dual-path networks losing realistic texture during structure enhancement or introducing artifacts while preserving texture, a cross-path guided dual-path anatomy-texture fusion network is proposed. The complex enhancement task is decoupled into two independent yet collaborative sub-tasks: anatomical structure reconstruction and structure-guided texture synthesis. High-quality ultrasound images are ultimately generated through adaptive fusion. The anatomical structure enhancement pathway, acting as the backbone of structure extraction, is responsible for recovering clear and accurate macroscopic anatomical structures consistent with pseudo-ultrasound labels from low-quality ultrasound images. The cross-path guided texture pathway is responsible for synthesizing realistic ultrasound tissue textures, with the key being that the generation process is guided and constrained by the anatomical structure pathway in real-time and at multiple scales. Finally, adaptive fusion is performed, adaptively weighting and fusing the clean structure images output from the first two pathways with the guided texture images. After training, the dual-path anatomy-texture fusion enhancement network, as an independent enhancer, directly converts low-quality ultrasound images input from any direction into high-quality enhanced images, significantly improving the spatial resolution and tissue contrast of ultrasound images. This makes it suitable for scenarios such as intraoperative real-time navigation, interventional therapy, and clinical diagnosis.
[0052] Example 2: This application also provides a system for an ultrasound image enhancement method based on multimodal image transfer, comprising: The registration module is used to acquire multimodal image data, including ultrasound image data and CT / MRI image data, and to spatially align the ultrasound image data and CT / MRI image data to obtain registered CT / MRI image data. The pseudo-ultrasound generation module is used to convert registered CT / MRI image data into pseudo-ultrasound image data with an ultrasound style. A dual-path anatomy-texture fusion enhancement network is used to enhance acquired ultrasound image data. The anatomical structure enhancement pathway extracts and reconstructs anatomical structures from the ultrasound image data, while the texture enhancement pathway generates guided texture maps that conform to anatomical constraints. Finally, the enhanced ultrasound image data is fused together. The anatomical structure enhancement pathway is trained using pseudo-ultrasound image data as the supervision signal, while the texture enhancement pathway is trained using the original ultrasound image data as the supervision signal.
[0053] Example 3: This invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the ultrasound image enhancement method based on multimodal image migration.
[0054] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.
Claims
1. An ultrasound image enhancement method based on multi-modal image migration, characterized in that, include: S1. Acquire multimodal imaging data, including ultrasound imaging data and CT / MRI imaging data; S2. Spatial alignment of ultrasound image data and CT / MRI image data is performed using the registration module to obtain registered CT / MRI image data; S3. The registered CT / MRI image data is converted into pseudo-ultrasound image data with an ultrasound style through the pseudo-ultrasound generation module. S4. The dual-path anatomy-texture fusion enhancement network includes an anatomical structure enhancement pathway and a texture enhancement pathway, which are jointly trained in an end-to-end manner. Ultrasound image data is input into the dual-path anatomy-texture fusion enhancement network. The generation process of the texture enhancement pathway is constrained through a cross-path guidance mechanism. The texture features are guided and constrained based on the anatomical structure enhancement pathway. The anatomical structure map output by the anatomical structure enhancement pathway and the guided texture map output by the texture enhancement pathway are weighted and fused through an adaptive fusion module to generate enhanced ultrasound image data. Among them, the anatomical structure enhancement pathway is trained with pseudo ultrasound image data as the supervision signal, and the texture enhancement pathway is trained with the original ultrasound image data as the supervision signal. S5. The trained dual-path anatomy-texture fusion enhancement network is used to enhance the ultrasound image data acquired during the operation. The anatomical structure map in the ultrasound image data is extracted and reconstructed through the anatomical structure enhancement pathway, and the guided texture map that conforms to the anatomical structure constraint is generated through the texture enhancement pathway. Finally, the enhanced ultrasound image data is fused to generate the enhanced ultrasound image data.
2. The ultrasound image enhancement method based on multimodal image transfer as described in claim 1, characterized in that S2 include: S21. Perform affine transformation registration on ultrasound image data and CT / MRI image data, and output the affine transformation matrix through affine transformation to correct spatial differences. S22. An elastic registration method based on coordinate convolution is adopted. The dense displacement field is predicted by the trained elastic deformation network. The affine-corrected CT / MRI image data is deformed according to the dense displacement field to obtain the registered CT / MRI image data that is pixel-level aligned with the ultrasound image data.
3. The ultrasound image enhancement method based on multimodal image transfer as described in claim 2, characterized in that, When constructing the elastic deformation network, coordinate information is introduced to generate a dense displacement field; the affine-corrected CT / MRI image data is deformed according to the dense displacement field to obtain deformed CT / MRI image data. The similarity between deformed CT / MRI and ultrasound images was evaluated using a loss function to optimize the elastic deformation network for image registration.
4. The ultrasound image enhancement method based on multimodal image transfer as described in claim 2 or 3, characterized in that, The elastic deformation network contains multiple coordinate convolutional layers for processing the spatial location information of voxels, including; Before processing each coordinate convolutional layer, the three-dimensional spatial coordinates (x, y, z) of each voxel in the input feature map are obtained, and the three-dimensional spatial coordinates (x, y, z) are normalized and scaled to the interval [-1, 1]. The normalized three-dimensional spatial coordinate data is used as an additional channel and concatenated with the original channel of the input feature map to generate a concatenated feature tensor. The concatenated feature tensors are convolved using coordinate convolution layers to generate a dense displacement field that is related to the voxel spatial location. ; 。 5. The ultrasound image enhancement method based on multimodal image transfer as described in claim 3, characterized in that, The loss function includes a normalized mutual information term, a structural similarity index term, and a gradient difference constraint term; The overall content similarity between deformed CT / MRI image data and ultrasound image data is calculated by normalizing the mutual information term. The degree of local structural matching between deformed CT / MRI and ultrasound imaging data is assessed using a structural similarity index. The edge differences between the deformed CT / MRI and ultrasound images in each direction are calculated using gradient difference constraints to force alignment of the edge structures.
6. The ultrasound image enhancement method based on multimodal image transfer as described in claim 1, characterized in that S3 include: A recurrent generative adversarial network architecture is used to generate pseudo-ultrasound image data with an ultrasound image style by using registered CT / MRI image data as the source domain input. The pseudo-ultrasound image data is converted back to the source domain image data by inverse transformation, and the cycle consistency loss is calculated to preserve the anatomical structure map. The generator and discriminator are trained using adversarial loss and cycle consistency loss, and the final pseudo-ultrasound image data is output.
7. The ultrasound image enhancement method based on multimodal image transfer as described in claim 6, characterized in that... The recurrent generative adversarial network architecture includes: The first generator is used to convert registered CT / MRI image data into pseudo ultrasound image data; The first discriminator is used to determine the similarity between the generated pseudo ultrasound image data and the real ultrasound image data; The second generator is used to convert pseudo-ultrasound image data back to the CT / MRI image data domain; The second discriminator is used to determine the similarity between the converted CT / MRI image data and the registered CT / MRI image data.
8. The ultrasound image enhancement method based on multimodal image transfer as described in claim 1, characterized in that... In S5, the anatomical structure maps extracted and reconstructed from ultrasound imaging data through the anatomical structure enhancement pathway include: A deep encoder-decoder structure is used to extract features from the raw ultrasound image data; Deep semantic features are extracted from raw ultrasound image data during the encoding stage; In the decoding stage, deep semantic features are upsampled to restore spatial structure information. Based on the spatial structure information, an anatomical structure map is generated. At the same time, multi-scale features at different levels are extracted as guide feature maps to be output to the texture enhancement layer path of the corresponding scale.
9. The ultrasound image enhancement method based on multimodal image transfer as described in claim 1, characterized in that... In S5, the generation of guided post-texture maps that conform to anatomical constraints through the texture enhancement pathway includes: During the decoding stage of the texture enhancement pathway, guided feature maps from the corresponding layers of the anatomical structure enhancement pathway are received; At the corresponding layer of the texture enhancement path, modulation parameters of the same size as the guiding feature map, including scaling factor γ and offset β, are predicted by a convolutional network based on the guiding feature map. The feature map of the texture enhancement path is subjected to pixel-by-pixel numerical transformation using modulation parameters to generate a guided texture map.
10. The ultrasound image enhancement method based on multimodal image transfer as described in claim 1, characterized in that... In S5, the fused and enhanced ultrasound image data includes: The structure map and the guided texture map are input into the adaptive fusion module; a spatial fusion weight map is learned through a convolutional network, and the spatial fusion weight map assigns a fusion weight to each pixel position; the structure map and the texture map are weighted and averaged according to the fusion weight to generate the enhanced ultrasound image data.
11. The ultrasound image enhancement method based on multimodal image transfer as described in claim 1, wherein its features In S4, the dual-path anatomy-texture fusion enhancement network is jointly trained in an end-to-end manner, and the loss function of the dual-path anatomy-texture fusion enhancement network is composed of three weighted parts, including; Lstruct, a structural pathway loss algorithm, is trained using pseudo-ultrasound image data as supervised labels to minimize the difference between the anatomical structure map output by the anatomical structure enhancement pathway and the supervised labels. The texture path loss Ltexture is trained using the original ultrasound image data as supervised labels, minimizing the difference between the guided texture map output by the texture path and the supervised labels. The fusion output loss Ltotal is trained using pseudo-ultrasound image data as supervised labels to minimize the difference between the final output enhanced ultrasound image data and the supervised labels.
12. A system used in the ultrasound image enhancement method based on multimodal image transfer as described in any one of claims 1 to 11, characterized in that, include: The data acquisition module is used to acquire multimodal image data, including ultrasound image data and CT / MRI image data; The registration module is used to spatially align ultrasound image data and CT / MRI image data to obtain registered CT / MRI image data. The pseudo-ultrasound generation module is used to convert registered CT / MRI image data into pseudo-ultrasound image data with an ultrasound style. A dual-path anatomy-texture fusion enhancement network is used to enhance acquired ultrasound image data. The anatomical structure enhancement pathway extracts and reconstructs anatomical structures from the ultrasound image data, while the texture enhancement pathway generates guided texture maps that conform to anatomical constraints. Finally, the enhanced ultrasound image data is fused together. The anatomical structure enhancement pathway is trained using pseudo-ultrasound image data as the supervision signal, while the texture enhancement pathway is trained using the original ultrasound image data as the supervision signal.
13. A computer-readable storage medium, characterized in that, It contains a computer program, which, when executed, implements the ultrasound image enhancement method based on multimodal image migration as described in any one of claims 1 to 11.