Virtual desktop ultrasonic image generation method based on handheld ultrasonic image

Abstract

The invention relates to the technical field of medical image generation processing, in particular to a virtual desktop ultrasonic image generation method based on a handheld ultrasonic image, and the method comprises the steps: constructing a training set and a test set; constructing a generator network; constructing a discriminator network; establishing a cyclic generative adversarial network; training the cyclic generative adversarial network; and outputting the virtual desktop ultrasonic image. According to the method, the palm ultrasonic image and the desk type ultrasonic image of the same part are matched and normalized, so that the uniformity and consistency of data are ensured, high-quality input is provided for subsequent network training, the diversity of a training set is enhanced through image expansion and normalization, the robustness and generalization ability of the network are improved, and the network training efficiency is improved. The method can better adapt to different input images, and in the data set division process, the reasonable training set and test set proportion is beneficial to avoiding over-fitting, so that the final model can be more practical and reliable in practical application.

Description

Technical Field

[0001] This invention relates to the field of medical image generation and processing technology, and in particular to a method for generating virtual desktop ultrasound images based on handheld ultrasound images. Background Technology

[0002] Ultrasound imaging is a non-ionizing radiation medical technology widely used in the examination of pregnant women and infants. Currently, ultrasound equipment on the market is mainly divided into desktop, portable, and handheld ultrasound devices. While handheld ultrasound devices are inexpensive and easy to carry, their image quality is relatively low, affecting the accuracy of clinical diagnosis. In recent years, deep learning technology, especially generative adversarial networks (GANs), has provided powerful support for medical image generation. Through deep learning, high-quality image conversion between different devices can be achieved, thereby improving the diagnostic value of low-quality images, expanding the application scenarios of handheld ultrasound, and further promoting its application in clinical practice.

[0003] Patent CN112990267B discloses a breast ultrasound imaging method, device, and storage medium based on a style transfer model. The method includes: acquiring a content-based breast ultrasound image of the object being tested, the content-based breast ultrasound image including at least one content-based breast image feature; automatically acquiring N style-based breast ultrasound images that are closest to the content-based breast ultrasound image, the style-based breast ultrasound image including at least one style-based breast ultrasound image feature and at least one style annotation feature; classifying the content-based breast image feature into the style-based breast ultrasound image feature classification; processing the input content-based breast ultrasound image and the N style-based breast ultrasound images using a trained style transfer neural network to obtain N style-transferred breast ultrasound images, the style-transferred breast ultrasound images including at least one converted style annotation feature and at least one converted breast ultrasound image feature; inputting the N style-transferred breast ultrasound images into a trained breast lesion classification network to obtain N classification results, averaging the N classification results to obtain the final breast lesion classification result.

[0004] Therefore, the breast ultrasound imaging method based on the style transfer model has the following problems: The method provides a dual-channel feature fusion network to realize style transfer of breast ultrasound images, but the method is only applicable to the conversion of imaging style, and has limited improvement on imaging quality. Moreover, the network structure is simple and the feature information used is limited. The style-transferred image is only used for the classification of breast lesions. Summary of the Invention

[0005] To address this, the present invention provides a virtual desktop ultrasound image generation method based on handheld ultrasound images, which overcomes the problems of insufficient diagnostic accuracy and limited application scenarios caused by the low imaging quality of handheld ultrasound devices in the prior art by using generative adversarial networks and self-attention mechanisms.

[0006] To achieve the above objectives, the present invention provides a method for generating virtual desktop ultrasound images based on handheld ultrasound images, comprising: Step S1: Acquire high-quality handheld ultrasound images and desktop ultrasound images, and perform preprocessing and data partitioning to construct a training set for network training and a test set for validation. Step S2: Construct a generator network based on nonlocal features according to the nonlocal feature extraction and self-attention enhancement mechanism; Step S3: Construct a discriminator network based on the multi-scale discrimination and global feature fusion mechanism; Step S4: Establish a cyclic generative adversarial network by combining the generator network and the discriminator network according to the bidirectional cyclic adversarial constraint and joint optimization mechanism; Step S5: Train the recurrent generative adversarial network using the images in the training set to obtain an optimized model; Step S6: Use the optimization model to process the handheld ultrasound images in the test set and output virtual desktop ultrasound images.

[0007] Further, step S1 includes: Step S11: Acquire handheld ultrasound images and desktop ultrasound images of the same location with consistent image pixels to form several matching image pairs; Step S12: Normalize all the matching image pairs according to a preset uniform scale so that the pixel values ​​of each matching image are in the range of [-1, 1], and then use rotation, scaling and cropping to expand in sequence to obtain several preprocessed images; Step S13: Divide all the preprocessed images into test sets and training sets using a random 1:9 ratio.

[0008] Further, step S2 includes: Step S21: Perform the first convolution operation on the input image, using a 3×3 convolution kernel to extract local features of 64 channels; Step S22: Embed a self-attention module on the local features, calculate the correlation weights between spatial locations of the feature map, and generate enhanced global features; Step S23: Perform two 3×3 convolution operations with a stride of 1 on the global features, and output deep semantic features with 128 channels and 256 channels in sequence. Step S24: Perform three deconvolution operations on the deep semantic features to gradually reduce the number of channels from 256 to 1, and output the target domain image through the Tanh activation function; Step S25: Repeat steps S21 to S24 to construct another generator to achieve the reverse domain transformation.

[0009] Further, in step S22, the structural order of the self-attention module is as follows: convolutional layer, global dependency enhancement layer, two convolutional layers and three deconvolutional layers, all with a kernel size of 3×3 and a stride of 1, and the number of feature maps for convolutional operations are 64, 128 and 256 respectively; the number of feature maps for deconvolutional operations are 128, 64 and 1 respectively.

[0010] Furthermore, the discriminator network structure constructed in step S3 is as follows: three convolutional layers, one fully connected layer, and one sigmoid activation function output. The kernel size of each convolutional layer is 3×3, the stride is 2, and the number of convolutional feature maps is 64, 128, and 256 respectively. The LeakyReLU activation function is used after each convolutional layer. The number of feature maps in the deconvolution operation is 128, 64, and 1 respectively.

[0011] Further, step S4 includes: Step S41: Construct a generator network GT for converting handheld ultrasound images to virtual desktop ultrasound images and a generator network GH for converting desktop ultrasound images to virtual handheld ultrasound images, wherein the generator network GT and the generator network GH have the same network structure. Step S42: Construct a discriminator network DT for distinguishing between virtual desktop ultrasound images and real desktop ultrasound images, and a discriminator network DH for distinguishing between virtual handheld ultrasound images and real handheld ultrasound images. Step S43: The generator network GT optimizes the generation of virtual desktop ultrasound images based on the feedback from the discriminator network DT, and the generator network GH optimizes the generation of virtual handheld ultrasound images based on the feedback from the discriminator network DH. Step S44: Alternately train multiple rounds until the virtual images output by the generator network GT and the generator network GH can no longer be distinguished as fake images by the discriminator, so as to establish the recurrent generative adversarial network.

[0012] Furthermore, in step S5, a hybrid loss function is used during network training. The hybrid loss function consists of four parts: adversarial loss, identity loss, cycle consistency loss, and structural similarity loss. The weight of the adversarial loss is 1.0, the weight of the cycle consistency loss is 10.0, the weight of the structural similarity loss is 5.0, and the weight of the identity loss is 0.5.

[0013] Furthermore, in step S5, the Adam algorithm is used to iteratively update the network model parameters during network training. In each iteration, 32 pairs of matching handheld ultrasound images and desktop ultrasound images from the training set are randomly selected as input, and the training period is 300 epochs.

[0014] Furthermore, the specific settings for network training include Adam algorithm parameter settings, data loading and iteration settings, training monitoring and verification settings, and hardware acceleration and parallel computing settings; The Adam algorithm parameter settings include initial learning rate, adjustment strategy, first moment estimation coefficients, second moment estimation coefficients, and numerical stability parameters. The initial learning rate is 0.0002, the adjustment strategy is to decay the learning rate every 100 epochs with a decay factor of 0.1, the first-order moment estimation coefficient is set to 0.5, the second-order moment estimation coefficient is set to 0.999, and the numerical stability parameter is set to 1e. -08 ; The data loading and iteration are set to randomly select 32 pairs of matching handheld ultrasound images and desktop ultrasound images from the training set as input in each iteration, and update the network model parameters using stochastic gradient descent. The training monitoring and verification is set up so that the loss function value on the verification set is calculated after each epoch during network training. If the verification loss no longer decreases after a preset number of consecutive epochs, the early stop mechanism is triggered to terminate the training in advance. The hardware acceleration and parallel computing are configured such that when the number of GPUs exceeds the preset number of GPUs, CUDA parallel computing technology is used to divide the training set data into multiple sub-batches and perform calculations on each GPU.

[0015] Furthermore, in step S6, the test data uses only the generator network GT in the network, inputting the handheld ultrasound image required for the test into the generator network GT, and outputting the virtual desktop ultrasound image.

[0016] Compared with existing technologies, the beneficial effects of this invention are that by combining generative adversarial networks (GANs) and nonlocal feature extraction techniques, it achieves high-quality conversion from handheld ultrasound images to virtual desktop ultrasound images. It fully utilizes global contextual information during image generation, effectively improving image detail and realism. Through multiple rounds of adversarial training, the generator and discriminator are continuously optimized, making the generated virtual images visually almost indistinguishable from real desktop ultrasound images. This not only improves the visualization effect of handheld ultrasound images but also provides image quality similar to desktop ultrasound in resource-limited environments, offering a more efficient and accurate solution for telemedicine, convenient diagnosis, and intelligent analysis of ultrasound images.

[0017] Furthermore, by matching and normalizing handheld and desktop ultrasound images of the same body part, data uniformity and consistency were ensured, providing high-quality input for subsequent network training. This processing step eliminated device differences between images, making the generated virtual desktop ultrasound images more accurate and realistic. Image augmentation and normalization enhanced the diversity of the training set, improved the network's robustness and generalization ability, and enabled it to better adapt to different input images. During the dataset partitioning process, a reasonable ratio of training to test sets helped avoid overfitting, improved the accuracy of model evaluation, and made the final model more practical and reliable in real-world applications.

[0018] Furthermore, by combining self-attention modules with convolutional operations, long-range dependencies in images can be effectively captured, thereby enhancing feature representation capabilities. This structure allows the generated virtual images to better preserve the details and textures of the original images and more accurately convert from handheld ultrasound images to desktop ultrasound images. Through multiple convolution and deconvolution operations, the model progressively extracts and optimizes deep semantic features, improving the precision and realism of image generation. This method not only improves the quality of image conversion but also enhances the model's adaptability and generalization ability, enabling it to better handle different types of input images, thus exhibiting higher stability and effectiveness in practical applications.

[0019] Furthermore, employing a self-attention module structure effectively enhances the generative network's ability to capture global dependencies in the image, thereby improving image detail and overall quality. Through the global dependency enhancement layer, the network can better understand and fuse long-distance features in the image, thus avoiding long-distance information that traditional convolutional neural networks might ignore. This ability to enhance features makes the generated virtual desktop ultrasound images visually more realistic and detailed, more accurately reflecting the details and structure of the desktop ultrasound image. In addition, the deconvolution operation progressively restores the spatial resolution of the image, helping to ensure that the generated image maintains high-quality output during reconstruction. In summary, the combination of the self-attention module structure enables the network to generate images more accurately and efficiently, contributing to improved diagnostic effectiveness and practicality of ultrasound images.

[0020] Furthermore, by constructing a discriminator network structure with three convolutional layers and employing 3×3 convolutional kernels and a stride of 2, local features in the input image can be effectively extracted, gradually deepening the understanding of image details. The LeakyReLU activation function is used after each convolutional layer, which helps improve the network's non-linear modeling ability, thereby enhancing the model's ability to recognize complex image patterns. The fully connected layers and the output of the sigmoid activation function in the network structure ensure that the discriminator can accurately distinguish between virtual and real images. In addition, the use of deconvolution operations effectively recovers image information, preserving the spatial structure and texture details of the image. This structural design not only improves the classification accuracy of the discriminator but also enhances the stability and robustness of the generative adversarial network, making the generated virtual images more realistic and reliable.

[0021] Furthermore, by constructing generator networks (GT and GH) for mutual conversion between different ultrasound image domains (handheld ultrasound images and desktop ultrasound images), high-quality image domain conversion can be achieved. The generator networks GT and GH share the same structural design, ensuring consistency and stability in cross-domain conversion. By constructing discriminator networks DT and DH, the differences between virtual and real images are effectively distinguished, thereby improving the realism of the generated images. During training, the generator networks continuously optimize the quality of virtual images through feedback from the discriminator, ensuring that the generated images gradually approach real images. In multiple rounds of alternating training, the game between the generator and the discriminator drives the continuous optimization of the generator networks until the generated virtual images can no longer be distinguished as fake images by the discriminator. This not only improves the quality and effect of image conversion but also enhances the network's generalization ability, ensuring its reliability and practicality in real-world applications.

[0022] Furthermore, by employing a hybrid loss function, combining adversarial loss, identity loss, cycle consistency loss, and structural similarity loss, the training process of the generator and discriminator can be comprehensively optimized, improving the image generation quality and the stability of the generation model. Adversarial loss ensures the realism of the generated images, preventing them from appearing overly "fake," while identity loss helps maintain the features of the original image, ensuring consistency in features between the transformed and original images. Cycle consistency loss forces the network to maintain consistency in image transformations in both directions, effectively preventing the loss of image information and error accumulation, ensuring that the generated virtual image retains the key details of the original image. Structural similarity loss further improves the structural similarity of the images, making the generated virtual image structurally closer to the real image. The reasonable weight allocation of each loss term makes the training process more balanced, and the final optimized model not only generates high-quality virtual images but also possesses good robustness and generalization ability.

[0023] Furthermore, iterative optimization using the Adam algorithm effectively accelerates the training process of the network model and improves convergence speed. The Adam algorithm automatically adjusts the learning rate of each parameter by combining first-order moment estimation and second-order moment estimation, reducing the complexity of manually adjusting the learning rate and improving training stability. Randomly selecting 32 pairs of matching images from the training set as input in each iteration increases data diversity, helping the model better learn the subtle differences between images and improving the generalization ability of the generative model. A training cycle of 300 epochs allows the model to be fully trained and the performance of the generator and discriminator to be gradually optimized, ensuring that the final output virtual image is of higher quality, with stronger expressiveness and realism, thus meeting the needs of practical applications.

[0024] Furthermore, the data loading and iteration settings utilize 32 pairs of matching images randomly selected from the training set, making the input data more diverse for each training iteration and enhancing the model's robustness. Stochastic gradient descent is employed to update network parameters, ensuring the effectiveness of each update and smooth model learning. The training monitoring and validation settings validate the model on the validation set at the end of each epoch, monitoring performance changes in real time. An early stopping mechanism effectively prevents overtraining and improves training efficiency. The hardware acceleration and parallel computing settings leverage GPU parallel computing technology to significantly accelerate large-scale data processing and model training efficiency, fully utilizing hardware resources to reduce computation time. Especially when there are sufficient GPUs, parallel data processing can be achieved, further improving training speed and large-scale data processing capabilities, thus making the training process more efficient and stable.

[0025] Furthermore, by using only the generator network (GT) to process handheld ultrasound images and convert them into virtual desktop ultrasound images during the testing phase, the system's efficiency and response speed can be significantly improved. Unlike the complex adversarial networks involved in the training process, the simplified processing during the testing phase reduces computational resource consumption, making the image generation process more efficient. Simultaneously, this process ensures that the generated virtual desktop ultrasound images accurately reflect the characteristics of desktop ultrasound images, meeting actual clinical needs, and eliminating the need for complex discriminator judgments, thus shortening processing time. This method not only reduces the complexity of system operation but also improves the real-time performance of image generation, facilitating the rapid acquisition of desired results in practical applications. Attached Figure Description

[0026] Figure 1 This is a flowchart of the virtual desktop ultrasound image generation method based on handheld ultrasound images in this embodiment; Figure 2 This is a flowchart of step S1 in this embodiment; Figure 3 This is a flowchart of step S2 in this embodiment; Figure 4 This is a flowchart of step S4 in this embodiment. Detailed Implementation

[0027] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0028] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0029] Please see Figure 1 As shown, it is a flowchart of the virtual desktop ultrasound image generation method based on handheld ultrasound images in this embodiment; This embodiment provides a method for generating virtual desktop ultrasound images based on handheld ultrasound images, including: Step S1: Acquire high-quality handheld ultrasound images and desktop ultrasound images, and perform preprocessing and data partitioning to construct a training set for network training and a test set for validation. Step S2: Construct a generator network based on nonlocal features according to the nonlocal feature extraction and self-attention enhancement mechanism; Step S3: Construct a discriminator network based on the multi-scale discrimination and global feature fusion mechanism; Step S4: Establish a cyclic generative adversarial network by combining the generator network and the discriminator network according to the bidirectional cyclic adversarial constraint and joint optimization mechanism; Step S5: Train the recurrent generative adversarial network using the images in the training set to obtain an optimized model; Step S6: Use the optimization model to process the handheld ultrasound images in the test set and output virtual desktop ultrasound images.

[0030] First, high-quality handheld and desktop ultrasound images are acquired, preprocessed, and divided into training and testing sets. Then, a generator network and a discriminator network based on non-local features are constructed, and combined using a recurrent generative adversarial network (GAN). Next, the network is trained using the training set to optimize the performance of the generator and discriminator, ultimately forming an optimized image generation model. Finally, this optimized model is used to process the handheld ultrasound images in the testing set to generate virtual desktop ultrasound images.

[0031] By combining Generative Adversarial Networks (GANs) and nonlocal feature extraction techniques, high-quality conversion from handheld ultrasound images to virtual desktop ultrasound images is achieved. Global contextual information is fully utilized during image generation, effectively improving image detail and realism. Through multiple rounds of adversarial training, the generator and discriminator are continuously optimized, making the generated virtual images visually almost indistinguishable from real desktop ultrasound images. This not only improves the visualization effect of handheld ultrasound images but also provides image quality similar to desktop ultrasound in resource-limited environments, offering a more efficient and accurate solution for telemedicine, convenient diagnosis, and intelligent analysis of ultrasound images.

[0032] Please continue reading. Figure 2 As shown, it is a flowchart of step S1 in this embodiment; Step S1 includes: Step S11: Acquire handheld ultrasound images and desktop ultrasound images of the same location with consistent image pixels to form several matching image pairs; Step S12: Normalize all the matching image pairs according to a preset uniform scale so that the pixel values ​​of each matching image are in the range of [-1, 1], and then use rotation, scaling and cropping to expand in sequence to obtain several preprocessed images; Step S13: Divide all the preprocessed images into test sets and training sets using a random 1:9 ratio.

[0033] A pre-defined uniform scale refers to scaling all matching images to the same size during image preprocessing to ensure consistency of input images during network training. The choice of uniform scale typically depends on the input requirements of the network model and the limitations of computational resources, and is usually set between 256×256 and 512×512 pixels. In this embodiment, the uniform scale is set to 256×256 pixels, which effectively balances image detail preservation and computational efficiency, ensuring that the image maintains good quality after scaling, while reducing computational burden and improving training speed.

[0034] By acquiring handheld and desktop ultrasound images of the same body part, ensuring consistent pixel settings, several matching image pairs were created. Next, these images were normalized according to a pre-defined uniform scale, scaling their pixel values ​​to the range of [-1, 1]. To increase data diversity, data augmentation techniques such as rotation, scaling, and cropping were employed to obtain multiple preprocessed images. Finally, following a random and 1:9 ratio, all preprocessed images were divided into test and training sets for subsequent model training and evaluation.

[0035] Specifically, the training and testing datasets are constructed with the following basic requirements: the handheld ultrasound images and desktop ultrasound images acquired through scanning must be from the same body part, with consistent image pixels, forming matched handheld and desktop ultrasound image pairs. Image preprocessing operations include: normalizing all images according to a uniform scale so that their pixel values ​​are within the range of [-1, 1], and using rotation, scaling, and cropping to augment the images. The data is divided randomly and according to a 1:9 ratio, with the image pairs forming the test set. and training set Test set It can be represented as: ,in This represents the i-th handheld ultrasound image. The training set represents the i-th desktop ultrasound image, with a total of I pairs of ultrasound images used for network testing. It can be represented as: ,in This represents the nth handheld ultrasound image. This represents the nth desktop ultrasound image. A total of N pairs of ultrasound images are used for network training.

[0036] By matching and normalizing handheld and desktop ultrasound images of the same body part, data uniformity and consistency were ensured, providing high-quality input for subsequent network training. This processing step eliminated device differences between images, making the generated virtual desktop ultrasound images more accurate and realistic. Image augmentation and normalization enhanced the diversity of the training set, improving the network's robustness and generalization ability, enabling it to better adapt to different input images. During dataset partitioning, a reasonable training and test set ratio helped avoid overfitting, improved the accuracy of model evaluation, and made the final model more practical and reliable in real-world applications.

[0037] Please continue reading. Figure 3 As shown, it is a flowchart of step S2 in this embodiment; Step S2 includes: Step S21: Perform the first convolution operation on the input image, using a 3×3 convolution kernel to extract local features of 64 channels; Step S22: Embed a self-attention module on the local features, calculate the correlation weights between spatial locations of the feature map, and generate enhanced global features; Step S23: Perform two 3×3 convolution operations with a stride of 1 on the global features, and output deep semantic features with 128 channels and 256 channels in sequence. Step S24: Perform three deconvolution operations on the deep semantic features to gradually reduce the number of channels from 256 to 1, and output the target domain image through the Tanh activation function; Step S25: Repeat steps S21 to S24 to construct another generator to achieve the reverse domain transformation.

[0038] Specifically, the structure of the generator network based on non-local features is as follows: First, a convolutional layer is used to obtain a preliminary feature map; then, the preliminary feature map is fed into a self-attention module to obtain a feature map fused using non-local similarity information; next, two convolutional layers are used to further extract the fused feature map; finally, three deconvolutional layers are used to generate the image. The kernel size of the convolutional layers and deconvolutional layers is 3×3, the stride is 1, and the number of feature maps for convolutional operations is 64, 128, and 256 respectively; the number of feature maps for deconvolution operations is 128, 64, and 1 respectively. All convolutions and deconvolutions are followed by activation functions; except for the last deconvolution which uses the Tanh activation function, the rest use the ReLU activation function.

[0039] The generator network extracts local features from the input image through multi-level convolution operations and introduces a self-attention module to model the spatial correlation of the feature maps, thereby enhancing the expressive power of global features. Furthermore, through deep semantic feature extraction and deconvolution operations, it gradually reconstructs the image details of the target domain and finally outputs a virtual image that is consistent with the distribution of the target domain (tabletop ultrasound image).

[0040] By combining self-attention modules with convolutional operations, long-range dependencies in images can be effectively captured, thereby enhancing feature representation capabilities. This structure allows the generated virtual images to better preserve the details and textures of the original images and more accurately convert from handheld ultrasound images to desktop ultrasound images. Through multiple convolution and deconvolution operations, the model progressively extracts and optimizes deep semantic features, improving the precision and realism of image generation. This method not only improves the quality of image conversion but also enhances the model's adaptability and generalization ability, enabling it to better handle different types of input images, thus exhibiting higher stability and effectiveness in practical applications.

[0041] Specifically, in step S22, the structure of the self-attention module is as follows: convolutional layer, global dependency enhancement layer, two convolutional layers and three deconvolutional layers, all with a kernel size of 3×3 and a stride of 1. The number of feature maps for convolutional operations are 64, 128 and 256 respectively; the number of feature maps for deconvolutional operations are 128, 64 and 1 respectively.

[0042] Specifically, the discriminator network structure constructed in step S3 is as follows: three convolutional layers, one fully connected layer, and one sigmoid activation function output. The kernel size of each convolutional layer is 3×3, the stride is 2, and the number of convolutional feature maps is 64, 128, and 256 respectively. The LeakyReLU activation function is used after each convolutional layer. The number of feature maps in the deconvolution operation is 128, 64, and 1 respectively.

[0043] The discriminator network employs a multi-level convolutional structure, which compresses the spatial size of the input image through progressive downsampling and extracts multi-scale features by combining the LeakyReLU activation function. Finally, it integrates global features through a fully connected layer and outputs a single-value discrimination result to distinguish the difference between the generated image and the real target domain image.

[0044] By constructing a discriminator network structure with three convolutional layers, employing 3×3 convolutional kernels and a stride of 2, local features in the input image can be effectively extracted, gradually deepening the understanding of image details. The LeakyReLU activation function is used after each convolutional layer, which helps improve the network's non-linear modeling ability, thereby enhancing the model's ability to recognize complex image patterns. The fully connected layers and the output of the sigmoid activation function in the network structure ensure that the discriminator can accurately distinguish between virtual and real images. Furthermore, the use of deconvolution operations effectively recovers image information, preserving the spatial structure and texture details of the image. This structural design not only improves the classification accuracy of the discriminator but also enhances the stability and robustness of the generative adversarial network, making the generated virtual images more realistic and reliable.

[0045] Please continue reading. Figure 4 As shown, it is a flowchart of step S4 in this embodiment; Step S4 includes: Step S41: Construct a generator network GT for converting handheld ultrasound images to virtual desktop ultrasound images and a generator network GH for converting desktop ultrasound images to virtual handheld ultrasound images, wherein the generator network GT and the generator network GH have the same network structure. Step S42: Construct a discriminator network DT for distinguishing between virtual desktop ultrasound images and real desktop ultrasound images, and a discriminator network DH for distinguishing between virtual handheld ultrasound images and real handheld ultrasound images. Step S43: The generator network GT optimizes the generation of virtual desktop ultrasound images based on the feedback from the discriminator network DT, and the generator network GH optimizes the generation of virtual handheld ultrasound images based on the feedback from the discriminator network DH. Step S44: Alternately train multiple rounds until the virtual images output by the generator network GT and the generator network GH can no longer be distinguished as fake images by the discriminator, so as to establish the recurrent generative adversarial network.

[0046] Specifically, the recurrent generative adversarial network based on nonlocal features includes two generators, GT and GH, with identical network structures as described in step 2. The generator GT utilizes handheld ultrasound images. Generate virtual desktop ultrasound images The generator GH utilizes desktop ultrasound images. Generate virtual handheld ultrasound images Meanwhile, the recurrent generative adversarial network based on nonlocal features includes two discriminators, DT and DH, with identical network structures, as described in step 3. Discriminator DT distinguishes between generated virtual desktop ultrasound images and real desktop ultrasound images, while discriminator DH distinguishes between generated virtual handheld ultrasound images and real handheld ultrasound images. The implementation process is as follows: first, the handheld ultrasound image... As input, the input generator GT generates virtual desktop ultrasound images. Then, the discriminator DT distinguishes and differentiates the generated virtual desktop ultrasound images. Compared with real desktop ultrasound images Next, using indistinguishable virtual desktop ultrasound images... As input, the generator GH is used to generate virtual handheld ultrasound images. Finally, the discriminator DH distinguishes and differentiates the generated virtual handheld ultrasound images. Compared to real handheld ultrasound images .

[0047] A bidirectional generator network (handheld → desktop, desktop → handheld) and a corresponding discriminator network are constructed. The authenticity of the generated image is constrained by adversarial loss, and the consistency of the input image and the reconstructed image is ensured by cycle-consistency loss, thus forming a closed-loop optimized adversarial training framework.

[0048] High-quality image domain conversion can be achieved by constructing generator networks (GT and GH) for mutual conversion between different ultrasound image domains (handheld ultrasound images and desktop ultrasound images). The generator networks GT and GH share the same structural design, ensuring consistency and stability in cross-domain conversion. By constructing discriminator networks DT and DH, the differences between virtual and real images are effectively distinguished, thereby improving the realism of the generated images. During training, the generator networks continuously optimize the quality of virtual images through feedback from the discriminator, ensuring that the generated images gradually approach real images. In multiple rounds of alternating training, the game between the generator and the discriminator drives the continuous optimization of the generator networks until the generated virtual images can no longer be distinguished as fake images by the discriminator. This not only improves the quality and effect of image conversion but also enhances the network's generalization ability, ensuring its reliability and practicality in real-world applications.

[0049] Specifically, in step S5, a hybrid loss function is used during network training. The hybrid loss function consists of four parts: adversarial loss, identity loss, cycle consistency loss, and structural similarity loss. The weight of the adversarial loss is 1.0, the weight of the cycle consistency loss is 10.0, the weight of the structural similarity loss is 5.0, and the weight of the identity loss is 0.5.

[0050] Specifically, as a further improvement of the present invention, in step 5, a hybrid loss function is used during network training. The hybrid loss function comprises four parts: adversarial loss, identity loss, cycle consistency loss, and structural similarity loss, wherein: Combating losses The calculation is as follows:

[0051] Loss of identity The calculation is as follows:

[0052] Cyclic consistent loss The calculation is as follows:

[0053] Structural similarity loss The calculation is as follows:

[0054] GT is a generator for generating virtual desktop ultrasound images from handheld ultrasound images, and GH is a generator for generating virtual handheld ultrasound images from desktop ultrasound images. These are real handheld ultrasound images. The data consists of real desktop ultrasound images, SSIM is the structural similarity calculation operator, and N is the total number of training images.

[0055] Finally, the hybrid loss function Represented as:

[0056] in, These are the weights for identity loss, cycle consistency loss, and structural similarity loss, respectively.

[0057] By employing a hybrid loss function, combining adversarial loss, identity loss, cycle consistency loss, and structural similarity loss, the training process of the generator and discriminator can be comprehensively optimized, improving image generation quality and the stability of the generation model. Adversarial loss ensures the realism of the generated images, preventing them from appearing overly "fake," while identity loss helps maintain the features of the original image, ensuring consistency in features between the transformed and original images. Cycle consistency loss forces the network to maintain consistency in image transformations in both directions, effectively preventing the loss of image information and error accumulation, ensuring that the generated virtual image retains key details of the original image. Structural similarity loss further improves the structural similarity of the images, making the generated virtual image structurally closer to the real image. The reasonable weight allocation of each loss term makes the training process more balanced, and the final optimized model not only generates high-quality virtual images but also possesses good robustness and generalization ability.

[0058] Specifically, in step S5, the Adam algorithm is used to iteratively update the network model parameters during network training. In each iteration, 32 pairs of matching handheld ultrasound images and desktop ultrasound images from the training set are randomly selected as inputs, and the training period is 300 epochs.

[0059] Iterative optimization using the Adam algorithm effectively accelerates the training process of the network model and improves convergence speed. By combining first-order and second-order moment estimation, the Adam algorithm automatically adjusts the learning rate of each parameter, reducing the complexity of manual learning rate adjustment and improving training stability. Randomly selecting 32 pairs of matching images from the training set as input in each iteration increases data diversity, helping the model better learn the subtle differences between images and improving the generalization ability of the generative model. A training cycle of 300 epochs allows the model to be fully trained and the performance of the generator and discriminator to be gradually optimized, ensuring higher quality, stronger expressiveness, and realism in the final output virtual images, thus meeting the needs of practical applications.

[0060] Specifically, the network training settings include Adam algorithm parameter settings, data loading and iteration settings, training monitoring and verification settings, and hardware acceleration and parallel computing settings; The Adam algorithm parameter settings include initial learning rate, adjustment strategy, first moment estimation coefficients, second moment estimation coefficients, and numerical stability parameters. The initial learning rate is 0.0002, the adjustment strategy is to decay the learning rate every 100 epochs with a decay factor of 0.1, the first-order moment estimation coefficient is set to 0.5, the second-order moment estimation coefficient is set to 0.999, and the numerical stability parameter is set to 1e. -08 ; The data loading and iteration are set to randomly select 32 pairs of matching handheld ultrasound images and desktop ultrasound images from the training set as input in each iteration, and update the network model parameters using stochastic gradient descent. The training monitoring and verification is set up so that the loss function value on the verification set is calculated after each epoch during network training. If the verification loss no longer decreases after a preset number of consecutive epochs, the early stop mechanism is triggered to terminate the training in advance. The hardware acceleration and parallel computing are configured such that when the number of GPUs exceeds the preset number of GPUs, CUDA parallel computing technology is used to divide the training set data into multiple sub-batches and perform calculations on each GPU.

[0061] By carefully configuring the Adam algorithm parameters, data loading and iteration strategies, training monitoring and verification mechanisms, and hardware acceleration and parallel computing, the efficiency and stability of the network training process were ensured. An initial learning rate of 0.0002, with a decay strategy every 100 epochs, effectively controlled the learning rate variation, avoiding overfitting or oscillations during training, thus accelerating model convergence. Reasonably set first and second moment estimation coefficients (0.5 and 0.999 respectively) improved the adaptability during optimization, ensuring the stability of parameter updates, while the numerical stability parameter 1e... -08 This effectively avoids the gradient explosion problem.

[0062] Data loading and iteration settings utilize 32 pairs of matching images randomly selected from the training set, making the input data more diverse for each training iteration and enhancing the model's robustness. Stochastic gradient descent is employed to update network parameters, ensuring the effectiveness of each update and smooth model learning.

[0063] The training monitoring and validation settings validate the validation set at the end of each epoch, monitor performance changes in real time during training, and the early stopping mechanism can effectively avoid overtraining and improve training efficiency.

[0064] Hardware acceleration and parallel computing setup significantly accelerates the efficiency of large-scale data processing and model training through GPU parallel computing technology, making full use of hardware resources to reduce computing time. Especially when there are enough GPUs, parallel data processing can be achieved, further improving training speed and large-scale data processing capabilities, thus making the training process more efficient and stable.

[0065] Specifically, in step S6, the test data uses only the generator network GT in the network. The handheld ultrasound image required for the test is input into the generator network GT, and the virtual desktop ultrasound image is output.

[0066] Specifically, only the generator GT in the network is used for testing data. The handheld ultrasound images required for the test are used. Input is fed into the generator GT, which outputs a virtual desktop ultrasound image. Simultaneously, the output virtual image can be compared with the real desktop ultrasound image. The generated virtual desktop ultrasound images were compared to evaluate their quality.

[0067] By using only the generator network (GT) to process handheld ultrasound images and convert them into virtual desktop ultrasound images during the testing phase, the system's efficiency and response speed can be significantly improved. Unlike the complex adversarial networks involved in the training process, the simplified processing during the testing phase reduces computational resource consumption, making the image generation process more efficient. Simultaneously, this process ensures that the generated virtual desktop ultrasound images accurately reflect the characteristics of desktop ultrasound images, meeting actual clinical needs, and eliminating the need for complex discriminator judgments, thus shortening processing time. This method not only reduces the complexity of system operation but also improves the real-time performance of image generation, facilitating the rapid acquisition of desired results in practical applications.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating virtual desktop ultrasound images based on handheld ultrasound images, characterized in that, include: Step S1: Acquire high-quality handheld ultrasound images and desktop ultrasound images, and perform preprocessing and data partitioning to construct a training set for network training and a test set for validation. Step S2: Construct a generator network based on nonlocal features according to the nonlocal feature extraction and self-attention enhancement mechanism; Step S3: Construct a discriminator network based on the multi-scale discrimination and global feature fusion mechanism; Step S4: Establish a cyclic generative adversarial network by combining the generator network and the discriminator network according to the bidirectional cyclic adversarial constraint and joint optimization mechanism; Step S5: Train the recurrent generative adversarial network using the images in the training set to obtain an optimized model; Step S6: Use the optimization model to process the handheld ultrasound images in the test set and output virtual desktop ultrasound images.

2. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, Step S1 includes: Step S11: Acquire handheld ultrasound images and desktop ultrasound images of the same location with consistent image pixels to form several matching image pairs; Step S12: Normalize all the matching image pairs according to a preset uniform scale so that the pixel values ​​of each matching image are in the range of [-1, 1], and then use rotation, scaling and cropping to expand in sequence to obtain several preprocessed images; Step S13: Divide all the preprocessed images into test sets and training sets using a random 1:9 ratio.

3. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, Step S2 includes: Step S21: Perform the first convolution operation on the input image, using a 3×3 convolution kernel to extract local features of 64 channels; Step S22: Embed a self-attention module on the local features, calculate the correlation weights between spatial locations of the feature map, and generate enhanced global features; Step S23: Perform two 3×3 convolution operations with a stride of 1 on the global features, and output deep semantic features with 128 channels and 256 channels in sequence. Step S24: Perform three deconvolution operations on the deep semantic features to gradually reduce the number of channels from 256 to 1, and output the target domain image through the Tanh activation function; Step S25: Repeat steps S21 to S24 to construct another generator to achieve the reverse domain transformation.

4. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, In step S22, the structure of the self-attention module is as follows: convolutional layer, global dependency enhancement layer, two convolutional layers and three deconvolutional layers. The kernel size of each convolutional layer is 3×3, the stride of each convolutional layer is 1, and the number of feature maps for the convolutional operation is 64, 128 and 256 respectively. The number of feature maps for the deconvolutional operation is 128, 64 and 1 respectively.

5. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, The discriminator network structure constructed in step S3 is as follows: three convolutional layers, one fully connected layer, and one sigmoid activation function output. The kernel size of each convolutional layer is 3×3, the stride is 2, and the number of convolutional feature maps is 64, 128, and 256 respectively. The LeakyReLU activation function is used after each convolutional layer. The number of feature maps in the deconvolution operation is 128, 64, and 1 respectively.

6. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, Step S4 includes: Step S41: Construct a generator network GT for converting handheld ultrasound images to virtual desktop ultrasound images and a generator network GH for converting desktop ultrasound images to virtual handheld ultrasound images, wherein the generator network GT and the generator network GH have the same network structure. Step S42: Construct a discriminator network DT for distinguishing between virtual desktop ultrasound images and real desktop ultrasound images, and a discriminator network DH for distinguishing between virtual handheld ultrasound images and real handheld ultrasound images. Step S43: The generator network GT optimizes the generation of virtual desktop ultrasound images based on the feedback from the discriminator network DT, and the generator network GH optimizes the generation of virtual handheld ultrasound images based on the feedback from the discriminator network DH. Step S44: Alternately train multiple rounds until the virtual images output by the generator network GT and the generator network GH can no longer be distinguished as fake images by the discriminator, so as to establish the recurrent generative adversarial network.

7. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, In step S5, a hybrid loss function is used during network training. The hybrid loss function consists of four parts: adversarial loss, identity loss, cycle consistency loss, and structural similarity loss. The weight of the adversarial loss is 1.0, the weight of the cycle consistency loss is 10.0, the weight of the structural similarity loss is 5.0, and the weight of the identity loss is 0.

5.

8. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, In step S5, the Adam algorithm is used to iteratively update the network model parameters during network training. In each iteration, 32 pairs of matching handheld ultrasound images and desktop ultrasound images in the training set are randomly selected as input, and the training period is 300 epochs.

9. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 8, characterized in that, The specific settings for network training include Adam algorithm parameter settings, data loading and iteration settings, training monitoring and verification settings, and hardware acceleration and parallel computing settings. The Adam algorithm parameter settings include initial learning rate, adjustment strategy, first moment estimation coefficients, second moment estimation coefficients, and numerical stability parameters. The initial learning rate is 0.0002, the adjustment strategy is to decay the learning rate every 100 epochs with a decay factor of 0.1, the first-order moment estimation coefficient is set to 0.5, the second-order moment estimation coefficient is set to 0.999, and the numerical stability parameter is set to 1e. -08 ; The data loading and iteration are set to randomly select 32 pairs of matching handheld ultrasound images and desktop ultrasound images from the training set as input in each iteration, and update the network model parameters using stochastic gradient descent. The training monitoring and verification is set up so that the loss function value on the verification set is calculated after each epoch during network training. If the verification loss no longer decreases after a preset number of consecutive epochs, the early stop mechanism is triggered to terminate the training in advance. The hardware acceleration and parallel computing are configured such that when the number of GPUs exceeds the preset number of GPUs, CUDA parallel computing technology is used to divide the training set data into multiple sub-batches and perform calculations on each GPU.

10. The method for generating virtual desktop ultrasound images based on handheld ultrasound images according to claim 1, characterized in that, In step S6, the test data uses only the generator network GT in the network. The handheld ultrasound image required for the test is input into the generator network GT, and the virtual desktop ultrasound image is output.

Images