A method and device for constructing a generative adversarial model for SWE image generation

Abstract

The present invention relates to a method and device for constructing a generative adversarial model for SWE image generation. The method comprises: preprocessing an acquired target image data set and dividing the data into a training set, a validation set, and a test set; constructing a generative adversarial model, wherein the generative adversarial model comprises a generator and a discriminator; using the BUMS image as input of the generator, the generator outputting a generated SWE image, extracting RGB three-channel images corresponding to the real SWE image and the generated SWE image, combining the RGB three-channel images, calculating a loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters according to the loss value by backpropagation; simultaneously using the generated SWE image and the real SWE image as input of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters according to the discrimination loss by backpropagation; and testing the generative adversarial model using the test set, and evaluating the generative adversarial model according to the test results.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for constructing a generative adversarial model for SWE image generation. Background Art

[0002] With the advancement of modern medicine, ultrasound elastography is widely used in clinical practice. Ultrasound elastography has evolved from the initial one-dimensional instantaneous ultrasound elastography to two-dimensional instantaneous elastography using acoustic radiation pulses, and the current shear wave elastography (SWE). SWE is achieved by emitting acoustic radiation to percuss the tissue to exert excitation. Using the Mach Cone principle, shear waves of sufficient intensity can be generated in the tissue. The shear waves are captured and tracked by an ultra-high-speed imaging system with a speed of up to 20,000 frames per second, achieving a measured shear wave velocity accuracy of 1 mm / s. The tissue elasticity map is displayed in real time using color coding technology. Since the shear wave propagation velocity varies in different tissues, SWE technology can be applied to the detection of lesions in organs such as the thyroid, breast, liver, and kidney.

[0003] The traditional encoder-decoder architecture maps input images to images of the same size, offering significant advantages in object detection and segmentation tasks. However, in SWE image generation, the encoder-decoder architecture lacks feedback on the generated image, hindering the generation of realistic and detailed images. Consequently, SWE images generated using only the encoder-decoder architecture exhibit blurriness. The emergence of generative adversarial networks (GANs) addresses this issue and has achieved promising results in various image generation tasks.

[0004] Existing generative adversarial networks (GANs) consist of a generator and a discriminator. The generator generates the target image, while the discriminator determines whether the input is a real image or a fake (generated image). However, directly applying this basic network structure to generate SWE images still suffers from poor performance. These issues primarily arise from the following: 1. Raw breast ultrasound (BMUS) images contain significant noise, which conventional generator networks cannot effectively detect, leading to insufficient feature extraction. 2. The red, blue, and green colors in breast color ultrasound (SWE) images represent breast tissue hardness information. Different regions of breast tissue (e.g., tumor areas, tumor margins, and normal tissue) have different hardness distributions, indicating an inherent correlation between the red, blue, and green colors. The color SWE images generated by conventional generators are independent of each other in their color channels, failing to explicitly model the inter-channel dependencies. 3. The convolution operation used in the discriminator fails to extract global image features, impacting discrimination accuracy.

[0005] The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered as having been recognized in any prior art. Summary of the Invention

[0006] In response to the technical problems existing in the prior art, the present invention provides a method and device for constructing a generative adversarial model for SWE image generation.

[0007] The technical solution of the present invention to solve the above technical problems is as follows:

[0008] In one aspect, the present invention provides a method for constructing a generative adversarial model for SWE image generation, comprising:

[0009] S1, preprocessing the acquired target image dataset and dividing it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images;

[0010] S2, constructing a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator;

[0011] S3, training and validating the generative adversarial model using the training set and validation set; the training and validation comprising: using the BUMS image as input to a generator, the generator outputting a generated SWE image, extracting RGB three-channel images corresponding to the real SWE image and the generated SWE image, combining the RGB three-channel images, calculating a loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters based on the loss value through backpropagation; and using the generated SWE image and the real SWE image as input to a discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters based on the discrimination loss through backpropagation;

[0012] S4: Test the generative adversarial model using the test set, and evaluate the generative adversarial model based on the test results.

[0013] Furthermore, the preprocessing of the acquired target image data set includes:

[0014] Synchronously selecting a region of interest (ROI) of the BUMS image and the real SWE image, and cropping the BUMS image and the real SWE image based on the region of interest (ROI);

[0015] For the cropped BUMS image and the real SWE image, the pixel value in the region of interest (ROI) is set to 255, and the pixel value in the non-region of interest (ROI) is set to 0 to obtain the corresponding MASK image;

[0016] The cropped BUMS image, the cropped real SWE image and the corresponding MASK image are integrated into one image in a preset order.

[0017] Furthermore, the cropping of the BUMS image and the real SWE image based on the region of interest ROI includes:

[0018] Generate a bounding rectangle containing the region of interest (ROI), and record the coordinates of the four vertices of the bounding rectangle;

[0019] The BUMS image and the real SWE image are cropped based on the coordinates of the four vertices of the circumscribed rectangular frame.

[0020] Furthermore, the generator is implemented based on the Unet++ network structure, including: a downsampling module, a feature fusion module and an upsampling module;

[0021] Using the downsampling module, downsampling the input BUMS image is performed to generate a multi-level feature map;

[0022] Utilizing the feature fusion module, performing a feature fusion operation on the multi-level feature map to generate feature fusion results at different levels;

[0023] By using the upsampling module, the feature fusion results of different levels are spliced with the feature maps of the corresponding levels and an upsampling operation is performed to map the BUMS image to a SWE image.

[0024] Furthermore, the discriminator includes multiple convolutional layers, each of which is connected to a CBAM module; the convolutional layer is used to extract features of the input image, and the CBAM module is used to weight the feature map in channel and spatial dimensions.

[0025] Furthermore, the CBAM module includes a channel attention module and a spatial attention module. The operations performed by the CBAM module are as follows:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] Where F is the input feature map of the CBAM module, Mc(F) represents the execution process of the channel attention module, Ms(F') represents the execution process of the spatial attention module, F'' represents the feature map output after processing by the CBAM module, MLP is a weight-sharing multi-layer perceptron, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, Conv The convolution kernel size is The convolution operation, Represents a matrix multiplication operation.

[0031] Furthermore, the loss value between the generated SWE image and the true SWE image includes image loss and discrimination loss; the image loss includes L1 loss and Vgg perceptual loss, and the discrimination loss is Gan loss; the calculation formula for the loss value between the generated SWE image and the true SWE image is as follows:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] Where N and i refer to the total number of pixels in the current image and the i-th pixel. To generate SWE images, is the real SWE image, is the RGB three-channel image generated by generating the SWE image, is an RGB three-channel image generated from a real SWE image, is the perceptual loss, To identify the loss, is the probability of the discriminator identifying the i-th pixel of the generated SWE image, is the identification criterion of the i-th pixel, is a matrix of all 1s, is the generator loss, are the L1 loss value weights for generating SWE images and R, G, and B three-channel images respectively;

[0040] Furthermore, the calculation formula of the discrimination loss of the discriminator for the input image is as follows:

[0041] ;

[0042] ;

[0043] ;

[0044] Where, They represent the discrimination loss of the discriminator for real SWE images and generated SWE images, and the discrimination standard of real SWE images is a matrix of all 1s, generating the identification criteria for the SWE image is an all-0 matrix.

[0045] Furthermore, evaluating the generative adversarial model based on the test results includes:

[0046] The structural similarity SSIM between the generated SWE image and the real SWE image is calculated according to the following formula:

[0047] ;

[0048] Where x and y represent the generated SWE image and the real SWE image respectively. are the average brightness values of the two images, are the contrast variances of the two images, is the structural covariance of the two images, are two constants;

[0049] The generative adversarial model is evaluated according to the size of the structural similarity SSIM value.

[0050] On the other hand, the present invention also provides a device for constructing a generative adversarial model for SWE image generation, comprising:

[0051] A raw data processing module preprocesses the acquired target image dataset and divides it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all contain BUMS images and real SWE images;

[0052] A model building module, which builds a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator;

[0053] A model training module is configured to train and verify the generative adversarial model using the training set and validation set; the BUMS image is used as the input of the generator, the generator outputs a generated SWE image, the loss value between the generated SWE image and the true SWE image is calculated, and the generator parameters are adjusted based on the loss value by backpropagation; the generated SWE image and the true SWE image are used as the input of the discriminator, the discrimination loss of the discriminator for the input image is calculated, and the discriminator parameters are adjusted based on the discrimination loss by backpropagation;

[0054] A model evaluation module tests the generative adversarial model using the test set and evaluates the generative adversarial model based on the test results.

[0055] The present invention has the following beneficial effects: it uses a generator to generate a color image and then establishes a connection between the generated image and the color channels by extracting images of the R, G, and B color channels of the color image. This method can more precisely control the details of each color channel, reduce color distortion and information loss, and improve the overall visual quality and realism of the image. The present invention uses a Unet++ network as a generator, and its rich skip connections alleviate the problem of insufficient feature extraction. In addition, the present invention uses an attention mechanism in the discriminator to weight the feature maps in the channel and spatial dimensions, preserving the global characteristics of the image to a certain extent.

[0056] Low-end ultrasounds lack SWE elastography (which can significantly improve the diagnostic efficacy of diseases such as thyroid and breast cancer), but are widely used in grassroots areas. The generation of high-fidelity color SWE images by this invention will help improve the accuracy of ultrasonic breast examinations in grassroots areas. The generated SWE can reduce the cost of patient examinations, save medical costs, and benefit patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flow chart of a method for constructing a generative adversarial model for SWE image generation provided by an embodiment of the present invention;

[0058] Figure 2 The original input image provided by the embodiment of the present invention;

[0059] Figure 3 A schematic diagram of an input image after data preprocessing provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the RGB three-channel generator model structure provided by an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of the structure of a discriminator model based on the attention mechanism provided in an embodiment of the present invention;

[0062] Figure 6 A visual comparison of SWE images generated by the method of the present invention and the Pix2Pix model provided in an embodiment of the present invention; (a) is a real BMUS image, (b) is a real SWE image, (c) is a SWE image generated by pix2pix, and (d) is a SWE image generated by the method proposed in this article;

[0063] Figure 7 A schematic diagram of the structure of a generative adversarial model construction device for SWE image generation provided by an embodiment of the present invention;

[0064] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0066] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the described features. In the description of this application, "plurality" means two or more, unless otherwise specifically specified.

[0067] In the description of this application, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art will recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0068] like Figure 1 As shown, an embodiment of the present invention provides a method for constructing a generative adversarial model for SWE image generation, comprising the following steps:

[0069] S1, preprocessing the acquired target image dataset and dividing it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all contain BUMS images and real SWE images.

[0070] When constructing a dataset, it is necessary to preprocess the dataset to eliminate the influence of images in different scenes and enhance the generalization ability of the model. In this embodiment, experts are first required to manually outline the region of interest (ROI) on the original BMUS image, and then crop the original BMUS image and SWE image around the ROI area. The original input image is as follows: Figure 1 Specifically,

[0071] First, save the coordinates of the four vertices of the original image and the coordinates of the suspected lesion region (ROI) as a JSON file. Then, crop the BMUS and SWE images based on the ROI. See Equations 1 and 2.

[0072] (1)

[0073] (2)

[0074] in Represents the matrix composed of four vertices in the left ROI area, Indicates the cropping operation of the fixed position of the left BMUS image, Represents the matrix composed of four vertices in the right ROI area, Indicates the cropping operation of the SWE image on the right at a fixed position.

[0075] Secondly, a black and white binary MASK image is generated based on the coordinates of the lesion area outlined by the expert, with the pixel value of the lesion area being 255 and the pixel value of the non-lesion area being 0.

[0076] Finally, the BMUS, SWE, and MASK images are sequentially integrated into one image for subsequent network training. Figure 3 shown.

[0077] In this embodiment, the breast ultrasound image dataset generated after preprocessing contains a total of 720 images, which are divided into a training set of 576 images, a validation set of 72 images, and a test set of 72 images.

[0078] S2, constructing a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator.

[0079] Existing generative adversarial networks (GANs) consist of a generator and a discriminator. The generator generates target images, while the discriminator determines whether the input is a real image or a fake (generated image). This basic network model is suitable for most image generation tasks, but it performs poorly when used to generate SWE images. Key limitations include the following: 1. The generator accepts a random noise vector, which reduces generation efficiency and quality and can lead to insufficient feature extraction. 2. The generated color SWE images are independent of each other in the channel dimension, failing to explicitly model inter-channel dependencies and failing to effectively generate highly realistic color SWE images. 3. The convolution operation used in the discriminator fails to extract global features, affecting discrimination accuracy. Therefore, improvements to existing GANs are needed.

[0080] Specifically, in this embodiment, the generator model is as follows Figure 4 As shown in the figure, it is implemented based on the Unet++ network structure, including: downsampling module, feature fusion module and upsampling module;

[0081] Using the downsampling module, downsampling the input BUMS image is performed to generate a multi-level feature map;

[0082] Utilizing the feature fusion module, performing a feature fusion operation on the multi-level feature map to generate feature fusion results at different levels;

[0083] By using the upsampling module, the feature fusion results of different levels are spliced with the feature maps of the corresponding levels and an upsampling operation is performed to map the BUMS image to a SWE image.

[0084] The discriminator model is Figure 5 As shown, it includes multiple convolutional layers, and each convolutional layer is connected to a CBAM module; the convolutional layer is used to extract the features of the input image, and the CBAM module is used to weight the feature map in the channel and spatial dimensions.

[0085] The CBAM module includes a channel attention module (CAM) and a spatial attention module (SAM). The operations performed by the CBAM module are as follows:

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] Where F is the input feature map of the CBAM module, Mc(F) represents the execution process of the channel attention module, Ms(F') represents the execution process of the spatial attention module, F'' represents the feature map output after processing by the CBAM module, MLP is a weight-sharing multi-layer perceptron, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, Conv The convolution kernel size is The convolution operation, Represents the matrix multiplication operation. The channel weight map is obtained by inputting the feature map F , then use Weight F to obtain feature map , and finally Weighted in the spatial dimension to obtain the final weighted graph .

[0091] S3. Training and validating the generative adversarial model using the training set and validation set; the training and validation include: using the BUMS image as the input of the generator, the generator outputting the generated SWE image, extracting the RGB three-channel images corresponding to the real SWE image and the generated SWE image, combining the RGB three-channel images, calculating the loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters by backpropagation according to the loss value; and simultaneously using the generated SWE image and the real SWE image as the input of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters by backpropagation according to the discrimination loss.

[0092] In a generative adversarial model, the generator is used to predict SWE images, and the discriminator is used to determine whether the input image is a true SWE image. The generator and discriminator are trained collaboratively to obtain high-quality SWE prediction images.

[0093] During the training and verification process of the generator, feature extraction is first performed on the input BMUS image and mapped to the generated SWE image. Then, the generated SWE image is extracted into an RGB three-channel image, and the real SWE image is extracted into an RGB three-channel image.

[0094] Specifically, feature extraction is performed on the input BMUS image and mapped to generate the SWE image, which includes three processes: downsampling, upsampling, and skip connection. The operational formula for extracting the feature map in the downsampling process is:

[0095] ;

[0096] in, is the input BMUS image, represents the BatchNorm2d layer, Represents the activation function ReLU, The convolution kernel size is The convolution operation. Represents the feature map obtained during the downsampling process. The operation of restoring the size of the feature map during the upsampling process is:

[0097] ;

[0098] in ConvTranspose2d represents the rank-transforming convolution operation, which can enlarge the width and height of the feature map. At the same time, the upsampling operation must be performed together with the feature fusion operation, which is also called a skip connection. Each feature fusion operation is:

[0099] ;

[0100] in The operation is to stitch multiple feature maps together. In this algorithm, feature fusion is performed by stitching channel dimensions, and then using The convolution kernel reduces the number of channels of the fused feature map.

[0101] Finally, the SWE image is generated by the following formula:

[0102] ;

[0103] in Represents the generation of SWE images.

[0104] The RGB three-channel image extraction formula is as follows:

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] in and These are the real SWE image and the RGB channel images of the generated SWE image. Extracting the three RGB channel images allows the deep learning model to learn more color details.

[0112] After extracting the RGB three-channel image, the loss value between the generated SWE image and the real SWE image is calculated by combining the RGB three-channel image, and the generator parameters are adjusted by backpropagation according to the loss value.

[0113] The loss between the generated SWE image and the true SWE image includes image loss and discrimination loss; the image loss includes L1 loss and Vgg perceptual loss, and the discrimination loss is Gan loss. The calculation formula for the loss between the generated SWE image and the true SWE image is as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] ;

[0120] ;

[0121] Where N and i refer to the total number of pixels in the current image and the i-th pixel; To generate SWE images, is the real SWE image, is the RGB three-channel image generated by generating the SWE image, It is an RGB three-channel image generated from a real SWE image; for perceived loss; It is the image identification loss, which acts on the generator to make the generated SWE image disturb the judgment of the discriminator; is the probability of the discriminator identifying the i-th pixel of the generated SWE image, is the identification criterion of the i-th pixel, is a matrix of all 1s, is the generator loss, The weights of the L1 loss values for generating the SWE image and the R, G, and B three-channel images are respectively used. The color distribution of the generated SWE image can be adjusted by adjusting the weight parameters. Specifically, in this embodiment, the initial value of the weight parameter is set to .

[0122] During the training and verification process of the discriminator, the original BMUS image and the real SWE image, as well as the original BMUS image and the generated SWE image are spliced respectively. The operation method is shown in the following formula:

[0123] .

[0124] The convolutional layer feature extraction formula is:

[0125] ;

[0126] Where x represents the input of the current layer, express The convolution operation, represents the InstanceNorm2d layer, Represents the Leaky ReLU activation function. Represents the calculated feature map.

[0127] The output of the discriminator is ;in, This is a feature padding operation that fills the edges of the feature map with 0 to adjust the size of the feature map.

[0128] The loss function of the discriminator is the mean square error loss, which is calculated as follows:

[0129] ;

[0130] ;

[0131] ;

[0132] in Represent the discrimination loss of the discriminator for real SWE images and generated SWE images, is the probability of the discriminator identifying the i-th pixel of the real SWE image, and the identification standard of the i-th pixel of the real image is , is a matrix of all 1s, and the identification criterion for the i-th pixel of the generated image is , is an all-0 matrix. is the loss of the discriminator, used for back propagation. The generator and discriminator in this algorithm model are trained simultaneously to obtain high-quality predicted images.

[0133] S4: Test the generative adversarial model using the test set, and evaluate the generative adversarial model based on the test results.

[0134] Structural Similarity (SSIM) is used to quantify the image generation effect. Structural similarity is a metric used to measure the similarity between two images. It takes into account three key aspects of the image: luminance, contrast, and structure. It provides an image similarity assessment method that is more consistent with human visual perception than traditional metrics.

[0135] Specifically, the structural similarity SSIM between the generated SWE image and the real SWE image is calculated according to the following formula:

[0136] ;

[0137] Where x and y represent the generated SWE image and the real SWE image respectively. are the average brightness values of the two images, representing the brightness components of the images, are the contrast variances of the two images, representing the contrast components of the images, is the structural covariance of the two images, which represents the structural components of the images. are two constants; usually .

[0138] The generative adversarial model is evaluated according to the size of the structural similarity SSIM value.

[0139] Pix2Pix is an image-to-image translation model based on cGANs, consisting of a generator and a discriminator. Table 1 shows the evaluation metrics of pix2pix and the proposed method on the test set.

[0140] Table 1

[0141]

[0142] Figure 6 This is a visual comparison of the SWE images generated by the method of the present invention and the Pix2Pix model.

[0143] Comparison shows that the existing pix2pix algorithm performs poorly in feature extraction. This method, using Unet++ as a generator, enhances feature extraction. It also incorporates channel attention and spatial attention into the discriminator network to weight the feature maps, preserving global features to a certain extent. Most importantly, the pix2pix algorithm ignores image color information when generating SWE images, resulting in unclear color. This method extracts R, G, and B color channel images from SWE images, establishing a connection between the SWE images and the color channels. By assigning different loss weights to the R, G, and B color channel images, the color distribution of the SWE images can be adjusted to produce vibrant SWE images.

[0144] like Figure 7 As shown, the present invention also provides a generative adversarial model construction device for SWE image generation, comprising:

[0145] A raw data processing module preprocesses the acquired target image dataset and divides it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all contain BUMS images and real SWE images;

[0146] A model building module, which builds a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator;

[0147] A model training module is configured to train and verify the generative adversarial model using the training set and validation set; the BUMS image is used as the input of the generator, the generator outputs a generated SWE image, the loss value between the generated SWE image and the true SWE image is calculated, and the generator parameters are adjusted based on the loss value by backpropagation; the generated SWE image and the true SWE image are used as the input of the discriminator, the discrimination loss of the discriminator for the input image is calculated, and the discriminator parameters are adjusted based on the discrimination loss by backpropagation;

[0148] A model evaluation module tests the generative adversarial model using the test set and evaluates the generative adversarial model based on the test results.

[0149] According to one aspect of the present disclosure, an electronic device is also disclosed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0150] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is also disclosed, wherein the storage medium stores computer instructions, and the computer instructions implement the above method when executed by a computer.

[0151] According to one aspect of the present disclosure, a computer program product is also disclosed, including a computer program, wherein:

[0152] The computer program implements the above method when executed by a processor.

[0153] refer to Figure 8 , a block diagram of an electronic device 600 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0154] like Figure 8 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0155] Multiple components within electronic device 600 are connected to I / O interface 605, including an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. Input unit 606 can be any type of device capable of inputting information into electronic device 600. Input unit 606 can receive input numeric or character information and generate key signal input related to user settings and / or function control of the electronic device. It can include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 607 can be any type of device capable of presenting information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 608 can include, but is not limited to, a magnetic disk or an optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks. It can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0156] The computing unit 601 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the image processing method. For example, in some embodiments, the image processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the image processing method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the image processing method by any other appropriate means (e.g., by means of firmware).

[0157] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0159] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0162] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0163] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0164] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. A method for constructing a generative adversarial model for SWE image generation, characterized in that: include: S1, preprocessing the acquired target image dataset and dividing it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images; S2, constructing a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator; S3, training and verifying the generative adversarial model using the training set and the validation set; The training and validation includes: using the BUMS image as input to a generator, the generator outputting a generated SWE image, extracting RGB three-channel images corresponding to the real SWE image and the generated SWE image, combining the RGB three-channel images, calculating a loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters based on the loss value through backpropagation; and using the generated SWE image and the real SWE image as input to a discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters based on the discrimination loss through backpropagation; S4: Test the generative adversarial model using the test set, and evaluate the generative adversarial model based on the test results.

2. The method according to claim 1, characterized in that The preprocessing of the acquired target image data set includes: Synchronously selecting a region of interest (ROI) of the BUMS image and the real SWE image, and cropping the BUMS image and the real SWE image based on the region of interest (ROI); For the cropped BUMS image and the real SWE image, the pixel value in the region of interest (ROI) is set to 255, and the pixel value in the non-region of interest (ROI) is set to 0 to obtain the corresponding MASK image; The cropped BUMS image, the cropped real SWE image and the corresponding MASK image are integrated into one image in a preset order.

3. The method according to claim 2, characterized in that The cropping of the BUMS image and the real SWE image based on the region of interest ROI includes: Generate a bounding rectangle containing the region of interest (ROI), and record the coordinates of the four vertices of the bounding rectangle; The BUMS image and the real SWE image are cropped based on the coordinates of the four vertices of the circumscribed rectangular frame.

4. The method according to claim 1, wherein The generator is implemented based on the Unet++ network structure, including: downsampling module, feature fusion module and upsampling module; Using the downsampling module, the input BUMS image is downsampled to generate a multi-level feature map; Utilizing the feature fusion module, performing a feature fusion operation on the multi-level feature map to generate feature fusion results at different levels; By using the upsampling module, the feature fusion results of different levels are spliced with the feature maps of the corresponding levels and an upsampling operation is performed to map the BUMS image to a SWE image.

5. The method according to claim 4, characterized in that The discriminator includes multiple convolutional layers, each of which is connected to a CBAM module; the convolutional layers are used to extract features of the input image, and the CBAM module is used to weight the feature map in channel and spatial dimensions.

6. The method according to claim 5, characterized in that The CBAM module includes a channel attention module and a spatial attention module. The operations performed by the CBAM module are as follows: ； ； ； ； Where F is the input feature map of the CBAM module, Mc(F) represents the execution process of the channel attention module, Ms(F') represents the execution process of the spatial attention module, F'' represents the feature map output after processing by the CBAM module, MLP is a weight-sharing multi-layer perceptron, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, Conv The convolution kernel size is The convolution operation, Represents a matrix multiplication operation.

7. The method according to claim 1, characterized in that The loss between the generated SWE image and the true SWE image includes image loss and discrimination loss; the image loss includes L1 loss and Vgg perceptual loss, and the discrimination loss is Gan loss. The calculation formula for the loss between the generated SWE image and the true SWE image is as follows: ； ； ； ； ； ； ； Where N and i refer to the total number of pixels in the current image and the i-th pixel. To generate SWE images, is the real SWE image, is the RGB three-channel image generated by generating the SWE image, is an RGB three-channel image generated from a real SWE image, is the perceptual loss, To identify the loss, is the probability of the discriminator identifying the i-th pixel of the generated SWE image, is the identification criterion of the i-th pixel, is a matrix of all 1s, is the generator loss, They are the L1 loss value weights for generating SWE images and R, G, and B three-channel images respectively.

8. The method according to claim 7, characterized in that The calculation formula of the discrimination loss of the discriminator for the input image is as follows: ； ； ； Where, They represent the discrimination loss of the discriminator for real SWE images and generated SWE images, and the discrimination standard of real SWE images is a matrix of all 1s, generating the identification criteria for the SWE image is an all-0 matrix.

9. The method according to claim 1, characterized in that The step of evaluating the generative adversarial model according to the test results includes: The structural similarity SSIM between the generated SWE image and the real SWE image is calculated according to the following formula: ； Where x and y represent the generated SWE image and the real SWE image respectively. are the average brightness values of the two images, are the contrast variances of the two images, is the structural covariance of the two images, are two constants; The generative adversarial model is evaluated according to the size of the structural similarity SSIM value.

10. A generative adversarial model construction device for SWE image generation, characterized in that: include: A raw data processing module preprocesses the acquired target image dataset and divides it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all contain BUMS images and real SWE images; A model building module, which builds a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator; A model training module is configured to train and verify the generative adversarial model using the training set and validation set; the BUMS image is used as the input of the generator, the generator outputs a generated SWE image, the loss value between the generated SWE image and the true SWE image is calculated, and the generator parameters are adjusted based on the loss value by backpropagation; the generated SWE image and the true SWE image are used as the input of the discriminator, the discrimination loss of the discriminator for the input image is calculated, and the discriminator parameters are adjusted based on the discrimination loss by backpropagation; A model evaluation module tests the generative adversarial model using the test set and evaluates the generative adversarial model based on the test results.

Images