Generative adversarial model construction method and device for SWE image generation

Abstract

The invention relates to a generative adversarial model construction method and device for SWE image generation. The method comprises the steps that an obtained target image data set is preprocessed and divided into a training set, a verification set and a test set; constructing a generative adversarial model, wherein the generative adversarial model comprises a generator and a discriminator; taking the BUMS image as the input of a generator, outputting a generated SWE image by the generator, extracting RGB three-channel images corresponding to a real SWE image and the generated SWE image, calculating a loss value between the generated SWE image and the real SWE image by combining the RGB three-channel images, and adjusting the parameters of the generator according to the loss value in a back propagation manner, so as to obtain the real SWE image. Meanwhile, taking the generated SWE image and the real SWE image as the input of a discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the parameters of the discriminator according to the back propagation of the discrimination loss; and testing the generative adversarial model by using the test set, and evaluating the generative adversarial model according to a test result.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for constructing a generative adversarial model for SWE image generation. Background Art

[0002] With the progress of modern medicine, ultrasound elastography is widely used in clinical practice. Ultrasound elastography has evolved from the initial one-dimensional instantaneous ultrasound elastography to two-dimensional instantaneous elastography using acoustic radiation pulses, and the current shear wave elastography (SWE). SWE is to generate shear waves of sufficient intensity in the tissue by emitting acoustic radiation to knock on the tissue and using the Mach Cone principle. The shear waves are captured and tracked by an ultra-high-speed imaging system of up to 20,000 frames / s, so that the accuracy of the measured shear wave velocity reaches 1mm / s, and the tissue elasticity map is displayed in real time using color coding technology. Since the shear wave propagation speed varies in different tissues, SWE technology can be used in the detection of lesions in organs such as the thyroid, breast, liver and kidney.

[0003] The traditional encoder-decoder structure can map the input image to an image of the same size, which has significant advantages in object detection and segmentation tasks. However, in the SWE image generation task, since the encoder-decoder structure cannot obtain "feedback" for the generated image and lacks a game process, it is impossible to generate more realistic and detailed images. Therefore, the SWE image generated using only the encoder-decoder structure has the defect of being blurred. The emergence of generative adversarial networks solves this problem and has achieved good results in various image generation tasks.

[0004] The existing generative adversarial network consists of a generator and a discriminator. The generator is used to generate the target image, and the discriminator is used to determine whether the input is a real image or a fake image (generated image). However, when this basic network structure is directly used to generate SWE images, there are still problems with poor results, mainly including the following points: 1. The original breast ultrasound image (BMUS) has a lot of noise, and the ordinary generator network cannot effectively identify the noise, resulting in insufficient feature extraction; 2. The red, blue, and green colors of the breast color ultrasound image (SWE) represent the hardness information of the breast tissue. The hardness distribution of different areas of the breast tissue (such as tumor area, tumor edge, normal tissue) is different, that is, there is an intrinsic connection between red, blue, and green. The color SWE images generated by the ordinary generator are independent of each other in color channels, and the dependency between channels is not explicitly modeled; 3. The convolution operation used by the discriminator cannot extract the global features of the image, which will affect the accuracy of identification.

[0005] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0006] In view of the technical problems existing in the prior art, the present invention provides a method and device for constructing a generative adversarial model for SWE image generation.

[0007] The technical solution of the present invention to solve the above technical problems is as follows: In one aspect, the present invention provides a method for constructing a generative adversarial model for SWE image generation, comprising: S1, preprocessing the acquired target image data set and dividing it into a training set, a validation set, and a test set; the target image data set includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images; S2, constructing a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator; S3, using the training set and the validation set to train and validate the generative adversarial model; the training and validation includes: taking the BUMS image as the input of the generator, the generator outputting the generated SWE image, extracting the RGB three-channel images corresponding to the real SWE image and the generated SWE image, respectively, combining the RGB three-channel images, calculating the loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters by back-propagation according to the loss value, and taking the generated SWE image and the real SWE image as the input of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters by back-propagation according to the discrimination loss; S4: Test the generative adversarial model using the test set, and evaluate the generative adversarial model based on the test results.

[0008] Furthermore, the preprocessing of the acquired target image data set includes: Synchronously selecting a region of interest ROI of the BUMS image and the real SWE image, and cropping the BUMS image and the real SWE image based on the region of interest ROI; For the cropped BUMS image and the real SWE image, the pixel value in the region of interest ROI is set to 255, and the pixel value of the non-region of interest ROI is set to 0, to obtain the corresponding MASK image; The cropped BUMS image, the cropped real SWE image and the corresponding MASK image are integrated into one image in a preset order.

[0009] Furthermore, the step of cropping the BUMS image and the real SWE image based on the region of interest ROI includes: Generate a bounding rectangle containing the region of interest ROI, and record the coordinates of four vertices of the bounding rectangle; The BUMS image and the real SWE image are cropped based on the coordinates of the four vertices of the circumscribed rectangular frame.

[0010] Furthermore, the generator is implemented based on the Unet++ network structure, including: a downsampling module, a feature fusion module and an upsampling module; Using the downsampling module, downsampling the input BUMS image to generate a multi-level feature map; Using the feature fusion module, a feature fusion operation is performed on the multi-level feature map to generate feature fusion results at different levels; By using the upsampling module, the feature fusion results of different levels are spliced ​​with the feature maps of the corresponding levels and an upsampling operation is performed to map the BUMS image to a SWE image.

[0011] Furthermore, the discriminator includes multiple convolutional layers, each of which is connected to a CBAM module; the convolutional layer is used to extract features of the input image, and the CBAM module is used to weight the feature map in channel and spatial dimensions.

[0012] Furthermore, the CBAM module includes a channel attention module and a spatial attention module, and the operations performed by the CBAM module are as follows: ; ; ; ; Where F is the input feature map of the CBAM module, Mc(F) represents the execution process of the channel attention module, Ms(F') represents the execution process of the spatial attention module, F'' represents the feature map output after being processed by the CBAM module, MLP is a weight-sharing multi-layer perceptron, AvgPool is an average pooling operation, MaxPool is a maximum pooling operation, Conv The convolution kernel size is The convolution operation, Represents a matrix multiplication operation.

[0013] Furthermore, the loss value between the generated SWE image and the real SWE image includes image loss and identification loss; the image loss includes L1 loss and Vgg perception loss, and the identification loss is Gan loss; the calculation formula of the loss value between the generated SWE image and the real SWE image is as follows: ; ; ; ; ; ; ; Where N and i refer to the total number of pixels in the current image and the i-th pixel. To generate SWE images, is the real SWE image, is the RGB three-channel image generated by generating the SWE image, is an RGB three-channel image generated from a real SWE image, is the perceived loss, To identify the loss, is the probability of the discriminator identifying the i-th pixel of the generated SWE image, is the identification criterion of the i-th pixel, is a matrix of all 1s, is the generator loss, They are the L1 loss value weights for generating SWE images and R, G, and B three-channel images respectively; Furthermore, the calculation formula of the discrimination loss of the discriminator for the input image is as follows: ; ; ; In the formula, They represent the discrimination loss of the discriminator for real SWE images and generated SWE images, and the discrimination standard of real SWE images is a matrix of all 1s, generating the identification criteria for the SWE image is a matrix of all 0s.

[0014] Further, the evaluating the generative adversarial model according to the test results includes: The structural similarity SSIM between the generated SWE image and the real SWE image is calculated according to the following formula: ; In the formula, x and y represent the generated SWE image and the real SWE image respectively. are the average brightness values ​​of the two images, are the contrast variances of the two images, is the structural covariance of the two images, are two constants; The generative adversarial model is evaluated according to the size of the structural similarity SSIM value.

[0015] On the other hand, the present invention also provides a generative adversarial model construction device for SWE image generation, comprising: The raw data processing module pre-processes the acquired target image data set and divides it into a training set, a validation set, and a test set; the target image data set includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images; A model building module, which builds a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator; A model training module, using the training set and the validation set to train and validate the generative adversarial model; using the BUMS image as the input of the generator, the generator outputs a generated SWE image, calculating the loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters by back-propagation according to the loss value; and using the generated SWE image and the real SWE image as the input of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters by back-propagation according to the discrimination loss; A model evaluation module tests the generative adversarial model using the test set and evaluates the generative adversarial model based on the test results.

[0016] The beneficial effect of the present invention is that the present invention generates a color image using a generator, and then establishes a connection between the generated image and the color channel by extracting images of the three color channels of R, G, and B of the color image. This method can more accurately control the details of each color channel, reduce color distortion and information loss, and improve the overall visual quality and realism of the image. The present invention uses the Unet++ network as a generator, and its rich jump connections alleviate the problem of insufficient feature extraction. In addition, the present invention uses an attention mechanism in the discriminator to weight the feature map in the channel and spatial dimensions, which retains the global features of the image to a certain extent.

[0017] Low-end ultrasound does not have the SWE elastography function (which can significantly improve the diagnostic efficiency of diseases such as thyroid and breast cancer), but is widely used in grassroots areas. The generation of high-fidelity color SWE images by the present invention will help improve the accuracy of ultrasonic breast examinations in grassroots areas; the generated SWE can reduce the cost of patient examinations, save medical costs, and benefit patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of a method for constructing a generative adversarial model for SWE image generation provided by an embodiment of the present invention; Figure 2 The original input image provided by the embodiment of the present invention; Figure 3 A schematic diagram of an input image after data preprocessing provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of the RGB three-channel generator model provided in an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a discriminator model based on an attention mechanism provided in an embodiment of the present invention; Figure 6 A visual comparison diagram of the SWE images generated by the method of the present invention and the Pix2Pix model provided in an embodiment of the present invention; wherein (a) is a real BMUS image, (b) is a real SWE image, (c) is a SWE image generated by pix2pix, and (d) is a SWE image generated by the method proposed in this article; Figure 7 A schematic diagram of the structure of a generative adversarial model building device for SWE image generation provided by an embodiment of the present invention; Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0020] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0021] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a method for constructing a generative adversarial model for SWE image generation, comprising the following steps: S1, preprocessing the acquired target image dataset and dividing it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images.

[0023] When constructing a dataset, the dataset needs to be preprocessed to eliminate the influence of the image in different scenes and enhance the generalization ability of the model. In this embodiment, experts are first required to manually outline the key region of interest (ROI) on the original BMUS image, and then crop the original BMUS image and SWE image around the ROI area. The original input image is as follows Figure 1 Specifically, First, save the coordinates of the four vertices of the original image and the coordinates of the suspected lesion area (ROI area) as a JSON file. Then crop the BMUS and SWE images according to the ROI area. See Formula 1 and Formula 2.

[0024] (1) (2) in Represents a matrix consisting of four vertices in the left ROI area. Indicates the cropping operation of the fixed position of the left BMUS image, Represents a matrix consisting of four vertices in the right ROI area. Indicates the cropping operation of the SWE image on the right at a fixed position.

[0025] Secondly, a black-and-white binary MASK image is generated based on the coordinates of the lesion area outlined by the expert, with the pixel value of the lesion area being 255 and the pixel value of the non-lesion area being 0.

[0026] Finally, the three images of BMUS, SWE and MASK are sequentially integrated into one image for subsequent network training. Figure 3 shown.

[0027] In this embodiment, the breast ultrasound image data set generated after preprocessing has a total of 720 images, which are divided into a training set of 576 images, a validation set of 72 images, and a test set of 72 images.

[0028] S2, constructing a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator.

[0029] The existing generative adversarial network consists of a generator and a discriminator. The generator is used to generate the target image, and the discriminator is used to determine whether the input is a real image or a fake image (generated image). This basic network model is suitable for most image generation tasks, but it is not effective when used to generate SWE images. The main shortcomings include the following: 1. The generator accepts random noise vectors, which reduces the generation efficiency and quality to a certain extent, and may lead to insufficient feature extraction; 2. The generated color SWE images are independent of each other in the channel dimension, and the dependency between channels is not explicitly modeled, so high-fidelity color SWE images cannot be effectively generated; 3. The convolution operation used by the discriminator cannot extract global features, which will affect the identification accuracy. Based on this, it is necessary to improve the existing generative adversarial network.

[0030] Specifically, in this embodiment, the generator model is as follows Figure 4 As shown, it is implemented based on the Unet++ network structure, including: downsampling module, feature fusion module and upsampling module; Using the downsampling module, downsampling the input BUMS image to generate a multi-level feature map; Using the feature fusion module, a feature fusion operation is performed on the multi-level feature map to generate feature fusion results at different levels; By using the upsampling module, the feature fusion results of different levels are spliced ​​with the feature maps of the corresponding levels and an upsampling operation is performed to map the BUMS image to a SWE image.

[0031] The discriminator model is Figure 5 As shown, it includes multiple convolutional layers, and each convolutional layer is connected to a CBAM module; the convolutional layer is used to extract the features of the input image, and the CBAM module is used to weight the feature map in the channel and spatial dimensions.

[0032] The CBAM module includes a channel attention module (CAM) and a spatial attention module (SAM). The operations performed by the CBAM module are as follows: ; ; ; ; Where F is the input feature map of the CBAM module, Mc(F) represents the execution process of the channel attention module, Ms(F') represents the execution process of the spatial attention module, F'' represents the feature map output after being processed by the CBAM module, MLP is a weight-sharing multi-layer perceptron, AvgPool is an average pooling operation, MaxPool is a maximum pooling operation, Conv The convolution kernel size is The convolution operation, Represents a matrix multiplication operation. The channel weight map is obtained by inputting the feature map F , then use Weight F to obtain the feature map , and finally Weighted in the spatial dimension to obtain the final weighted graph .

[0033] S3, using the training set and the validation set to train and validate the generative adversarial model; the training and validation includes: taking the BUMS image as the input of the generator, the generator outputting the generated SWE image, extracting the RGB three-channel images corresponding to the real SWE image and the generated SWE image, respectively, combining the RGB three-channel images, calculating the loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters by back-propagation according to the loss value, and taking the generated SWE image and the real SWE image as the input of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters by back-propagation according to the discrimination loss.

[0034] In the generative adversarial model, the generator is used to predict the SWE image, and the discriminator is used to determine whether the input image is a real SWE image. The generator and the discriminator are trained together to obtain high-quality SWE prediction images.

[0035] During the training and verification process of the generator, feature extraction is first performed on the input BMUS image and mapped to the generated SWE image. Then, the generated SWE image is extracted into an RGB three-channel image, and the real SWE image is extracted into an RGB three-channel image.

[0036] Specifically, feature extraction is performed on the input BMUS image and mapped to generate the SWE image, including three processes: downsampling, upsampling, and skip connection. The operation formula for extracting the feature map in the downsampling process is: ; in, is the input BMUS image, represents the BatchNorm2d layer, represents the activation function ReLU, The convolution kernel size is The convolution operation. represents the feature map obtained during the downsampling process. The operation of restoring the size of the feature map during the upsampling process is: ; in ConvTranspose2d represents the rank-transforming convolution operation, which can enlarge the width and height of the feature map. At the same time, the upsampling operation must be performed together with the feature fusion operation, which is also called skip connection. Each feature fusion operation is: ; in The operation is to splice multiple feature maps together. In this algorithm, feature fusion is performed by splicing channel dimensions, and then using The convolution kernel reduces the number of channels of the fused feature map.

[0037] Finally, the SWE image is generated by the following formula: ; in Represents the generation of SWE images.

[0038] The RGB three-channel image extraction formula is as follows: ; ; ; ; ; ; in and They are the real SWE image and the RGB channel images of the generated SWE image. Extracting the images of the three RGB channels allows the deep learning model to learn more color details.

[0039] After extracting the RGB three-channel image, the loss value between the generated SWE image and the real SWE image is calculated by combining the RGB three-channel image, and the generator parameters are adjusted by backpropagation according to the loss value.

[0040] The loss value between the generated SWE image and the real SWE image includes image loss and discrimination loss; the image loss includes L1 loss and Vgg perception loss, and the discrimination loss is Gan loss; the calculation formula of the loss value between the generated SWE image and the real SWE image is as follows: ; ; ; ; ; ; ; Where N and i refer to the total number of pixels and the i-th pixel in the current image; To generate SWE images, is the real SWE image, is the RGB three-channel image generated by generating the SWE image, It is an RGB three-channel image generated from a real SWE image; For perceived loss; It is the image identification loss, which acts on the generator to make the generated SWE image disturb the judgment of the discriminator; is the probability of the discriminator identifying the i-th pixel of the generated SWE image, is the identification criterion of the i-th pixel, is a matrix of all 1s, is the generator loss, are the L1 loss value weights for generating the SWE image and the R, G, and B three-channel images, respectively. The color distribution of the generated SWE image can be adjusted by adjusting the weight parameters. Specifically, in this embodiment, the initial value of the weight parameter is set to .

[0041] During the training and verification process of the discriminator, the original BMUS image and the real SWE image as well as the original BMUS image and the generated SWE image are spliced ​​respectively, and the operation method is shown in the following formula: .

[0042] The convolutional layer feature extraction formula is: ; Where x represents the input of the current layer, express The convolution operation, represents the InstanceNorm2d layer, Represents the Leaky ReLU activation function. Represents the calculated feature map.

[0043] The output of the discriminator is ;in, This is a feature padding operation that fills the edge of the feature map with 0 to adjust the size of the feature map.

[0044] The loss function of the discriminator is mean square error loss, which is calculated as follows: ; ; ; in Respectively represent the discrimination loss of the discriminator for the real SWE image and the generated SWE image, is the probability of the discriminator identifying the ith pixel of the real SWE image, and the identification criterion of the ith pixel of the real image is , is a matrix of all 1s, and the identification criterion for the i-th pixel of the generated image is , is a matrix of all 0s. is the loss of the discriminator, which is used for back propagation. The generator and discriminator in this algorithm model are trained simultaneously to obtain high-quality predicted images.

[0045] S4: Test the generative adversarial model using the test set, and evaluate the generative adversarial model based on the test results.

[0046] The generation effect of images is quantified using Structural Similarity (SSIM). Structural similarity is an indicator used to measure the similarity between two images. It takes into account three key aspects of the image: luminance, contrast, and structure, and provides an image similarity evaluation method that is more in line with human visual perception than traditional indicators.

[0047] Specifically, the structural similarity SSIM between the generated SWE image and the real SWE image is calculated according to the following formula: ; In the formula, x and y represent the generated SWE image and the real SWE image respectively. are the average brightness values ​​of the two images, representing the brightness components of the images. are the contrast variances of the two images, representing the contrast components of the images. is the structural covariance of the two images, indicating the structural components of the images, are two constants; usually .

[0048] The generative adversarial model is evaluated according to the size of the structural similarity SSIM value.

[0049] Pix2Pix is ​​an image-to-image conversion model based on cGANs, which consists of two parts: a generator and a discriminator. Table 1 shows the evaluation indicators of pix2pix and the method proposed in this paper on the test set.

[0050] Table 1

[0051] Figure 6 This is a visualization comparison diagram of the SWE images generated by the method of the present invention and the Pix2Pix model.

[0052] According to the comparison, the existing pix2pix algorithm has poor effect in feature extraction. The present invention enhances the effect of feature extraction based on Unet++ as a generator, and adds channel attention and spatial attention to the discriminator network to weight the feature map, which retains the global features to a certain extent. Most importantly, the pix2pix algorithm ignores the color information of the image when generating the SWE image, resulting in unclear color of the generated image. The present invention extracts the three color channel images of R, G, and B from the SWE image to establish a connection between the SWE image and the color channel, and then assigns different loss weights to the three color channel images of R, G, and B to adjust the color distribution of the SWE image to generate a SWE image with vivid colors.

[0053] like Figure 7 As shown, the present invention also provides a generative adversarial model construction device for SWE image generation, comprising: The raw data processing module pre-processes the acquired target image data set and divides it into a training set, a validation set, and a test set; the target image data set includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images; A model building module, which builds a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator; A model training module, using the training set and the validation set to train and validate the generative adversarial model; using the BUMS image as the input of the generator, the generator outputs a generated SWE image, calculating the loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters by back-propagation according to the loss value; and using the generated SWE image and the real SWE image as the input of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters by back-propagation according to the discrimination loss; A model evaluation module tests the generative adversarial model using the test set and evaluates the generative adversarial model based on the test results.

[0054] According to one aspect of the present disclosure, an electronic device is also disclosed, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0055] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is also disclosed, wherein the storage medium stores computer instructions, and the computer instructions implement the above method when executed by a computer.

[0056] According to one aspect of the present disclosure, a computer program product is also disclosed, including a computer program, wherein: The computer program implements the above method when executed by a processor.

[0057] refer to Figure 8, a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0058] like Figure 8 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0059] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600. The input unit 606 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0060] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0061] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0062] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0063] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0064] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0065] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0066] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0067] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0068] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for constructing a generative adversarial model for SWE image generation, characterized in that: include: S1, preprocessing the acquired target image dataset and dividing it into a training set, a validation set, and a test set; the target image dataset includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images; S2, constructing a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator; S3, training and verifying the generative adversarial model using the training set and the verification set; The training and verification includes: using the BUMS image as an input of the generator, the generator outputs a generated SWE image, extracting RGB three-channel images corresponding to the real SWE image and the generated SWE image, combining the RGB three-channel images, calculating the loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters by back-propagation according to the loss value, and using the generated SWE image and the real SWE image as inputs of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters by back-propagation according to the discrimination loss; S4: Test the generative adversarial model using the test set, and evaluate the generative adversarial model based on the test results.

2. The method according to claim 1, characterized in that The preprocessing of the acquired target image data set includes: Synchronously selecting a region of interest ROI of the BUMS image and the real SWE image, and cropping the BUMS image and the real SWE image based on the region of interest ROI; For the cropped BUMS image and the real SWE image, the pixel value in the region of interest ROI is set to 255, and the pixel value of the non-region of interest ROI is set to 0, to obtain the corresponding MASK image; The cropped BUMS image, the cropped real SWE image and the corresponding MASK image are integrated into one image in a preset order.

3. The method according to claim 2, characterized in that The step of cropping the BUMS image and the real SWE image based on the region of interest ROI includes: Generate a bounding rectangle containing the region of interest ROI, and record the coordinates of four vertices of the bounding rectangle; The BUMS image and the real SWE image are cropped based on the coordinates of the four vertices of the circumscribed rectangular frame.

4. The method according to claim 1, characterized in that: The generator is implemented based on the Unet++ network structure, including: a downsampling module, a feature fusion module and an upsampling module; Using the downsampling module, downsampling the input BUMS image to generate a multi-level feature map; Using the feature fusion module, a feature fusion operation is performed on the multi-level feature map to generate feature fusion results at different levels; By using the upsampling module, the feature fusion results of different levels are spliced ​​with the feature maps of the corresponding levels and an upsampling operation is performed to map the BUMS image to a SWE image.

5. The method according to claim 4, characterized in that The discriminator includes multiple convolutional layers, each of which is connected to a CBAM module; the convolutional layers are used to extract features of the input image, and the CBAM module is used to weight the feature map in channel and spatial dimensions.

6. The method according to claim 5, characterized in that The CBAM module includes a channel attention module and a spatial attention module. The operations performed by the CBAM module are as follows: ； ； ； ； Where F is the input feature map of the CBAM module, Mc(F) represents the execution process of the channel attention module, Ms(F') represents the execution process of the spatial attention module, F'' represents the feature map output after being processed by the CBAM module, MLP is a weight-sharing multi-layer perceptron, AvgPool is an average pooling operation, MaxPool is a maximum pooling operation, Conv The convolution kernel size is The convolution operation, Represents a matrix multiplication operation.

7. The method according to claim 1, characterized in that The loss value between the generated SWE image and the real SWE image includes image loss and discrimination loss; the image loss includes L1 loss and Vgg perception loss, and the discrimination loss is Gan loss; the calculation formula of the loss value between the generated SWE image and the real SWE image is as follows: ； ； ； ； ； ； ； Where N and i refer to the total number of pixels in the current image and the i-th pixel. To generate SWE images, is the real SWE image, is the RGB three-channel image generated by generating the SWE image, is an RGB three-channel image generated from a real SWE image, is the perceived loss, To identify the loss, is the probability of the discriminator identifying the i-th pixel of the generated SWE image, is the identification criterion of the i-th pixel, is a matrix of all 1s, is the generator loss, They are the L1 loss value weights for generating SWE images and R, G, and B three-channel images respectively.

8. The method according to claim 7, characterized in that The calculation formula of the discrimination loss of the discriminator for the input image is as follows: ； ； ； In the formula, They represent the discrimination loss of the discriminator for real SWE images and generated SWE images, and the discrimination standard of real SWE images is a matrix of all 1s, generating the identification criteria for the SWE image is a matrix of all 0s.

9. The method according to claim 1, characterized in that: The step of evaluating the generative adversarial model according to the test results includes: The structural similarity SSIM between the generated SWE image and the real SWE image is calculated according to the following formula: ； In the formula, x and y represent the generated SWE image and the real SWE image respectively. are the average brightness values ​​of the two images, are the contrast variances of the two images, is the structural covariance of the two images, are two constants; The generative adversarial model is evaluated according to the size of the structural similarity SSIM value.

10. A generative adversarial model construction device for SWE image generation, characterized in that: include: The raw data processing module pre-processes the acquired target image data set and divides it into a training set, a validation set, and a test set; the target image data set includes BUMS images and real SWE images, and the training set, validation set, and test set all include BUMS images and real SWE images; A model building module, which builds a generative adversarial model, wherein the generative adversarial model includes a generator and a discriminator; A model training module, using the training set and the validation set to train and validate the generative adversarial model; using the BUMS image as the input of the generator, the generator outputs a generated SWE image, calculating the loss value between the generated SWE image and the real SWE image, and adjusting the generator parameters by back-propagation according to the loss value; and using the generated SWE image and the real SWE image as the input of the discriminator, calculating the discrimination loss of the discriminator for the input image, and adjusting the discriminator parameters by back-propagation according to the discrimination loss; A model evaluation module tests the generative adversarial model using the test set and evaluates the generative adversarial model based on the test results.

Images