An image generation method, a terminal device, and a storage medium

Abstract

The embodiment of the application discloses an image generation method, a terminal device and a storage medium, the image generation method comprises the following steps: inputting a random vector into a generator, sequentially passing through a full connection layer, a reconstruction reshape layer and a convolution layer, and obtaining a first feature map; performing multiple upsampling on the first feature map after processing by a double-path connection block (DPB), and obtaining multiple second feature maps; inputting the multiple second feature maps and a sample dermatitis lesion image into discriminators of a generative adversarial network in sequence, performing multiple maximum pooling after processing by convolution layers and the DPB in sequence, obtaining a third feature map, and processing the third feature map by a full connection layer to obtain a decision result; training the generator by using the decision result and a preset loss function, obtaining a trained generator; and generating a target dermatitis lesion image by using the trained generator.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image generation method, terminal device, and storage medium. Background Technology

[0002] With the application and development of artificial intelligence (AI) technology in the medical field, a large number of medical images are usually needed for medical AI research. However, medical images are difficult to obtain due to privacy, ethical and other issues.

[0003] Specific dermatitis lesion images are characterized by clear morphology, distinct texture features, clear boundaries, and the ability to combine multiple lesions. They can be used to generate medical images that meet medical requirements through image generation technology. However, in current technologies, tens of thousands of dermatitis lesion images are usually required as samples for training when generating specific dermatitis lesion images. Currently, the number of specific dermatitis lesion sample images that can be used in medical AI research is relatively small because they contain sensitive information that is not easy to obtain. Summary of the Invention

[0004] In view of this, embodiments of this application aim to provide an image generation method, terminal device, and storage medium that can efficiently generate a large number of detailed and medically compliant images of specific dermatitis lesions using a small number of samples.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an image generation method, the method comprising:

[0007] A random vector is input into the generator and passed through a fully connected layer, a reshape layer, and a convolutional layer in sequence to obtain a first feature map. The first feature map is then processed by a dual-path connection block (DPB) and upsampled multiple times to obtain multiple second feature maps.

[0008] The multiple second feature maps and sample dermatitis lesion images are sequentially input into the discriminator of the generative adversarial network. After passing through convolutional layers and DPB processing, multiple max pooling operations are performed to obtain the third feature map. The third feature map is then processed by a fully connected layer to obtain the decision result.

[0009] The generator is trained using the decision result and a preset loss function to obtain a trained generator; the trained generator is then used to generate a target dermatitis lesion image.

[0010] Secondly, embodiments of this application provide a terminal device, the terminal device comprising:

[0011] The input unit is used to input random vectors into the generator, which then passes them sequentially through a fully connected layer, a reshape layer, and a convolutional layer to obtain a first feature map. The first feature map is then processed by a dual-path connection block (DPB) and upsampled multiple times to obtain multiple second feature maps. These multiple second feature maps and a sample dermatitis lesion image are then sequentially input into the discriminator of the generative adversarial network (GAN). After passing through a convolutional layer and DPB, multiple max-pooling operations are performed to obtain a third feature map. This third feature map is then processed by a fully connected layer to obtain a decision result.

[0012] The training unit is used to train the generator using the decision result and a preset loss function to obtain the trained generator.

[0013] A generation unit is used to generate a target dermatitis lesion image using the trained generator.

[0014] Thirdly, embodiments of this application provide a terminal device, the terminal device including: a processor, a memory, and a communication bus; the processor executes the running program stored in the memory to implement the above-mentioned image generation method.

[0015] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described image generation method.

[0016] This application provides an image generation method, terminal device, and storage medium. The method includes: inputting a random vector into a generator, passing it sequentially through a fully connected layer, a reshape layer, and a convolutional layer to obtain a first feature map; processing the first feature map using a dual-path connection block (DPB) and then performing multiple upsampling operations to obtain multiple second feature maps; sequentially inputting the multiple second feature maps and a sample dermatitis lesion image into a discriminator of a generative adversarial network (GAN), passing it sequentially through a convolutional layer and a DPB, and then performing multiple max pooling operations to obtain a third feature map; processing the third feature map through a fully connected layer to obtain a decision result; training the generator using the decision result and a preset loss function to obtain a trained generator; and using the trained generator to generate a target dermatitis lesion image. Using the above implementation scheme, in the image generation process, random vectors are sequentially input into the fully connected layer, reshape layer, and convolutional layer in the generator to obtain the first feature map. The first feature map is then processed by DPB. In DPB, grouped convolution is used to reduce the number of parameters, which effectively improves the training speed of the model. In addition, both the generator and the discriminator use a dual-path network (DPN) as the backbone network for feature extraction, which can effectively learn the features of dermatitis lesion images and improve the realism of the generated images. Thus, a large number of detailed and medically compliant specific dermatitis lesion images can be efficiently generated using a relatively small number of existing samples. Attached Figure Description

[0017] Figure 1 A flowchart of an image generation method provided in this application embodiment Figure 1 ;

[0018] Figure 2 This is a schematic diagram illustrating the implementation process of the dual-path connector block (DPB) provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram illustrating the implementation process of grouped convolution provided in an embodiment of this application;

[0020] Figure 4 A schematic diagram illustrating the process of processing feature maps using a self-attention mechanism, as provided in an embodiment of this application.

[0021] Figure 5 A flowchart of an image generation method provided in this application embodiment Figure 2

[0022] Figure 6 A schematic diagram of the structure of a terminal device 1 provided in this application embodiment. Figure 1 ;

[0023] Figure 7 A schematic diagram of the structure of a terminal device 1 provided in this application embodiment. Figure 2 . Detailed Implementation

[0024] To gain a more detailed understanding of the features and technical content of the embodiments of this application, the technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments. The accompanying drawings are for reference only and are not intended to limit the embodiments of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0026] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first / second / third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein.

[0027] Currently, in existing technologies, when using GAN networks and their improved versions for image generation, the following problems still exist:

[0028] (1) The GAN network currently in use usually requires a large number of samples for training. To generate specific dermatitis lesion images of 512*512 size with clear details and meeting medical requirements, tens of thousands of images are usually needed as samples for GAN network training.

[0029] (2) In the currently used GAN networks, the generator and discriminator usually use ResNet as the backbone network for feature extraction. However, when using ResNet for feature extraction, the features extracted in previous layers are reused, resulting in high redundancy in the extracted features. Moreover, ResNet has too many structural parameters and too many model parameters, resulting in excessive training time.

[0030] (3) Usually, when taking pictures of specific dermatitis lesions, they are mostly taken in well-lit places, which will cause interference such as highlights in the pictures of specific dermatitis lesions. When the specific dermatitis lesion pictures with interference such as highlights are directly input into the network for image generation, the highlight interference on the specific dermatitis lesion pictures will be used in the subsequent generated images, resulting in low authenticity of the generated images.

[0031] (4) For images of specific dermatitis lesions, the texture features are the color and texture of the lesion area, while the structural and geometric features are the shape of the lesion area. The GAN network currently used can learn the texture features of sample dermatitis lesion images during training, but it is not easy to learn the specific structural and geometric features of specific dermatitis lesion images. After multiple rounds of training, the GAN network currently used is prone to only learning the texture features of the lesion area of ​​specific dermatitis lesions and ignoring the specific structural and geometric features, which makes the generated dermatitis lesion images easy to be disconnected from the surrounding skin and does not meet medical requirements.

[0032] (5) The GAN networks currently in use usually use ordinary classification networks as discriminators, such as the SoftMax classifier. However, ordinary classification networks cannot separate the features of real and fake specific dermatitis lesions when making real and fake distinctions, resulting in poor model robustness, model collapse, and poor quality of generated images.

[0033] (6) The GAN networks currently in use usually train the model directly without optimizing it, resulting in a long training time and low efficiency.

[0034] To address the aforementioned problems, embodiments of this application provide an image generation method, such as... Figure 1 As shown, the image generation method mainly includes the following steps:

[0035] S101. Input the random vector into the generator, and pass it through the fully connected layer, the reshape layer and the convolutional layer in sequence to obtain the first feature map; after the first feature map is processed by the dual-path connection block DPB, perform multiple upsampling to obtain multiple second feature maps.

[0036] In this embodiment of the application, by inputting a random vector into the generator of the generative adversarial network, the random vector input into the generator is passed through the fully connected layer and the reshape layer in the generator in sequence, and then through 32 3*3 convolutional kernels, so that the size of the obtained feature map is transformed to 16*16*32.

[0037] It should be noted that the random vector can be input noise data, and the specific choice can be made according to the actual situation. This application does not impose any specific restrictions.

[0038] In this embodiment, Generative Adversarial Networks (GANs) are unsupervised deep learning models that utilize computer-generated data. Through a game-like learning process between a generative model and a discriminative model, the GAN ultimately produces outputs that closely resemble realistic images. A GAN mainly consists of two parts: a generator and a discriminator. The generator takes noise or sample data as input and outputs an image, aiming to make the discriminator unable to distinguish between real and fake images. The discriminator judges the authenticity of the input image, aiming to identify fake images generated by the generator as much as possible. The generator and discriminator constitute a dynamic adversarial process. During this adversarial process, the images generated by the generator become increasingly closer to real images, while the discriminator's ability to distinguish between real and fake images also improves. Ideally, the generator can generate images that are indistinguishable from real images, while the discriminator struggles to determine whether the generated images are genuine. After training the GAN, an ideal generative model can be obtained, capable of generating indistinguishable images.

[0039] In this embodiment of the application, after obtaining a feature map of size 16*16*32, the obtained feature map of size 16*16*32 is processed by two dual path blocks (DPB). The feature map after DPB processing is then upsampled 5 times to obtain 5 upsampled feature maps. The size of the feature map is expanded to 32 times the original size, which is 512*512. The feature maps obtained after 5 upsampling can be named G1, G2, G3, G4 and G5.

[0040] It should be noted that the naming method of the upsampled feature map can be selected according to the actual situation, and no specific limitation is made in this application.

[0041] It should be noted that in this embodiment, Dual Path Networks (DPN) are used as the backbone network of the generator for feature extraction. The DPN network mainly combines the advantages of the dense block in the deep network DenseNet and the Resblock in the residual network ResNet to form the dual path connection block DPB.

[0042] It should be noted that the structure and implementation principle of the dual-path connector block (DPB) are as follows: Figure 2As shown, x is used as the input to the DPB network structure. On one hand, the input x is processed through two convolutional layers and the ReLU activation function to obtain the convolutional function F(x). The first d feature maps of F(x) are then input into the adder. The first d feature maps of F(x) can be represented as F(x)[:d], where d is a hyperparameter used to determine the number of new features to be utilized. On the other hand, the input x is processed through residual learning to obtain the first d feature maps x[:d] of x. After x[:d] is input into the adder, x[d:] and F(x)[d:] are calculated in the adder. The sum of x[d:] and F(x)[d:] is then concatenated and input into the cascade. On the other hand, the input x is processed through two convolutional layers and the ReLU activation function to obtain the convolutional function F(x), which is then processed through dense connections to obtain F(x)[d:]. The obtained F(x)[d:] is then input into the cascade. After that, the input x is processed through dense connections to obtain x[d:], which is then directly input into the cascade for calculation. Finally, before the Rectified Linear Unit (ReLU) processing, the output of the cascade is: y = G([x[:d]F(x)[:d],F(x)[d:]+x[d:]]).

[0043] It should be noted that G is the activation function of the ReLU linear correction unit, F is the convolutional layer function, and x is the input of the two-path connected block. Before ReLU, there are three parts: the first part is the densely connected part x[:d] of the original input x; the second part is the densely connected part F(x)[:d] after convolution; and the third part is the sum of the residual connected part x[d:] of the original input x and the residual part F(x)[d:] after convolution, using an adder.

[0044] In this embodiment of the application, in the dual-path connection block (DPB), a grouped convolution operation is performed on the convolutional layer containing the 3*3 convolutional kernel, such as... Figure 3 As shown, the input dimension of the convolutional layer is divided into 8 groups, with each group having only 1 / 8 of the number of input channels. In this case, only 1 / 8 of the number of input channels is needed to convolve each group, and then the groups are concatenated along the channel dimension, so that the number of parameters of the convolutional layer is reduced to 1 / 8 of the number of ungrouped convolutions.

[0045] It should be noted that using grouped convolutions can significantly reduce the number of parameters, improve training speed, and under the same network structure, the performance of grouped convolutions is significantly improved compared to ungrouped convolutions.

[0046] S102. Multiple second feature maps and sample dermatitis lesion images are sequentially input into the discriminator of the generative adversarial network. After passing through convolutional layers and DPB processing, multiple max pooling operations are performed to obtain the third feature map. The third feature map is then processed by a fully connected layer to obtain the decision result.

[0047] In this embodiment, after obtaining multiple 512*512 feature maps generated by the Generative Adversarial Network (GAN) generator and acquiring sample dermatitis lesion images, the multiple 512*512 feature maps and the acquired sample dermatitis lesion images are input into the discriminator. After passing through 32 3*3 convolutional kernels, the feature map size is transformed to 512*512*32. After two DPB processes, five max pooling processes are performed, and the size of the feature map after max pooling is changed to 16*16*32. The max pooled feature maps can be named D1, D2, D3, D4, and D5. Finally, the obtained feature maps are processed through a fully connected layer.

[0048] It should be noted that the dermatitis lesion images can be selected from an existing database or taken by photographing the dermatitis lesions. Specifically, the choice can be made according to the actual situation, and no specific limitation is made in this application.

[0049] It should be noted that the Generative Adversarial Network (GAN) discriminator uses DPN as the backbone network, and the method used for DPB processing is the same as that used in the generator, so it will not be repeated here.

[0050] In this embodiment, the obtained feature maps are combined through a fully connected layer and output to the output layer. After being judged by a classification network, the dermatitis lesion image generated by the generator can be compared with the sample dermatitis lesion image, and two judgment results can be obtained: it can be real or it can be fake.

[0051] In another embodiment of this application, to address the issue that in the prior art, simple classification networks, such as softmax classifiers, are typically used as discriminators in GAN networks, ordinary classification networks divide the input image into two categories: the original image and the generated image. Training is then performed on these two categories to enable the discriminator to correctly classify the input image. However, classifying dermatitis lesion images differs from classifying regular images in that the distinction between generated specific dermatitis lesion images and dermatitis lesion sample images does not lie in certain obvious features; medical experience is usually required for judgment. Therefore, the discriminative power required to determine whether generated specific dermatitis lesion images can be used for extracting feature maps in medical research is significantly higher than the requirements for classifying regular images. Therefore, this application uses a discriminator based on contrastive learning to replace ordinary classification networks for classifying generated specific dermatitis lesion images.

[0052] In this embodiment, the fully connected layer includes a multilayer perceptron. The multilayer perceptron processes the third feature map to obtain feature vectors corresponding to the third feature map. The cosine distance between any two feature vectors is calculated. Based on the cosine distance between any two feature vectors, a first cosine distance sum corresponding to the generated image and a second cosine distance sum corresponding to the sample image are determined. The first cosine distance sum is compared with a first preset threshold to obtain a first comparison result. The second cosine distance sum is compared with a second preset threshold to obtain a second comparison result. The first comparison result and the second comparison result are determined as the decision result.

[0053] In this embodiment, 2N images are used as input to the discriminator backbone network. The 2N images include N dermatitis lesion sample images and N generated images. A multilayer perceptron with one hidden layer is added after the backbone network of the discriminator. After the images in the discriminator are processed by the backbone network, the 2N input images will eventually generate 2N feature maps, and the final output will be a 128-dimensional feature vector.

[0054] In this embodiment, after obtaining a 128-dimensional feature vector, each feature vector is regularized to become a unit vector, and each feature vector lies on a hypersphere with a radius of 1. The resulting feature vector can be represented as {z1, z2, ..., z...} 2N}

[0055] In this embodiment, after obtaining the feature vectors corresponding to 2N feature maps, the cosine distance between every two feature vectors is calculated using cosine distance, and the sum of the first cosine distances corresponding to the N generated images and the sum of the second cosine distances corresponding to the N dermatitis lesion sample images are determined. By comparing the sum of the first cosine distances corresponding to the N generated images with a first preset threshold and comparing the sum of the second cosine distances corresponding to the N dermatitis lesion sample images with a second preset threshold, a comparison result is obtained. The comparison result is determined as the decision result of the decision maker. If the comparison result is that the images belong to the same class, the decision result is true; if the comparison result is that the images belong to different classes, the decision result is false.

[0056] It should be noted that images with a larger sum of cosine distances belong to the same class, while images with a smaller sum of cosine distances belong to different classes.

[0057] It should be noted that the first preset threshold and the second preset threshold can be selected according to the actual situation, and no specific limitation is made in this application.

[0058] For example, for any input image i: Given the class label, among the remaining 2N-1 input images, there are N-1 images belonging to the same class as input image i. Let's assume one of these images is image j. Input image i and input image j both originate from the same class of input images. Because they belong to the same class, we want the feature vectors of input image i and input image j to be as close as possible. For the other N images (excluding image i), since they belong to different classes of input images, we want their feature vectors to be as far away from the feature vector of image i as possible.

[0059] After calculating the cosine distance between any two feature vectors, the sum of the cosine distances corresponding to the N generated images and the sum of the cosine distances corresponding to the N dermatitis lesion sample images are calculated. The calculated pre-selected distances are compared with the first and second preset distance thresholds, respectively. When the sum of the cosine distances is greater than the preset distance, the input images are determined to be images of the same type; when the cosine distances are less than the sum of the preset distances, the input images are determined to be images of different types.

[0060] It should be noted that when performing inference after training, the multilayer perceptron is replaced by a single linear layer.

[0061] S103. Using the decision result and the preset loss function, train the generator to obtain the trained generator; use the trained generator to generate the target dermatitis lesion image.

[0062] In this embodiment of the application, after obtaining the judgment result, a random input is given to the generator of the GAN, and the degree of difference between the output of the discriminator and the real data is judged by the loss function. Based on the loss function, the parameters of the generator are continuously optimized to obtain the optimized generator.

[0063] In the embodiments of this application, the loss function used in the discriminator can be expressed by the following formulas (1) and (2):

[0064]

[0065]

[0066] It should be noted that, in the embodiments of this application, the loss function means that, for any input image i: the sum of the cosine distances between the feature vectors of all other input images belonging to the same class as input image i and the feature vector of input image i, should be as large as possible; the sum of the cosine distances between the feature vectors of all other input images not belonging to the same class as input image i and the feature vector of input image i, should be as small as possible.

[0067] It should be noted that, compared with other loss functions, the contrastive loss used in this application can maintain a relatively stable average loss error for different degrees of image damage, and can completely suppress image damage caused by the enhancement of a small number of sample images.

[0068] In this embodiment of the application, after the generator is trained using the loss function, the discriminator of the target dermatitis lesion image generated by the generator can no longer distinguish between real and fake images. The generator trained at this time is used to generate the target dermatitis lesion image.

[0069] It should be noted that this application uses an Nvidia GPU with 16GB of video memory for training, and performs Ristretto fixed-point floating-point quantization on the entire GA network, converting the parameters to 8 bits. This reduces the model size to one-quarter and improves efficiency by about 8 times. After 5000 training rounds, 512*512 images are generated, ultimately achieving the goal of efficiently generating a large number of detailed images of specific dermatitis lesions that meet medical requirements.

[0070] In this embodiment, a discriminator based on contrastive learning is used. From the perspective of network training objectives and results, the feature vectors of images of the same type are made close together, while the feature vectors of different types are made far apart. The entire discriminator network can be likened to a hypersphere, causing features of different types to cluster towards different points. This greatly increases the image discrimination in the classification results. It can achieve high accuracy even with a small amount of data and shows good robustness. Compared with traditional classification networks, which are similar to a hyperplane and do not create a large distance between features of different categories, the discriminative power of the features is significantly lower than that of the discriminator using contrastive learning.

[0071] It is understood that in the image generation method provided in this application embodiment, during the image generation process, after the random vector is sequentially input into the fully connected layer, reshape layer and convolutional layer in the generator to obtain the first feature map, the first feature map is processed by DPB. In DPB, grouped convolution is used to reduce the number of parameters, which effectively improves the training speed of the model. In addition, the generator and discriminator both use the dual-path network DPN as the backbone network of the generator and discriminator for feature extraction, which can effectively learn the features of dermatitis lesion images and improve the realism of the generated images. Thus, it is possible to efficiently generate a large number of specific dermatitis lesion images with clear details and meeting medical requirements using the existing small number of samples.

[0072] Optionally, after obtaining multiple second feature maps, before sequentially inputting the multiple second feature maps and sample dermatitis lesion image data into the discriminator of the generative adversarial network, each second feature map can be sequentially input into multiple preset convolutions to obtain multiple third feature maps. By fusing and filtering the multiple third feature maps, each second feature map is converted into a corresponding second feature map containing target feature data. The target feature data is the feature data representing the junction area between the lesion area and the normal skin area.

[0073] In this embodiment of the application, a self-attention mechanism module is used to process the obtained multiple second feature maps, such as... Figure 4 As shown, three 8*1 convolution kernels are preset, namely q convolution, k convolution and v convolution. Taking feature map G1 as an example, feature map G1 of size c*w*h is sequentially input into the three preset convolution kernels q convolution, k convolution and v convolution. After three convolution processes, three three-dimensional feature maps of size (8*c)*w*h are obtained after extracting features from each channel 8 times, which are denoted as q1, k1 and v1 respectively.

[0074] In this embodiment of the application, after obtaining the three-dimensional feature maps q1, k1, and v1 of size (8*c)*w*h, the three-dimensional feature maps q1, k1, and v1 of size (8*c)*w*h are processed by a reshape layer to convert them into two-dimensional feature maps q1, k1, and v1 of size (8*c)*(w*h). The converted feature map k1 is then matrix transposed, and the two-dimensional feature map q1 and the transposed k1 are multiplied by a dot product. The matrix is ​​then normalized by a normalized exponential function softmax classifier and a normalization layer to obtain a matrix a1 of size (w*h)*(w*h).

[0075] In this embodiment, after obtaining a matrix a1 of size (w*h)*(w*h), the transformed two-dimensional feature map v1 is multiplied by a1 to obtain a matrix v2 of size (8*c)*(w*h). The obtained matrix v2 of size (8*c)*(w*h) is reshaped to obtain a three-dimensional feature map V3 of size (8*c)*w*h. The obtained V3 is then subjected to dimensionality reduction convolution to obtain a three-dimensional feature map of size c*w*h after processing by a self-attention mechanism. The obtained three-dimensional feature map of size c*w*h contains target feature data, which is the target feature region of the area where the lesion area and the normal skin area meet.

[0076] It should be noted that, in the embodiments of this application, the obtained second feature maps G2, G3, G4, and G5 are processed to obtain a second feature map containing feature data of the area connecting the lesion area and the normal skin area. This can be achieved by processing feature map G1. The specific implementation process will not be described here.

[0077] In this embodiment of the application, after obtaining the second feature map containing feature data of the area where the lesion area and the normal skin area meet, multiple second feature maps and dermatitis lesion image data are sequentially input into the discriminator of the generative adversarial network. Alternatively, multiple second feature maps containing target feature data and dermatitis lesion image data can be sequentially input into the discriminator of the generative adversarial network.

[0078] In this embodiment, multiple second feature maps containing target feature data and dermatitis lesion image data are sequentially input into the discriminator of the generative adversarial network. After being processed by convolutional layers and DPB, multiple max pooling operations are performed to obtain a third feature map. The third feature map is then processed by a fully connected layer to obtain a decision result. The specific implementation process is the same as the implementation process of step S102 above, and will not be repeated here.

[0079] In this embodiment, the obtained second feature maps G1, G2, G3, G4, and G5 are processed sequentially using a self-attention mechanism, which effectively improves the overall effect of the generated image without significantly affecting the network size and training time.

[0080] In this embodiment, the self-attention mechanism utilizes the data information of all pixels in the image to quickly obtain the global geometric features of the image by directly calculating the relationship between any two pixels, effectively learning the structure and overall features of the entire image. Furthermore, the self-attention mechanism uses multiple convolutional kernels, which can combine and filter multiple features at each pixel to obtain the most significant and highly relevant features from multiple global features. During training, the image can be trained towards the most relevant features. The multiple convolutional kernels used can better extract the features of the region where the lesion area and the normal skin area meet, effectively improving the realism of the generated image. The self-attention mechanism can also utilize information from distant image regions, and each position can combine information from similar or related regions to ensure the regional consistency of the generated image.

[0081] Optionally, before sequentially inputting multiple second feature maps and sample dermatitis lesion images into the discriminator of the generative adversarial network, image segmentation can be performed on the initial sample dermatitis lesion image to obtain a first image region and a second image region corresponding to the initial sample dermatitis lesion image; wherein, the first image region is the region including dermatitis lesion image data, and the second image region is the region excluding dermatitis lesion image data; using predefined specular component coefficients, specular reflection components in the first image region are removed to obtain the first image region after specular removal; using predefined diffuse reflection component coefficients, diffuse reflection components in the second image region are determined; and the second image region after specular removal is generated based on the diffuse reflection components; dermatitis lesion image data is generated using the first image region after specular removal and the second image region after specular removal.

[0082] In this embodiment, the dermatitis lesion sample images are usually taken by mobile phones or other devices under bright light. Most of the captured dermatitis lesion images have highlight areas. Using dermatitis lesion images with highlight areas for training will affect the training or subsequent research. If dermatitis lesion images with highlight areas are directly used for image generation, the highlight areas in the captured images will continue to exist in the generated images. Therefore, when training the network, in order to avoid the influence of some interference factors on the generated images, it is necessary to perform a highlight removal preprocessing operation on the dermatitis lesion sample images.

[0083] In this embodiment of the application, when performing highlight removal preprocessing on the initial dermatitis lesion sample image, it is first necessary to perform image segmentation on the initial sample dermatitis lesion image to obtain the first image region and the second image region corresponding to the initial sample dermatitis lesion image.

[0084] In this embodiment of the application, image segmentation of the initial sample dermatitis lesion image can be performed by locating the lesion area from the initial dermatitis lesion image, extracting the first dermatitis lesion image from the initial dermatitis lesion image data with the lesion area as the center, separating the first dermatitis lesion image into RGB three-channel image data, generating an image mask based on the RGB three-channel image data, and using the image mask to divide the initial dermatitis lesion image data into a first image region and a second image region.

[0085] In this embodiment of the application, the lesion area is first located from the initial dermatitis lesion image. The lesion area is then used as the center to extract a first dermatitis lesion image with a size of 512*512 pixels.

[0086] It should be noted that, in order to avoid the problem of deformation of the dermatitis lesion sample image in the input discriminator, which would cause the lesion area of ​​the image to also be deformed, the dermatitis lesion sample image needs to be converted into an image of size 512*512 first.

[0087] In this embodiment, after obtaining the first dermatitis lesion image, the first dermatitis lesion image is image-separated by three channels: R, G, and B, to obtain an image corresponding to each channel. The three images corresponding to the three channels are then filtered using different filters of different sizes and parameters to determine the texture of each image, resulting in filtered images. The average value of all filtered images is calculated, and the threshold is obtained by using the maximum inter-class variance method. After segmenting the images, a mask mask1 is obtained. The initial dermatitis lesion image is processed using mask mask1, which can effectively separate the dermatitis lesion area from the non-dermatitis lesion area, resulting in the first image area and the second image area.

[0088] It should be noted that the first image region can be a region that includes dermatitis lesion image data, and the second image region can be a region that does not include dermatitis lesion image data; the first image region can also be a region that does not include dermatitis lesion image data, and the second image region can be a region that includes dermatitis lesion image data. Specifically, the selection can be made according to the actual situation, and no specific limitation is made here.

[0089] It should be noted that when performing image segmentation on the initial sample dermatitis lesion image, other image segmentation methods besides those in this application can be used. Specifically, the method can be selected according to the actual situation, and no specific limitation is made in this application.

[0090] In this embodiment of the application, after segmenting the initial sample dermatitis lesion image region to obtain a first image region and a second image region, the first image region and the second image region are respectively subjected to highlight removal processing.

[0091] In this embodiment of the application, for example, it is assumed that the first image region is the dermatitis lesion image data region, and the second image region is the region that does not include the dermatitis lesion image data. When performing highlight removal processing on the first image region including the dermatitis lesion region, the specular reflection component of the highlight in the dermatitis lesion region is usually large. Therefore, for the first image region, it is necessary to remove the specular reflection component in the region. First, the maximum chromaticity value Amax and the minimum chromaticity value Amin of all pixels in the dermatitis lesion region are calculated using formulas (3), (4) and (5). The calculation formulas are as follows:

[0092] Ai(x)=Ci(x) / (R+G+B) (3)

[0093] Amax(x)=max(Ai(x)) (4)

[0094] Amin(x)=min(Ai(x)) (5)

[0095] Where Ci(x) represents the {R, G, B} value of the pixel at position x, and Ai(x) represents the pixel chromaticity value.

[0096] In this embodiment, after calculating the maximum chromaticity value Amax and the minimum chromaticity value Amin, C(x) is defined as max(R(x),G(x),B(x)), which is the maximum pixel value among the three channels of the pixel at x; the specular component coefficient is defined as G(x), and Gi(x) is defined as the specular component coefficient at the x-pixel position of each of the three channels. The specular component at the x-pixel in the dermatitis lesion image area is calculated using formulas (6), (7), and (8). The calculation formulas are as follows:

[0097] Gi(x)=(Ai(x)-Amin(x)) / (1-3*Amin(x)) (6)

[0098] G(x) = max(Gi(x)) (7)

[0099] GV(x)=(G(x)*(R(x)+G(x)+B(x))-C(x)) / (3*G(x)-1) (8)

[0100] In this embodiment, after calculating the highlight component at pixel x, the pixel value in the three channels at pixel x in the dermatitis lesion image region is replaced with the value of Ci(x)-GV(x), and then the three-channel images are merged to obtain the first image region containing the dermatitis lesion image region after highlight removal processing.

[0101] In this embodiment of the application, when performing specular removal processing on the second image region that does not include the dermatitis lesion image data region, the specular reflection component is usually small for the specular highlights of the region that does not include the dermatitis lesion image data region, so it is necessary to retain the diffuse reflection component inside the second image region. First, the maximum and minimum chromaticity values ​​of the second image region that does not include the dermatitis lesion image data region are calculated using the above formulas (3), (4), and (5). After calculating the maximum and minimum chromaticity values ​​of the second image region, the diffuse reflection component coefficient is defined as B(x). The diffuse reflection component at pixel x in the second image region that does not include the dermatitis lesion image region is calculated using formulas (9), (10), and (11). The calculation formula is as follows:

[0102] Bi(x)=1-(Amax(x)-Ai(x)) / (3*Amax(x)-1) (9)

[0103] B(x) = max(Bi(x)) (10)

[0104] BV(x)=(max(B(x)*(R(x)+G(x)+B(x)),Ci(x))) / (3*B(x)-1) (11)

[0105] Where Bi(x) is the diffuse reflection component at pixel x.

[0106] In this embodiment of the application, after calculating the diffuse reflection component at pixel x in the second image region, the pixel value of the three channels corresponding to pixel x in the second image region can be replaced by the diffuse reflection component value, and then the three channels are merged to obtain the second image region after specular removal.

[0107] In this embodiment of the application, after performing highlight removal processing on the first image region and the second image region respectively, an initial dermatitis lesion image after highlight removal processing is obtained, and the initial dermatitis lesion image after highlight removal processing is used to generate a dermatitis lesion image.

[0108] Optionally, after obtaining multiple second feature maps, before sequentially inputting the multiple second feature maps and sample dermatitis lesion image data into the discriminator of the generative adversarial network, the multiple second feature maps and sample dermatitis lesion image data can be image-shifted by a preset angle to obtain shifted image data; and / or, the multiple second feature maps and sample dermatitis lesion image data can be image-cropped to obtain cropped image data; and / or, the multiple second feature maps and sample dermatitis lesion image data can be image-rotated by a preset rotation angle to obtain rotated image data; and / or, the multiple second feature maps and sample dermatitis lesion image data can be band-stopped to obtain filtered image data.

[0109] In the embodiments of this application, when the amount of image data is insufficient, image data enhancement is usually performed to increase the amount of image data.

[0110] In this embodiment of the application, when performing data augmentation on an image, multiple second feature maps and dermatitis lesion image data input to the discriminator can be offset by eight different preset angles with an offset probability of 0.25 and an offset distance within the range of (0, 0.1) of the shortest side of the entire image, to obtain offset image data.

[0111] It should be noted that the eight different preset angles are 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°.

[0112] It should be noted that the preset angle, the probability of offset occurrence, and the offset distance can be selected according to the actual situation, and no specific limitations are made in this application.

[0113] In another embodiment of this application, when performing data augmentation on an image, multiple second feature maps and dermatitis lesion image data input into the discriminator can be cropped in four different directions with a cropping probability of 0.25 and a cropping length of random values ​​within the range of (0, 0.1) of the shortest side of the entire image, to obtain cropped image data.

[0114] It should be noted that the four different directions can be 0°, 90°, 180°, and 270°.

[0115] It should be noted that the probability of cutting, the cutting length, and the cutting direction can be selected according to the actual situation, and are not specifically limited in this application.

[0116] In another embodiment of this application, when performing data augmentation on an image, the multiple second feature maps and dermatitis lesion image data input to the discriminator can be rotated to the left or right by a certain angle with a rotation probability of 0.25 and a rotation angle within the range of (0, 15 degrees) to obtain rotated image data.

[0117] It should be noted that the direction of rotation can be left, right, or other directions; no specific limitation is made in this application.

[0118] It should be noted that the probability of rotation and the rotation angle can be selected according to the actual situation, and no specific limitations are made in this application.

[0119] In another embodiment of this application, when performing data enhancement on the image, with a filtering occurrence probability of 0.25, multiple second feature maps and dermatitis lesion image data input to the discriminator can be subjected to band-stop filtering. First, the image is subjected to Fourier transform, and a ring region with a radius ranging from 0.23 times the image side length to 0.55 times the image side length is set as the band-stop. The image is then subjected to band-stop filtering and inverse Fourier transform. Under the action of the band-stop filter with the above parameters, the image can effectively retain the texture and color-related information of the lesion, and the filtered image data is obtained.

[0120] It should be noted that the filtering probability and the range of band-stop can be selected according to the actual situation, and no specific limitations are made in this application.

[0121] It should be noted that the above-mentioned image enhancement methods can process images individually or in combination. The specific method can be selected according to the actual situation, and no specific limitation is made in this application.

[0122] In this embodiment of the application, after obtaining the enhanced image data corresponding to multiple feature maps and dermatitis lesion image data, multiple second feature maps and dermatitis lesion image data, and / or offset image data, and / or cropped image data, and / or rotated image data, and / or filtered image data are sequentially input into the discriminator of the generative adversarial network.

[0123] In this embodiment of the application, multiple second feature maps and dermatitis lesion image data, and / or offset image data, and / or cropped image data, and / or rotated image data, and / or filtered image data are sequentially input into the discriminator of the generative adversarial network, and then processed by the above step S102, which will not be repeated here.

[0124] In this embodiment of the application, by performing image shifting and / or image cropping and / or image rotation and / or band-stop filtering on multiple second feature maps and sample dermatitis lesion image data input to the discriminator, the problem of the prior art, which only adjusts the illumination, saturation, and contrast of the dermatitis lesion image input to the discriminator to increase image data, will be overcome. The trained discriminator will then identify the enhanced part of the dermatitis lesion image as real information, resulting in a deviation in the training direction of the generator.

[0125] In this embodiment of the application, the use of a low-probability data augmentation method can increase the amount of image data and enrich the details of the generated dermatitis lesion image without changing key information such as the morphology, texture, and morphology of the area where the lesion meets the skin in the image, thus effectively alleviating the problem of insufficient initial dermatitis lesion sample image data.

[0126] Based on the above embodiments, this application provides an image generation method, such as... Figure 5 As shown, the specific steps include:

[0127] Step 1: Input the random vector into the generator of the generative adversarial network;

[0128] Step 2: Process the feature map generated by the generative adversarial network generator through a self-attention mechanism to obtain the processed feature map;

[0129] Step 3: Perform image enhancement on the feature map generated in Step 2 and the sample dermatitis lesion image;

[0130] Step 4: Input the feature map generated after image enhancement and the sample dermatitis lesion image into the discriminator for judgment;

[0131] Step 5: Train the generator based on the discrimination results and loss function to obtain the trained generator;

[0132] Step 6: Use the trained generator to generate images of the target dermatitis lesions.

[0133] Based on the above embodiments in this application, the following technical effects can be achieved:

[0134] (1) In this application, deep learning is used to generate images of specific dermatitis lesions, and a large number of lesion images that meet medical requirements and have high resolution are generated using a small number of samples, which solves the problem of lack of specific dermatitis lesion images in current medical AI research.

[0135] (2) In this application, a DPN network architecture is used as the backbone network for the generator and discriminator for feature extraction. This network architecture uses dual path blocks, which combine the advantages of dense blocks and res blocks, making it easier to explore new features. This addresses the problem that existing technologies typically use ResNet as the backbone network for feature extraction, but the residual channels are not good at exploring new features, and the number of model parameters is too large.

[0136] (3) In the dual path block, grouped convolution significantly reduces the number of parameters and improves the training speed. Moreover, under the same network structure, the effect of this grouped convolution is better than that of ungrouped convolution. It improves the richness of feature extraction while reducing the number of model parameters and improving the training speed of the model.

[0137] (4) This application uses a self-attention mechanism. First, this mechanism can utilize information from all locations to quickly obtain the global geometric features of the image by directly calculating the relationship between any two pixels in the image, effectively learning the structure and overall features of the entire image. Furthermore, this mechanism uses multiple convolutional kernels, especially those that can better extract features from the area where the lesion and normal skin meet, guiding training towards the most relevant features. Simultaneously, this mechanism can utilize information from distant areas, with each location combining information from similar or related areas to ensure regional consistency in the generated image. Finally, this mechanism is combined with a DPN structure that integrates the advantages of dense blocks and res blocks, further enhancing the backbone network's ability to effectively extract features, thus improving the quality and realism of the generated image. This addresses the common problem that generators typically use ordinary deconvolutional networks to generate images from input vectors. However, currently used generators often learn the texture features of samples during training but struggle to learn specific structural and geometric features, resulting in poor image quality.

[0138] (5) This application uses a highlight removal method tailored to the characteristics of different regions for real skin lesion images of atopic dermatitis. This method can effectively remove highlight areas without affecting the lesions and other skin areas, reducing or even eliminating the impact of highlights on the image, so that highlight interference will not flow into the generated image, thus improving the quality of the generated image. This addresses the issue that atopic dermatitis lesion images are usually taken in bright light, which can cause highlight interference in the image. Existing preprocessing methods for computer-aided diagnosis of skin disease images often simply perform color normalization and equalization. Such preprocessing methods cannot effectively remove interference information such as highlights; instead, they can confuse useful information with interference information, thus reducing the quality of the real image.

[0139] (6) This application not only performs data augmentation on the real images entering the discriminator, but also performs the same data augmentation on the generated images entering the discriminator. This can prevent the discriminator from misclassifying the augmented parts of the real images as real information during the training process, which would lead to a deviation in the training direction of the generator. By using a band-stop filter to augment the images of specific dermatitis lesions, the morphology of the non-lesion areas of the skin is amplified without changing key information such as the morphology, texture, and the morphology of the lesion and the skin junction in the image. The low-probability augmentation can increase the amount of data and enrich the details of the generated images without destroying the essence of the image, effectively alleviating the problem of insufficient original sample size.

[0140] (7) This application uses a discriminator based on contrastive learning, and uses cosine distance in the loss function to calculate the distance between vectors. From the perspective of the purpose and results of network training, it differs from classification networks. By making feature vectors of the same class close together and feature vectors of different classes far apart, the entire network can resemble a hypersphere, causing different types of features to cluster towards different points, greatly increasing the discriminative power. Even with limited data, it can achieve high accuracy and demonstrates good robustness. Traditional classification networks are often used as discriminators, but their purpose is classification. They resemble a hyperplane and do not have the constraint of separating features of different classes, resulting in significantly lower discriminative power between different classes. This is not suitable for tasks like true / false discrimination where features are not as concrete as in classification tasks.

[0141] (8) This application uses the Ristretto fixed-point floating-point quantization model, which reduces the model size and improves training efficiency.

[0142] Based on the above embodiments, another embodiment of this application provides a terminal device 1, such as... Figure 6 As shown, the terminal device 1 includes:

[0143] Input unit 10 is used to input random vectors into the generator, which then passes through a fully connected layer, a reshape layer, and a convolutional layer to obtain a first feature map. The first feature map is then processed by a dual-path connection block (DPB) and upsampled multiple times to obtain multiple second feature maps. The multiple second feature maps and the sample dermatitis lesion image are then input into the discriminator of the generative adversarial network, which passes through a convolutional layer and DPB, and then performs multiple max pooling operations to obtain a third feature map. The third feature map is then processed by a fully connected layer to obtain a decision result.

[0144] Training unit 11 is used to train the generator using the decision result and a preset loss function to obtain the trained generator.

[0145] The generation unit 12 is used to generate target dermatitis lesion images using the trained generator.

[0146] Optionally, the input unit 10 is further configured to sequentially input each second feature map into multiple preset convolutions to obtain multiple third feature maps.

[0147] Optionally, terminal device 1 may further include: a conversion unit,

[0148] The conversion unit is used to convert each second feature map into a second feature map containing target feature data by fusing and filtering the plurality of third feature maps; the target feature data is feature data representing the junction area between the lesion area and the normal skin area.

[0149] Optionally, the input unit 10 is further configured to sequentially input multiple second feature maps containing target feature data and the dermatitis lesion image data into the discriminator of the generative adversarial network.

[0150] Optionally, terminal device 1 may further include: a segmentation unit,

[0151] The segmentation unit is used to segment the initial sample dermatitis lesion image to obtain a first image region and a second image region corresponding to the initial sample dermatitis lesion image; the first image region is the region that includes dermatitis lesion image data, and the second image region is the region that does not include dermatitis lesion image data.

[0152] Optionally, terminal device 1 may further include: a rejection unit.

[0153] The removal unit is used to remove the specular reflection component in the first image region using a predefined specular component coefficient, so as to obtain the first image region after specular removal.

[0154] Optionally, terminal device 1 may further include: a determining unit,

[0155] The determination unit is used to determine the diffuse reflection component in the second image region using predefined diffuse reflection component coefficients.

[0156] Optionally, the generation unit 12 is further configured to generate a second image region after specular removal based on the diffuse reflection component; and to generate dermatitis lesion image data using the first image region after specular removal and the second image region after specular removal.

[0157] Optionally, terminal device 1 may further include: a positioning unit,

[0158] The localization unit is used to locate the lesion area from the initial sample dermatitis lesion image.

[0159] Optionally, terminal device 1 may further include: an interception unit,

[0160] The cropping unit is used to crop the first dermatitis lesion image from the initial sample dermatitis lesion image data, with the lesion area as the center.

[0161] Optionally, terminal device 1 may further include: a separation unit,

[0162] The separation unit is used to separate the first dermatitis lesion image into RGB three-channel image data.

[0163] Optionally, the generation unit 12 is further configured to generate an image mask based on RGB three-channel image data.

[0164] Optionally, terminal device 1 may further include: a partitioning unit,

[0165] The segmentation unit is used to divide the initial sample dermatitis lesion image data into a first image region and a second image region using an image mask.

[0166] Optionally, terminal device 1 may further include: an offset unit,

[0167] The offset unit is used to offset multiple second feature maps and sample dermatitis lesion image data by a preset angle to obtain offset image data.

[0168] Optionally, the terminal device 1 may further include: a trimming unit.

[0169] The cropping unit is used to crop multiple second feature maps and sample dermatitis lesion image data to obtain cropped image data.

[0170] Optionally, terminal device 1 may further include: a rotating unit,

[0171] The rotation unit is used to rotate multiple second feature maps and sample dermatitis lesion image data according to a preset rotation angle to obtain rotated image data.

[0172] Optionally, terminal device 1 may further include: a filtering unit,

[0173] The filtering unit is used to perform band-stop filtering on multiple second feature maps and sample dermatitis lesion image data to obtain filtered image data.

[0174] Optionally, the input unit 10 is also used to sequentially input multiple second feature maps and sample dermatitis lesion image data, and / or offset image data, and / or cropped image data, and / or rotated image data, and / or filtered image data into the discriminator of the generative adversarial network.

[0175] Optionally, terminal device 1 may further include: a processing unit,

[0176] The processing unit is used to process the third feature map using a multilayer perceptron to obtain the feature vector corresponding to the third feature map.

[0177] Optionally, terminal device 1 may further include: a computing unit,

[0178] The computational unit is used to calculate the cosine distance between any two eigenvectors.

[0179] Optionally, the determining unit is further configured to determine the first cosine distance sum corresponding to the generated image and the second cosine distance sum corresponding to the sample image based on the cosine distance between every two feature vectors.

[0180] Optionally, terminal device 1 may further include: a comparison unit,

[0181] The comparison unit is used to compare the first cosine distance sum with a first preset threshold to obtain a first comparison result; and to compare the second cosine distance sum with a second preset threshold to obtain a second comparison result.

[0182] Optionally, the determining unit is further configured to determine the first comparison result and the second comparison result as the decision result.

[0183] This application provides a terminal device that inputs a random vector into a generator, which then passes it sequentially through a fully connected layer, a reshape layer, and a convolutional layer to obtain a first feature map. The first feature map is then processed by a dual-path connection block (DPB) and upsampled multiple times to obtain multiple second feature maps. These multiple second feature maps and a sample dermatitis lesion image are sequentially input into a discriminator of a generative adversarial network (GAN). After passing through a convolutional layer and DPB, multiple max-pooling operations are performed to obtain a third feature map. This third feature map is then processed by a fully connected layer to obtain a decision result. The generator is trained using the decision result and a preset loss function to obtain a trained generator. The trained generator is then used to generate a target dermatitis lesion image. Therefore, the terminal device proposed in this application, during the image generation process, sequentially inputs random vectors into a fully connected layer, a reshape layer, and a convolutional layer in the generator to obtain a first feature map. The first feature map is then processed by DPB. In DPB, grouped convolution is used to reduce the number of parameters, effectively improving the training speed of the model. Furthermore, both the generator and the discriminator use a dual-path network (DPN) as the backbone network for feature extraction, which can effectively learn the features of dermatitis lesion images and improve the realism of the generated images. Thus, it can efficiently generate a large number of detailed and medically compliant specific dermatitis lesion images using a relatively small number of existing samples.

[0184] Figure 7 This is a schematic diagram of the composition structure of another terminal device 1 provided in an embodiment of this application. In practical applications, based on the same disclosed concept of the above embodiments, such as... Figure 7 As shown, the terminal device 1 in this embodiment includes a processor 13, a memory 14, and a communication bus 15.

[0185] In specific embodiments, the input unit 10, training unit 11, generation unit 12, conversion unit, segmentation unit, elimination unit, determination unit, positioning unit, truncating unit, separation unit, partitioning unit, offset unit, cropping unit, rotation unit, filtering unit, processing unit, calculation unit, and comparison unit can be implemented by a processor 13 located on the terminal device 1. The processor 13 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic devices used to implement the above processor functions can also be other types; this embodiment does not specifically limit them.

[0186] In this embodiment, the communication bus 15 is used to establish a connection between the processor 13 and the memory 14; when the processor 13 executes the running program stored in the memory 14, it implements the following image generation method:

[0187] A random vector is input into the generator and passed sequentially through a fully connected layer, a reshape layer, and a convolutional layer to obtain a first feature map. The first feature map is then processed by a dual-path connection block (DPB) and upsampled multiple times to obtain multiple second feature maps. These multiple second feature maps and the sample dermatitis lesion image are sequentially input into the discriminator of the generative adversarial network (GAN). After passing through a convolutional layer and DPB, multiple max-pooling operations are performed to obtain a third feature map. This third feature map is then processed by a fully connected layer to obtain a decision result. The generator is trained using the decision result and a preset loss function to obtain a trained generator. The trained generator is then used to generate the target dermatitis lesion image.

[0188] Furthermore, the processor 13 is also used to sequentially input each second feature map into multiple preset convolutions to obtain multiple third feature maps; by fusing and filtering the multiple third feature maps, each second feature map is converted into a second feature map containing target feature data; the target feature data is feature data representing the junction area between the lesion area and the normal skin area.

[0189] Furthermore, the processor 13 is also used to sequentially input multiple second feature maps containing target feature data and the dermatitis lesion image data into the discriminator of the generative adversarial network.

[0190] Furthermore, the processor 13 is also used to perform image segmentation on the initial sample dermatitis lesion image to obtain a first image region and a second image region corresponding to the initial sample dermatitis lesion image; the first image region is a region including dermatitis lesion image data, and the second image region is a region excluding dermatitis lesion image data; using a predefined specular component coefficient, the specular reflection component in the first image region is removed to obtain the first image region after specular removal; using a predefined diffuse reflection component coefficient, the diffuse reflection component in the second image region is determined; and the second image region after specular removal is generated based on the diffuse reflection component; and the dermatitis lesion image data is generated using the first image region after specular removal and the second image region after specular removal.

[0191] Furthermore, the processor 13 is also used to locate the lesion area from the initial sample dermatitis lesion image; to extract the first dermatitis lesion image from the initial sample dermatitis lesion image data with the lesion area as the center; to separate the first dermatitis lesion image into RGB three-channel image data; to generate an image mask based on the RGB three-channel image data; and to use the image mask to divide the initial sample dermatitis lesion image data into a first image region and a second image region.

[0192] Furthermore, the processor 13 is also configured to perform image offset on the multiple second feature maps and sample dermatitis lesion image data according to a preset angle to obtain offset image data; and / or, perform image cropping on the multiple second feature maps and sample dermatitis lesion image data to obtain cropped image data; and / or, perform image rotation on the multiple second feature maps and sample dermatitis lesion image data according to a preset rotation angle to obtain rotated image data; and / or, perform band-stop filtering on the multiple second feature maps and sample dermatitis lesion image data to obtain filtered image data.

[0193] Furthermore, the processor 13 is also used to sequentially input multiple second feature maps and sample dermatitis lesion image data, and / or offset image data, and / or cropped image data, and / or rotated image data, and / or filtered image data into the discriminator of the generative adversarial network.

[0194] Furthermore, the processor 13 is also configured to process the third feature map using a multilayer perceptron to obtain the feature vector corresponding to the third feature map; calculate the cosine distance between every two feature vectors; and determine the first cosine distance sum corresponding to the generated image and the second cosine distance sum corresponding to the sample image based on the cosine distance between every two feature vectors; compare the first cosine distance sum with a first preset threshold to obtain a first comparison result; compare the second cosine distance sum with a second preset threshold to obtain a second comparison result; and determine the first comparison result and the second comparison result as the decision result.

[0195] Based on the above embodiments, this application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in a terminal device. The computer program implements the data processing method described above.

[0196] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image generation method, characterized in that, The method includes: A random vector is input into the generator and passed through a fully connected layer, a reshape layer, and a convolutional layer in sequence to obtain a first feature map. The first feature map is then processed by a dual-path connection block (DPB) and upsampled multiple times to obtain multiple second feature maps. The multiple second feature maps and sample dermatitis lesion images are sequentially input into the discriminator of the generative adversarial network. After passing through convolutional layers and DPB processing, multiple max pooling operations are performed to obtain the third feature map. The third feature map is then processed by a fully connected layer to obtain the decision result. The generator is trained using the decision result and a preset loss function to obtain a trained generator; the trained generator is then used to generate a target dermatitis lesion image. The processing of the DPB includes: The input of DPB is processed sequentially through a first convolutional layer, a first activation function, and a second convolutional layer to obtain a fourth feature map; the fourth feature map is then processed by a dense connection module and input into a cascade, the input including the first feature map or the plurality of second feature maps and the sample dermatitis lesion image; The input is processed by the residual learning module to obtain the fifth feature map; The fourth feature map and the fifth feature map are processed sequentially by an adder and a splicing module, and the processed splicing result is input into the cascade. The sixth feature map obtained after the input is processed by the dense connection module is input into the cascade. The fourth feature map, the splicing result, and the sixth feature map are processed sequentially through the cascade and the second activation function to obtain the feature map after DPB processing.

2. The method according to claim 1, characterized in that, After obtaining multiple second feature maps, and before sequentially inputting the multiple second feature maps and the sample dermatitis lesion image into the discriminator of the generative adversarial network, the method further includes: Each second feature map is sequentially input into multiple preset convolutions to obtain multiple third feature maps; By fusing and filtering the multiple third feature maps, each second feature map is converted into a second feature map containing target feature data; the target feature data is feature data representing the junction area between the lesion area and the normal skin area. The step of sequentially inputting the plurality of second feature maps and sample dermatitis lesion images into the discriminator of the generative adversarial network includes: Multiple second feature maps containing target feature data and the sample dermatitis lesion image are sequentially input into the discriminator of the generative adversarial network.

3. The method according to claim 1, characterized in that, Before sequentially inputting the plurality of second feature maps and sample dermatitis lesion images into the discriminator of the generative adversarial network, the method further includes: The initial sample dermatitis lesion image is segmented to obtain a first image region and a second image region corresponding to the initial sample dermatitis lesion image; the first image region is a region that includes dermatitis lesion image data, and the second image region is a region that does not include dermatitis lesion image data; Using predefined specular component coefficients, the specular reflection component in the first image region is removed to obtain the first image region after specular removal. Using predefined diffuse reflection component coefficients, the diffuse reflection component in the second image region is determined; and the second image region after specular removal is generated based on the diffuse reflection component. The sample dermatitis lesion image is generated using the first image region after highlight removal and the second image region after highlight removal.

4. The method according to claim 3, characterized in that, The step of segmenting the initial sample dermatitis lesion image to obtain a first image region and a second image region corresponding to the initial sample dermatitis lesion image includes: Locate the lesion area from the initial sample dermatitis lesion image; Centered on the lesion area, a first dermatitis lesion image is extracted from the initial sample dermatitis lesion image; The first dermatitis lesion image is separated into RGB three-channel image data; An image mask is generated based on the RGB three-channel image data, and the initial sample dermatitis lesion image is divided into the first image region and the second image region using the image mask.

5. The method according to claim 1, characterized in that, After obtaining multiple second feature maps, and before sequentially inputting the multiple second feature maps and the sample dermatitis lesion image into the discriminator of the generative adversarial network, the method further includes: The multiple second feature maps and the sample dermatitis lesion image are shifted at a preset angle to obtain the shifted image data; And / or, crop the plurality of second feature maps and the sample dermatitis lesion image to obtain cropped image data; And / or, rotate the plurality of second feature maps and the sample dermatitis lesion image by a preset rotation angle to obtain rotated image data; And / or, band-stop filtering is performed on the plurality of second feature maps and the sample dermatitis lesion image to obtain filtered image data; The step of sequentially inputting the plurality of second feature maps and sample dermatitis lesion images into the discriminator of the generative adversarial network includes: The plurality of second feature maps and sample dermatitis lesion images, and / or the offset image data, and / or the cropped image data, and / or the rotated image data, and / or the filtered image data are sequentially input into the discriminator of the generative adversarial network.

6. The method according to claim 1, characterized in that, The fully connected layer includes a multilayer perceptron. The third feature map is processed by the fully connected layer to obtain a decision result, including: The third feature map is processed using the multilayer perceptron to obtain the feature vector corresponding to the third feature map; Calculate the cosine distance between every two feature vectors; and determine the first cosine distance sum corresponding to the generated image and the second cosine distance sum corresponding to the sample image based on the cosine distance between every two feature vectors; The first cosine distance sum is compared with a first preset threshold to obtain a first comparison result; the second cosine distance sum is compared with a second preset threshold to obtain a second comparison result. The first comparison result and the second comparison result are determined as the judgment result.

7. A terminal device, characterized in that, The terminal device includes: The input unit is used to input random vectors into the generator, which then passes them sequentially through a fully connected layer, a reshape layer, and a convolutional layer to obtain a first feature map. The first feature map is then processed by a dual-path connection block (DPB) and upsampled multiple times to obtain multiple second feature maps. These multiple second feature maps and a sample dermatitis lesion image are then sequentially input into the discriminator of the generative adversarial network (GAN). After passing through a convolutional layer and DPB, multiple max-pooling operations are performed to obtain a third feature map. This third feature map is then processed by a fully connected layer to obtain a decision result. The training unit is used to train the generator using the decision result and a preset loss function to obtain the trained generator. A generation unit is used to generate a target dermatitis lesion image using the trained generator. The input unit is further configured to process the DPB input sequentially through a first convolutional layer, a first activation function, and a second convolutional layer to obtain a fourth feature map; and to process the fourth feature map through a dense connection module and input it into a cascade, wherein the input includes the first feature map or the plurality of second feature maps and the sample dermatitis lesion image; The input is processed by the residual learning module to obtain the fifth feature map; the fourth feature map and the fifth feature map are processed by the adder and the concatenation module in sequence, and the processed concatenation result is input into the cascade. The sixth feature map obtained after the input is processed by the dense connection module is input into the cascade. The fourth feature map, the splicing result, and the sixth feature map are processed sequentially through the cascade and the second activation function to obtain the feature map after DPB processing.

8. The terminal device according to claim 7, characterized in that, The input unit is also used to sequentially input each second feature map into multiple preset convolutions to obtain multiple third feature maps; The conversion unit is used to convert each second feature map into a second feature map containing target feature data by fusing and filtering the plurality of third feature maps; the target feature data is feature data representing the junction area between the lesion area and the normal skin area. The input unit is also used to sequentially input multiple second feature maps containing target feature data and the sample dermatitis lesion image into the discriminator of the generative adversarial network.

9. A terminal device, characterized in that, The terminal device includes: a processor, a memory, and a communication bus; when the processor executes the running program stored in the memory, it implements the functionality described in claim 1. The method described in any one of the 6 methods.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements as described in claim 1. The method described in any one of the 6 methods.

Images