A skin lesion image segmentation method and system based on semantic editing

Abstract

A kind of skin lesion image segmentation method and system based on semantic editing, first, skin lesion image dataset is divided, and training set, verification set and test set are obtained, then skin lesion image dataset is preprocessed, then the generative adversarial network is constructed, the generative adversarial network includes generator and discriminator, and the generative adversarial network is trained, and the trained generator and discriminator are obtained, then the semantic editing model is constructed, and the trained generator is used to generate pseudo skin lesion image, the semantic editing model is trained using pseudo skin lesion image, and the trained semantic editing model is obtained, and the performance of semantic editing model is evaluated, finally the trained semantic editing model is used for skin lesion image segmentation.The present application overcomes the deficiencies such as relying on a large number of artificial labeling, poor segmentation effect, can make the boundary of skin image segmentation clearer, accurate, and has good robustness.

Description

Technical Field

[0001] This invention relates to the technical field of medical image segmentation, and in particular to a method and system for segmenting skin lesion images based on semantic editing. Background Technology

[0002] Observing under a microscope to find pathological features is time-consuming, laborious, and tedious. Excessive time spent in this step is equivalent to wasting more time in the entire diagnostic process. Therefore, a fully automated method is needed to rapidly segment tissues in dermatopathology images, helping doctors to quickly find sensitive features for diagnosing relevant skin diseases. However, skin lesion images often exhibit characteristics such as varying lesion sizes, irregular shapes and edges, and complex background information. They also often show low contrast and blurred boundaries between lesion areas and normal skin. Therefore, accurately segmenting lesion areas is a very challenging task.

[0003] In recent years, deep learning has achieved significant success in the field of medical image segmentation, outperforming traditional segmentation algorithms in many organ segmentation tasks. Deep learning relies on training with a large number of manually annotated images; however, in medical image segmentation, achieving precise pixel-level manual annotation is difficult because it requires domain-knowledgeable experts to invest considerable time and effort, resulting in high-quality annotated data costs and becoming a major obstacle to the development of deep learning algorithms. Therefore, unsupervised learning, which does not require manual annotation, is gaining increasing attention for medical image segmentation.

[0004] Existing unsupervised image segmentation methods, such as edge detection algorithms and region growing algorithms, while not requiring image labels for training, rely on manually designed features and parameters, resulting in cumbersome segmentation steps and insufficient robustness. When the contrast between lesion areas and normal skin is not obvious, the segmentation effect is poor, and even incorrect segmentation is prone to occur. Therefore, how to achieve efficient and accurate segmentation of pathological images while reducing reliance on manual annotation is a pressing technical problem to be solved in this field. Summary of the Invention

[0005] To address the issues of poor segmentation performance and cumbersome steps in unsupervised image segmentation methods, this invention proposes a skin lesion image segmentation method and system based on semantic editing. By generating pseudo-labels for samples from semantically edited images, the pseudo-labels are used to segment skin lesion images, resulting in segmentation performance that is closer to the actual results and clearer, more accurate segmented boundaries.

[0006] According to a first aspect of the present invention, a method for segmenting skin lesion images based on semantic editing is provided, comprising:

[0007] S1. Divide the skin lesion image dataset into training set, validation set and test set. The skin lesion images in the training set are unlabeled, while the skin lesion images in the validation set and test set are labeled.

[0008] S2. Preprocess the skin lesion image dataset;

[0009] S3. Construct a generative adversarial network, which includes a generator and a discriminator. The generator is used to convert noise vectors into pseudo-skin lesion images, and the discriminator is used to identify whether the input image is a pseudo-skin lesion image.

[0010] S4. Input the training set into the generative adversarial network and train the generative adversarial network until the generator loss is minimized and the discriminator loss is maximized. The trained generator and discriminator are obtained, and the training ends.

[0011] S5. Construct a semantic editing model, which includes a trained generator;

[0012] S6. Generate pseudo-skin lesion images using the trained generator, and train the semantic editing model using the pseudo-skin lesion images. In each round of training, evaluate the performance of the semantic editing model using the validation set, and select the best-performing semantic editing model as the trained semantic editing model. Input the test set into the trained semantic editing model, and evaluate the performance of the semantic editing model based on the obtained data.

[0013] S7. Use the trained semantic editing model for skin lesion image segmentation.

[0014] Preferably, in step S2, the preprocessing step includes:

[0015] S21. Segment all skin lesion images in the training set into images of the same size and compress them into zip format;

[0016] S22. Segment all skin lesion images in the test set and validation set into images of the same size, and perform noise reduction processing on all skin lesion images.

[0017] Based on the above technical means, the skin lesion images in the dataset are processed into data that is easy for the network model to process.

[0018] Preferably, step S4 includes:

[0019] Construct the objective function of the generative adversarial network, the expression of which is:

[0020]

[0021] Where V(D, G) is a binary classification cross-entropy function, G represents the generator, and D represents the discriminator. This means maximizing the discriminator loss while minimizing the generator loss, where x is sampled from the real data distribution P. data The noise vector z is sampled from the prior distribution P. z E x～Pdata [logD(x)] represents the mathematical expectation of the discriminator's correct classification probability of a real skin lesion image, E x～Pdata Used to measure the loss of the discriminator E represents the mathematical expectation of the probability that the discriminator misclassifies an image of a pseudo-skin lesion. z～Pz Used to measure the loss of the generator.

[0022] Based on the above techniques, an objective function for the generative adversarial network is constructed, and the losses of the generator and discriminator are quantified by the mathematical expectation of the classification probability, which facilitates subsequent optimization of the generative adversarial network based on the loss.

[0023] Preferably, step S4 includes:

[0024] S41. Input a random noise vector z sampled from a prior distribution into the generator to obtain pseudo-skin lesion image data;

[0025] S42. Input the pseudo-skin lesion image data x1 into the discriminator, obtain the discriminator's discrimination result, and calculate the loss E of the discrimination result. x～Pdata [logD(x1)]: Input a random skin lesion image x2 from the training set into the discriminator, obtain the discriminator's discrimination result, and calculate the loss of the discrimination result.

[0026] S43. Repeat steps S41-S42, and use stochastic gradient descent and adaptive moment estimation to optimize and update the weights and biases of the generative adversarial network until the loss of the discrimination result is minimized when a fake skin lesion image is input into the discriminator, and the loss of the discrimination result is maximized when a skin lesion image in the training set is input into the discriminator. Then, end the training and output the trained generative adversarial network.

[0027] Based on the above technical means, the generative adversarial network is trained using unlabeled skin lesion images as the training set, thereby achieving unsupervised learning and eliminating the dependence on manually labeled data.

[0028] Preferably, in step S5, the semantic editing model further includes: a direction matrix A, a reconstructor R, and a generator in a generative adversarial network, wherein the direction matrix A is an orthogonal matrix of order d, where d is the latent spatial dimension of the generative adversarial network, and the content of each column is the change direction parameter of the generative adversarial network, and the direction matrix A is randomly generated by a Gaussian distribution; the reconstructor R is a ResNet-18 network.

[0029] Each column in the orientation matrix A corresponds to a direction of change. After the image passes through the orientation matrix A, its different features will be highlighted, thus facilitating semantic editing.

[0030] Preferably, in step S6, the step of training the semantic editing model includes:

[0031] S611. Input the noise vector z into the trained generator to obtain the pseudo-skin lesion image I1; the noise vector z is a noise vector sampled from the prior distribution;

[0032] S612. Select a random number k as the direction index, where k specifies the column of the direction matrix A. Select another random number ε as the shift amplitude, and transform the noise vector z into z + A(εe). k The transformed noise vector z is input into the generator in the generative adversarial network to obtain the pseudo-skin lesion image I2;

[0033] S613. Input the pseudo-skin lesion images I1 and I2 into the reconstructor R, and the reconstructor R obtains the predicted change direction index. and predicted shift amplitude

[0034] S614. Construct a loss function that minimizes the semantic editing model, and substitute parameters k into the loss function. ε、 The reconstructor R and the direction matrix A are updated using the adaptive moment estimation method.

[0035] The expression for minimizing the loss function is:

[0036]

[0037] Among them, L cl (·, ·) represent categorical terms, using the cross-entropy function, L r (·,·) represents the regression term, using the Dice coefficient, and λ is the weight coefficient;

[0038] S615. Set the iteration count n, and repeat steps S611-S614 until the number of iterations reaches n. End the training and obtain the reconstructor R, the orientation matrix A, the shift magnitude ε, and the change direction index k. The k-th column of the orientation matrix A, multiplied by the shift magnitude ε, is used to remove the background direction h. bg .

[0039] Based on the above techniques, the orientation matrix A and the reconstructor R are jointly optimized. Therefore, the columns of A are sought in the minimization process, making the corresponding image transformations easier to distinguish from each other, thus simplifying the classification problem of the reconstructor. Furthermore, the regression term Lr makes the discovered orientation shifts have a continuous effect, thereby preventing sudden changes.

[0040] Preferably, in step S6, the step of calculating the performance of the semantic editing model includes:

[0041] S621. After each training iteration of the semantic editing model, input validation set data into the semantic editing model obtained in this training round. The semantic editing model outputs predicted values. Evaluate the predicted values ​​and the real data in the validation set using the Dice coefficient. After the semantic editing model completes training, save the semantic editing model with the highest Dice coefficient as the final semantic editing model. The expression for the Dice coefficient is:

[0042]

[0043] Where, pred represents the set of predicted values ​​output by the semantic editing model, true is the set of true values, 2×(pred∩true) represents the intersection of the true values ​​and the predicted values ​​output by the semantic editing model, and pred∪true represents the number of elements in the true values ​​and the predicted values.

[0044] S622. Input the test set into the final model of the obtained semantic editing model. The semantic editing model outputs the predicted value. Then, evaluate the predicted value with the real data of the test set using the Dice coefficient to obtain the performance of the semantic editing model.

[0045] Based on the above technical means, the segmentation accuracy of the semantic editing model is evaluated by the Dice coefficient, thereby measuring the performance of the semantic editing model through accuracy, and the model with the best performance is saved in this process.

[0046] Preferably, in step S7, the specific steps for using the trained semantic editing model for skin lesion image segmentation include:

[0047] S71. Obtain the skin lesion image to be segmented, generate an Ireal distribution based on the skin lesion image to be segmented, input a noise vector z that follows the Ireal distribution into the generator, and obtain a pseudo skin lesion image;

[0048] S72. Perform a background removal direction transformation on the noise vector z to obtain z+h bg Input vector z+h to the generator bg This yields an image with background removed to remove pseudo-skin lesions;

[0049] S73. Set the segmentation threshold θ for the semantic editing model, compare the average pixel intensity of the three color channels of the background-removed pseudo-skin lesion image with the segmentation threshold θ, and label the background-removed pseudo-skin lesion image. The labeling expression is:

[0050] Mask(G(z)) = [G(z+h bg )<θ],

[0051] After labeling, pixels smaller than the segmentation threshold in the background-removed pseudo-skin lesion image are labeled, while pixels larger than the segmentation threshold are discarded; the labeled background-removed pseudo-skin lesion image is a pseudo-label.

[0052] S74. Input pseudo-labels and pseudo-skin lesion images into the segmentation network to obtain semantically edited skin lesion images. The segmentation network is the U-net segmentation network.

[0053] Based on the above techniques, feature extraction is performed on the input skin lesion image by extracting a noise vector z that follows an Ireal distribution. Then, background removal is performed to make the lesion area in the skin lesion image more prominent. By comparing with a segmentation threshold, normal skin in the skin lesion image is filtered out to obtain semantic segmentation labels. The labels are then used as masks and input together with the image features into the U-net segmentation network. The U-net segmentation network can extract features from the input image and concatenate the extracted features to achieve the segmentation purpose and obtain a semantically edited skin lesion image.

[0054] Preferably, step S72 includes:

[0055] S721. Take N noise vectors z and perform background removal direction transformation. The shift magnitude of the N background removal directions increases in an arithmetic sequence. Input the N transformed vectors into the generator to obtain N background removal pseudo-skin lesion images.

[0056] S722. Calculate the average value of N background-removed pseudo-skin lesion images, and use the average value of the N background-removed pseudo-skin lesion images as the new background-removed pseudo-skin lesion image.

[0057] Based on the above techniques, the background removal of pseudo-skin lesion images is smoothed to make the segmentation effect more accurate.

[0058] According to another aspect of the present invention, a skin lesion image segmentation system based on semantic editing is provided, comprising:

[0059] The data partitioning module is used to partition the skin lesion image dataset into training set, validation set and test set. The skin lesion images in the training set are unlabeled, while the skin lesion images in the validation set and test set are labeled.

[0060] The data processing module is used to preprocess the skin lesion image dataset;

[0061] A network construction module is used to construct a generative adversarial network, which includes a generator and a discriminator. The generator is used to convert noise vectors into fake skin lesion images, and the discriminator is used to identify whether the input image is a fake skin lesion image.

[0062] The network training module is used to input the training set into the generative adversarial network and train the generative adversarial network until the generator loss is minimized and the discriminator loss is maximized, thus obtaining the trained generator and discriminator and ending the training.

[0063] A segmentation building block is used to construct a semantic editing model, which includes a trained generator;

[0064] The segmentation training module is used to generate pseudo-skin lesion images using the trained generator. These pseudo-skin lesion images are then used to train the semantic editing model. During each training round, the performance of the semantic editing model is evaluated using a validation set. After training, the best-performing semantic editing model is selected as the trained semantic editing model. The test set is then input into the trained semantic editing model, and the performance of the semantic editing model is evaluated based on the obtained data.

[0065] The image segmentation module is used to segment skin lesion images using the trained semantic editing model.

[0066] Compared to existing technologies, this invention constructs a Generative Adversarial Network (GAN) and searches for interpretable semantic directions in the GAN's latent space, thereby enabling fully unsupervised image segmentation. By utilizing the background removal directions in the GAN's latent space, the foreground and background of the generator-generated image can be effectively distinguished, resulting in a suitable segmentation mask. This mask is then used to segment skin lesion images. This invention overcomes the shortcomings of relying heavily on manual annotation during training and poor segmentation results. It enables segmentation of dermoscopic images to more closely approximate real-world results, with clearer and more accurate segmented boundaries, and exhibits better robustness. Attached Figure Description

[0067] Figure 1This is a schematic diagram illustrating the steps of the semantic editing-based skin lesion image segmentation method proposed in this embodiment of the invention.

[0068] Figure 2 This diagram illustrates the training process of the semantic editing model proposed in this embodiment of the invention.

[0069] Figure 3 This diagram illustrates the impact of semantic editing direction on images in an embodiment of the present invention.

[0070] Figure 4 This diagram illustrates the semantic editing process proposed in this embodiment of the invention.

[0071] Figure 5 This diagram illustrates the segmentation results of skin lesion images in an embodiment of the present invention.

[0072] Figure 6 This is a schematic diagram illustrating the structure of the skin lesion image segmentation system based on semantic editing proposed in this embodiment of the invention. Detailed Implementation

[0073] This invention provides a method and system for segmenting skin lesion images based on semantic editing, which can effectively improve detection performance.

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention and are only used for illustrative purposes, and should not be construed as limiting the present patent. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The accompanying drawings are only for illustrative purposes and should not be construed as limiting the present patent; in order to better illustrate the embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent actual dimensions. The description of directions such as "up" and "down" is not a limitation of the present patent; it is understandable for those skilled in the art that some well-known content may be omitted in the drawings; the description of positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0075] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] Example 1

[0077] This embodiment provides a skin lesion image segmentation method based on semantic editing. See [link to previous document]. Figure 1 ,include:

[0078] S1. Divide the skin lesion image dataset into training set, validation set and test set. The skin lesion images in the training set are unlabeled, while the skin lesion images in the validation set and test set are labeled.

[0079] S2. Preprocess the skin lesion image dataset;

[0080] S3. Construct a generative adversarial network, which includes a generator and a discriminator. The generator is used to convert noise vectors into pseudo-skin lesion images, and the discriminator is used to identify whether the input image is a pseudo-skin lesion image.

[0081] S4. Input the training set into the generative adversarial network and train the generative adversarial network until the generator loss is minimized and the discriminator loss is maximized. The trained generator and discriminator are obtained, and the training ends.

[0082] S5. Construct a semantic editing model, which includes a trained generator;

[0083] S6. Generate pseudo-skin lesion images using the trained generator, and train the semantic editing model using the pseudo-skin lesion images. In each round of training, evaluate the performance of the semantic editing model using the validation set, and select the best-performing semantic editing model as the trained semantic editing model. Input the test set into the trained semantic editing model, and evaluate the performance of the semantic editing model based on the obtained data.

[0084] S7. Use the trained semantic editing model for skin lesion image segmentation.

[0085] Step S2 includes:

[0086] S21. Segment all skin lesion images in the training set into images of the same size and compress them into zip format;

[0087] S22. Segment all skin lesion images in the test set and validation set into images of the same size, and perform noise reduction processing on all skin lesion images.

[0088] In this embodiment, the skin lesion images in the training set are all cropped to 256×256 pixels and compressed into zip format for input into the generative adversarial network. The skin lesion images in the validation and test sets are all cropped to 128×128 pixels, and noise such as hair and ruler scales are removed.

[0089] Step S4 includes:

[0090] Construct the objective function of the generative adversarial network, the expression of which is:

[0091]

[0092] Where V(D, G) is a binary classification cross-entropy function, G represents the generator, and D represents the discriminator. This means maximizing the discriminator loss while minimizing the generator loss, where x is sampled from the real data distribution P. data The noise vector z is sampled from the prior distribution P. z E x～Pdata [logD(x)] represents the mathematical expectation of the discriminator's correct classification probability of a real skin lesion image, E x～Pdata Used to measure the loss of the discriminator E represents the mathematical expectation of the probability that the discriminator misclassifies an image of a pseudo-skin lesion. z～Pz Used to measure the loss of the generator.

[0093] The core idea of ​​Generative Adversarial Networks (GANs) models originates from Nash equilibrium in game theory, and includes a generator and a discriminator. The generator aims to continuously capture the probability distribution of real data in the training set, and then generate fake data by inputting random noise; while the discriminator aims to correctly distinguish whether the input data comes from real data or data generated by the generator.

[0094] In this embodiment, the generative adversarial network model is the StyleGAN2 network. During training, 2000 training set data are fed into the StyleGAN2 network for training, Adam is used as the optimizer, the batch size is 16, and the number of training iterations is 9000 epochs.

[0095] The ultimate goal of this loss function is to minimize the KL divergence between the generated distribution and the true distribution. The first term of the expression represents the optimization of the discriminator. When the input is real sample data, the output value is expected to be as large as possible, that is, the prediction result of the real sample is as close to 1 as possible. The second term of the formula is random noise z input to the generator. The noise generates pseudo data G(z) through the generator. The discriminator is expected to make the error judgment result 1-D(G(z)) of the pseudo data as small as possible.

[0096] Example 2

[0097] Based on Example 1, the generative adversarial network is trained according to the objective function of the generative adversarial network.

[0098] Step S4 includes:

[0099] S41. Input a random noise vector z sampled from a prior distribution into the generator to obtain pseudo-skin lesion image data;

[0100] S42. Input the pseudo-skin lesion image data x1 into the discriminator, obtain the discriminator's discrimination result, and calculate the loss E of the discrimination result. x～Pdata [logD(x1)]: Input a random skin lesion image x2 from the training set into the discriminator, obtain the discriminator's discrimination result, and calculate the loss of the discrimination result.

[0101] S43. Repeat steps S41-S42, and use stochastic gradient descent and adaptive moment estimation to optimize and update the weights and biases of the generative adversarial network until the loss of the discrimination result is minimized when a fake skin lesion image is input into the discriminator, and the loss of the discrimination result is maximized when a skin lesion image in the training set is input into the discriminator. Then, end the training and output the trained generative adversarial network.

[0102] In step S5, the semantic editing model further includes: a direction matrix A, a reconstructor R, and a generator in a generative adversarial network. The direction matrix A is an orthogonal matrix of order d, where d is the latent spatial dimension of the generative adversarial network, and the content of each column is the change direction parameter of the generative adversarial network. The direction matrix A is randomly generated by a Gaussian distribution. The reconstructor R is a ResNet-18 network.

[0103] The orientation matrix A is randomly generated using a Gaussian distribution. Each column in the orientation matrix A corresponds to a direction of change. The Gaussian distribution can yield more semantically meaningful directions. After the image passes through the orientation matrix A, different features of the image will be highlighted, thus facilitating semantic editing.

[0104] The reconstructor R is used to reconstruct the orientation index k and shift magnitude ε. The reconstructor R is an 18-layer network model based on ResNet. The main feature of ResNet18 is the introduction of the concept of residual blocks. In the residual blocks, skip connections can directly connect the input to the output, allowing the network to learn residual information, thereby improving feature extraction and processing.

[0105] In step S6, see Figure 2 The steps for training a semantic editing model include:

[0106] S611. Input the noise vector z into the trained generator to obtain the pseudo-skin lesion image I1; the noise vector z is a noise vector sampled from the prior distribution;

[0107] The noise vector z sampled from the prior distribution is the latent code, which can influence the features and style of the generated samples. By interpolating, manipulating, or controlling the latent code in the latent space, samples with different features, styles, or attributes can be generated.

[0108] S612. Select a random number k as the direction index, where k specifies the column of the direction matrix A. Select another random number ε as the shift amplitude, and transform the noise vector z into z + A(εe). k The transformed noise vector z is input into the generator in the generative adversarial network to obtain the pseudo-skin lesion image I2;

[0109] S613. Input the pseudo-skin lesion images I1 and I2 into the reconstructor R, and the reconstructor R obtains the predicted change direction index. and predicted shift amplitude

[0110] S614. Construct a loss function that minimizes the semantic editing model, and substitute parameters k into the loss function. ε、 The reconstructor R and the direction matrix A are updated using the adaptive moment estimation method.

[0111] The expression for minimizing the loss function is:

[0112]

[0113] Among them, L cl (·, ·) represent categorical terms, using the cross-entropy function, L r (·,·) represents the regression term, using the Dice coefficient, and λ is the weight coefficient;

[0114] S615. Set the iteration count n, and repeat steps S611-S614 until the number of iterations reaches n. End the training and obtain the reconstructor R, the orientation matrix A, the shift magnitude ε, and the change direction index k. The k-th column of the orientation matrix A, multiplied by the shift magnitude ε, is used to remove the background direction h. bg .

[0115] The direction index k is sampled from the uniform distribution U[1, K], and the shift amplitude ε is sampled from the uniform distribution U[-6, 6].

[0116] During training, the orientation matrix A and the reconstructor R are jointly optimized. The columns of A are sought in the minimization process to make the corresponding image transformations easier to distinguish from each other, thus simplifying the classification problem of the reconstructor. Furthermore, the regression term Lr ensures that the discovered orientation shifts have a continuous effect, thereby preventing abrupt changes.

[0117] See Figure 3 After the image is moved in the background removal direction, the foreground object, i.e. the lesion area, remains unchanged, while the background pixels turn white to facilitate segmentation.

[0118] In this embodiment, the training iterations are 30,000 epochs, the initial learning rate is 0.005, and the decay rate is 0.2 after 4,000 iterations.

[0119] Step S6, the step of calculating the performance of the semantic editing model, includes:

[0120] S621. After each training iteration of the semantic editing model, input validation set data into the semantic editing model obtained in this training round. The semantic editing model outputs predicted values. Evaluate the predicted values ​​and the real data in the validation set using the Dice coefficient. After the semantic editing model completes training, save the semantic editing model with the highest Dice coefficient as the final semantic editing model. The expression for the Dice coefficient is:

[0121]

[0122] Where, pred represents the set of predicted values ​​output by the semantic editing model, true is the set of true values, 2×(pred∩true) represents the intersection of the true values ​​and the predicted values ​​output by the semantic editing model, and pred∪true represents the number of elements in the true values ​​and the predicted values.

[0123] S622. Input the test set into the final model of the obtained semantic editing model. The semantic editing model outputs the predicted value. Then, evaluate the predicted value with the real data of the test set using the Dice coefficient to obtain the performance of the semantic editing model.

[0124] The segmentation accuracy of the semantic editing model is evaluated using the Dice coefficient, thereby measuring the performance of the semantic editing model through accuracy. In this process, the best-performing model is saved, and then the performance of the best-performing model is measured using a test set.

[0125] In step S7, see Figure 4 The specific steps for using the trained semantic editing model for skin lesion image segmentation include:

[0126] S71. Obtain the skin lesion image to be segmented, generate an Ireal distribution based on the skin lesion image to be segmented, input a noise vector z that follows the Ireal distribution into the generator, and obtain a pseudo skin lesion image;

[0127] S72. Perform a background removal direction transformation on the noise vector z to obtain z+h bg Input vector z+h to the generator bg This yields an image with background removed to remove pseudo-skin lesions;

[0128] S73. Set the segmentation threshold θ for the semantic editing model, compare the average pixel intensity of the three color channels of the background-removed pseudo-skin lesion image with the segmentation threshold θ, and label the background-removed pseudo-skin lesion image. The labeling expression is:

[0129] Mask(G(z)) = [G(z+h bg )<θ],

[0130] After labeling, pixels smaller than the segmentation threshold in the background-removed pseudo-skin lesion image are labeled, while pixels larger than the segmentation threshold are discarded; the labeled background-removed pseudo-skin lesion image is a pseudo-label.

[0131] S74. Input pseudo-labels and pseudo-skin lesion images into the segmentation network to obtain semantically edited skin lesion images. The segmentation network is the U-net segmentation network.

[0132] U-net segmentation network is a fully convolutional neural network model that has outstanding performance in medical image processing. It can extract features from input images and concatenate the extracted features. It can fuse shallow and deep features of images to achieve segmentation. In addition, U-net segmentation network performs data augmentation on the input data during the calculation process, so that it can achieve good accuracy even with a small dataset.

[0133] Feature extraction is performed on the input skin lesion image by extracting a noise vector z that follows an Ireal distribution. Background removal is then applied to make the lesion areas more prominent. Normal skin in the image is filtered out by comparing the vector with a segmentation threshold, resulting in semantic segmentation labels. These labels are then used as masks along with the image features and input into the U-Net segmentation network. The U-Net network extracts features from the input image and concatenates the extracted features to achieve segmentation. See [link to relevant documentation]. Figure 5 This yields semantically edited images of skin lesions.

[0134] Step S72 includes:

[0135] S721. Take N noise vectors z and perform background removal direction transformation. The shift magnitude of the N background removal directions increases in an arithmetic sequence. Input the N transformed vectors into the generator to obtain N background removal pseudo-skin lesion images.

[0136] S722. Calculate the average value of N background-removed pseudo-skin lesion images, and use the average value of the N background-removed pseudo-skin lesion images as the new background-removed pseudo-skin lesion image.

[0137] In this embodiment, five images with continuously varying displacement amplitudes are selected for processing in each training session. The displacement amplitudes are 22, 24, 26, 28, and 30, respectively. The average value of the five images is used to smooth the background removal image of the pseudo-skin lesion, resulting in more accurate segmentation.

[0138] Example 3

[0139] This embodiment provides a skin lesion image segmentation system based on semantic editing. See [link to documentation]. Figure 6 ,include:

[0140] The data partitioning module is used to partition the skin lesion image dataset into training set, validation set and test set. The skin lesion images in the training set are unlabeled, while the skin lesion images in the validation set and test set are labeled.

[0141] The data processing module is used to preprocess the skin lesion image dataset;

[0142] A network construction module is used to construct a generative adversarial network, which includes a generator and a discriminator. The generator is used to convert noise vectors into fake skin lesion images, and the discriminator is used to identify whether the input image is a fake skin lesion image.

[0143] The network training module is used to input the training set into the generative adversarial network and train the generative adversarial network until the generator loss is minimized and the discriminator loss is maximized, thus obtaining the trained generator and discriminator and ending the training.

[0144] A segmentation building block is used to construct a semantic editing model, which includes a trained generator;

[0145] The segmentation training module is used to generate pseudo-skin lesion images using the trained generator. These pseudo-skin lesion images are then used to train the semantic editing model. During each training round, the performance of the semantic editing model is evaluated using a validation set. After training, the best-performing semantic editing model is selected as the trained semantic editing model. The test set is then input into the trained semantic editing model, and the performance of the semantic editing model is evaluated based on the obtained data.

[0146] The image segmentation module is used to segment skin lesion images using the trained semantic editing model.

[0147] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for segmenting skin lesion images based on semantic editing, characterized in that, Includes the following steps: S1. Divide the skin lesion image dataset into training set, validation set and test set. The skin lesion images in the training set are unlabeled, while the skin lesion images in the validation set and test set are labeled. S2. Preprocess the skin lesion image dataset; S3. Construct a generative adversarial network, which includes a generator and a discriminator. The generator is used to convert noise vectors into pseudo-skin lesion images, and the discriminator is used to identify whether the input image is a pseudo-skin lesion image. S4. Input the training set into the generative adversarial network and train the generative adversarial network until the generator loss is minimized and the discriminator loss is maximized. The trained generator and discriminator are obtained, and the training ends. S5. Construct a semantic editing model, which includes a trained generator; In step S5, the semantic editing model further includes: a direction matrix A and a reconstructor R, wherein the direction matrix A is an orthogonal matrix of order d, where d is the latent spatial dimension of the generative adversarial network, and the content of each column is the change direction parameter of the generative adversarial network; the direction matrix A is randomly generated by a Gaussian distribution; and the reconstructor R is a ResNet-18 network. S6. Generate pseudo-skin lesion images using the trained generator, and train the semantic editing model using the pseudo-skin lesion images. In each round of training, evaluate the performance of the semantic editing model using the validation set, and select the best-performing semantic editing model as the trained semantic editing model. Input the test set into the trained semantic editing model, and evaluate the performance of the semantic editing model based on the obtained data. S7. Use the trained semantic editing model for skin lesion image segmentation.

2. The skin lesion image segmentation method based on semantic editing according to claim 1, characterized in that, In step S2, the preprocessing step includes: S21. Segment all skin lesion images in the training set into images of the same size and compress them into zip format; S22. Segment all skin lesion images in the test set and validation set into images of the same size, and perform noise reduction processing on all skin lesion images.

3. The skin lesion image segmentation method based on semantic editing according to claim 2, characterized in that, In step S4, Construct the objective function of the generative adversarial network, the expression of which is: Where V(D,G) is a binary cross-entropy function, G represents the generator, and D represents the discriminator. This means maximizing the discriminator loss while minimizing the generator loss, where x is sampled from the real data distribution P. data The noise vector z is sampled from the prior distribution P. z , This represents the mathematical expectation of the probability that the discriminator correctly classifies a real skin lesion image. Used to measure the loss of the discriminator This represents the mathematical expectation of the probability that the discriminator misclassifies images of pseudo-skin lesions. Used to measure the loss of the generator.

4. The skin lesion image segmentation method based on semantic editing according to claim 3, characterized in that, In step S4, the training set is input into the generative adversarial network, and the process of training the generative adversarial network is as follows: S41. Input several skin lesion images from the training set into the discriminator and train the discriminator; S42. Input a random noise vector sampled from a prior distribution into the generator to obtain pseudo-skin lesion image data; S43. Input the pseudo-skin lesion image data x1 into the discriminator, obtain the discriminator's discrimination result, and calculate the loss of the discrimination result. A random skin lesion image x2 from the training set is input into the discriminator to obtain the discriminator's discrimination result, and the loss of the discrimination result is calculated. S44. Repeat steps S41-S43, and use stochastic gradient descent and adaptive moment estimation to optimize and update the weights and biases of the generative adversarial network until the loss of the discrimination result is minimized when a fake skin lesion image is input into the discriminator, and the loss of the discrimination result is maximized when a skin lesion image in the training set is input into the discriminator. Then, end the training and output the trained generative adversarial network.

5. The skin lesion image segmentation method based on semantic editing according to claim 4, characterized in that, In step S6, the steps for training the semantic editing model include: S611. Input the noise vector z into the trained generator to obtain the pseudo-skin lesion image I1; the noise vector z is a noise vector sampled from the prior distribution; S612. Select a random number k as the direction index, where k specifies the column of the direction matrix A. Select another random number ε as the shift amplitude, and transform the noise vector z into z + A(εe). k The transformed noise vector z is input into the generator in the generative adversarial network to obtain the pseudo-skin lesion image I2; S613. Input the pseudo-skin lesion images I1 and I2 into the reconstructor R, and the reconstructor R obtains the predicted change direction index. and predicted shift magnitude S614. Construct a loss function that minimizes the semantic editing model, and substitute parameters k into the loss function. ε、 The reconstructor R and the direction matrix A are updated using the adaptive moment estimation method. The expression for minimizing the loss function is: Among them, L cl (·, ·) represent categorical terms, using the cross-entropy function, L r (·,·) represents the regression term, using the Dice coefficient, and λ is the weight coefficient; S615. Set the iteration count n, and repeat steps S611-S614 until the number of iterations reaches n. End the training and obtain the reconstructor R, the orientation matrix A, the shift magnitude ε, and the change direction index k. The k-th column of the orientation matrix A, multiplied by the shift magnitude ε, is used to remove the background direction h. bg .

6. The skin lesion image segmentation method based on semantic editing according to claim 5, characterized in that, Step S6, the step of calculating the performance of the semantic editing model, includes: S621. After each training iteration of the semantic editing model, input validation set data into the semantic editing model obtained in this training round. The semantic editing model outputs predicted values. Evaluate the predicted values ​​and the real data in the validation set using the Dice coefficient. After the semantic editing model completes training, save the semantic editing model with the highest Dice coefficient as the final semantic editing model. The expression for the Dice coefficient is: Where, pred represents the set of predicted values ​​output by the semantic editing model, true is the set of true values, 2×(pred∩true) represents the intersection of the true values ​​and the predicted values ​​output by the semantic editing model, and pred∪true represents the number of elements in the true values ​​and the predicted values. S622. Input the test set into the final model of the obtained semantic editing model. The semantic editing model outputs the predicted value. Then, evaluate the predicted value with the real data of the test set using the Dice coefficient to obtain the performance of the semantic editing model.

7. The skin lesion image segmentation method based on semantic editing according to claim 6, characterized in that, In step S7, the specific steps for using the trained semantic editing model for skin lesion image segmentation include: S71. Obtain the skin lesion image to be segmented, generate an Ireal distribution based on the skin lesion image to be segmented, input a noise vector z that follows the Ireal distribution into the generator, and obtain a pseudo skin lesion image; S72. Perform a background removal direction transformation on the noise vector z to obtain z+h bg Input vector z+h to the generator bg This yields an image with background removed to remove pseudo-skin lesions; S73. Set the segmentation threshold θ for the semantic editing model, compare the average pixel intensity of the three color channels of the background-removed pseudo-skin lesion image with the segmentation threshold θ, and label the background-removed pseudo-skin lesion image. The labeling expression is: Mask(G(z))=[G(z+h bg )<θ], After labeling, pixels smaller than the segmentation threshold in the background-removed pseudo-skin lesion image are labeled, while pixels larger than the segmentation threshold are discarded; the labeled background-removed pseudo-skin lesion image is a pseudo-label. S74. Input pseudo-labels and pseudo-skin lesion images into the segmentation network to obtain semantically edited skin lesion images. The segmentation network is the U-net segmentation network.

8. The skin lesion image segmentation method based on semantic editing according to claim 7, characterized in that, Step S72 includes: S721. Take N noise vectors z and perform background removal direction transformation. The shift magnitude of the N background removal directions increases in an arithmetic sequence. Input the N transformed vectors into the generator to obtain N background removal pseudo-skin lesion images. S722. Calculate the average value of N background-removed pseudo-skin lesion images, and use the average value of the N background-removed pseudo-skin lesion images as the new background-removed pseudo-skin lesion image.

9. A skin lesion image segmentation system based on semantic editing, the system being used to implement the skin lesion image segmentation method based on semantic editing as described in claim 1, characterized in that, include: The data partitioning module is used to partition the skin lesion image dataset into training set, validation set and test set. The skin lesion images in the training set are unlabeled, while the skin lesion images in the validation set and test set are labeled. The data processing module is used to preprocess the skin lesion image dataset; A network construction module is used to construct a generative adversarial network, which includes a generator and a discriminator. The generator is used to convert noise vectors into fake skin lesion images, and the discriminator is used to identify whether the input image is a fake skin lesion image. The network training module is used to input the training set into the generative adversarial network and train the generative adversarial network until the generator loss is minimized and the discriminator loss is maximized, thus obtaining the trained generator and discriminator and ending the training. A segmentation building block is used to construct a semantic editing model, which includes a trained generator; The segmentation training module is used to generate pseudo-skin lesion images using the trained generator. These pseudo-skin lesion images are then used to train the semantic editing model. During each training round, the performance of the semantic editing model is evaluated using a validation set. After training, the best-performing semantic editing model is selected as the trained semantic editing model. The test set is then input into the trained semantic editing model, and the performance of the semantic editing model is evaluated based on the obtained data. The image segmentation module is used to segment skin lesion images using the trained semantic editing model.

Images