Image editing-based vitiligo lesion region segmentation method and device, and electronic equipment

Abstract

The application provides a vitiligo lesion area segmentation method and device based on image editing and electronic equipment, and belongs to the technical field of image processing. The method comprises the following steps: contrast learning pre-training of dual-mode vitiligo data; data set expansion based on Poisson image editing mode; pre-processing of the expanded data set is input into a segmentation neural network for training; the model parameters based on the above segmentation training are used as the baseline segmentation model of the semi-supervised framework of the student-teacher model, and the unmarked data and the marked data are jointly trained; the prediction result of the unmarked data input into the teacher model is refined to obtain a pseudo label as the corresponding label to train the network; and the segmentation network obtains the segmentation result of the vitiligo lesion area. Thus, the pre-trained segmentation network is used, and a reliable neural network model is obtained by using image editing to automatically segment the vitiligo lesion area. This process does not need to rely on manual work, and the segmentation efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a planar image segmentation method, apparatus, and electronic device. Background Technology

[0002] Vitiligo is a common acquired depigmenting skin disease, mainly characterized by white patches on the skin and mucous membranes, with some patients also experiencing whitening of the hair. The global prevalence of vitiligo is 0.5%-2%. Unfortunately, there are currently no specific auxiliary examinations for diagnosing vitiligo; diagnosis relies primarily on medical history and clinical manifestations. Therefore, the development and progression of the white patches becomes a crucial basis for diagnosis. Diagnosing fully developed, typical vitiligo lesions is relatively easy. However, in the early stages, many patients present with atypical clinical manifestations, merely a lightening of the skin color, and the white patches are only noticeable within a few days to a few weeks. In such a short time, it is difficult to observe the development and evolution of the white patches. Even experienced doctors find it difficult to differentiate it from other hypopigmentation disorders. The diagnosis of skin diseases is mainly based on the morphology of the skin lesions. Dermatologists make diagnoses by observing various primary and secondary lesions. However, visual observation is highly subjective; doctors with different levels of experience may reach different conclusions. Light conditions, the stage and evolution of the lesions, and whether treatment has been administered can all influence a doctor's judgment. Artificial intelligence technology based on deep learning can precisely solve this problem. Artificial intelligence (AI) technology uses deep learning models to automatically segment skin diseases, answering questions such as what lesions are present in images and where they are located. The results of segmentation algorithms can effectively assist doctors in clinical diagnosis and assessment of disease severity. For example, the International Skin Imaging Collaboration uses multi-dimensional and multimodal skin image data as raw material. Therefore, AI algorithms have significant advantages in dermatology. Studies have confirmed that in skin tumor scenarios, the performance of a "human-machine collaboration" model is superior to that of AI algorithms alone or clinicians, reflecting the importance of combining human expertise with computer algorithms from a clinical perspective. Summary of the Invention

[0003] To address the problems existing in the prior art, embodiments of the present invention provide a method, apparatus, and electronic device for segmenting vitiligo lesions.

[0004] In a first aspect, the present invention provides a method for segmenting vitiligo lesion regions based on image editing, comprising:

[0005] After acquiring medical image data, we use professional annotation tools to annotate the medical image data and obtain the corresponding annotations for the medical images. We then perform self-supervised multimodal data pre-training learning based on image pairs composed of human skin images in natural scenes and human skin images under medical Wood's lamp illumination.

[0006] To augment the limited amount of expert-annotated data, an image editing method based on the Poisson equation was used to edit the vitiligo image data. By randomly sampling the foreground lesion region of different source images, the target image region was edited to obtain the edited result image. In this way, more natural and diverse vitiligo image data was obtained, while expanding the scale of the dataset.

[0007] After data preprocessing, the expanded dataset is input into the segmentation neural network to obtain the semantic segmentation prediction results of the segmentation network and save the trained model parameters.

[0008] Based on the saved model parameters, a relatively reliable prediction result can be obtained. Therefore, the model parameters trained based on the above segmentation can be used as a baseline segmentation model and added to a semi-supervised framework called the student-teacher model. Unlabeled data is input into the teacher model to obtain prediction results. After further adaptive refinement, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area.

[0009] The image editing method described herein is based on the Poisson image editing principle to edit the vitiligo lesion area in the original image, and randomly selects different lesion areas to edit into a new target image, thereby obtaining a large amount of labeled data.

[0010] In some embodiments, the image-edit-based vitiligo lesion region segmentation method involves augmenting and enhancing expert-annotated data through image editing before segmenting the vitiligo lesion region to obtain the predicted result output by the model. This includes:

[0011] For a small amount of expert-annotated data, the data was augmented. The image editing method based on the Poisson equation was used to edit the vitiligo image data. By randomly sampling the foreground lesion area of ​​different source images, the target image area was edited to obtain the edited result image.

[0012] The edited image is input into the segmentation neural network to obtain the semantic segmentation prediction result of the segmentation network and save the trained model parameters;

[0013] In this embodiment, the image editing process consists of four inputs: a source image, its corresponding annotation image, a target image, and a corresponding annotation image of the human body region in the target image. In a single editing process, the source image and its corresponding annotation image are input first. A connected component detection algorithm is applied to the annotation image to segment different annotation regions into multiple independent connected regions. Subsequently, several connected regions are randomly sampled from the corresponding regions in the source image; each connected region serves as the basic unit for performing the editing operation. Next, feasible regions in the target image are searched for editing. For each connected region to be edited, only feasible regions are selected for image editing, ensuring that the edited region does not exceed the human body region or areas covered by clothing, etc. Finally, the annotations corresponding to the connected regions and the annotations of the target image are merged to complete one image editing operation.

[0014] In some embodiments, the pre-trained model refers to a neural network pre-trained for a segmentation task, including:

[0015] The self-supervised pre-training learning based on the bimodal vitiligo data provides an easily convergent pre-trained neural network for the original segmentation network.

[0016] The expanded dataset obtained based on image editing is further trained on the weights of the pre-trained model using a supervised segmentation task to obtain a pre-trained neural network.

[0017] The model weights trained based on the segmentation task are further trained and converged into a pre-trained neural network in a semi-supervised framework of a student-teacher model, using unlabeled data.

[0018] In some embodiments, the semi-supervised framework of the student-teacher model includes two identically constructed models—a student model and a teacher model—with different parameters, including:

[0019] The teacher model and the student model's initial model weights are loaded onto the neural network model trained on the expanded dataset obtained based on image editing. Meanwhile, the teacher model's parameters are not backpropagated with gradients; instead, their model parameters are obtained by weighting the original model parameters with the updated parameters of the student model and updating them with momentum.

[0020] The unlabeled data is input into the teacher model to obtain semantic prediction results. After further adaptive refinement screening, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. Thus, the pseudo-labels of the unlabeled data are updated by the output of the teacher network.

[0021] The unlabeled data is input into the student model and supervised segmentation training is performed using the aforementioned labels. The final semantic prediction result is then input into the student network for updating.

[0022] In order to prevent the model from overfitting and resulting in inaccurate results during the training process, the teacher-student model is trained together with unlabeled data and labeled data in a semi-supervised framework. That is, the prediction results of labeled data are also input into the student network for updating.

[0023] In some embodiments, the step of preprocessing the unlabeled data and then inputting it into the teacher network to obtain pseudo-labels for subsequent training includes:

[0024] After the unlabeled data is input into the teacher model to obtain the final feature matrix, it passes through a fully connected network layer and an activation layer to obtain the semantic prediction result;

[0025] The semantic prediction results are further refined by removing low-confidence prediction result pixels, and the final result is used as a pseudo-label for unlabeled data.

[0026] The refined selection strategy is based on the prior knowledge that the network's prediction performance will gradually improve, and will be adaptively adjusted with each training iteration.

[0027] In some embodiments, the step of preprocessing the labeled data and inputting it into the student network to obtain its predicted labels for supervised training includes:

[0028] After the unlabeled and labeled data are input into the teacher model to obtain the final feature matrix, they are passed through a fully connected network layer and an activation layer to obtain the semantic prediction result.

[0029] The unlabeled data pseudo-labels are generated by the teacher model and trained together with labeled data in order to prevent the model from overfitting and resulting in inaccurate results.

[0030] The gradient backpropagation of the teacher model is frozen, and its model parameters are obtained by weighting the original parameters of the teacher model and the parameters of the student model after each batch of training.

[0031] In some embodiments, the data preprocessing includes techniques commonly used in image segmentation tasks, including:

[0032] The pixel values ​​of the input image are processed uniformly. First, tensor quantization and regularization are performed to map the pixel values ​​to a uniform range, which facilitates neural network training and speeds up convergence.

[0033] Random and centered cropping of the input image increases the diversity of the dataset information composition while ensuring that the overall foreground area of ​​the image is not excessively affected.

[0034] The input image is randomly flipped and rotated. Without changing the original information of the image, the difficulty of the segmentation task is increased by changing the viewpoint and angle, which helps the neural network learn better features.

[0035] Gaussian blurring and color transformation are applied to the input image. Without changing the original resolution of the image, the segmentation task is increased by mapping the color space and adding noise, without changing the lesion area, thus promoting the neural network to learn better features.

[0036] In some embodiments, the specifications of the labeled data include:

[0037] During the annotation process, the corresponding bimodal images should be cross-referenced and compared, and three vitiligo medical experts should be invited to manually perform pixel-level annotations. Then, the dataset should be cross-checked to ensure the quality of the annotation.

[0038] If the annotation results are inconsistent, the consensus reached should be re-examined and then considered the final annotated data. The same annotation rules should be followed for all three experts.

[0039] For obvious lesion areas, the label map should completely cover them while minimizing the inclusion of normal skin areas. For suspicious areas, compare the modal images with those under Wood's lamp and natural light to determine whether they belong to lesion areas as much as possible. If there is uncertainty on the part of the annotator regarding a specific area, it is recommended not to annotate it to avoid compromising the quality of the annotation.

[0040] Secondly, the present invention also provides a vitiligo lesion area segmentation device, comprising:

[0041] The pre-training module uses images of vitiligo under Wood's lamp and corresponding images of vitiligo under natural light, labeled by experts, to obtain images of the vitiligo lesion areas to be segmented and their corresponding prediction results. Based on the fact that the same lesion areas exist between the two modal data, self-supervised comparative learning is performed to pre-train the segmentation network to obtain a pre-trained neural network that is easy to converge and train.

[0042] The image editing module expands the limited amount of expert-annotated data by using a Poisson-based image editing method to edit vitiligo image data. By randomly sampling the foreground lesion regions of different source images, the target image region is edited to obtain the edited result image. This method yields more natural and diverse vitiligo image data while expanding the dataset size. After data preprocessing, the expanded dataset is input into the segmentation neural network to obtain the semantic segmentation prediction results of the segmentation network and save the trained model parameters.

[0043] The semi-supervised training module can obtain relatively reliable prediction results based on the saved model parameters. Therefore, the model parameters trained based on the above segmentation can be used as the baseline segmentation model and added to the semi-supervised framework called the student-teacher model. Unlabeled data is input into the teacher model to obtain prediction results. After further adaptive refinement, pseudo-labels with higher confidence are obtained and used as labels for unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area.

[0044] The image editing method described herein is based on the Poisson image editing principle to edit the vitiligo lesion area in the original image, and randomly selects different lesion areas to edit into a new target image, thereby obtaining a large amount of labeled data.

[0045] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vitiligo lesion area segmentation method as described above.

[0046] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vitiligo lesion area segmentation method as described above.

[0047] This invention provides a method, apparatus, and electronic device for segmenting vitiligo lesion regions based on image editing. The method involves pre-training the image data (comparatively labeled with expert annotations) using a comparative learning approach. Data augmentation using Poisson image editing is employed to increase the dataset size, diversity, and information richness. The augmented dataset is preprocessed and input into a segmentation neural network to obtain prediction results for vitiligo lesion images. The model parameters trained using this segmentation method serve as the baseline segmentation model in a semi-supervised framework of student and teacher models. Unlabeled and labeled data are then trained together in the semi-supervised framework, with unlabeled data input into the teacher model to obtain prediction results. Further refinement yields pseudo-labels with higher confidence, which are then used as labels to train the network for the unlabeled data. Finally, the model converges to obtain the segmentation result for the vitiligo lesion region. Thus, this invention utilizes a pre-trained segmentation network and image editing for data augmentation to obtain reliable segmentation results. This method effectively utilizes information from a large amount of unlabeled data to achieve automatic segmentation and improve performance. This process does not rely on manual intervention, thus improving segmentation efficiency. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is one of the flowcharts illustrating the method for segmenting vitiligo lesion areas provided by the present invention;

[0050] Figure 2 The present invention provides an image of a vitiligo lesion under natural light and its corresponding image under a medical Wood's lamp.

[0051] Figure 3 This is a flowchart illustrating the image enhancement method based on Poisson image editing provided by the present invention;

[0052] Figure 4 This is a schematic diagram illustrating the principle of obtaining Poisson image editing results provided by the present invention;

[0053] Figure 5 This is a schematic diagram illustrating the visual effect of the image enhancement method based on Poisson image editing provided by the present invention;

[0054] Figure 6 This is a schematic diagram showing the results of the segmentation of vitiligo lesion areas provided by the present invention;

[0055] Figure 7 This is the second flowchart illustrating the method for segmenting vitiligo lesion areas according to the present invention;

[0056] Figure 8 This is a schematic diagram of the planar diagram segmentation device provided by the present invention;

[0057] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0059] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1This is one of the flowcharts illustrating the vitiligo lesion area segmentation method provided by the present invention. The execution entity for each step in this method can be a vitiligo lesion area segmentation device. This device can be implemented through software and / or hardware, and can be integrated into an electronic device. The electronic device can be a terminal device (such as a smartphone, personal computer, learning machine, etc.), a server (such as a local server or cloud server, or a server cluster, etc.), a processor, or a chip, etc. Figure 1 As shown, the method may include the following steps:

[0060] Step 101: The vitiligo images under Wood's lamp and the corresponding vitiligo images under natural light are labeled by experts to obtain the images of the vitiligo lesion areas to be segmented and their corresponding prediction results. Based on the fact that there are the same lesion areas between the two modal data, self-supervised comparative learning is performed to pre-train the segmentation network to obtain a pre-trained neural network that is easy to converge and train.

[0061] It should be noted that in this embodiment, experts annotated the images by comparing two modalities, and performed self-supervised learning based on the bimodal data. Both methods were based on the premise that vitiligo areas are more clearly visible under medical Wood's lamp light. Figure 2 The image shown is a vitiligo image under natural light, a vitiligo image under Wood's lamp light, an image with manually annotated labels, and a segmentation result prediction image obtained in step 102.

[0062] Specifically, self-supervised learning employs the BYOL algorithm without using negative sample pairs, and its loss function is... Defined as ,in These represent two different sets of parameters, corresponding to two networks with the same model structure but different parameters in this training framework. For parameters The regularized output of the defined network. For parameters The regularized output of the defined network. It is the L2 norm, used to calculate the Euclidean length of a vector. It calculates the dot product between two vectors, and this training method is used to... Figure 2 Information from different modalities of images is unified into a single segmentation pre-trained neural network.

[0063] Step 102: For a small amount of expert-annotated data, data augmentation is performed. The vitiligo image data is edited using an image editing method based on the Poisson equation. By randomly sampling the foreground lesion region of different source images, the target image region is edited to obtain the edited result image. In this way, more natural and diverse vitiligo image data is obtained, and the scale of the dataset is expanded. After data preprocessing, the expanded dataset is input into the segmentation neural network to obtain the semantic segmentation prediction result of the segmentation network and save the trained model parameters.

[0064] In this embodiment, the segmentation training model is performed on the augmented dataset after the image is edited. Its original model parameters are loaded with the model parameters pre-trained by self-supervised contrastive learning in step 101 above. Based on this, the semantic prediction results of the segmentation network are more accurate, and the model converges faster.

[0065] In this embodiment, the process of image editing based on the Poisson equation for vitiligo image data is as follows: Figure 3 As shown, our image editing process consists of four inputs: a source image, its corresponding annotation image, a target image, and a corresponding annotation image of the human body region within the target image. The human body region can be simply outlined or segmented using existing computer vision methods. In a single editing cycle, the source image and its corresponding annotation image are input first. A connected component detection algorithm is applied to the annotation image to segment different annotation regions into multiple independent connected regions. Subsequently, several connected regions are randomly sampled from the corresponding regions in the source image; each connected region serves as the basic unit for performing the editing operation. Next, feasible regions in the target image are searched for editing. For each connected region to be edited, we only select feasible regions for image editing, ensuring that the edited region does not exceed the human body region or areas covered by clothing, etc. Finally, the annotations corresponding to the connected regions and the annotations in the target image are merged to complete one image editing operation.

[0066] In this embodiment, the principle of the image editing method based on the Poisson equation is as follows: Figure 4 As shown, image synthesis is the process of inserting a source image S into a region Ψ of a target image D, the boundary of which can be represented by ∂Ψ. We denote the known scalar function defined in the target image as f, and the unknown scalar function defined in the synthesized image region Ψ as the function f. Therefore, only Let f be an unknown function representing a portion of the synthetic region to be solved. Furthermore, we desire that the boundary of the synthetic region be as smooth as possible, without tearing or unnatural conditions, which requires f on the boundary ∂Ψ and f''''''''''''''''''''''""""""," which's ," and f' ... The values ​​are equal. However, in image editing tasks, this simple approach can blur image boundaries. Therefore, an efficient solution to this problem is a more complex differential equation, which modifies the problem by introducing further constraints in the form of a guiding field. This leads to a variational problem: ∇ represents the gradient operator. A vector field called the guiding field represents the gradient of the source image in the image synthesis task. Therefore, by copying the pixels of the source image S to the Ψ region of the target image D, while ensuring that the median of the boundary region ∂Ψ is consistent with that of the target image, the synthesized image D can be obtained. This process is represented by the variational equation, which can be solved using the Poisson equation under Dirichlet boundary conditions: . represents the Laplace operator, and div represents the divergence of the vector.

[0067] The goal of the image editing is to adapt the gradient of the fusion region to the target image while maintaining consistent values ​​at the boundaries. This requires choosing an appropriate v to guide the process. In other words, Figure 4 The image gradient within the Ψ region shown should be selected from either the target image D or the source image S. For image editing on small datasets, not only the total number of images but also their realism must be considered, which is the purpose of introducing Poisson image editing. The image editing strategy employs two methods. Strategy one, called the source image gradient mode, determines the gradient of the target region solely based on the source image during image editing. Sometimes, the style differences between the source and target images are significant, and the gradient information of the source image cannot be fully utilized. Directly editing these two images would result in an unnatural and unrealistic image. The solution is to set v to the gradient of the source image S when the gradient of the source image S is stronger than that of the target image D; otherwise, it is set to the gradient of the target image D. Strategy two, called the mixed gradient mode, obtains a more natural image by mixing information from the target and source images. This preserves some information from the gradients of both the target and original images and linearly adjusts the proportions according to their respective gradient strengths. Since human skin images are relatively similar, the gradient information of the source image can be used in most cases. Therefore, we use both the source image gradient and the mixed gradient strategy simultaneously to increase the richness of the samples. As shown on the right side of Figure 3, we demonstrated the editing results of the same connected component labels under two strategies by fixing the random seed. In this case, their labeled images are identical. Figure 5 This is a schematic diagram illustrating the visual effect of the image enhancement method based on Poisson image editing provided by the present invention;

[0068] Step 103: Based on the saved model parameters, a relatively reliable prediction result can be obtained. Therefore, the model parameters trained based on the above segmentation can be used as the baseline segmentation model and added to a semi-supervised framework called the student-teacher model. Unlabeled data is input into the teacher model to obtain the prediction result. After further adaptive refinement screening, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area.

[0069] Confidence-based pseudo-labeling methods can leverage information from unlabeled data to improve model performance when sufficient labeled data samples are lacking. Figure 6 The visualization results of the proposed method are shown in the following order: vitiligo data map, expert-annotated map, segmentation prediction result map of the baseline model, segmentation prediction result map after adding image editing data augmentation, and segmentation prediction result map after adding semi-supervised training.

[0070] Specifically, the adaptive fine-tuning filtering strategy is as follows: ; ;otherwise ignore.

[0071] in This represents the labeled image corresponding to the unlabeled data i. This corresponds to the probability value of the j-th pixel in image i after softmax activation, where c represents the number of output categories in the last layer of the segmentation network. If neither of these two conditions is met, the pixel is ignored and does not participate in gradient calculation. As the network performance gradually improves during training, the predicted pseudo-labels become increasingly reliable. To explain this, we linearly adjust the threshold, calculated as follows:

[0072]

[0073]

[0074] Based on the verification of multiple experimental results, the filtering threshold was set. and It gradually loosens over time. Variable and This represents the upper and lower bounds of the highest and lowest values ​​obtained during the acquisition of the high threshold. Similarly, and The threshold represents the corresponding constraint, epoch represents the maximum number of training epochs, and t represents the remaining training epochs. Since the performance of a neural network gradually improves with increasing training iterations, based on this prior knowledge, the range of threshold selection is broadened to achieve better performance.

[0075] The semi-supervisory framework is Figure 7 The teacher-student framework shown is a classic semi-supervised learning method, where the teacher model parameters are the continuous average of the student model parameters. In some embodiments, the semi-supervised framework of the student-teacher model includes two identically constructed models—a student model and a teacher model—with different parameters, including:

[0076] The teacher model and the student model's initial model weights are loaded onto the neural network model trained on the expanded dataset obtained based on image editing. Meanwhile, the teacher model's parameters are not backpropagated with gradients; instead, their model parameters are obtained by weighting the original model parameters with the updated parameters of the student model and updating them with momentum.

[0077] The unlabeled data is input into the teacher model to obtain semantic prediction results. After further adaptive refinement screening, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. Thus, the pseudo-labels of the unlabeled data are updated by the output of the teacher network.

[0078] The unlabeled data is input into the student model and supervised segmentation training is performed using the aforementioned labels. The final semantic prediction result is then input into the student network for updating.

[0079] In order to prevent the model from overfitting and resulting in inaccurate results during the training process, the teacher-student model is trained together with unlabeled data and labeled data in a semi-supervised framework. That is, the prediction results of labeled data are also input into the student network for updating.

[0080] In some embodiments, the pre-trained model refers to a neural network pre-trained for a segmentation task, including:

[0081] The self-supervised pre-training learning based on the aforementioned bimodal vitiligo data provides the original segmentation network with a pre-trained neural network that is easy to converge. Figure 7 shown The representative model needs to load the self-supervised pre-trained model weight parameters when training the segmentation task;

[0082] The expanded dataset obtained based on image editing is further trained on the weights of the pre-trained model using a supervised segmentation task to obtain a pre-trained neural network.

[0083] The model weights trained based on the segmentation task are further trained and converged into a pre-trained neural network in a semi-supervised framework of the student-teacher model using unlabeled data. Figure 7 shown Representative teacher model The weight parameters of the student model, which represent the model, need to be loaded during the training of the semi-supervised learning framework, based on the model weight parameters obtained from the supervised segmentation task training.

[0084] For the final test dataset, the model weight parameters trained by the semi-supervised framework can be loaded, and the data to be tested can be input into the neural network to obtain the predicted segmentation results.

[0085] In some embodiments, the step of preprocessing the unlabeled data and then inputting it into the teacher network to obtain pseudo-labels for subsequent training includes:

[0086] After the unlabeled data is input into the teacher model to obtain the final feature matrix, it passes through a fully connected network layer and an activation layer to obtain the semantic prediction result;

[0087] The semantic prediction results are further refined by removing low-confidence prediction result pixels, and the final result is used as a pseudo-label for unlabeled data.

[0088] The refined selection strategy is based on the prior knowledge that the network's prediction performance will gradually improve, and will be adaptively adjusted with each training iteration.

[0089] In some embodiments, the step of preprocessing the labeled data and inputting it into the student network to obtain its predicted labels for supervised training includes:

[0090] After the unlabeled and labeled data are input into the teacher model to obtain the final feature matrix, they are passed through a fully connected network layer and an activation layer to obtain the semantic prediction result.

[0091] The unlabeled data pseudo-labels are generated by the teacher model and trained together with labeled data in order to prevent the model from overfitting and resulting in inaccurate results.

[0092] The gradient backpropagation of the teacher model is frozen, and its model parameters are determined solely by the teacher model. Original parameters and student model parameters It is obtained by weighting after each batch of training.

[0093] Specifically,

[0094] in Let represent the parameters of the student model at time t. Let represent the parameters of the teacher model at time t. Therefore, the parameters of the teacher model are obtained by the moving average of the model parameters at the previous time t-1 and the parameters of the student model at time t.

[0095] In some embodiments, the data preprocessing includes techniques commonly used in image segmentation tasks, including:

[0096] The pixel values ​​of the input image are processed uniformly. First, tensor quantization and regularization are performed to map the pixel values ​​to a uniform range, which facilitates neural network training and speeds up convergence.

[0097] Random and centered cropping of the input image increases the diversity of the dataset information composition while ensuring that the overall foreground area of ​​the image is not excessively affected.

[0098] The input image is randomly flipped and rotated. Without changing the original information of the image, the difficulty of the segmentation task is increased by changing the viewpoint and angle, which helps the neural network learn better features.

[0099] Gaussian blurring and color transformation are applied to the input image. Without changing the original resolution of the image, the segmentation task is increased by mapping the color space and adding noise, without changing the lesion area, thus promoting the neural network to learn better features.

[0100] In some embodiments, the specifications of the labeled data include:

[0101] During the annotation process, the corresponding bimodal images should be cross-referenced and compared, and three vitiligo medical experts should be invited to manually perform pixel-level annotations. Then, the dataset should be cross-checked to ensure the quality of the annotation.

[0102] If the annotation results are inconsistent, the consensus reached should be re-examined and then considered the final annotated data. The same annotation rules should be followed for all three experts.

[0103] For obvious lesion areas, the label map should completely cover them while minimizing the inclusion of normal skin areas. For suspicious areas, compare the modal images with those under Wood's lamp and natural light to determine whether they belong to lesion areas as much as possible. If there is uncertainty on the part of the annotator regarding a specific area, it is recommended not to annotate it to avoid compromising the quality of the annotation.

[0104] In this embodiment, the IoU and Dice metrics are used to calculate the accuracy of the segmentation results. , ;

[0105] in, The image shows the segmentation results of the vitiligo lesion area. gt is the actual segmentation result. The higher both indicators are, the more accurate the segmentation result.

[0106] Furthermore, in this embodiment, the corresponding TP (True Positives), FP (False Positives), and FN (False Negatives) are calculated based on the segmentation results of the vitiligo lesion area map and the actual segmentation results.

[0107] In this context, TP stands for the number of matching logs, FP stands for the number of logs in the segmentation results of the vitiligo lesion area map that do not match the actual segmentation results, and FN stands for the number of logs in the segmentation results of the actual segmentation results that do not match the segmentation results of the vitiligo lesion area map.

[0108] Figure 8 This is a schematic diagram of the structure of the vitiligo lesion area segmentation device provided by the present invention, as shown below. Figure 8 As shown, the device includes:

[0109] The pre-training module 810 uses images of vitiligo under Wood's lamp and corresponding images of vitiligo under natural light, which are labeled by experts to obtain images of the vitiligo lesion areas to be segmented and their corresponding prediction results. Based on the fact that there are the same lesion areas between the two modal data, self-supervised comparative learning is performed to pre-train the segmentation network to obtain a pre-trained neural network that is easy to converge and train.

[0110] The image editing module 820 expands the limited amount of expert-annotated data by using a Poisson-based image editing method to edit vitiligo image data. It edits the target image region by randomly sampling the foreground lesion region of different source images, resulting in an edited image. This method yields more natural and diverse vitiligo image data while expanding the dataset size. After data preprocessing, the expanded dataset is input into a segmentation neural network to obtain the semantic segmentation prediction results of the segmentation network and save the trained model parameters.

[0111] The semi-supervised training module 830 can obtain relatively reliable prediction results based on the saved model parameters. Therefore, the model parameters trained based on the above segmentation can be used as a baseline segmentation model and added to a semi-supervised framework called the student-teacher model. Unlabeled data is input into the teacher model to obtain prediction results. After further adaptive refinement screening, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area.

[0112] The image editing method described herein is based on the Poisson image editing principle to edit the vitiligo lesion area in the original image, and randomly selects different lesion areas to edit into a new target image, thereby obtaining a large amount of labeled data.

[0113] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a planar diagram partitioning method, which includes:

[0114] After acquiring medical image data, we use professional annotation tools to annotate the medical image data and obtain the corresponding annotations for the medical images. We then perform self-supervised multimodal data pre-training learning based on image pairs composed of human skin images in natural scenes and human skin images under medical Wood's lamp illumination.

[0115] To augment the limited amount of expert-annotated data, an image editing method based on the Poisson equation was used to edit the vitiligo image data. By randomly sampling the foreground lesion region of different source images, the target image region was edited to obtain the edited result image. In this way, more natural and diverse vitiligo image data was obtained, while expanding the scale of the dataset.

[0116] After data preprocessing, the expanded dataset is input into the segmentation neural network to obtain the semantic segmentation prediction results of the segmentation network and save the trained model parameters.

[0117] Based on the saved model parameters, a relatively reliable prediction result can be obtained. Therefore, the model parameters trained based on the above segmentation can be used as a baseline segmentation model and added to a semi-supervised framework called the student-teacher model. Unlabeled data is input into the teacher model to obtain prediction results. After further adaptive refinement, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area.

[0118] The image editing method described herein is based on the Poisson image editing principle to edit the vitiligo lesion area in the original image, and randomly selects different lesion areas to edit into a new target image, thereby obtaining a large amount of labeled data.

[0119] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vitiligo lesion region segmentation method provided by the above methods, the method comprising:

[0121] After acquiring medical image data, we use professional annotation tools to annotate the medical image data and obtain the corresponding annotations for the medical images. We then perform self-supervised multimodal data pre-training learning based on image pairs composed of human skin images in natural scenes and human skin images under medical Wood's lamp illumination.

[0122] To augment the limited amount of expert-annotated data, an image editing method based on the Poisson equation was used to edit the vitiligo image data. By randomly sampling the foreground lesion region of different source images, the target image region was edited to obtain the edited result image. In this way, more natural and diverse vitiligo image data was obtained, while expanding the scale of the dataset.

[0123] After data preprocessing, the expanded dataset is input into the segmentation neural network to obtain the semantic segmentation prediction results of the segmentation network and save the trained model parameters.

[0124] Based on the saved model parameters, a relatively reliable prediction result can be obtained. Therefore, the model parameters trained based on the above segmentation can be used as a baseline segmentation model and added to a semi-supervised framework called the student-teacher model. Unlabeled data is input into the teacher model to obtain prediction results. After further adaptive refinement, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area.

[0125] The image editing method described herein is based on the Poisson image editing principle to edit the vitiligo lesion area in the original image, and randomly selects different lesion areas to edit into a new target image, thereby obtaining a large amount of labeled data.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for segmenting vitiligo lesion regions based on image editing, characterized in that, include: Step 1: After acquiring medical image data, use professional annotation tools to annotate the medical image data to obtain the corresponding annotations for the medical images. The medical image data includes human skin images in natural scenes and human skin images under medical Wood's lamp illumination. Based on the image pairs composed of the acquired human skin images in natural scenes and human skin images under medical Wood's lamp illumination, perform self-supervised multimodal data pre-training learning to obtain a pre-trained neural network that is easy to converge. Step 2: For a small amount of expert-annotated data, data augmentation is performed. Image editing methods based on the Poisson equation are used to edit the vitiligo image data. By randomly sampling the foreground lesion regions of different source images, the target image region is edited to obtain the edited result image. After data preprocessing, the augmented dataset is input into the segmentation neural network to obtain the semantic segmentation prediction result of the segmentation neural network. The trained model parameters are saved. The segmentation training model is performed on the augmented dataset after the image is edited. Its original model parameters are the model parameters pre-trained by self-supervised contrastive learning in step 1. Step 3: Based on the trained model parameters as the baseline segmentation model, it is added to the semi-supervised framework of the student and teacher models. Unlabeled data is input into the teacher model to obtain prediction results. After further adaptive refinement, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area. Step 3 further includes: after the unlabeled data and labeled data are input into the teacher model to obtain the final feature matrix, they are passed through a fully connected network layer and an activation layer to obtain the semantic prediction result; The unlabeled data pseudo-labels are generated by the teacher model and trained together with labeled data. The gradient backpropagation of the teacher model is frozen, and its model parameters are obtained by weighting the original parameters of the teacher model and the parameters of the student model after each batch of training.

2. The method for segmenting vitiligo lesion regions based on image editing according to claim 1, characterized in that, Step 2 also includes: The image editing process consists of four inputs: the source image, the corresponding labeled image, the target image, and the corresponding labeled image of the human body region in the target image. In a single editing cycle, the source image and its corresponding labeled image are input first. A connected component detection algorithm is applied to the labeled image to segment different labeled regions into multiple independent connected regions. Then, several connected regions are randomly sampled from the corresponding regions in the source image. Each connected region serves as the basic unit for performing the editing operation. Next, feasible regions in the target image are searched for editing. For each connected region to be edited, only feasible regions are selected for image editing, ensuring that the edited region does not exceed the human body region or the area covered by clothing. Finally, the labels corresponding to the connected regions and the labels in the target image are merged to complete one image editing operation.

3. The method for segmenting vitiligo lesion regions based on image editing according to claim 1, characterized in that, Self-supervised pre-training learning based on bimodal vitiligo data provides an easily convergent pre-trained neural network for the original segmentation network. Based on the expanded dataset obtained through image editing, the pre-trained neural network weights are further trained using a supervised segmentation task to obtain the trained model parameters. Based on the trained model parameter weights, the model is further trained and converged in a semi-supervised framework of student and teacher models using unlabeled data, resulting in the final converged model.

4. The method for segmenting vitiligo lesion regions based on image editing according to claim 3, characterized in that, The semi-supervised framework of the student-teacher model in step 3 includes two identically constructed models—a student model and a teacher model—with different parameters. The teacher model and the student model are initialized with weights loaded onto a neural network model trained on an expanded dataset obtained by image editing based on the Poisson equation. Meanwhile, the teacher model's parameters are not backpropagated with gradients; instead, their model parameters are obtained by weighting the original model parameters with the updated parameters of the student model and updating them with momentum. The unlabeled data is input into the teacher model to obtain semantic prediction results. After further adaptive refinement screening, pseudo-labels with higher confidence are obtained as labels for the unlabeled data to train the network. The pseudo-labels for the unlabeled data are updated by the output of the teacher network.

5. The method for vitiligo lesion region segmentation based on image editing according to claim 4, further comprising inputting unlabeled data into a teacher network to obtain pseudo-labels for use in semi-supervised framework training of a student-teacher model, wherein the semi-supervised framework training of the student-teacher model includes: The unlabeled data is input into the teacher model to obtain the final feature matrix, and then passes through a fully connected network layer and an activation layer to obtain the semantic prediction result. The semantic prediction results are further refined by a screening strategy to remove low-confidence prediction result pixels, and the final result is used as a pseudo-label for unlabeled data. The refined selection strategy is based on the prior knowledge that the network's prediction performance will gradually improve, and will be adaptively adjusted with each training iteration.

6. The method for segmenting vitiligo lesion regions based on image editing according to claim 1, further comprising: During the annotation process, the corresponding bimodal images were cross-referenced and compared, and three vitiligo medical experts were invited to manually perform pixel-level annotations. The datasets were then cross-checked to ensure the quality of the annotations.

7. A device for segmenting vitiligo lesion areas, characterized in that, include: The pre-training module uses images of vitiligo under Wood's lamp and corresponding images of vitiligo under natural light, labeled by experts, to obtain images of the vitiligo lesion areas to be segmented and their corresponding prediction results. Based on the fact that the same lesion areas exist between the two modal data, self-supervised comparative learning is performed to pre-train the segmentation network to obtain a pre-trained neural network that is easy to converge. The image editing module augments a small amount of expert-annotated data. It then uses a Poisson-based image editing method to edit vitiligo image data. By randomly sampling the foreground lesion regions of different source images, the target image region is edited to obtain the edited result image. After data preprocessing, the augmented dataset is input into a segmentation neural network to obtain the semantic segmentation prediction results of the segmentation network and save the trained model parameters. The segmentation training model is performed on the augmented dataset after image editing, and its original model parameters are the model parameters pre-trained by self-supervised contrastive learning in the pre-training module. The semi-supervised training module uses the trained model parameters as a baseline segmentation model and adds it to a semi-supervised framework called the student-teacher model. Unlabeled data is input into the teacher model to obtain prediction results. After further adaptive refinement, pseudo-labels with higher confidence are obtained and used as labels for the unlabeled data to train the network. At the same time, in order to prevent the model from overfitting, unlabeled data and labeled data are trained together in the semi-supervised framework to obtain the final converged model and finally obtain the segmentation result of the vitiligo lesion area. The semi-supervised training module further includes: after the unlabeled and labeled data are input into the teacher model to obtain the final feature matrix, they are passed through a fully connected network layer and an activation layer to obtain the semantic prediction result; The unlabeled data pseudo-labels are generated by the teacher model and trained together with labeled data. The gradient backpropagation of the teacher model is frozen, and its model parameters are obtained by weighting the original parameters of the teacher model and the parameters of the student model after each batch of training.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image editing-based vitiligo lesion region segmentation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image editing-based vitiligo lesion region segmentation method as described in any one of claims 1 to 6.

Images