A method and system for generating virtual dermoscopic images

Abstract

The present invention provides a virtual dermoscopic image generation method and system, including the steps of preprocessing, labeling, constructing a first segmentation model, a second segmentation model, pairing segmentation masks, pairing clinical skin images and dermoscopic images, training an unpaired dermoscopic generation model and a virtual dermoscopic image generation model, inputting a clinical skin image acquired in real time to obtain a corresponding virtual dermoscopic image, etc. In the case where it is difficult to collect dermoscopic images in practice and only clinical skin images can be provided, a virtual dermoscopic image can be generated by the model in the present invention, thereby providing more perspectives for clinicians in remote diagnosis and treatment, and improving the diversity, accuracy and efficiency of remote diagnosis and treatment disease screening.

Description

Technical Field

[0001] The present invention belongs to the field of clinical skin image generation, and in particular relates to a dermatoscopic image generation method and system. Background Art

[0002] In the diagnosis of skin diseases, clinical images and dermoscopic images are both important diagnostic bases and the basis for diagnosing skin diseases. However, in telemedicine scenarios, it is often difficult for primary hospitals or patients themselves to obtain dermoscopic images, and clinicians have to rely solely on clinical images for preliminary screening of diseases, resulting in a decrease in the accuracy and efficiency of initial screening, which has a great impact on patients' medical treatment. Under limited conditions, improving the screening accuracy of telemedicine is of great significance for patients in remote areas.

[0003] At present, some studies have achieved image generation between different modalities through generative networks. For example, some studies have proposed using ultrasound images to generate virtual elastic ultrasound images to assist diagnosis and improve diagnostic accuracy. However, in the generation of virtual dermoscopic images, due to the problems of offset and deformation between clinical images and corresponding dermoscopic images, the use of ultrasound image virtual generation still cannot solve the problem and it is difficult to achieve accurate image generation. In addition, for the case of relatively large skin lesions, the existing technology has not proposed a good solution to the registration problem of the corresponding areas in the dermoscopic image and the clinical image. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the existing technology is unable to accurately match and generate dermoscopic images and clinical images. In order to overcome the deficiencies and defects mentioned in the above background technology, a virtual dermoscopic image generation method and system are provided.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A method for generating a virtual dermoscopic image, the method comprising:

[0007] S1, preprocessing clinical skin images and dermoscopic images;

[0008] S2. Annotate the segmentation masks corresponding to the skin lesions in the clinical skin images and the dermoscopic images to form a clinical skin image dataset and a dermoscopic image dataset; pair the clinical skin images and the dermoscopic images to obtain a paired dataset and an unpaired dataset;

[0009] Divide the paired data set and the unpaired data set into a paired training set and an unpaired training set, as well as a paired test set and an unpaired test set;

[0010] S3. Constructing a first segmentation model of clinical skin image lesion sites according to the clinical skin image dataset, wherein the first segmentation model takes the clinical skin image as input and takes the clinical skin image segmentation mask corresponding to the skin lesion site as output;

[0011] S4. Constructing a second segmentation model for dermoscopic image lesion sites according to the dermoscopic image dataset, wherein the second segmentation model takes the dermoscopic image as input and takes the dermoscopic image segmentation mask corresponding to the skin lesion site as output;

[0012] S5, pairing the clinical skin image segmentation mask and the dermoscopic image segmentation mask obtained in S3 and S4 to obtain a paired segmentation mask, and then pairing the clinical skin image and the dermoscopic image corresponding to the paired segmentation mask to obtain a paired image;

[0013] S6. According to the unpaired data set, train and test the unpaired dermatoscope generation model, and then retrain the unpaired dermatoscope generation model according to the paired images obtained in S5 to obtain a virtual dermatoscope image generation model;

[0014] The virtual dermoscopic image generation model takes clinical skin images as input and virtual dermoscopic images as output;

[0015] S7. Testing the virtual dermoscopy image generation model;

[0016] S8. Input the clinical skin image acquired in real time into the virtual dermoscopic image generation model to obtain the virtual dermoscopic image corresponding to the current clinical skin image.

[0017] Preferably, the preprocessing in step S1 includes: retaining clinical skin images and dermoscopic images containing skin lesions, and screening out other images.

[0018] Preferably, in step S2, the segmentation mask corresponding to the skin lesion site in the clinical skin image and the dermoscopic image is labeled, and the major axis and the minor axis of the segmentation mask are labeled.

[0019] Preferably, the pairing of the clinical skin image segmentation mask and the dermoscopic image segmentation mask obtained in S3 and S4 in step S5 specifically includes: distinguishing the long axis and short axis of the clinical skin image and the dermoscopic image obtained in S2, and adjusting and pairing the clinical skin image segmentation mask and the dermoscopic image segmentation mask by one or more methods of rotation, translation or scaling according to the correspondence between the long axis and the short axis of the two.

[0020] Preferably, the step S4 of pairing the clinical skin images corresponding to the paired segmentation masks specifically includes: adjusting and pairing the clinical skin images corresponding to the paired segmentation masks by one or more methods of rotation, translation or scaling.

[0021] Preferably, the training of the unpaired dermatoscope generation model in step S6 specifically includes:

[0022] In the first step, two pairs of generators and discriminators are constructed for unpaired clinical skin images and dermoscopic images respectively;

[0023] Unpaired clinical skin images or dermoscopic images are respectively input into the generator to generate virtual images, and the virtual images are respectively input into the discriminator together with real clinical skin images or dermoscopic images for discrimination supervision; thereby, the virtual dermoscopic images generated by the training generator are difficult to be discriminated by the discriminator.

[0024] In the second step, the clinical image is input into the dermoscopic image generator to obtain a virtual dermoscopic image, the virtual dermoscopic image is input into the clinical image generator to obtain a virtual clinical image, and the virtual clinical image and the original clinical skin image are input into the discriminator together for discrimination supervision; the unpaired dermoscopic generation model is obtained by training.

[0025] In the second step, the two pairs of generators obtained in the previous step are nested; the purpose of nested training is to achieve the goal of generating matching images as much as possible in the case of mismatches.

[0026] Preferably, the retraining of the unpaired dermatoscope generation model according to the paired images in step S6 specifically includes:

[0027] The clinical skin image in the paired image of step S5 is input into the unpaired dermoscopy generation model to retrain the model, so as to obtain a virtual dermoscopy image, and then the virtual dermoscopy image is supervised by the dermoscopy image in the paired image.

[0028] Under the same technical concept, the present invention also provides a virtual dermoscopic image generation system, the generation system includes a processor, and the processor includes the following modules:

[0029] Image collection and preprocessing module: used to collect clinical skin images and dermoscopic images, and perform data cleaning, screening and preprocessing;

[0030] Annotation module: used to annotate the segmentation masks corresponding to the skin lesions in the clinical skin images and dermoscopic images to form clinical skin image datasets and dermoscopic image datasets; pair the clinical skin images and dermoscopic images to obtain paired datasets and unpaired datasets;

[0031] Clinical skin image lesion site segmentation model training module: used to construct a first segmentation model of clinical skin image lesion sites based on a clinical skin image dataset; the first segmentation model takes a clinical skin image as input and takes a clinical skin image segmentation mask corresponding to the skin lesion site as output;

[0032] Dermoscopic image lesion site segmentation model training module: used to construct a second segmentation model for dermoscopic image lesion sites based on the dermoscopic image dataset, wherein the second segmentation model takes the dermoscopic image as input and takes the dermoscopic image segmentation mask corresponding to the skin lesion site as output;

[0033] Pairing module: used to pair the clinical skin image segmentation mask and the dermoscopic image segmentation mask to obtain a paired segmentation mask, and then pair the clinical skin image and the dermoscopic image corresponding to the paired segmentation mask to obtain a paired image;

[0034] Virtual dermoscopy image generation model training module: used to train an unpaired dermoscopy generation model based on an unpaired data set, and then retrain the unpaired dermoscopy generation model based on paired images to obtain a virtual dermoscopy image generation model;

[0035] The virtual dermoscopic image generation model takes clinical skin images as input and virtual dermoscopic images as output;

[0036] The unpaired dermatoscope generation model and the virtual dermatoscope image generation model were tested based on the paired test set and the unpaired test set respectively;

[0037] Virtual dermoscopic image generation module: used to input the clinical skin image acquired in real time into the virtual dermoscopic image generation model to obtain the virtual dermoscopic image corresponding to the current clinical skin image.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) A virtual dermoscopic image generation method is proposed. The method can use historical clinical skin images and dermoscopic images to train clinical skin image segmentation models, dermoscopic image segmentation models, and virtual dermoscopic image generation models. In the case where it is difficult to collect dermoscopic images in practice and only clinical skin images can be provided, virtual dermoscopic images can be generated by the model of the present invention, thereby providing more perspectives for clinicians in remote diagnosis and treatment, and improving the diversity, accuracy and efficiency of remote diagnosis and treatment disease screening;

[0040] (2) The virtual dermoscopy generation model designed by the present invention adds supervised training of paired data on the basis of the unpaired model, thereby improving the authenticity of the virtual dermoscopy images generated by the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a schematic diagram of the overall process of the virtual dermoscopic image generation method of the present invention;

[0043] Figure 2 It is a schematic diagram of data set division and annotation in step S2 of the virtual dermoscopic image generation method of the present invention;

[0044] Figure 3 It is a schematic diagram of segmentation annotation and major and minor axis annotation in step S2 of the virtual dermoscopic image generation method of the present invention;

[0045] Figure 4 4 is a schematic diagram of the pairing process of the clinical skin image segmentation mask and the dermoscopic image segmentation mask in step S4 of the virtual dermoscopic image generation method of the present invention;

[0046] Figure 5 is a schematic diagram of an unpaired image generation model in step S6 of the virtual dermoscopic image generation method of the present invention;

[0047] Figure 6 It is a schematic diagram of a virtual dermoscopic image generation model in step S6 of the virtual dermoscopic image generation method of the present invention. DETAILED DESCRIPTION

[0048] In order to facilitate the understanding of the present invention, the present invention will be described more comprehensively and meticulously below in conjunction with the accompanying drawings and preferred embodiments of the present invention, but the protection scope of the present invention is not limited to the following specific embodiments.

[0049] Unless otherwise defined, all professional terms used below have the same meanings as those generally understood by those skilled in the art. The professional terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention.

[0050] Unless otherwise specified, various raw materials, reagents, instruments and equipment used in the present invention can be purchased from the market or prepared by existing methods.

[0051] The device for acquiring clinical skin images in the embodiment of the present invention includes, but is not limited to, a mobile phone, a tablet or a professional medical photography device.

[0052] Example 1

[0053] This embodiment provides a method for generating a virtual dermatoscopy image, and the overall flow chart is as follows: Figure 1 As shown, the specific steps include:

[0054] 1) Perform data cleaning, screening and preprocessing:

[0055] All clinical images and dermoscopic images were cleaned, screened and preprocessed, and the two images containing skin lesions were retained, and the rest were screened out;

[0056] 2) Divide the data set and annotate the data. The diagram of data set division and annotation is as follows: Figure 2 As shown:

[0057] The annotation includes but is not limited to clinical skin images and their corresponding segmentation masks, dermoscopic images and their corresponding segmentation masks; after the segmentation mask is annotated, the long and short axes of the segmentation mask are further annotated; the schematic diagram of the segmentation annotation and the long and short axis annotation is as shown in Figure 3 As shown;

[0058] After the annotation is completed, based on the annotation of the long axis and short axis of the segmentation mask, according to the correspondence between the long axis and the short axis, the paired clinical skin images and dermoscopic images are adjusted and paired by one or more methods of rotation, translation or scaling to form a paired data set, and the remaining images are unpaired data sets;

[0059] Afterwards, the paired data set and the unpaired data set were divided into paired training set and unpaired training set, as well as paired test set and unpaired test set in a ratio of 8:2;

[0060] 3) Based on the images and annotations in the clinical skin image dataset, a first segmentation model of the lesion site of the clinical skin image is constructed:

[0061] The clinical skin image dataset is divided into a training dataset and a validation set in a ratio of 8:2;

[0062] Based on the training data set, the first segmentation model of the lesion site of clinical skin images was constructed and verified on the validation set;

[0063] The first segmentation model takes the clinical skin image as input and the clinical skin image segmentation mask corresponding to the skin lesion site as output;

[0064] 4) Based on the images and annotations in the dermoscopic image dataset, a second segmentation model of the lesion area of ​​the dermoscopic image is constructed:

[0065] The dermoscopic image dataset was divided into a training dataset and a validation set in a ratio of 8:2;

[0066] Based on the training data set, a second segmentation model for the lesion area of ​​the dermatoscopic image is constructed and verified on the validation set;

[0067] The second segmentation model takes the dermoscopic image as input and the dermoscopic image segmentation mask corresponding to the skin lesion site as output;

[0068] 5) Pairing the images based on the segmented clinical skin image and the segmentation mask of the dermoscopic image. The schematic diagram of the pairing process of the clinical skin image segmentation mask and the dermoscopic image segmentation mask is as follows: Figure 4 As shown:

[0069] First, the clinical skin image segmentation mask obtained in step 3) and step 4) is paired with the dermoscopic image segmentation mask. Based on the annotation of the major axis and the minor axis of the segmentation mask in S2, according to the correspondence between the major axis and the minor axis, the clinical skin image segmentation mask and the dermoscopic image segmentation mask are adjusted and paired by one or more methods of rotation, translation or scaling to obtain a paired segmentation mask;

[0070] Then, the clinical skin image and the dermoscopic image corresponding to the paired segmentation mask are adjusted and paired by one or more methods of rotation, translation or scaling to obtain a paired image;

[0071] 6) Constructing an unpaired dermatoscope generation model and a virtual dermatoscope image generation model:

[0072] First, based on the unpaired dataset, the unpaired dermatoscope generation model is trained. The schematic diagram of the unpaired image generation model is as follows: Figure 5 As shown;

[0073] In the first step, two pairs of generators and discriminators are constructed for unpaired clinical skin images and dermoscopic images respectively;

[0074] For clinical skin images:

[0075] Input the unpaired clinical skin image into the generator to generate a virtual image, and input the virtual image and the real clinical skin image into the discriminator for discrimination supervision;

[0076] For dermoscopic images:

[0077] The unpaired dermoscopic image is input into the generator to generate a virtual image, and the virtual image and the real dermoscopic image are input into the discriminator for discrimination supervision;

[0078] In the second step, the two pairs of generators obtained in the previous step are nested: the clinical image is input into the dermoscopic image generator to obtain a virtual dermoscopic image, the virtual dermoscopic image is input into the clinical image generator to obtain a virtual clinical image, and the virtual clinical image and the original clinical skin image are input into the discriminator together for discrimination supervision; the unpaired dermoscopic generation model is obtained through training;

[0079] Afterwards, based on the unpaired dermoscopy generation model, the paired images are used to retrain the unpaired dermoscopy generation model to obtain a virtual dermoscopy image generation model. The schematic diagram of the virtual dermoscopy image generation model is shown in FIG. Figure 6 As shown;

[0080] To ensure the authenticity of the generated virtual dermoscopic image, the virtual dermoscopic image generation model is supervised: the clinical skin image in the paired image is input into the unpaired dermoscopic generation model to retrain the model to obtain the virtual dermoscopic image, and then the virtual dermoscopic image is supervised by the dermoscopic image in the paired image;

[0081] 7) Testing the unpaired dermatoscope generation model and the virtual dermatoscope image generation model: testing the unpaired dermatoscope generation model and the virtual dermatoscope image generation model based on the paired test set and the unpaired test set respectively;

[0082] 8) Acquire clinical skin images in real time;

[0083] 9) Inputting the clinical skin image acquired in real time into the paired and unpaired image generation model to obtain the virtual dermoscopic image corresponding to the current image.

[0084] The virtual dermoscopic image generation system used in this embodiment includes a processor, and the processor includes the following modules:

[0085] Image collection and preprocessing module: used to collect clinical skin images and dermoscopic images, and perform data cleaning, screening and preprocessing;

[0086] Annotation module: used to annotate the segmentation masks corresponding to the skin lesions in the clinical skin images and dermoscopic images to form clinical skin image datasets and dermoscopic image datasets; pair the clinical skin images and dermoscopic images to obtain paired datasets and unpaired datasets;

[0087] Clinical skin image lesion site segmentation model training module: used to construct a first segmentation model of clinical skin image lesion sites based on a clinical skin image dataset; the first segmentation model takes a clinical skin image as input and takes a clinical skin image segmentation mask corresponding to the skin lesion site as output;

[0088] Dermoscopic image lesion site segmentation model training module: used to construct a second segmentation model for dermoscopic image lesion sites based on the dermoscopic image dataset, wherein the second segmentation model takes the dermoscopic image as input and takes the dermoscopic image segmentation mask corresponding to the skin lesion site as output;

[0089] Pairing module: used to pair the clinical skin image segmentation mask and the dermoscopic image segmentation mask to obtain a paired segmentation mask, and then pair the clinical skin image and the dermoscopic image corresponding to the paired segmentation mask to obtain a paired image;

[0090] Virtual dermoscopy image generation model training module: used to train an unpaired dermoscopy generation model based on an unpaired data set, and then retrain the unpaired dermoscopy generation model based on paired images to obtain a virtual dermoscopy image generation model;

[0091] The virtual dermoscopic image generation model takes clinical skin images as input and virtual dermoscopic images as output;

[0092] The unpaired dermatoscope generation model and the virtual dermatoscope image generation model were tested based on the paired test set and the unpaired test set respectively;

[0093] Virtual dermoscopic image generation module: used to input the clinical skin image acquired in real time into the virtual dermoscopic image generation model to obtain the virtual dermoscopic image corresponding to the current clinical skin image.

Claims

1. A method for generating a virtual dermoscopic image, characterized in that: The method comprises: S1, preprocessing clinical skin images and dermoscopic images; S2, annotating the long axis and short axis of the segmentation mask corresponding to the skin lesion site in the clinical skin image and the dermoscopic image to form a clinical skin image dataset and a dermoscopic image dataset; pairing the clinical skin image and the dermoscopic image to obtain a paired dataset and an unpaired dataset; S3. Constructing a first segmentation model of clinical skin image lesion sites according to the clinical skin image dataset, wherein the first segmentation model takes the clinical skin image as input and takes the clinical skin image segmentation mask corresponding to the skin lesion site as output; S4. Constructing a second segmentation model for dermoscopic image lesion sites according to the dermoscopic image dataset, wherein the second segmentation model takes the dermoscopic image as input and takes the dermoscopic image segmentation mask corresponding to the skin lesion site as output; S5. Based on the annotation of the major axis and the minor axis of the segmentation mask in S2, according to the correspondence between the major axis and the minor axis, the clinical skin image segmentation mask and the dermoscopic image segmentation mask are adjusted and paired by one or more methods of rotation, translation and scaling to obtain a paired segmentation mask, and then the clinical skin image and the dermoscopic image corresponding to the paired segmentation mask are paired by one or more methods of rotation, translation and scaling to obtain a paired image; S6. According to the unpaired data set, train and test the unpaired dermatoscope generation model, and then retrain the unpaired dermatoscope generation model according to the paired images obtained in S5 to obtain a virtual dermatoscope image generation model; The virtual dermoscopic image generation model takes clinical skin images as input and virtual dermoscopic images as output; S7. Testing the virtual dermoscopy image generation model; S8. Input the clinical skin image acquired in real time into the virtual dermoscopic image generation model to obtain the virtual dermoscopic image corresponding to the current clinical skin image.

2. The method according to claim 1, characterized in that The training and testing of the unpaired dermoscopy generation model in step S6 specifically includes: In the first step, two pairs of generators and discriminators are constructed for unpaired clinical skin images and dermoscopic images respectively; Inputting unpaired clinical skin images or dermoscopic images in the unpaired data set into the generator to generate virtual images, and inputting the virtual images together with real clinical skin images or dermoscopic images into the discriminator for discrimination supervision; In the second step, the clinical image is input into the dermoscopic image generator to obtain a virtual dermoscopic image, the virtual dermoscopic image is input into the clinical image generator to obtain a virtual clinical image, and the virtual clinical image and the original clinical skin image are input into the discriminator together for discrimination supervision; the unpaired dermoscopic generation model is obtained by training.

3. The method according to claim 1, characterized in that The step S6 of retraining the unpaired dermatoscope generation model according to the paired images obtained in S5 specifically includes: The clinical skin image in the paired image of step S5 is input into the unpaired dermoscopy generation model to retrain the model, so as to obtain a virtual dermoscopy image, and then the virtual dermoscopy image is supervised by the dermoscopy image in the paired image.

4. A virtual dermoscopic image generation system for executing the virtual dermoscopic image generation method according to any one of claims 1 to 3, characterized in that: The generation system includes a processor, and the processor includes the following modules: Image collection and preprocessing module: used to collect clinical skin images and dermoscopic images, and perform data cleaning, screening and preprocessing; Annotation module: used to annotate the segmentation masks corresponding to the skin lesions in the clinical skin images and dermoscopic images to form clinical skin image datasets and dermoscopic image datasets; pair the clinical skin images and dermoscopic images to obtain paired datasets and unpaired datasets; Clinical skin image lesion site segmentation model training module: used to construct a first segmentation model of clinical skin image lesion sites based on a clinical skin image dataset; the first segmentation model takes a clinical skin image as input and takes a clinical skin image segmentation mask corresponding to the skin lesion site as output; Dermoscopic image lesion site segmentation model training module: used to construct a second segmentation model for dermoscopic image lesion sites based on the dermoscopic image dataset, wherein the second segmentation model takes the dermoscopic image as input and takes the dermoscopic image segmentation mask corresponding to the skin lesion site as output; Pairing module: used to pair the clinical skin image segmentation mask and the dermoscopic image segmentation mask to obtain a paired segmentation mask, and then pair the clinical skin image and the dermoscopic image corresponding to the paired segmentation mask to obtain a paired image; Virtual dermoscopy image generation model training module: used to train an unpaired dermoscopy generation model based on an unpaired data set, and then retrain the unpaired dermoscopy generation model based on paired images to obtain a virtual dermoscopy image generation model; The virtual dermoscopic image generation model takes clinical skin images as input and virtual dermoscopic images as output; The unpaired dermatoscope generation model and the virtual dermatoscope image generation model were tested based on the paired test set and the unpaired test set respectively; Virtual dermoscopic image generation module: used to input the clinical skin image acquired in real time into the virtual dermoscopic image generation model to obtain the virtual dermoscopic image corresponding to the current clinical skin image.

Images