Prediction model training method for acquiring blood perfusion map of keloid

Abstract

The invention relates to a prediction model training method for acquiring blood perfusion maps of keloids, which comprises the following steps of: acquiring a sample data set, preprocessing the sample data set, acquiring a training data set comprising a training set and a test set, and samples of which each sample is pix2pix, including a true general image and a true blood perfusion map to which the true general image belongs; a true segmentation image of the general image; respectively inputting the three images of each sample into four corresponding generators in a pre-constructed TripleGAN neural network, and acquiring total loss information by three discriminators based on the outputs of the four generators and the corresponding images in the samples; and further training and testing the TripleGAN neural network to obtain the trained TripleGAN neural network, and taking a generator for receiving the general image and outputting the blood perfusion map in the TripleGAN neural network as a trained prediction model. According to the method, the problems that existing medium scale evaluation is inaccurate, and LSCI evaluation is high in cost and wastes time and labor are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a prediction model training method for obtaining a blood perfusion graph of a keloid, a prediction method for blood perfusion results of the keloid and an electronic device. BACKGROUND

[0002] In the diagnosis and treatment process of keloids, accurate assessment of the severity of its progression is necessary for reminding patients to seek medical intervention, efficacy monitoring, and objectively comparing the efficacy of different intervention measures.

[0003] At present, most of the researches on keloid assessment at home and abroad are mainly based on scale assessment, and the most commonly used assessment standard is the Vancouver Scar Scale (VSS). They mainly perform subjective evaluation from aspects such as pigmentation, vascular dilation, thickness, flexibility and roughness, and the VSS has the defects of observer dependence and poor repeatability of observation.

[0004] Blood perfusion is an objective indicator for keloid assessment, but previous studies on blood flow evaluation in keloids are not consistent. The reason may be that most current studies focus on the number of blood vessels, and the blood vessels in pathological scars (including hypertrophic scars and keloids) may present as endothelial proliferation, lumen stenosis or even occlusion, so the number of blood vessels may not accurately reflect the blood perfusion state in pathological keloids. Therefore, measuring and studying the blood perfusion in pathological keloids as a whole has become a current hotspot.

[0005] Laser speckle contrast imaging (LSCI) is a relatively mature technology based on speckle contrast analysis in recent years. It is an innovative method for evaluating tissue blood perfusion with high resolution and fast scanning time, which can provide objective facts and non-contact measurements of blood perfusion in a specific area. LSCI has been used to evaluate blood flow in patients with microvascular diseases, port-wine stains and burn keloids. Its specific advantages include high image resolution, fast imaging speed, large scanning range and low spatial variability. At present, there are few studies on the application of LSCI to explore the blood perfusion level in the field of keloids. Although LSCI has become an objective method for evaluating tissue blood perfusion, on the one hand, many hospitals do not have this LSCI blood flow evaluation instrument, and on the other hand, it still takes a lot of time to apply LSCI to evaluate keloids. Among them, the outlining of keloids is the most time-consuming, especially for multiple irregular keloids, the outlining time is much higher than that of single regular keloids, which brings inconvenience to clinical work.

[0006] Therefore, there is an urgent need for a scheme for automatically and objectively predicting blood perfusion results of keloids using gross images. SUMMARY

[0007] (I) Technical problems to be solved

[0008] In view of the above-mentioned defects and deficiencies of the prior art, the present application provides a prediction model training method for obtaining a blood perfusion map of a keloid, a prediction method for blood perfusion results of a keloid, and an electronic device, which solves the problems of inaccurate scale assessment and high cost and time-consuming and laborious assessment using LSCI in the prior art.

[0009] (II) Technical solutions

[0010] In order to achieve the above-mentioned purpose, the main technical solutions adopted by the present application include:

[0011] In a first aspect, the present application provides a prediction model training method for obtaining a blood perfusion map of a keloid, comprising:

[0012] Obtaining a sample data set, each sample of the sample data set comprising: a gross image with at least one keloid, a blood perfusion map corresponding to the gross image; a gross image with a keloid boundary outlined;

[0013] Preprocessing the sample data set to obtain a training data set comprising a training set and a test set, the training set comprising paired samples, each paired sample being a pix2pix sample comprising: a true gross image, a true blood perfusion map to which the true gross image belongs; a true segmentation image of the gross image;

[0014] Inputting the three images of each sample into the corresponding four generators of a pre-constructed TripleGAN neural network, and obtaining total loss information based on the outputs of the four generators and the corresponding images in the sample by three discriminators of the TripleGAN neural network;

[0015] Training and testing the TripleGAN neural network based on the training set, the test set, and a target value of the total loss information pre-set, obtaining the trained TripleGAN neural network, and taking the generator in the TripleGAN neural network that receives the gross image and outputs the blood perfusion map as the trained prediction model.

[0016] Optionally, the inputting the three images of each sample into the corresponding four generators of the pre-constructed TripleGAN neural network comprises:

[0017] For each paired sample:

[0018] The true anatomy image is input to the first generator G I2P , and a first blood perfusion map is output;

[0019] The true blood perfusion map is input to the second generator G P2I , and a first anatomy image is output;

[0020] The true anatomy image is input to the third generator G I2S , and a first segmentation image is output;

[0021] The true segmentation image is input to the fourth generator G S2I , and a second anatomy image is output;

[0022] The first blood perfusion map is input to the first generator G I2P , and a third anatomy image is output;

[0023] The first anatomy image is input to the second generator G P2I , and a second blood perfusion map is output;

[0024] The first segmentation image is input to the third generator G I2S , and a fourth anatomy image is output;

[0025] The second anatomy image is input to the fourth generator G S2I , and a second segmentation image is output.

[0026] Optionally, the three discriminators of the TripleGAN neural network obtain total loss information based on the outputs of the four generators and the corresponding images in the samples, including:

[0027] After each sample is input to the generator, the discriminator loss, the consistency loss, and the cycle loss are obtained respectively;

[0028] The total loss information L includes the discriminator loss L D to which the four generators belong, the cycle loss L cyc to which the four generators belong, and the two consistency losses L cons ;

[0029] L=L D (G I2S ,D S ,I,S)+L D (G S2I ,D I ,S,I)+L D (G P2I ,D I ,P,I)+L D (G I2P ,D P ,I,P)+L cyc (G I2S ,GS2I I)+L cyc (G S2I I)+L I2S I)+L

[0030] L cyc I)+L I2P I)+L P2I I)+L cyc I)+L P2I I)+L I2P I)+L cons I)+L I2P I)+L

[0031] L cons I)+L S2I I)+L P2I I)+L

[0032] wherein, G I2P represents the first generator, G P2I represents the second generator, G I2S represents the third generator, and G S2I represents the fourth generator;

[0033] D S represents the first discriminator, D I represents the second discriminator, and D P represents the third discriminator;

[0034] I, S and P respectively correspond to a gross image, a segmentation map and a blood perfusion map;

[0035] G P2I (P) represents an output result of inputting P to G P2I , and e is the base of the logarithm.

[0036] Optionally, the sample data set is preprocessed to obtain a training data set including a training set and a test set, comprising:

[0037] obtaining historical diagnosis and treatment information of a patient ID to which the gross image belongs;

[0038] based on the historical diagnosis and treatment information, removing the gross image and the blood perfusion map of the gross image in the sample data set which has scar treatment record information and incomplete epidermis information, and

[0039] removing the gross image and the blood perfusion map of the gross image in the sample data set which has specified defect information in the historical diagnosis and treatment information;

[0040] cropping and adopting boundary registration for all remaining gross images and blood perfusion maps to obtain a true gross image of pix2pix and a true blood perfusion map to which the true gross image belongs;

[0041] and the automatic recognition processing is performed on the general image with the outlined keloid boundary to obtain a binary true segmentation image;

[0042] The training set includes paired samples, and the test set includes paired samples and unpaired samples.

[0043] The true general image and the true blood perfusion map in the unpaired samples are not paired.

[0044] Optionally, the pre-constructed TripleGAN neural network includes four generators and three discriminators, each generator is an adversarial neural network, the training and test of the TripleGAN neural network are performed based on the training set, the test set and the target value of the preset total loss information, the trained TripleGAN neural network is obtained,

[0045] The phase training method is adopted, the unpaired samples are input for training as the first phase, the paired samples are input for training as the second phase, the first phase and the second phase are alternately performed, and each cycle is greater than 200 rounds.

[0046] There is no consistency loss in the loss in the training of the unpaired samples.

[0047] Optionally, the blood perfusion map of each paired sample is obtained by performing LSCI on the general image in the sample to obtain the blood perfusion map, and the blood perfusion value corresponding to the blood perfusion map is obtained by LSCI.

[0048] In the training of the prediction model, further comprising:

[0049] For the paired sample, the blood perfusion value calculated by the prediction model based on the output blood perfusion map is obtained, and the blood perfusion value is compared with the blood perfusion value in the paired sample.

[0050] The trained prediction model can output the blood perfusion map and the blood perfusion value.

[0051] In a second aspect, the embodiments of the present application further provide a prediction method for blood perfusion results of keloids, comprising:

[0052] Obtaining a general image with a specified size and a keloid to be analyzed;

[0053] Inputting the general image into the trained generator to obtain the blood perfusion result output by the generator.

[0054] The trained generator is a generator obtained by the training method of the prediction model for obtaining the blood perfusion map of the keloid according to any one of claims 1 to 6.

[0055] Optionally, the general image is input into the trained generator to obtain the blood perfusion result output by the generator, including:

[0056] If the prediction interface receives an instruction triggered by a user for predicting the blood perfusion map, the general image is input into the trained generator to obtain the blood perfusion map output by the generator, and the general image and the predicted blood perfusion map are displayed for comparison on the prediction interface.

[0057] If the prediction interface receives an instruction triggered by a user for predicting the blood perfusion value, the general image is input into the trained generator to obtain the blood perfusion map output by the generator, and the blood perfusion value is automatically generated based on the blood perfusion map, the blood perfusion value being the average value of blood flow pixels within the boundary range of the keloid.

[0058] The general image, the predicted blood perfusion map, and the blood perfusion value are displayed on the prediction interface, and the boundary of the keloid is displayed in the blood perfusion map.

[0059] In a third aspect, an electronic device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program stored in the memory and executing the steps of the prediction method of the blood perfusion result of a keloid according to any of the second aspect, or executing the steps of the training method of the prediction model for obtaining the blood perfusion map of a keloid according to any of the first aspect.

[0060] (III) Advantages

[0061] The prediction model training method for obtaining the blood perfusion map of a keloid provided by the present application has the advantages that: the paired samples in the training data set are used to train the pre-constructed neural network with four generators and three discriminators, and then the trained generator is obtained as the prediction model, which facilitates the subsequent output of the blood perfusion map using the general image, improves the judgment efficiency, ensures the accuracy, reduces the cost of the existing LSCI, and solves the defects of high cost and long evaluation time of the LSCI in the prior art, and remote evaluation cannot be realized. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 A flowchart of the prediction model training method for obtaining the blood perfusion map of a keloid provided by the present application is shown in the figure;

[0063] Figure 2 A flowchart of the prediction method of the blood perfusion result of a keloid provided by the present application is shown in the figure;

[0064] Figure 3A andFigure 3B Architectural diagram of TripleGAN neural network provided by the embodiment of the application respectively;

[0065] Figure 4 Schematic diagram of the laser speckle contrast imaging based on gross photographs and model prediction in the embodiment of the application;

[0066] Figure 5 Schematic diagram of the predicted value and the measured value of the model in the pre-treatment and post-treatment conditions in the embodiment of the application; DETAILED DESCRIPTION

[0067] In order to better explain the application, so as to be understood, the application is described in detail in combination with the specific embodiments and the accompanying drawings.

[0068] Scars are the natural healing products after skin injury, specifically, the normal repair process of replacing normal skin with fibrous tissue after skin injury, which usually presents as flat or slightly raised pale pink marks, gradually softening over time, and the range does not exceed the original wound.

[0069] Keloids are a pathological proliferation that can continue to expand and exceed the original injury range, which belongs to an abnormal proliferation, presenting as red or dark red lumps, hard and tough in texture, often accompanied by itching or stinging, and spreading to the surrounding normal skin, commonly seen in high-tension areas such as chest, back and shoulders, and related to genetic, immune abnormalities and other factors.

[0070] TripleGAN is a triple generative adversarial network, which belongs to an improved GAN model, including four generators G and three discriminators D in the training stage.

[0071] Embodiment one

[0072] As shown in the embodiment, the method can be implemented on any server for training a prediction model for predicting the blood perfusion results of keloids on gross images, and the method can include the following steps: Figure 1

[0073] 100, obtaining a sample data set, each sample of the sample data set including: a gross image with at least one keloid, a blood perfusion image corresponding to the gross image, and a gross image with an outlined keloid boundary.

[0074] In actual preparation of the sample data set, a sample pair of gross image-blood perfusion image can be constructed to generate a pixel-based image generation data set.

[0075] The gross image in the sample pair of the embodiment can be an image obtained from a hospital with the consent of the patient, and the blood perfusion image is a blood perfusion image paired with the gross image output by LSCI. ​

[0076] 200. Preprocess the sample dataset to obtain a training dataset including a training set and a test set. The training set includes paired samples, each paired sample being a pix2pix sample (i.e., a sample belonging to pixel-level alignment), including: a true gross image, the true blood flow perfusion map to which the true gross image belongs, and the true segmentation image of the gross image.

[0077] For example, you can first obtain the historical medical information of the patient ID to which each gross image belongs;

[0078] Based on historical medical information, gross images containing keloid treatment records, incomplete epidermal information, and blood perfusion maps of such gross images were removed from the sample dataset.

[0079] Remove gross images and blood perfusion maps of the specified defects from the historical medical information in the sample dataset;

[0080] All remaining gross images and blood perfusion maps are cropped and registered using boundary registration to obtain pix2pix true gross images and the true blood perfusion maps to which the true gross images belong.

[0081] The system automatically identifies and processes the general image with the keloid boundary to obtain a binarized true segmentation image.

[0082] The training set includes paired samples, and the test set includes paired samples and unpaired samples;

[0083] The true gross images and true blood flow perfusion maps in the unpaired samples are not paired.

[0084] It should be noted that during the training phase, in order to better distinguish between paired samples belonging to the training dataset and the generator output during the training process, the paired samples are all distinguished as "true" during training. That is, all images belonging to the paired samples are considered true images input to the generator.

[0085] 300. Input the three images of each sample into the four generators in the pre-constructed TripleGAN neural network, and the three discriminators of the TripleGAN neural network obtain the total loss information based on the output of the four generators and the corresponding images in the sample.

[0086] The pre-constructed TripleGAN neural network includes four generators and three discriminators, each generator is an adversarial neural network, the TripleGAN neural network is trained and tested based on the training set, the test set and the target value of the preset total loss information, and the trained TripleGAN neural network is obtained. In actual application, a validation set can also be set separately, which is processed according to actual conditions, and the embodiment does not limit it.

[0087] It should be noted that in the embodiment, a phased training method can be used, that is, first inputting unpaired samples for training as the first phase, and then inputting paired samples for training as the second phase, the first phase and the second phase are alternately performed, and each cycle is greater than 200 rounds.

[0088] There is no consistency loss in the loss in the unpaired sample training.

[0089] That is, the input of the TripleGAN neural network in the embodiment includes a general image (Image), a blood perfusion image (Perfusion image) and a scar segmentation image (Segmentation). Figure 3A and Figure 3B as shown.

[0090] The TripleGAN neural network mainly includes four generators and three discriminators. The four generators are a general image to blood perfusion image generator G_I2P, a blood perfusion image to general image generator G_P2I, a general image to segmentation image generator G_I2S, and a segmentation image to general image generator G_S2I. Among them, the real image will generate a fake image after passing through the generator, and the fake image will get a cycle image (rec) after passing through the opposite generator.

[0091] There are three types of loss functions, namely the discriminant loss function (that is, the loss in true and false discrimination), the consistency loss function and the cycle loss function. For each fake image (such as fake_SI) generated by passing through a generator once, and the original image (real_I) of this category, it will be input into the discriminator for judgment, and its loss function L D For the cycle image (such as rec_S) obtained by passing through the generator once again for the fake image (such as fake_SI), the difference between the cycle image and the original image is calculated to obtain the loss function L CSince the input includes paired samples, meaning some samples can achieve a pixel-level correspondence between the image and blood perfusion, a consistency loss function is calculated for the generated fake image (e.g., fake_P) and its corresponding original image (real_P). Simultaneously, for the paired input images, a consistency loss function is also calculated between fake_SI and fake_PI.

[0092] It is worth noting that the consistency loss function will not be calculated when unpaired images are used as input.

[0093] 400. Based on the training set, test set, and the target value of the pre-set total loss information, the TripleGAN neural network is trained and tested to obtain the trained TripleGAN neural network. The generator in the TripleGAN neural network that receives the gross image and outputs the blood flow perfusion map is used as the prediction model after training.

[0094] Furthermore, in this embodiment, the perfusion map of each paired sample in steps 100 and 200 above is obtained by performing LSCI on the gross image within the sample to obtain the perfusion map; and the perfusion value corresponding to the perfusion map is obtained by LSCI.

[0095] At this point, training the prediction model in step 400 further includes:

[0096] For paired samples, the blood perfusion value calculated by the prediction model based on the output blood perfusion map is obtained, and the blood perfusion value is compared with the blood perfusion value in the paired samples; the trained prediction model can output blood perfusion map and blood perfusion value.

[0097] In this embodiment, the blood perfusion value is obtained based on the pixel values ​​within a specified area of ​​the blood perfusion map.

[0098] That is, the prediction model can generate fine-grained blood perfusion images. By combining the blood perfusion map with the gross image, the blood perfusion is divided into zones (e.g., distinguishing between scar and non-scar areas; the scar area can be divided into fine grids, and pixel values ​​are calculated for each grid) to obtain the blood perfusion values ​​inside the keloid and the blood perfusion values ​​of the normal skin surrounding the keloid. Specifically, in an optional application, the scar boundary in the blood perfusion map can also be manually delineated to obtain the blood perfusion value.

[0099] The prediction model training method of the embodiment trains the neural network with four generators and three discriminators in advance by using the paired samples in the training data set, thereby obtaining the trained generator as the prediction model, which facilitates subsequent direct use of the large-scale image to output the blood perfusion map, improves the judgment efficiency, ensures the accuracy, reduces the cost of the existing LSCI, and solves the defects of high cost and long evaluation time of the LSCI and the inability to realize remote evaluation of the prior art.

[0100] In an alternative implementation, for better understanding of the process of step 300 described above, the following will be described in detail in combination with Figure 3A and Figure 3B

[0101] 301, for each paired sample: the true large-scale image is input into the first generator G-I2P, and the first blood perfusion map is output;

[0102] The true blood perfusion map is input into the second generator G-P2I, and the first large-scale image is output;

[0103] The true large-scale image is input into the third generator G-I2S, and the first segmentation image is output;

[0104] The true segmentation image is input into the fourth generator G-S2I, and the second large-scale image is output;

[0105] The first blood perfusion map is input into the first generator G-I2P, and the third large-scale image is output;

[0106] The first large-scale image is input into the second generator G-P2I, and the second blood perfusion map is output;

[0107] The first segmentation image is input into the third generator G-I2S, and the fourth large-scale image is output;

[0108] The second large-scale image is input into the fourth generator G-S2I, and the second segmentation image is output.

[0109] In the embodiment, Figure 3A only the modules of the convolutional layer, the BatchNorm layer and the ReLU activation function of one generator are shown, and the flow processing of the skip connection, the up-sampling layer and the down-sampling layer is shown. Meanwhile, in Figure 3A the information and flow processing of the convolutional layer, the normalization layer and the ReLU activation function in the discriminator are shown, which are only shown for illustration.

[0110] In Figure 3B ​In the present embodiment, the same color generator is one, and for better illustration of the input and output, it uses two same colors to represent, its input is generally image (Image, referred to as I), segmentation (Segmentation, referred to as S), perfusion (Perfusion, referred to as P), real represents true, the graph from the sample, fake represents false, represents the graph output by a certain generator, rec represents cycle (that is, after two image generation).

[0111] 302, after each sample input to the generator, respectively, get the discriminator loss, consistency loss and cycle loss;

[0112] The total loss information L includes: four generator belonging to the discriminator loss L D , four generator belonging to the cycle loss L cyc And two consistency losses L cons ;

[0113] L = L D (G I2S ,D S ,I,S) + L D (G S2I ,D I ,S,I) + L D (G P2I ,D I ,P,I) + L D (G I2P ,D P ,I,P) + L cyc (G I2S ,G S2I ,I) + L cyc (G S2I ,G I2S ,S) + L

[0114] L cyc (G I2P ,G P2I ,I) + L cyc (G P2I ,G I2P ,P) + L cons (G I2P ,I,P) + L

[0115] L cons (G S2I ,S,G P2I (P));

[0116] Wherein, G I2P represents the first generator, G P2I represents the second generator, G I2S represents the third generator, and G S2Idenotes the fourth generator;

[0117] D S denotes the first discriminator, D I denotes the second discriminator, D P denotes the third discriminator;

[0118] I, S, P respectively correspond to the general image, the segmentation image and the blood perfusion image;

[0119] G P2I (P) denotes inputting P to G P2I the output result of G; the base number e of log.

[0120] During the entire training process, the four generators are converged to a deployable state based on the total loss information.

[0121] In the embodiment, the four generators are driven to converge to a directly deployable available state through dynamic weight integration of the above-mentioned triple loss.

[0122] Embodiment two

[0123] The method of the embodiment can be implemented on any server or central server. After training of the prediction model, the prediction model can be sent to any medical client such as a PAD or a mobile terminal, or other self-service detection terminal, to realize prediction of the blood perfusion image of the general image with a scar. The prediction model training method of the embodiment can include the following steps:

[0124] Step 1, data acquisition

[0125] The sample pairs of the general images are obtained. All the general images can be obtained by shooting through a mobile terminal such as a smart phone. In the embodiment, the lesion can be clearly seen when the mobile terminal is shot. The magnification is not limited in the embodiment.

[0126] The blood perfusion image and the blood perfusion value (measured in perfusion units, PUs, mL / 100g / min) of the pathological scar are measured for each general image using LSCI and the software matched with the system, thereby obtaining the sample pairs of the general images.

[0127] Step 2, data preprocessing

[0128] In the embodiment, the historical diagnosis and treatment information of the patient ID to which the general image belongs can be obtained in advance, and then the historical diagnosis and treatment information is screened, such as removing the general image and the blood perfusion image of the general image in the sample data set with scar treatment record information and incomplete epidermis information, and removing the general image and the blood perfusion image of the general image in the sample data set with specified defect information in the historical diagnosis and treatment information.

[0129] The data standards of the general images obtained in this embodiment are: 1) consistent with the clinical diagnosis of pathological scars, containing one or more lesions; 2) the patient's pathological scars have not been treated before, including surgery, radiotherapy, laser, corticosteroid injection, compression, cryotherapy, etc.; 3) the lesion epidermis is complete, and there is no obvious sign of infection such as ulceration and pus on the surface; 4) the patient has no systemic diseases such as important organ disease history and active autoimmune disease. It is explained here that the above-mentioned method is used to screen samples in the training data set, and no screening of the general images to be analyzed is required in the test stage.

[0130] Then, each image in the sample is cropped, for example, the general image is cropped to a square with a side length of 3 times the width of the scar, and the acne scar is as possible as possible in the middle of the image.

[0131] Further, the scar boundary in the general image and the blood perfusion map is manually outlined, and a boundary registration-based method is used, that is, the boundary outlined in the general photo is scaled, translated and rotated to coincide with the boundary in the blood perfusion map. In this way, the registration between the general image and the blood perfusion map is completed, thereby constructing a pix2pix-based training sample. In this embodiment, the pix2pix-based paired sample is used, and the purpose is to supervise the value of each pixel in the model training, to ensure the accuracy and accuracy of the model output.

[0132] In order to better obtain the data set for training, the applicant retrospectively collected 948 lesion photos of pathological scars and 758 blood perfusion maps from January 2018 to December 2021, and constructed 156 pairs of paired samples (i.e. the registration sample pairs after segmentation, scaling, translation and rotation of the lesion photo coincide with the boundary of the blood perfusion map). Among the 156 pairs of paired samples, 116 pairs are used as the training set of the blood perfusion prediction model, and the other 40 pairs are used as the test set.

[0133] Each of the 156 pairs of paired samples includes: a general photo, a blood perfusion map.

[0134] In this embodiment, paired samples need to be constructed in the data preprocessing stage.

[0135] The segmented image in the paired sample in this embodiment can be a binary image, which displays 1 and 1 pixel points, 1 representing the pixel points of acne scars, and 0 representing the pixel points of non-acne scars.

[0136] Step 3, constructing a TripleGAN neural network and training based on the constructed TripleGAN neural network

[0137] The TripleGAN neural network includes four generators and three discriminators; the task of the generator is to generate realistic images to confuse the discriminators, and the task of the discriminators is to distinguish the generated images from the original images; through mutual confrontation between the generators, the images are more realistic.

[0138] The four generators are a general image to perfusion image generator G_I2P, a perfusion image to general image generator G_P2I, a general image to segmentation image generator G_I2S, and a segmentation image to general image generator G_S2I.

[0139] The four generators are a general image to perfusion image generator G_I2P, a perfusion image to general image generator G_P2I, a general image to segmentation image generator G_I2S, and a segmentation image to general image generator G_S2I.

[0140] Wherein, the real image (real) is generated by the generator to generate a fake image, and the fake image is generated by the opposite generator to obtain a cycle image (rec). In FIG. 3, real includes real_I, Real_S, and Real_P. I, S, and P correspond to general images, segmentation images, and perfusion images, respectively. fake includes fake_I, fake_S, and fake_P, and rec includes rec_I, rec_S, and rec_P. FIG. 3 shows the network architecture required for the entire training.

[0141] After the model is trained, in the use stage, only the generator G_I2P can be used to input the general image (Image) to generate the perfusion image (Perfusion). The remaining generators and discriminators can not be used.

[0142] In the training process of the embodiment, there are three types of loss functions, namely, a discriminator loss function, a consistency loss function, and a cycle loss function.

[0143] For each fake image (such as fake_SI) generated by a generator, and the original image (real_I) of this category, they are input into the discriminator for judgment, and the loss function LD is calculated.

[0144] For the cycle image (such as rec_S) obtained by inputting the fake image (such as fake_SI) into the generator again, the difference between the cycle image and the original image is calculated to obtain the loss function LC.

[0145] Since the input includes paired samples, that is, part of the samples can realize pixel-level correspondence from images to perfusion, for the generated fake image (such as fake_P) and the corresponding original image (real_P), the consistency loss function is calculated.

[0146] At the same time, for the input paired sample image, the consistency loss function between fake SI and fake PI will also be calculated. It is worth noting that for the non-paired image as input, the consistency loss function will not be calculated. The specific loss function is as follows:

[0147] For the cycle loss L cyc ;

[0148] L cyc (G1,G2,X)=E x∈X [||G2(G1(x))-x||],X∈{I,S};

[0149] L cyc (G1,G2,X)=E x∈X [||norm(G2(G1(x)))-norm(x)||],X∈{P};

[0150] For the discriminator loss L D :

[0151] L D (G,D y ,X,Y)=E x∈X [log(1-D y (G(x)))]+E y∈Y [log(D y (y))],X,Y∈{I,S,P};

[0152] For the consistency loss L cons ;

[0153] L cons (G,X,Y)=E x,y∈X,Y [‖G(x)-y‖],X∈{I,S,P},Y∈{I,S},X≠Y;

[0154] L cons (G,X,Y)=E x,y∈X,Y [‖norm(G(x))-norm(y)‖],X∈{I,S},Y∈{P};

[0155] For the overall loss L

[0156] L=L D (G I2S ,D S ,I,S)+L2(G S2I ,D I ,S,I)+L D (G P2I ,D I ,P,I)+L D (G I2PD P ,I,P)+L cyc (G I2S ,G S2I ,I)+L cyc (G S2I ,G I2S ,S)+

[0157] L cyc (G I2P ,G P2I ,I)+L cyc (G P2I ,G I2P ,P)+L cons (G I2P ,I,P)+

[0158] L cons (G S3I ,S,G P2I (P));

[0159] wherein G1, G2 both represent different generators, G1(x) represents the result output after inputting x to the generator G1, G2(G1(x)) represents the result output after inputting G1(x) to the generator G2; E represents expectation, and norm represents normalization.

[0160] D y (y) represents the result after inputting y to the discriminator D y , D y (G(x)) represents the result after inputting G(x) to D y .

[0161] the base number e of log, I, S, P respectively correspond to the general image, the segmentation image and the blood perfusion image;

[0162] It is particularly pointed out that in the embodiment, the sample containing the unpaired and paired general image and blood perfusion image, and the sample further includes a segmentation image (or called a scar segmentation image), that is, the aforementioned binary image.

[0163] In the embodiment, a phased training method is adopted, at the beginning of each epoch, firstly, the unpaired sample is input for training, and then all the paired samples are input for training, and the training of the two stages is alternately performed. The main parameters include setting the epoch to 200 rounds.

[0164] Step 4, model evaluation

[0165] The blood perfusion value reflects the proliferation of blood vessels and the growth activity of scars, and in the embodiment, the blood perfusion value can be used to evaluate the prediction model. Figure 4The first column is the general image, the second column is the output result of LSCI, and the third column is the output result of the prediction model. By comparing the results, it is found that the results meet the expectations. In order to better evaluate the model, the relative blood perfusion value is used for evaluation.

[0166] Relative blood perfusion value: During the study, it was found that the image generation model was also affected by the color and brightness of the same scar photo. Previous literature studies have shown that using relative blood perfusion values to evaluate scars is more meaningful than absolute blood perfusion evaluation, that is, calculating the blood perfusion difference between the scar and the surrounding skin is more meaningful than calculating the blood perfusion value inside the scar. Therefore, it is considered to calculate the relative blood perfusion index: first calculate the blood perfusion value inside the scar, then calculate the normal skin within the radius distance from the scar according to the size of the scar, and take it as the normal skin blood flow value of the scar. The relative blood perfusion value is obtained by calculating the difference between the two. The blood perfusion values in the foregoing examples are all relative blood perfusion values.

[0167] In addition, in practical applications, shooting conditions can significantly affect model performance, therefore, the model needs to have high robustness.

[0168] In this embodiment, the following three tasks are designed to evaluate the stability of the prediction model:

[0169] 1) Consistency of blood perfusion values under different viewing angles;

[0170] 2) Consistency of blood perfusion values under different light conditions (bright, dim);

[0171] 3) Effect of photo blur degree on blood perfusion value;

[0172] Robustness test data consists of 100 images taken from each of the four videos, a total of 400 images. The video shooting method is continuous recording from a 45° angle to a 135° angle in front of the patient's scar. For each of the images taken, the following processing is performed: light adjustment: adjust the brightness to 0.8 to 1.2 times the original brightness; blur processing: use Gaussian blur method to blur the image to simulate possible image quality degradation in reality.

[0173] The Pearson correlation coefficient between the pathological scar blood perfusion value of the test set sample and the true measured value is calculated, and the absolute error between the two is calculated. The test results show that the correlation coefficient between the predicted relative blood perfusion value and the actual relative blood perfusion value is 0.761, and the blood perfusion error is 9.5 PU, indicating that the model can relatively accurately predict the blood perfusion of the scar area, has higher robustness, and indicates that there is a strong correlation between the relative blood perfusion value prediction model of the scar and the true relative blood perfusion value measured by LSCI. The model can relatively accurately predict the blood perfusion value of the scar in the photo.

[0174] In addition, Figure 5 The results of the predicted blood perfusion value, the blood perfusion value measured by LSCI before treatment, and the blood perfusion value measured by LSCI after treatment are compared, and it is found that the output results of the prediction model are relatively stable and have high accuracy. Figure 5 The prediction model is shown in the middle of the prediction value and the measured value before and after treatment, which reaches a correlation of 0.64 and 0.76, respectively.

[0175] Stability evaluation is used for different perspectives, and stability is defined as: subtracting the mean value of the full perspective result from the prediction result of each perspective, and calculating the fluctuation caused by the change of the perspective. The smaller the fluctuation, the more stable the model.

[0176] Anti-interference ability evaluation is used under light and blur interference, and anti-interference ability is defined as: comparing the results under interference condition with the average value of the results under non-interference condition. If the absolute value of the difference is small, the model shows strong anti-interference ability; on the contrary, if the difference value is large and the fluctuation is significant, the anti-interference ability of the model is weak.

[0177] Example Three

[0178] The embodiment provides a prediction method for blood perfusion results of a keloid, which can be implemented in any lightweight client, such as an APP or a small program of a mobile terminal, or a client of a doctor or a client of a self-service device, and the like, and the prediction model can be arranged to implement the prediction method of the application. The prediction method of the embodiment can include:

[0179] A1, obtaining a general image of a keloid to be analyzed with a specified size;

[0180] A2, inputting the general image into the trained generator to obtain the blood perfusion result output by the generator;

[0181] The trained generator is a generator obtained by the training method of the prediction model for obtaining the blood perfusion map of the keloid according to any of the preceding embodiments.

[0182] The step A2 can further include:

[0183] A21, if the prediction interface receives a user-triggered instruction for predicting a blood perfusion map, the general image is input into the trained generator, a blood perfusion map output by the generator is obtained, and the general image and the predicted blood perfusion map are compared and displayed on the prediction interface;

[0184] A22, if the prediction interface receives a user-triggered instruction for predicting a blood perfusion value, the general image is input into the trained generator, a blood perfusion map output by the generator is obtained, and a blood perfusion value is automatically generated based on the blood perfusion map, the blood perfusion value being an average value of blood flow pixels within a boundary range of the keloid;

[0185] and the general image, the predicted blood perfusion map, and the blood perfusion value are displayed on the prediction interface; and the blood perfusion map displays the boundary of the keloid. In this embodiment, the boundary of the keloid can be recognized by an automatic recognition model or manually outlined, and is selected according to actual conditions, which is not limited in this embodiment.

[0186] The prediction method of this embodiment is simple and portable, and can effectively reduce the cost and save time of doctors, and at the same time, reduce the time cost of patients, and is convenient for popularization and use.

[0187] In addition, the embodiment also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program stored in the memory and executes the steps of the prediction method of the blood perfusion result of the keloid according to any of Embodiment Three, or executes the steps of the training method of the prediction model for obtaining the blood perfusion map of the keloid according to any of Embodiment One or Embodiment Two.

[0188] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0189] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions.

[0190] It should be noted that the description uses terms like first, second, third, etc. These should not be interpreted as implying any order or sequence unless explicitly stated. These terms are merely used to identify different components.

[0191] Furthermore, it is noted that the description of the present application is made in relation to specific embodiments thereof, and that the terms used are to be construed in an indicative and not in a limiting sense. The description is made in relation to the embodiments with reference to the drawings, wherein:

[0192] While the preferred embodiments of the application have been described, it should be understood that various modifications and changes can be made by those skilled in the art which follow in the spirit of the application and the scope of the appended claims. There is no intention that the application be limited to the embodiments described. Accordingly, the application is intended to embrace all such alterations, modifications and variations which fall within the scope of the application, including licensed or customary uses of the same, and that it includes all such alternatives, modifications and variations as come within the scope of the appended claims and their equivalents.

[0193] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for training a predictive model to obtain blood flow perfusion maps of keloids, characterized in that, include: Obtain a sample dataset, wherein each sample in the sample dataset includes: a gross image of at least one keloid, a blood perfusion map corresponding to the gross image; and a gross image outlining the boundary of the keloid. The sample dataset is preprocessed to obtain a training dataset including a training set and a test set. The training set includes paired samples, each paired sample being a pix2pix sample, including: a true gross image, the true blood flow perfusion map to which the true gross image belongs, and the true segmentation image of the gross image. The three images of each sample are input into the four generators in the pre-built TripleGAN neural network, and the three discriminators of the TripleGAN neural network obtain the total loss information based on the output of the four generators and the corresponding images in the sample. Based on the training set, test set, and pre-set target value of total loss information, the TripleGAN neural network is trained and tested to obtain the trained TripleGAN neural network. The generator in the TripleGAN neural network that receives the gross image and outputs the blood flow perfusion map is used as the prediction model after training.

2. The training method according to claim 1, characterized in that, The step of inputting the three images of each sample into the four corresponding generators in the pre-constructed TripleGAN neural network includes: For each paired sample: The true macro image is input into the first generator G. I2P Output the first blood flow perfusion map; The true blood flow perfusion map is input into the second generator G. P2I Output the first general image; The true macro image is input into the third generator G. I2S Output the first segmented image; The true segmented image is input into the fourth generator G. S2I Output the second large image; The first blood flow perfusion map is input into the first generator G. I2P Output the third large image; The first general image is input into the second generator G. P2I Output the second blood flow perfusion map; The first segmented image is input into the third generator G. I2S Output the fourth large image; The second large image is input into the fourth generator G. S2I Output the second segmented image.

3. The training method according to claim 2, characterized in that, The TripleGAN neural network's three discriminators obtain total loss information based on the outputs of the four generators and the corresponding images in the samples, including: After each sample is input into the generator, the discriminator loss, consistency loss, and cycle loss are obtained respectively. The total loss information L includes: the discriminator loss L of the four generators. D The cycle loss L of the four generators cyc and two consistency losses L cons ; L=L D (G I2S ,D S ,I,S)+L D (G S2I ,D I ,S,I)+L D (G P2I ,D I ,P,I)+L D (G I2P ,D P ,I,P)+L cyc (G I2S ,G S2I ,I)+L cyc (G S2I ,G I2S ,S)+L cyc (G I2P ,G P2I ,I)+L cyc (G P2I ,G I2P ,P)+L cons (G I2P ,I,P)+L cons (G S2I ,S,G P2I (P)); Among them, G I2P Represents the first generator, G P2I Indicates the second generator, G I2S Represents the third generator, G S2I Indicates the fourth generator; D S Indicates the first discriminator, D I Indicates the second discriminator, D P This represents the third discriminator; I, S, and P correspond to the gross image, segmentation image, and blood perfusion image, respectively. G P2I (P) indicates that P is input into G. P2I The output result; the base e of the logarithm.

4. The training method according to claim 1, characterized in that, The sample dataset is preprocessed to obtain a training dataset including a training set and a test set, including: Retrieve the historical medical information of the patient ID to which the gross image belongs; Based on historical medical information, gross images containing keloid treatment records, incomplete epidermal information, and blood perfusion maps of such gross images were removed from the sample dataset. Remove gross images and blood perfusion maps of the specified defects from the historical medical information in the sample dataset; All remaining gross images and blood perfusion maps are cropped and registered using a boundary registration method to obtain pix2pix true gross images and the true blood perfusion maps to which the true gross images belong; The system automatically identifies and processes the general image with the keloid boundary to obtain a binarized true segmentation image. The training set includes paired samples, and the test set includes paired samples and unpaired samples; The true gross images and true blood flow perfusion maps in the unpaired samples are not paired.

5. The training method according to claim 1, characterized in that, The pre-built TripleGAN neural network includes four generators and three discriminators. Each generator is an adversarial neural network. The TripleGAN neural network is trained and tested based on the training set, the test set, and a pre-defined target value for total loss information to obtain the trained TripleGAN neural network. A phased training method is adopted. First, unpaired samples are input for training as the first phase, and then paired samples are input for training as the second phase. The first and second phases are alternated, with each cycle consisting of more than 200 rounds. There is no consistency loss in the loss for training unpaired samples.

6. The training method according to claim 1, characterized in that, The perfusion map of each paired sample is obtained by performing LSCI on the gross image of the sample. And obtain the blood perfusion value corresponding to the blood perfusion map through LSCI; Training the prediction model also includes: For paired samples, obtain the blood perfusion value calculated by the prediction model based on the output blood perfusion map, and compare the blood perfusion value with the blood perfusion value in the paired samples; The trained prediction model can output blood perfusion maps and blood perfusion values.

7. A method for predicting blood perfusion results in keloids, characterized in that, include: Obtain a gross image of the keloid to be analyzed at a specified size; The general image is input into the trained generator to obtain the blood perfusion results output by the generator; The trained generator is a generator obtained based on the training method of the prediction model for obtaining blood perfusion maps of keloids as described in any one of claims 1 to 6.

8. The prediction method according to claim 7, characterized in that, The general image is input into the trained generator to obtain the blood perfusion results output by the generator, including: If the prediction interface receives a user-triggered instruction to predict the blood perfusion map, the gross image is input into the trained generator to obtain the blood perfusion map output by the generator, and the gross image and the predicted blood perfusion map are compared and displayed on the prediction interface. If the prediction interface receives a user-triggered instruction to predict blood perfusion values, the general image is input into the trained generator to obtain the blood perfusion map output by the generator, and the blood perfusion value is automatically generated based on the blood perfusion map. The blood perfusion value is the average value of blood flow pixels within the boundary range of the keloid. The prediction interface displays the general image, the predicted blood perfusion map, and the blood perfusion value; the blood perfusion map shows the boundary of the keloid.

9. An electronic device, characterized in that, include: The device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory and performs the steps of the method for predicting the blood perfusion results of keloids according to any one of claims 7 and 8; or, performs the steps of the training method for a prediction model for obtaining blood perfusion maps of keloids according to any one of claims 1 to 6.

Images