A skin condition alert system to assist in follow-up visits

Abstract

This invention provides a skin condition alert system for assisting follow-up visits, belonging to the field of assisted follow-up visits. The skin condition alert system includes: a skin image acquisition unit for acquiring original images and determining target skin images based on the original images; a lesion detection unit for detecting candidate lesion boxes in the target skin image based on a target detection model; the target detection model is obtained by pre-training a CenterNet network using a first training sample set; a classification unit for determining the skin disease type at each lesion candidate box based on the original image and the lesion candidate boxes in the original image, using a dual-channel classification model; the dual-channel classification model is obtained by pre-training a convolutional neural network using a second training sample set; and an alert unit for determining the severity value of the skin condition based on the skin disease type at each lesion candidate box. This system can better monitor the patient's skin condition, dynamically adjust medication regimens, and improve follow-up visit efficiency.

Description

Technical Field

[0001] This invention relates to the field of assisted follow-up visits, and in particular to a skin condition warning system for assisted follow-up visits. Background Technology

[0002] In many medical fields, such as orthodontics and liver cancer treatment, current follow-up visits are mostly conducted in-person at hospitals / clinics, which significantly restricts regular check-ups due to time and space constraints. Therefore, established follow-up monitoring and management systems exist, typically comprising a camera and a processing unit. The camera is taken by the patient using specialized hardware. The processing unit then uses image algorithms such as image sharpening and 3D model reconstruction to process the images. After processing, the system returns the processed image and suggested follow-up appointments to the doctor.

[0003] However, in the field of dermatology, there is currently no such follow-up management system. Since most skin diseases are characterized by chronicity, mild symptoms, and recurrent flare-ups, an offline follow-up system for patients could significantly reduce the burden on hospitals and allow them to gain a dynamic understanding of the progression of patients' conditions.

[0004] In the treatment of skin diseases, most are chronic, mild, and recurrent, so patients often do not require hospitalization. Even when their condition worsens, they often choose to change hospitals rather than seek post-treatment feedback. Therefore, hospital departments cannot conduct long-term follow-up observations, and the monitoring of treatment effectiveness is often insufficient. This prevents them from providing targeted long-term treatment to patients or improving the quality of treatment based on the effectiveness of past cases. Developing new follow-up tools for skin diseases can facilitate the return of skin disease cases, which has significant medical and research value for dermatology. Currently, many other departments use specialized equipment for follow-up research. However, there is currently no follow-up system specifically for skin diseases. Summary of the Invention

[0005] The purpose of this invention is to provide a skin condition alert system to assist in follow-up visits, which can monitor the patient's skin condition in real time, dynamically adjust the medication plan, and improve the efficiency of follow-up visits.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A skin condition alert system to assist in follow-up visits includes:

[0008] A skin image acquisition unit is used to acquire an original image and determine a target skin image based on the original image;

[0009] The lesion detection unit, connected to the skin image acquisition unit, is used to detect lesion candidate boxes in the target skin image based on the target detection model; the target detection model is obtained by pre-training the CenterNet network using a first training sample set; the first training sample set includes multiple first sample images and lesion candidate boxes in each first sample image;

[0010] A classification unit, connected to the lesion detection unit, is used to determine the skin disease type at each lesion candidate box based on the original image and the lesion candidate boxes in the original image, using a dual-channel classification model. The dual-channel classification model is obtained by pre-training a convolutional neural network using a second training sample set. The second training sample set includes multiple second sample images, lesion candidate boxes in each second sample image, and the actual skin disease type of each lesion candidate box.

[0011] The warning unit, connected to the classification unit, is used to determine the severity value of the skin condition based on the type of skin disease at each lesion candidate box.

[0012] Optionally, the skin image acquisition unit includes:

[0013] The image acquisition module is used to acquire raw images;

[0014] An image quality inspection module, connected to the image acquisition module, is used to determine whether the brightness of the original image meets the set range and whether the sharpness of the edges of the original image is greater than the set threshold. If so, the original image is retained; otherwise, the original image is re-acquired.

[0015] A skin recognition module, connected to the image module, is used to determine the target skin image from the original image based on a skin segmentation model; the skin segmentation model is obtained by training Unet using a third training sample set; the third training sample set includes multiple third sample images and labels of skin regions in each third sample image.

[0016] Optionally, the image quality inspection module uses an edge operator to extract the edges of the original image and determines the sharpness of the edges.

[0017] Optionally, the skin image acquisition unit further includes:

[0018] A timing module, connected to the image acquisition module, is used to control the image acquisition module to acquire raw images according to a set time period.

[0019] Optionally, the skin condition alert system for assisting follow-up visits further includes:

[0020] The sample acquisition unit is used to acquire the second training sample set;

[0021] The prediction unit, connected to the sample acquisition unit, is used to input the second sample image and the lesion candidate boxes in the second sample image into a convolutional neural network for any second sample image to obtain the prediction type of each lesion candidate box;

[0022] The training unit, connected to the prediction unit, is used to determine the loss function based on the predicted type of each lesion candidate box and the actual skin disease type in each second sample image, and to train the convolutional neural network using stochastic gradient descent and back-derivative methods based on the loss function until the loss function converges, thereby obtaining a dual-channel classification model.

[0023] Optionally, the loss function is the cross-entropy loss function.

[0024] Optionally, the skin disease types include blackheads, whiteheads, acne, papules, nodules, pustules, acne scars, and cysts.

[0025] Optionally, the skin condition alert system for assisting follow-up visits further includes:

[0026] A medical advice unit, connected to the warning unit, is used to determine corresponding medical advice based on the severity value of the skin condition and a knowledge graph; the knowledge graph is a mapping table between the severity value and the corresponding medical advice.

[0027] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0028] The skin image acquisition unit acquires raw images and determines target skin images based on these raw images. The lesion detection unit detects candidate lesion boxes in the target skin image based on a target detection model. The classification unit determines the skin disease type at each candidate lesion box based on the raw image and the candidate lesion boxes in the raw image, using a dual-channel classification model. The alert unit determines the severity value of the skin condition based on the skin disease type at each candidate lesion box. The severity value can better monitor the patient's skin condition, dynamically adjust the medication plan, and thus improve the efficiency of follow-up visits. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the skin condition warning system for assisting follow-up visits according to the present invention.

[0031] Symbol explanation:

[0032] Skin image acquisition unit-1, lesion detection unit-2, classification unit-3, warning unit-4, medical advice unit-5. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The purpose of this invention is to provide a skin condition alert system to assist in follow-up visits. By returning the severity value of the skin condition, doctors can dynamically adjust the medication plan to achieve targeted medication and obtain data on the changes in the patient's condition throughout the entire course of the disease, thereby improving the efficiency of follow-up visits.

[0035] Most dermatological diagnoses do not require specialized pathological slides; a photograph of the skin surface is sufficient for diagnosis. This invention can track the progression of a patient's specific dermatological condition, helping hospitals collect and monitor the dynamic changes in the patient's disease, thereby providing more targeted long-term treatment plans and improving the department's research capabilities.

[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, the skin condition warning system for assisting follow-up visits of the present invention includes: a skin image acquisition unit 1, a lesion detection unit 2, a classification unit 3, and a warning unit 4.

[0038] The skin image acquisition unit 1 is used to acquire the original image and determine the target skin image based on the original image.

[0039] Specifically, the skin image acquisition unit 1 includes: an image acquisition module, an image quality inspection module, and a skin recognition module.

[0040] The image acquisition module is used to acquire raw images.

[0041] The image quality inspection module is connected to the image acquisition module. The image quality inspection module determines whether the brightness of the original image meets a set range and whether the sharpness of the edges of the original image exceeds a set threshold. If so, the original image is retained; otherwise, the original image is re-acquired. Specifically, the image quality inspection module uses an edge operator to extract the edges of the original image and determines the sharpness of the edges. If the image boundaries are not sharp enough, it is determined to be a blurry image to prevent the image from being so blurry that lesions cannot be detected.

[0042] Specifically, in order to ensure the quality of the input image, the brightness of the image in the facial area should not be too bright or too dark, so as to ensure that the overall brightness of the image is within the range that is easy for subsequent model recognition. Otherwise, images with insufficient lighting or overexposed lighting will make it difficult to implement the subsequent target detection model and dual-channel classification model. Therefore, this invention achieves quality control of lighting by detecting the brightness of the original image.

[0043] The skin recognition module is connected to the image and is used to determine the target skin image from the original image based on a skin segmentation model. The skin segmentation model is obtained by training Unet using a third training sample set. The third training sample set includes multiple third sample images and labels for skin regions in each third sample image. The skin recognition module prevents users from mistakenly uploading non-skin images.

[0044] The loss function of the skin segmentation model is a variation of cross-entropy:

[0045] E = Σ x∈Ω w(x)log(p l(x) (x));

[0046] Where E is the loss function value of the skin segmentation model, Ω is the third training sample set, w(x) is a coefficient related to the overlap area, and p l(x) The probability that the third sample image x output by the skin segmentation model is of the positive class.

[0047] The image recognition module and skin recognition module can obtain clear images of the patient's skin lesions under appropriate lighting conditions from different angles.

[0048] Furthermore, the skin image acquisition unit 1 also includes a timing module. The timing module is connected to the image acquisition module and is used to control the image acquisition module to acquire raw images according to a set time period.

[0049] Specifically, for patients attending follow-up appointments, the timed module will send medication reminders to patients at pre-set intervals according to the patient and doctor's settings, while simultaneously reminding them to complete a skin test to monitor their current skin condition. Patients can also choose to initiate an additional skin test themselves when facial skin conditions flare up. It's worth noting that in the field of dermatology, skin tests are generally not used for follow-up appointments. For follow-up appointments, placing the post-diagnosis medication and skin test procedures together can reduce patients' resistance to information collection. After receiving the medication and test reminders, patients take a photo of the affected area, i.e., the original image, following the suggested shooting angle.

[0050] The lesion detection unit 2 is connected to the skin image acquisition unit 1. The lesion detection unit 2 is used to detect lesion candidate boxes (including lesion locations) in the target skin image based on the target detection model. The target detection model is obtained by pre-training the CenterNet network using a first training sample set. The first training sample set includes multiple first sample images and lesion candidate boxes in each first sample image.

[0051] Classification unit 3 is connected to lesion detection unit 2. Classification unit 3 is used to determine the skin disease type at each lesion candidate box based on the original image and the lesion candidate boxes in the original image, using a dual-channel classification model. The dual-channel classification model is obtained by pre-training a convolutional neural network using a second training sample set. The second training sample set includes multiple second sample images, lesion candidate boxes in each second sample image, and the actual skin disease type of each lesion candidate box.

[0052] The dual-channel classification model uses local information (candidate lesion boxes) and global information (original image) to derive classification results through forward inference, thus confirming the type of skin disease. For each candidate lesion box, it can be input into the dual-channel classification model along with the original image to obtain its classification result. For example, after inputting the target skin image into the target detection model, three candidate lesion boxes are drawn, which are the suspected lesion areas output by the model. Then, each candidate lesion box is input into the dual-channel classification model along with the original image to obtain the final result.

[0053] The warning unit 4 is connected to the classification unit 3. The warning unit 4 is used to determine the severity value of the skin condition based on the type of skin disease at each lesion candidate box.

[0054] Regarding the training of the dual-channel classification model, the skin condition warning system for assisting follow-up visits of this invention also includes: a sample acquisition unit, a prediction unit, and a training unit.

[0055] The sample acquisition unit is used to acquire the second training sample set.

[0056] The prediction unit is connected to the sample acquisition unit. The prediction unit is used to input the second sample image and the lesion candidate boxes in the second sample image into the convolutional neural network for any second sample image to obtain the prediction type of each lesion candidate box.

[0057] The training unit is connected to the prediction unit. The training unit determines a loss function based on the predicted type of each lesion candidate box and the actual skin disease type in each second sample image. Based on this loss function, it trains the convolutional neural network using stochastic gradient descent and backpropagation until the loss function converges, resulting in a dual-channel classification model. The loss function is the cross-entropy loss function.

[0058]

[0059] Where N is the number of second sample images, M is the number of lesion candidate box types, and pic is the sample probability output by the model when the i-th sample is classified as c. When the i-th sample is correctly classified as c, yic is 1, otherwise it is 0.

[0060] In this embodiment, the skin disease types include blackheads, whiteheads, acne, papules, nodules, pustules, acne scars, and cysts.

[0061] To further improve the efficiency of follow-up visits, the skin condition alert system for assisting follow-up visits of this invention also includes a medical advice unit 5. The medical advice unit 5 is connected to the alert unit 4, and is used to determine corresponding medical advice based on the severity value of the skin condition and a knowledge graph. The knowledge graph is a mapping table between severity values ​​and corresponding medical advice.

[0062] After obtaining the location and classification results of the lesion candidate boxes, this invention calculates the severity of the condition based on the number of papules, nodules, pustules, acne scars, and cysts through weighted calculations. Furthermore, it queries a knowledge graph using the current values ​​of various lesion signs and returns corresponding medical suggestions to the patient. Simultaneously, this invention also sends the location and classification results of each lesion candidate box to the doctor for treatment reference. By returning the processed severity values ​​and past medication regimens, doctors can better monitor the patient's skin condition and dynamically adjust the medication regimen to achieve targeted treatment. The knowledge graph is a mapping table based on the severity of different features. When whiteheads and blackheads are severe, the database containing articles about whiteheads and blackheads is automatically retrieved.

[0063] The severity score is calculated using a modified GAGS (Global Acne Grading System). One point is awarded for blackheads, whiteheads, or pimples; two points for papules; three points for pustules; and four points for cysts. The score is then multiplied by a weighted number of locations: nose (1 point), forehead, cheeks, and chin (2 points), and other areas (3 points). A total score of 1-18 indicates mild acne, 19-30 moderate, 31-38 severe, and 39 or higher extremely severe.

[0064] Compared to GAGS, this invention removes the weighting based on location and introduces the number of acne scars and the size of cysts as weights. The number of acne scars is smoothed using a logarithmic function, while the size of the cysts is calculated by taking the square root of the value. The final formula for calculating the severity score is:

[0065]

[0066] Where x1 to x4 represent the number of blackheads / whiteheads / acne, papules, nodules, and pustules, respectively; y1 to y2 represent the areas of acne scars and cysts, respectively; and α1 to α4 and β1 to β4 are adjustment coefficients. The threshold is set empirically.

[0067] The entire follow-up consultation process can be completed online without having to visit the hospital, which greatly improves the efficiency of follow-up consultations.

[0068] For example, a patient with severe acne, unable to visit the hospital daily for various reasons, chooses to monitor their condition online. The patient uploads images of the affected areas daily, or when the condition worsens, based on the reminders provided by this invention. The invention then provides the doctor's previous medication recommendations and initiates a skin disease detection process. The uploaded images undergo quality control, followed by a target detection model to identify candidate lesion boxes. A dual-channel classification model then returns the lesion classification results. For instance, this invention might identify three lesions in the patient: cysts, acne scars, and pustules. Finally, the invention summarizes the suspected lesion results into numerical values ​​and sends them to the doctor, prompting them to develop a targeted and dynamic treatment plan.

[0069] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the system and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A skin condition alert system for assisting follow-up visits, characterized in that, The skin condition alert system for assisting follow-up visits includes: A skin image acquisition unit is used to acquire an original image and determine a target skin image based on the original image; the skin image acquisition unit includes: The image acquisition module is used to acquire raw images; An image quality inspection module, connected to the image acquisition module, is used to determine whether the brightness of the original image meets the set range and whether the sharpness of the edges of the original image is greater than the set threshold. If so, the original image is retained; otherwise, the original image is re-acquired. A skin recognition module, connected to the image quality inspection module, is used to determine a target skin image from the original image based on a skin segmentation model; the skin segmentation model is obtained by training Unet using a third training sample set; the third training sample set includes multiple third sample images and labels of skin regions in each third sample image; A timing module, connected to the image acquisition module, is used to control the image acquisition module to acquire raw images according to a set time period; The lesion detection unit, connected to the skin image acquisition unit, is used to detect lesion candidate boxes in the target skin image based on the target detection model; the target detection model is obtained by pre-training the CenterNet network using a first training sample set; the first training sample set includes multiple first sample images and lesion candidate boxes in each first sample image; A classification unit, connected to the lesion detection unit, is used to determine the skin disease type at each lesion candidate box based on the original image and the lesion candidate boxes in the original image, using a dual-channel classification model. The dual-channel classification model is obtained by pre-training a convolutional neural network using a second training sample set. The second training sample set includes multiple second sample images, lesion candidate boxes in each second sample image, and the actual skin disease type of each lesion candidate box. The alert unit, connected to the classification unit, is used to determine the severity value of the skin condition based on the skin disease type at each lesion candidate box; the formula for calculating the severity value is: Where x1 to x4 are the number of blackheads / whiteheads / acne, papules, nodules, and pustules, respectively; y1 to y2 are the areas of acne scars and cysts, respectively; and α1 to α4 and β1 to β4 are adjustment coefficients.

2. The skin condition alert system for assisting follow-up visits according to claim 1, characterized in that, The image quality inspection module uses an edge operator to extract the edges of the original image and determines the sharpness of the edges.

3. The skin condition alert system for assisting follow-up visits according to claim 1, characterized in that, The skin condition alert system for assisting follow-up visits also includes: The sample acquisition unit is used to acquire the second training sample set; The prediction unit, connected to the sample acquisition unit, is used to input the second sample image and the lesion candidate boxes in the second sample image into a convolutional neural network for any second sample image to obtain the prediction type of each lesion candidate box; The training unit, connected to the prediction unit, is used to determine the loss function based on the predicted type of each lesion candidate box and the actual skin disease type in each second sample image, and to train the convolutional neural network using stochastic gradient descent and back-derivative methods based on the loss function until the loss function converges, thereby obtaining a dual-channel classification model.

4. The skin condition alert system for assisting follow-up visits according to claim 3, characterized in that, The loss function is the cross-entropy loss function.

5. The skin condition alert system for assisting follow-up visits according to claim 1, characterized in that, The skin conditions mentioned include blackheads, whiteheads, acne, papules, nodules, pustules, acne scars, and cysts.

6. The skin condition alert system for assisting follow-up visits according to claim 1, characterized in that, The skin condition alert system for assisting follow-up visits also includes: A medical advice unit, connected to the warning unit, is used to determine corresponding medical advice based on the severity value of the skin condition and a knowledge graph; the knowledge graph is a mapping table between the severity value and the corresponding medical advice.

Images