A dermatological classification system incorporating pictorial information

Abstract

This invention provides a skin disease classification system combining image and text information, belonging to the field of dermatology diagnosis and treatment. The system includes: an image and text acquisition unit for acquiring image information and medical text information of the target skin; the medical text information includes disease descriptions and past medical history; an image feature determination unit for performing target detection on the image information, determining candidate lesion boxes in the image information, classifying each candidate lesion box, and determining image features; a text feature determination unit for performing noun entity recognition and encoding on the medical text information, determining multiple text features; and a classification unit for determining the skin disease type of the target skin based on the image and text features and a text-image classification model; the text-image classification model is obtained by pre-training a support vector machine using a first training sample set. By combining image and medical text information, the accuracy of skin disease classification is improved.

Description

Technical Field

[0001] This invention relates to the field of dermatology diagnosis and treatment, and in particular to a dermatology classification system that combines graphic and textual information. Background Technology

[0002] Developing new diagnostic and assessment tools for skin diseases can empower primary care dermatologists and general practitioners with the clinical experience of top dermatologists, enabling tiered diagnosis and treatment of skin diseases, which has significant medical and scientific research value.

[0003] Currently, the diagnosis and treatment of skin diseases mainly uses image-based algorithms. However, relying solely on images for the diagnosis and treatment of skin diseases does not take into account the patient's own description of the symptoms, such as information that cannot be observed in the images, such as physical sensations and past medical history. As a result, the accuracy of skin disease classification is not high enough and cannot meet the current needs of patients. Summary of the Invention

[0004] The purpose of this invention is to provide a skin disease classification system that combines textual and graphic information, which can improve the accuracy of skin disease classification.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A skin disease classification system combining textual and image information includes:

[0007] The image and text acquisition unit is used to acquire image information and medical text information of the target skin; the medical text information includes disease description information and past medical history.

[0008] An image feature determination unit, connected to the image and text acquisition unit, is used to perform target detection on the image information, determine lesion candidate boxes in the image information, classify each lesion candidate box, and determine image features; the image features include the probability that each lesion candidate box belongs to each type of skin disease;

[0009] A text feature determination unit, connected to the image and text acquisition unit, is used to perform noun entity recognition and encoding on the medical text information to determine multiple text features;

[0010] The classification unit is connected to the image feature determination unit and the text feature determination unit respectively, and is used to determine the skin disease type of the target skin based on the image features and the text features and the image-text classification model; the image-text classification model is obtained by training a support vector machine in advance using a first training sample set; the first training sample set includes multiple sample features and the skin disease type corresponding to each sample feature.

[0011] Optionally, the image feature determination unit includes:

[0012] The target detection module, connected to the image acquisition unit, is used to perform target detection on the image information and determine the candidate bounding boxes of lesions in the image information;

[0013] An image classification module, connected to the target detection module, is used to determine image features based on the image information and lesion candidate boxes in the image information, using a dual-channel classification model. The dual-channel classification model is obtained by training a convolutional neural network using a second training sample set. The second training sample set includes multiple first sample images, lesion candidate boxes in each first sample image, and the skin disease category of each lesion candidate box.

[0014] Optionally, the target detection module performs target detection on the image information based on a lesion detection model; the lesion detection model is obtained by training CenterNet in advance using a third training sample set; the third training sample set includes multiple second sample images and lesion candidate boxes in each second sample image.

[0015] Optionally, the text feature determination unit includes:

[0016] An entity recognition module, connected to the image and text acquisition unit, is used to perform noun entity recognition on the medical text information using the FLAT model to obtain multiple noun entities;

[0017] A mapping module, connected to the entity recognition module, is used to map each noun entity to a medical entity; the medical entity is an entity with medical significance.

[0018] The encoding module, connected to the mapping module, is used to perform one-hot encoding on each medical entity to obtain multiple corresponding text features.

[0019] Optionally, the mapping module includes:

[0020] The vector embedding submodule, connected to the entity recognition module, is used to perform sentence vector embedding on each noun entity to transform each noun entity into a corresponding vector, thereby obtaining the sentence vector corresponding to each noun entity.

[0021] The distance calculation submodule, connected to the vector embedding submodule, is used to calculate the cosine distance between any sentence vector and each medical entity vector in the medical entity database; the medical entity database includes multiple medical entities and the corresponding medical entity vectors of each medical entity.

[0022] The entity determination submodule, connected to the distance calculation submodule, is used to identify the medical entity corresponding to the medical entity vector with the smallest cosine distance to the sentence vector as the medical entity of the sentence vector.

[0023] Optionally, SentenceBERT can be used to embed sentence vectors for each noun entity.

[0024] Optionally, the classification unit includes:

[0025] The merging module is connected to the image feature determination unit and the text feature determination unit respectively, and is used to merge the image features and text features into a single feature vector;

[0026] The type determination module, connected to the merging module, is used to determine the skin disease type of the target skin based on the feature vector and an image-text classification model; the sample features in the first training sample set are vectors.

[0027] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0028] The image and text acquisition unit acquires image information and medical text information of the target skin. The medical text information includes disease description information and past medical history. The image feature determination unit performs target detection on the image information, determines the candidate lesion boxes in the image information, classifies each candidate lesion box, and determines image features (the probability of each candidate lesion box belonging to each skin disease). The text feature determination unit performs noun entity recognition and encoding on the medical text information, and determines multiple text features. Based on the image and text features and the image and text classification model, the classification unit determines the skin disease type of the target skin, thereby improving the accuracy of skin disease classification. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the modules of the skin disease classification system that combines textual and graphic information according to the present invention;

[0031] Figure 2 This is a schematic diagram of a dual-channel classification model;

[0032] Figure 3 A schematic diagram illustrating the overall process of classifying skin diseases.

[0033] Symbol explanation:

[0034] Image and text acquisition unit-1, image feature determination unit-2, text feature determination unit-3, classification unit-4. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] The purpose of this invention is to provide a skin disease classification system that combines image and text information to classify patient skin, thereby improving classification accuracy.

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1 As shown, the skin disease classification system combining image and text information of the present invention includes: an image and text acquisition unit 1, an image feature determination unit 2, a text feature determination unit 3, and a classification unit 4. In this embodiment, the image and text acquisition unit 1, the image feature determination unit 2, the text feature determination unit 3, and the classification unit 4 are stored in a memory and run by a processor.

[0039] The image and text acquisition unit 1 is used to acquire image information and medical text information of the target skin; the medical text information includes symptom description information and past medical history. In this embodiment, image information refers to the image containing the lesion corresponding to the case. Medical text information refers to the patient's chief complaint information collected by the doctor, which is the patient's own description of the symptom, including information such as physical sensations and past medical history that cannot be observed in the image.

[0040] Image feature determination unit 2 is connected to image acquisition unit 1. Image feature determination unit 2 is used to perform target detection on the image information, determine lesion candidate boxes in the image information, classify each lesion candidate box, and determine image features. The image features include the probability that each lesion candidate box belongs to each type of skin disease.

[0041] Furthermore, the image feature determination unit 2 includes: a target detection module and an image classification module.

[0042] The target detection module is connected to the image acquisition unit 1. The target detection module performs target detection on the image information to determine candidate lesion boxes in the image information. Specifically, target detection is performed on the image information based on a lesion detection model. The lesion detection model is obtained by training CenterNet using a third training sample set. The third training sample set includes multiple second sample images and candidate lesion boxes in each second sample image.

[0043] The image classification module is connected to the target detection module. The image classification module determines image features based on the image information and lesion candidate boxes within the image information, using a dual-channel classification model. The dual-channel classification model is obtained by training a convolutional neural network using a second training sample set. The second training sample set includes multiple first sample images, lesion candidate boxes in each first sample image, and the skin disease category of each lesion candidate box.

[0044] like Figure 2 As shown, the dual-channel classification model includes a first convolutional neural network layer, a second convolutional neural network layer, and a linear network layer. The lesion candidate box is input into the first convolutional neural network layer, and the image information is input into the second convolutional neural network layer, resulting in two vectors. The two vectors output by the convolutional neural network layer are multiplied by a dot, and then passed through the linear network layer to output the probability of the skin disease.

[0045] Regarding image information, this invention, building upon previous image-based diagnostic models, incorporates not only local lesion information but also global background information. Lesion information consists of candidate lesion boxes detected and filtered by the lesion detection model. It's important to note that, for accuracy, while the algorithm proposes candidate lesion boxes, the correct boxes are ultimately selected by the doctor or patient. Background information includes the original image uploaded by the patient, covering areas outside the lesion. This background information allows the dual-channel classification model to understand the location and illumination of the lesion. To balance lesion location and local texture information, this invention simultaneously inputs both the candidate lesion boxes and the original image into the dual-channel classification model. "Dual-channel" means that the inputs of both are connected in parallel at a lower-level network level. The dual-channel classification model can be trained using collected case records and images, employing a gradient descent method with backward differentiation. By simultaneously incorporating both local and global lesion information, the dual-channel classification model achieves a significant performance improvement compared to the single-channel classification model.

[0046] The text feature determination unit 3 is connected to the image and text acquisition unit 1. The text feature determination unit 3 is used to perform noun entity recognition and encoding on the medical text information and determine multiple text features.

[0047] Furthermore, the text feature determination unit 3 includes: an entity recognition module, a mapping module, and an encoding module.

[0048] The entity recognition module is connected to the image and text acquisition unit 1. The entity recognition module uses the FLAT (Flat-Lattice Transformer) model to perform noun entity recognition on the medical text information, obtaining multiple noun entities. This invention extracts effective noun entities from retrospective clinical texts using name entity recognition technology. The FLAT model is a transformer-based model with added structural design for Chinese word segmentation.

[0049] The mapping module is connected to the entity recognition module and is used to map each noun entity to a medical entity. The medical entity is an entity with medical significance. Since the extracted noun entities are relatively scattered, the mapping module uses a mapping table to map the organized noun entities to entities with medical significance.

[0050] Specifically, the mapping module includes a vector embedding submodule, a distance calculation submodule, and an entity determination submodule.

[0051] The vector embedding submodule is connected to the entity recognition module. This submodule is used to perform sentence vector embedding on each noun entity, transforming each noun entity into a corresponding vector to obtain the sentence vector for each noun entity. In this embodiment, SentenceBERT is used to perform sentence vector embedding on each noun entity.

[0052] The distance calculation submodule is connected to the vector embedding submodule. The distance calculation submodule is used to calculate the cosine distance between any given sentence vector and each medical entity vector in the medical entity database. The medical entity database includes multiple medical entities and their corresponding medical entity vectors.

[0053] The entity determination submodule is connected to the distance calculation submodule. The entity determination submodule is used to identify the medical entity corresponding to the medical entity vector with the smallest cosine distance to the sentence vector as the medical entity of the sentence vector. Alternatively, if the cosine distance between the sentence vector and a medical entity vector in the medical entity database is less than a set threshold, then the medical entity corresponding to this medical entity vector is identified as the medical entity of the sentence vector.

[0054] The encoding module is connected to the mapping module. The encoding module is used to perform one-hot encoding on each medical entity to obtain multiple corresponding text features.

[0055] For example, after processing with the FLAT model, if the patient's description contains the word "raised," a vector representation of "raised" is obtained. Then, the word "papule" is found in the medical entity database that is semantically closest to "raised." If the cosine distance between the two vector representations is less than a set threshold, it indicates that "raised" and "papule" are semantically very similar and can be classified as the same word. Through this method, approximately five hundred entities with medical significance in dermatology can be identified. Afterward, this invention encodes the text name entities to obtain text features presented in one-hot vector form.

[0056] Classification unit 4 is connected to both image feature determination unit 2 and text feature determination unit 3. Classification unit 4 is used to determine the skin disease type of the target skin based on the image features and text features, using an image-text classification model. The image-text classification model is obtained by training a support vector machine using a first training sample set. The first training sample set includes multiple sample features and the skin disease type corresponding to each sample feature. The overall process of skin disease classification in this invention is as follows: Figure 3 As shown. Skin conditions include blackheads, whiteheads, pimples, papules, nodules, pustules, acne scars, cysts, acne vulgaris, etc.

[0057] Furthermore, classification unit 4 includes a merging module and a type determination module. The merging module is connected to both the image feature determination unit 2 and the text feature determination unit 3, and is used to merge the image features and the text features into a single feature vector. The type determination module is connected to the merging module, and is used to determine the skin disease type of the target skin based on the feature vector and an image-text classification model. The sample features in the first training sample set are vectors.

[0058] As a single-layer classification model, Support Vector Machines (SVMs) can output numerical values ​​of the importance of each feature while classifying the vectors. This solves the problem of poor interpretability in current classification models. Ultimately, it can simultaneously obtain the skin disease classification results of cases, the probability distribution values ​​of cases for each skin disease, and the proportion of each image text feature used as the basis for classification.

[0059] Skin disease diagnosis requires not only combining skin lesion images but also crucial consultation information. This invention extracts consultation information from the medical narrative and combines it with skin lesion images. By incorporating textual information, the graphic-text model achieves significant performance improvements compared to pure image or pure text models. Compared to most current skin disease diagnosis models that rely solely on image data, this invention utilizes both medical text and image information, thus achieving better model performance and better aligning with doctors' diagnostic logic.

[0060] In clinical trials, using pathological results as the gold standard, this invention significantly outperforms the performance of junior dermatologists, with overall performance falling between the average performance of mid-level and senior dermatologists. Furthermore, compared to models using purely neural networks, using a single-layer support vector machine also achieves better interpretability. In medical practice, interpretability helps doctors better understand the model's decision outcomes.

[0061] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the system and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A skin disease classification system combining textual and graphic information, characterized in that, The skin disease classification system that combines text and image information includes: The image and text acquisition unit is used to acquire image information and medical text information of the target skin; the medical text information includes disease description information and past medical history. An image feature determination unit, connected to the image and text acquisition unit, is used to perform target detection on the image information, determine lesion candidate boxes in the image information, classify each lesion candidate box, and determine image features; the image features include the probability that each lesion candidate box belongs to each type of skin disease; The image feature determination unit includes: The target detection module, connected to the image acquisition unit, is used to perform target detection on the image information based on the lesion detection model, and determine the lesion candidate boxes in the image information; the lesion detection model is obtained by training CenterNet in advance using a third training sample set; the third training sample set includes multiple second sample images and lesion candidate boxes in each second sample image; An image classification module, connected to the target detection module, is used to determine image features based on the image information and lesion candidate boxes in the image information, using a dual-channel classification model. The dual-channel classification model is obtained by training a convolutional neural network using a second training sample set. The second training sample set includes multiple first sample images, lesion candidate boxes in each first sample image, and the skin disease category of each lesion candidate box. The dual-channel classification model includes a first convolutional neural network layer, a second convolutional neural network layer, and a linear network layer. The lesion candidate boxes are input into the first convolutional neural network layer, and the image information is input into the second convolutional neural network layer to obtain two vectors. The two vectors output by the convolutional neural network layer are multiplied by a dot and then passed through the linear network layer to output the probability of the skin disease. A text feature determination unit, connected to the image and text acquisition unit, is used to perform noun entity recognition and encoding on the medical text information to determine multiple text features; The classification unit is connected to the image feature determination unit and the text feature determination unit respectively, and is used to determine the skin disease type of the target skin based on the image features and the text features and the image-text classification model; the image-text classification model is obtained by training a support vector machine in advance using a first training sample set; the first training sample set includes multiple sample features and the skin disease type corresponding to each sample feature.

2. The skin disease classification system combining textual and graphic information according to claim 1, characterized in that, The text feature determination unit includes: An entity recognition module, connected to the image and text acquisition unit, is used to perform noun entity recognition on the medical text information using the FLAT model to obtain multiple noun entities; A mapping module, connected to the entity recognition module, is used to map each noun entity to a medical entity; the medical entity is an entity with medical significance. The encoding module, connected to the mapping module, is used to perform one-hot encoding on each medical entity to obtain multiple corresponding text features.

3. The skin disease classification system combining textual and graphic information according to claim 2, characterized in that, The mapping module includes: The vector embedding submodule, connected to the entity recognition module, is used to perform sentence vector embedding on each noun entity to transform each noun entity into a corresponding vector, thereby obtaining the sentence vector corresponding to each noun entity. The distance calculation submodule, connected to the vector embedding submodule, is used to calculate the cosine distance between any sentence vector and each medical entity vector in the medical entity database; the medical entity database includes multiple medical entities and the corresponding medical entity vectors of each medical entity. The entity determination submodule, connected to the distance calculation submodule, is used to identify the medical entity corresponding to the medical entity vector with the smallest cosine distance to the sentence vector as the medical entity of the sentence vector.

4. The skin disease classification system combining textual and graphic information according to claim 3, characterized in that, SentenceBERT is used to embed sentence vectors for each noun entity.

5. The skin disease classification system combining textual and graphic information according to claim 1, characterized in that, The classification unit includes: The merging module is connected to the image feature determination unit and the text feature determination unit respectively, and is used to merge the image features and text features into a single feature vector; The type determination module, connected to the merging module, is used to determine the skin disease type of the target skin based on the feature vector and an image-text classification model; the sample features in the first training sample set are vectors.

Images