A skin multimodal data processing method and system

Abstract

The present invention relates to the technical field of skin information processing, and discloses a skin multimodal data processing method and system, wherein the method firstly supplements and amplifies the missing attribute values ​​of medical history data; performs clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein the clinical image amplification method includes skin clinical image amplification based on disease consistency and skin clinical image amplification based on skin common composition transformation. In this way, by supplementing and amplifying medical history data and amplifying skin clinical images, the problems of incompleteness of multimodal data and small amount of data can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of skin information processing, and in particular to a skin multimodal data processing method and system. Background Art

[0002] Using deep learning technology to build a skin lesion type recognition model has become a current research hotspot. However, existing methods mainly focus on single image modality data, while ignoring the important link of comprehensive analysis of medical history information and image data by skin clinicians in the actual diagnosis process. Therefore, building a skin lesion type recognition model based on single image data does not fully meet the actual clinical use. Using multimodal data, that is, "image + medical history" to build a skin lesion type recognition model can fully explore and utilize the complementarity between different types of data. At present, the main difficulty in building a skin lesion type recognition model lies in the integrity of the data. The multimodal data of skin diseases mainly include medical history data in text form, such as various types of information such as blood pressure, weight, age, and skin clinical image data taken by conventional mobile phones or cameras. However, due to various factors, data of some modalities may be missing, or some attribute values ​​in the medical history modality may be missing, resulting in relatively few sets of paired multimodal data. This situation seriously affects the application of multimodal data and the performance of multimodal diagnostic models.

[0003] It can be seen that the skin lesion type recognition model constructed based on single image data in the existing technology cannot fully meet the actual clinical usage, and the multimodal data in the existing technology has the problems of incompleteness and small data volume, which makes it difficult to effectively support the construction of a high-performance skin lesion type recognition model. Summary of the invention

[0004] The present invention provides a skin multimodal data processing method and system to solve the problems of incompleteness and small data volume of multimodal data in the prior art, and the problem that it is difficult to effectively support the construction of a high-performance skin lesion type recognition model.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0006] In a first aspect, the present invention provides a method for processing skin multimodal data, comprising:

[0007] S1: Collect the medical history data of the patient to be analyzed and the corresponding skin clinical images;

[0008] S2: supplementing and amplifying missing attribute values ​​of the medical history data based on the medical history data;

[0009] S3: Perform clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein the method of performing clinical image amplification includes skin clinical image amplification based on disease consistency and skin clinical image amplification based on transformation of common skin components. In a second aspect, the present application provides a skin multimodal data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0010] Beneficial effects:

[0011] The skin multimodal data processing method provided by the present invention first supplements and amplifies the missing attribute values ​​of the medical history data; performs clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein the clinical image amplification method includes skin clinical image amplification based on disease consistency and skin clinical image amplification based on skin common composition transformation. In this way, by supplementing and amplifying the medical history data and amplifying the skin clinical images, the problems of incompleteness and small amount of multimodal data can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a skin multimodal data processing method according to a preferred embodiment of the present invention;

[0013] Figure 2 A schematic diagram of attribute value weight calculation in a preferred embodiment of the present invention;

[0014] Figure 3 A schematic diagram of a method for calculating the distance between medical history data in a preferred embodiment of the present invention;

[0015] Figure 4 A schematic diagram of supplementing and amplifying missing values ​​of medical history data in a preferred embodiment of the present invention;

[0016] Figure 5 This is a schematic diagram of skin clinical image amplification based on disease consistency according to a preferred embodiment of the present invention, wherein: Figure 5 (a) is the clinical image C of patient A with disease C n , Figure 5 (b) is the clinical image C of patient B with disease C m ;

[0017] Figure 6 This is a schematic diagram of skin clinical image amplification based on common skin composition transformation according to a preferred embodiment of the present invention, wherein: Figure 6 (a) is the original clinical image. Figure 6(b) is a schematic diagram of common normal components on the skin surface. Figure 6 (c)- Figure 6 (f) are all enlarged images. Figure 6 (g) is a schematic diagram of the color bar of normal skin color. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0020] See also Figure 1 , the present application provides a skin multimodal data processing method, comprising:

[0021] S1: Collect the medical history data of the patient to be analyzed and the corresponding skin clinical images;

[0022] S2: Supplement and expand the missing attribute values ​​of medical history data based on medical history data;

[0023] S3: Perform clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein, the clinical image amplification methods include skin clinical image amplification based on disease consistency and skin clinical image amplification based on common skin composition transformation.

[0024] In this embodiment, the attribute values ​​of the medical history data refer to blood pressure, weight, medication intake, and the like.

[0025] It is worth noting that the attribute values ​​of the medical history data are missing. In this application, the missing values ​​are first supplemented. When supplementing, multiple appropriate supplementary values ​​can be generated, so that amplification is achieved simultaneously. Paired matching means that after completing the supplementation and amplification of the medical history data, one medical history data is paired with one clinical image.

[0026] The above-mentioned skin multimodal data processing method supplements and amplifies the missing attribute values ​​of the medical history data; performs clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein the clinical image amplification method includes skin clinical image amplification based on disease consistency and skin clinical image amplification based on skin common composition transformation. In this way, by supplementing and amplifying the medical history data and amplifying the skin clinical images, the problems of incompleteness and small amount of multimodal data can be solved.

[0027] The steps of the above method are described in detail below with a complete embodiment.

[0028] 1. Supplement and expand missing attribute values ​​of medical history data.

[0029] 1.1 Building a medical history database in Represents a piece of medical history data. Represents the attribute value of the medical history data; Indicates the missing value of the attribute. If Represents attribute value Missing, if Represents attribute value exist.

[0030] 1.2. Construct a random forest model based on the medical history data x n The importance of the attribute value is given a weight, and the information gain of each attribute value when dividing the nodes in all decision trees is calculated, and then these values ​​are averaged across all decision trees to obtain the weight of each attribute. like Figure 2 shown.

[0031] 1.3. Combined attribute value weight w n , calculate medical history data The similar distance between other medical history data in database D is used to obtain the distance set The calculation method is as follows: Figure 3 As shown; from the distance set Select the first k nearest neighbor samples from the nearest neighbor distance to form a nearest neighbor distance set

[0032]

[0033] In the formula, w n represents the weight, s i and n Represents the medical history data x i and x n The missing value of the attribute in d n Represents medical history data x i and x n ∞ represents infinity, N represents the total number of medical history data, and n represents the nth piece of data.

[0034] 1.4. In some feasible implementations, the medical history data x n The attribute values ​​mainly include two types: discrete values ​​and continuous values. Discrete values ​​include gender, and continuous values ​​include age.

[0035] For discrete attribute values Assume that it has B possible values, recorded as Follows multinomial distribution in Represents attribute value The probability of occurrence is based on k nearest neighbor sample data and attribute values. Calculate the probability value based on the frequency of occurrence The multinomial distribution satisfies the following relationship:

[0036]

[0037] Among them, among them, Indicates the attribute value. Represents a calculated property value exist Number of occurrences.

[0038] For continuous attribute values Assume that it follows a Gaussian distribution where μ d and σ d The distribution represents the mean and variance, and μ is calculated based on k nearest neighbor sample data d and σ d The value of the score satisfies the following relationship:

[0039]

[0040]

[0041] 1.5. Obtain medical history data After the attribute value statistical distribution characteristics are obtained, the missing attribute values ​​are calculated by randomly sampling m times from the statistical distribution. The specific method is as follows: for discrete attribute values, the mode of the m sampled values ​​is used as the missing attribute value; for continuous attribute values, the average of the m sampled values ​​is used as the missing attribute value, and the medical history data after the missing values ​​are supplemented is obtained.

[0042] 1.6. Repeat step 1.5 to obtain multiple missing attribute values ​​through random sampling Constitute T pieces of medical history data x n Complete amplification data x n2 ,x n2 ,...,x nT ,like Figure 4 shown.

[0043] 2. Amplification of skin clinical images.

[0044] After completing the supplementation and amplification of missing values ​​in the medical history data, the skin clinical images corresponding to the medical history data need to be amplified to form paired multimodal data. The amplification of skin clinical images in this application includes: skin clinical images based on disease consistency and skin clinical images based on common skin composition transformation.

[0045] 2.1. Skin clinical image augmentation based on disease consistency.

[0046] A skin clinical image mainly contains three tissue types, which are normal skin tissue, perilesional tissue and lesional area tissue. Perilesional and lesional areas of the same disease in different patients are highly similar; normal skin of all patients is highly similar. When exchanging the same tissue area in clinical images of the same disease in different patients, the disease type of the exchanged clinical images remains unchanged.

[0047] Clinical image of disease C in patient A n A certain area of ​​any irregular shape is randomly cut out from the skin lesion area of ​​patient B and compared with the clinical image C of patient B. m The same-shaped regions of equal area in the skin lesion area are exchanged, and during the exchange process, the randomly cropped regions are randomly amplified, including sharpening, smoothing, adjusting contrast and brightness to enhance the texture and color characteristics of the skin; exchanging the clinical image C n and C m The same area and shape of the skin lesions, which have been randomly amplified; exchange clinical image C n and C m The same area and shape of the normal skin tissue area, which has been randomly image-amplified. n and C mThe equal-area regions of the same tissue type in the image are respectively obtained to obtain the amplified images C' with the disease category unchanged. n and C' m .like Figure 5 As shown, Figure 5 (a) is the clinical image C of patient A with disease C n , Figure 5 (b) is the clinical image C of patient B with disease C m .

[0048] 2.2. Skin clinical image augmentation based on common skin component transformations.

[0049] Common normal components of the skin surface include hair, wrinkles, and pigmentation. Different patients with the same disease have different skin colors and different distributions of hair, wrinkles, and pigmentation. Changes in skin color and common normal tissue components of the skin surface such as hair, wrinkles, and pigmentation do not change the type of disease in the clinical picture.

[0050] 2.2.1. According to the distribution relationship between age, gender and skin color, normal skin color strips for different age segments of different genders are constructed; at the same time, the shape elements of common normal components of the skin surface, such as hair, wrinkles and pigmentation, are collected;

[0051] 2.2.2. Based on the original clinical image C n The age information in the corresponding medical history data is adjusted within the corresponding age range by referring to the normal skin color bars of different genders. n The normal skin color in the original clinical image C n Randomly add the shape elements of the common normal components of the skin surface to obtain an augmented image with the disease type unchanged. Figure 6 As shown, Figure 6 (a) is the original clinical image. Figure 6 (b) is a schematic diagram of common normal components on the skin surface. Figure 6 (c)- Figure 6 (f) are all enlarged images. Figure 6 (g) is a schematic diagram of the color bar of normal skin color.

[0052] It should be understood that the above-mentioned skin multimodal data processing method can be applied to the scenario of skin lesion type identification, which is only used as an example and not as a limitation.

[0053] The embodiment of the present application also provides a skin multimodal data processing system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program. The skin multimodal data processing system can implement various embodiments of the above method and achieve the same beneficial effects, which will not be described in detail here.

[0054] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for processing skin multimodal data, characterized in that: include: S1: Collect the medical history data of the patient to be analyzed and the corresponding skin clinical images; S2: supplementing and amplifying missing attribute values ​​of the medical history data based on the medical history data; S3: Performing clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein the clinical image amplification method includes skin clinical image amplification based on disease consistency and skin clinical image amplification based on transformation of common skin components; the common skin components include hair, wrinkles and pigmentation; The S2 includes: S21: constructing a medical history database according to the medical history data, wherein each medical history data in the medical history database includes a plurality of attribute values, and the medical history data includes blood pressure, weight, and medication status; S22: Build a random forest model and calculate the weight of each attribute value in the medical history data; S23: According to the weight of the attribute value, the similarity distance between each medical history data and other medical history data in the database is calculated to obtain a distance set, and the first K nearest neighbor samples with the closest distance are selected from the distance set to form a nearest neighbor sample set; S24: determining statistical distribution characteristics of different types of attribute values ​​of the medical history data according to the medical history data of K nearest neighbor samples in the nearest neighbor sample set; S25: randomly sampling m times from the statistical distribution according to the statistical distribution characteristics, and calculating the missing attribute values ​​according to the m sampling results; repeating the random sampling operation T times to obtain multiple missing attribute values, and supplementing the missing attribute values ​​to obtain T pieces of augmented medical history data; The skin clinical image expansion based on disease consistency includes: The skin clinical image is divided into normal skin tissue area, adjacent skin lesion tissue area and skin lesion area; Clinical image of disease C in patient A n A certain area of ​​any irregular shape is randomly cut out from the skin lesion area of ​​patient B and compared with the clinical image C of patient B. m The same-shaped regions of equal area in the lesion area are exchanged, and during the exchange process, the randomly cropped regions are randomly amplified, including sharpening, smoothing, adjusting contrast and brightness of the image to enhance the texture and color characteristics of the skin; exchanging the clinical image C n and C m The same area and shape of the skin lesions, which have been randomly amplified; exchange clinical image C n and C m The normal skin tissue area with equal area and shape and random image amplification is randomly exchanged by C n and C m The equal-area regions of the same tissue type in the image are respectively obtained to obtain the amplified images C' with the disease category unchanged. n and C' m ; The skin clinical image augmentation based on the transformation of common skin components includes: According to the distribution relationship between age, gender and skin color, normal skin color strips for different age segments of different genders are constructed; and the shape elements of common normal components on the skin surface are collected; According to the original clinical image C n The gender and age information in the corresponding medical history data is used to refer to the normal skin color bars of different genders, and the original clinical image C is adjusted within the corresponding age group. n The normal skin color in the original clinical image C n Shape elements of common normal components of the skin surface are randomly added to obtain an augmented image in which the disease type remains unchanged.

2. The skin multimodal data processing method according to claim 1, characterized in that: The S22 includes: Build a random forest model, calculate the information gain of each attribute value when dividing nodes in all decision trees, average these values ​​​​in all decision trees, and get the weight of each attribute value.

3. The skin multimodal data processing method according to claim 1, characterized in that: The distance set in S23 satisfies the following relationship: in, In the formula, w n represents the weight, s i and n Represents the medical history data x i and x n The missing value of the attribute in d n Represents medical history data x i and x n ∞ represents infinity, N represents the total number of medical history data, and n represents the nth piece of data.

4. The skin multimodal data processing method according to claim 1, characterized in that: The types of attribute values ​​of the medical history data include discrete type and continuous type, and S24 includes: For discrete attribute values Assume that it has B values, recorded as Follows multinomial distribution in Represents attribute value The probability of occurrence is based on K nearest neighbor sample data and attribute values. Calculate the probability value based on the frequency of occurrence The multinomial distribution satisfies the following relationship: in, Indicates the attribute value. Represents a calculated property value exist Number of occurrences; For continuous attribute values Assume that it follows a Gaussian distribution where μ d and σ d Represents the mean and variance respectively, and μ is calculated based on K nearest neighbor sample data. d and σ d The value of the score satisfies the following relationship:

5. The skin multimodal data processing method according to claim 4, characterized in that: When calculating the missing attribute value based on the m sampling results, for discrete attribute values, the mode of the m sampling values ​​is used as the missing attribute value; for continuous attribute values, the average of the m sampling values ​​is used as the missing attribute value.

6. A skin multimodal data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 5 are implemented.

Images