Skin multi-modal data processing method and system

Abstract

The present invention relates to the technical field of skin information processing. Disclosed are a skin multi-modal data processing method and system. The method comprises: firstly, supplementing and augmenting missing attribute values of medical history data; and performing clinical image augmentation on a skin clinical image corresponding to the supplemented and augmented medical history data, so as to form paired and matched multi-modal data, wherein the manner of performing clinical image augmentation includes skin clinical image augmentation based on disease consistency and skin clinical image augmentation based on common skin composition transformation. In this way, by supplementing and augmenting the medical history data and augmenting the skin clinical image, the problems of incompleteness and less data volume of the multi-modal data can be solved.

Description

A skin multimodal data processing method and system Technical Field

[0001] The present invention relates to the technical field of skin information processing, and in particular to a skin multimodal data processing method and system. Background Art

[0002] Using deep learning techniques to construct skin lesion type recognition models has become a current research hotspot. However, existing methods primarily focus on single-modality image data, overlooking the crucial step of comprehensive analysis, combining medical history information with image data, during the actual diagnosis process by dermatologists. Therefore, skin lesion type recognition models based on single image data are not fully adapted to actual clinical scenarios. Leveraging multimodal data—that is, combining "images" with medical history—to construct skin lesion type recognition models can fully exploit the complementary nature of different data types. Currently, the primary challenge in constructing skin lesion type recognition models lies in data integrity. Multimodal data for skin diseases primarily consists of text-based medical history data, such as blood pressure, weight, and age, as well as clinical skin images captured using conventional mobile phones or cameras. However, due to various factors, data from certain modalities may be missing, or some attribute values ​​within the medical history modality may be missing, resulting in a relatively limited supply of complete sets of paired multimodal data. This severely impacts the application of multimodal data and the performance of multimodal diagnostic models.

[0003] It can be seen that the skin lesion type recognition model constructed based on single image data in the existing technology cannot fully meet the actual clinical usage, and the multimodal data in the existing technology has problems of incompleteness and small data volume, which makes it difficult to effectively support the construction of a high-performance skin lesion type recognition model. Summary of the Invention

[0004] The present invention provides a skin multimodal data processing method and system to solve the problems of incompleteness and small data volume of multimodal data in the prior art, which makes it difficult to effectively support the construction of a high-performance skin lesion type recognition model.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for processing skin multimodal data, comprising:

[0007] S1: Collect the medical history data and corresponding skin clinical images of the patient to be analyzed;

[0008] S2: supplementing and amplifying missing attribute values ​​of the medical history data based on the medical history data;

[0009] S3: Perform clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein the clinical image amplification method includes skin clinical image amplification based on disease consistency and skin clinical image amplification based on transformation of common skin components. In a second aspect, the present application provides a skin multimodal data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above. Beneficial effects:

[0010] The skin multimodal data processing method provided by this invention first supplements and amplifies missing attribute values ​​in medical history data. Then, clinical image amplification is performed on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data. Clinical image amplification methods include disease consistency-based skin clinical image amplification and skin clinical image amplification based on transformations of common skin components. This method, by supplementing and amplifying medical history data and amplifying skin clinical images, can address the issues of incompleteness and limited data volume in multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG1 is a flow chart of a method for processing skin multimodal data according to a preferred embodiment of the present invention;

[0012] FIG2 is a schematic diagram of attribute value weight calculation according to a preferred embodiment of the present invention;

[0013] FIG3 is a schematic diagram of a method for calculating the distance between medical history data according to a preferred embodiment of the present invention;

[0014] FIG4 is a schematic diagram of supplementing and amplifying missing values ​​in medical history data according to a preferred embodiment of the present invention;

[0015] FIG5 is a schematic diagram of skin clinical image amplification based on disease consistency according to a preferred embodiment of the present invention, wherein FIG5(a) is a clinical image C of disease C of patient A. n , Figure 5(b) shows the clinical image C of patient B with disease C m ;

[0016] Figure 6 is a schematic diagram of skin clinical image amplification based on transformation of common skin components according to a preferred embodiment of the present invention, wherein Figure 6(a) is the original clinical image, Figure 6(b) is a schematic diagram of common normal components on the skin surface, Figures 6(c) to 6(f) are all amplified images, and Figure 6(g) is a schematic diagram of normal skin color strips. DETAILED DESCRIPTION

[0017] The following is a clear and complete description of the technical solutions of the present invention. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0018] Unless otherwise defined, the technical or scientific terms used in the present invention shall have the usual meanings understood by persons of ordinary skill in the field to which the present invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "a" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0019] Referring to FIG1 , the present application provides a method for processing multimodal skin data, including:

[0020] S1: Collect the medical history data and corresponding skin clinical images of the patient to be analyzed;

[0021] S2: Supplement and expand missing attribute values ​​of medical history data based on medical history data;

[0022] S3: Perform clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein, the methods of performing clinical image amplification include skin clinical image amplification based on disease consistency and skin clinical image amplification based on common skin composition transformation.

[0023] In this embodiment, the attribute values ​​of the medical history data refer to blood pressure, weight, medication intake, and the like.

[0024] It's worth noting that the attribute values ​​of medical history data are inherently missing. This application first supplements the missing values. During this supplementation process, multiple appropriate supplementary values ​​can be generated, thereby achieving simultaneous amplification. Paired matching means that after completing this supplementation and amplification of medical history data, one piece of medical history data is paired with one clinical image.

[0025] The aforementioned skin multimodal data processing method supplements and amplifies missing attribute values ​​in medical history data. Clinical image amplification is then performed on the skin clinical images corresponding to the supplemented and amplified medical history data, forming paired multimodal data. Clinical image amplification methods include disease consistency-based skin clinical image amplification and skin clinical image amplification based on transformations of common skin components. This approach, by supplementing and amplifying medical history data and amplifying skin clinical images, addresses the issues of incompleteness and limited data volume in multimodal data.

[0026] The steps of the above method are described in detail below with a complete embodiment.

[0027] 1. Supplement and expand missing attribute values ​​of medical history data.

[0028] 1.1 Building a medical history database in Represents a piece of medical history data. Indicates the attribute value of the medical history data; Indicates the missing value of the attribute. If Represents attribute value Missing, if Represents attribute value exist.

[0029] 1.2. Construct a random forest model based on the medical history data x n The importance of the attribute value is given a weight, and the information gain of each attribute value when dividing the nodes in all decision trees is calculated, and then these values ​​are averaged across all decision trees to obtain the weight of each attribute. As shown in Figure 2.

[0030] 1.3. Combined attribute value weight w n , calculate medical history data The similar distance between the medical history data and other medical history data in database D is used to obtain the distance set The calculation method is shown in the following formula and Figure 3; from the distance set Select the first k nearest neighbor samples from the nearest neighbor distance to form a nearest neighbor distance set

[0031] Where w n represents the weight, s i and s n Represents medical history data x i and x n The missing value of the attribute in d n Represents medical history data x i and x nThe distance between them, ∞ represents infinity, N represents the total number of medical history data, and n represents the nth data.

[0032] 1.4. In some feasible implementations, the medical history data x n The attribute values ​​mainly include two types: discrete values ​​and continuous values. Discrete values ​​include gender, and continuous values ​​include age.

[0033] For discrete attribute values Assume that it has B possible values, recorded as Follows multinomial distribution in Represents attribute value The probability of occurrence is based on k nearest neighbor sample data and attribute values. Calculate the probability value based on the frequency of occurrence The multinomial distribution satisfies the following relationship:

[0034] Among them, among them, Represents the attribute value, Represents a calculated property value exist Number of occurrences.

[0035] For continuous attribute values Assume it follows a Gaussian distribution where μ d and σ d The distribution represents the mean and variance, and μ is calculated based on k nearest neighbor sample data. d and σ d The value of the score satisfies the following relationship:

[0036] 1.5. Obtaining medical history data After the attribute value statistical distribution characteristics are obtained, the missing attribute value is calculated by randomly sampling m times from the statistical distribution. The specific method is: for discrete attribute values, the mode of the m sampled values ​​is used as the missing attribute value; for continuous attribute values, the average of the m sampled values ​​is used as the missing attribute value, and the medical history data after the missing values ​​are supplemented is obtained.

[0037] 1.6. Repeat step 1.5 to obtain multiple missing attribute values ​​through random sampling Constitute T pieces of medical history data x n Complete amplification data x n2 ,x n2 ,...,x nT , as shown in Figure 4.

[0038] 2. Amplification of skin clinical images.

[0039] After completing the supplementation and amplification of missing values ​​in the medical history data, the skin clinical images corresponding to the medical history data need to be amplified to form paired multimodal data. The amplification of skin clinical images in this application includes: skin clinical images based on disease consistency and skin clinical images based on common skin composition transformation.

[0040] 2.1. Skin clinical image augmentation based on disease consistency.

[0041] A clinical skin image primarily contains three tissue types: normal skin, perilesional tissue, and lesional tissue. The perilesional and lesional tissues of different patients with the same disease are highly similar; normal skin from all patients is also highly similar. Swapping the same tissue region between clinical images of the same disease from different patients maintains the same disease type in the resulting images.

[0042] Clinical image of disease C in patient A n Randomly cut out an irregular shape area of ​​a certain size from the skin lesion area and compare it with the clinical image C of patient B. m The same-shaped areas of equal area in the lesion area are exchanged, and random image amplification is performed on the randomly cropped areas during the exchange process, including sharpening, smoothing, adjusting contrast and brightness to enhance the texture and color characteristics of the skin; exchanging clinical image C n and C m The same area and shape of the skin lesion, which has been randomly amplified; exchange clinical image C n and C m The normal skin tissue area has the same area and shape, and has been randomly image-amplified. n and C m The equal area areas of the same tissue type in the image are respectively obtained to obtain the amplified images C' with the disease category unchanged. n and C' m As shown in Figure 5, Figure 5(a) is the clinical image C of patient A with disease C. n , Figure 5(b) shows the clinical image C of patient B with disease C m .

[0043] 2.2. Skin clinical image augmentation based on common skin component transformations.

[0044] Common, normal components of the skin surface include hair, wrinkles, and pigmentation. Different patients with the same disease may have varying skin color and distribution of hair, wrinkles, and pigmentation. Changes in skin color or normal tissue components such as hair, wrinkles, and pigmentation do not alter the clinical picture of the disease.

[0045] 2.2.1. Based on the relationship between age, gender, and skin color distribution, construct normal skin color strips for different age segments and genders. At the same time, collect the shape elements of common normal components of the skin surface, such as hair, wrinkles, and pigmentation.

[0046] 2.2.2. Based on the original clinical image C n The age information in the corresponding medical history data is used to refer to the normal skin color bars of different genders, and the original clinical image C is adjusted within the corresponding age group. n The skin color of normal skin in the original clinical image C n By randomly adding shape elements of common normal components on the skin surface, an augmented image is obtained while maintaining the same disease type. This is shown in Figure 6, where Figure 6(a) is the original clinical image, Figure 6(b) is a schematic diagram of common normal components on the skin surface, Figures 6(c)-6(f) are augmented images, and Figure 6(g) is a schematic diagram of the color bar of normal skin.

[0047] It should be understood that the above-mentioned skin multimodal data processing method can be applied to the scenario of skin lesion type identification, which is only an example and not a limitation.

[0048] The present application also provides a multimodal skin data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned method are implemented. This multimodal skin data processing system can implement various embodiments of the aforementioned method and achieve the same beneficial effects, which are not described in detail here.

[0049] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for processing multimodal skin data, characterized in that: include: S1: Collect the medical history data and corresponding skin clinical images of the patient to be analyzed; S2: supplementing and amplifying missing attribute values ​​of the medical history data based on the medical history data; S3: performing clinical image amplification on the skin clinical images corresponding to the supplemented and amplified medical history data to form paired multimodal data; wherein the clinical image amplification includes skin clinical image amplification based on disease consistency and skin clinical image amplification based on transformation of common skin components; the common skin components include hair, wrinkles, and pigmentation; The S2 includes: S21: constructing a medical history database based on the medical history data, wherein each medical history data in the medical history database includes multiple attribute values, and the medical history data includes blood pressure, weight, and medication status; S22: Build a random forest model and calculate the weight of each attribute value in the medical history data; S23: Calculate the similarity distance between each medical history data and other medical history data in the database according to the weight of the attribute value to obtain a distance set, and select the top K nearest neighbor samples with the closest distance from the distance set to form a nearest neighbor sample set; S24: determining statistical distribution characteristics of different types of attribute values ​​of the medical history data based on the K nearest neighbor sample medical history data in the nearest neighbor sample set; S25: Randomly sample m times from the statistical distribution according to the statistical distribution characteristics, and calculate the missing attribute values ​​based on the m sampling results; repeat the random sampling operation T times to obtain multiple missing attribute values, and supplement the missing attribute values ​​to obtain T pieces of augmented medical history data; The skin clinical image expansion based on disease consistency includes: The skin clinical image is divided into normal skin tissue area, perilesional tissue area and lesional area; Clinical image of disease C in patient A n Randomly cut out an irregular shape area of ​​a certain size from the skin lesion area and compare it with the clinical image C of patient B. m The same-shaped areas of equal area in the lesion area are exchanged, and random image amplification is performed on the randomly cropped areas during the exchange process, including sharpening, smoothing, adjusting contrast and brightness to enhance the texture and color characteristics of the skin; exchanging clinical image C n and C m The same area and shape of the skin lesion, which has been randomly amplified; exchange clinical image C n and C m The normal skin tissue area with equal area and same shape and random image amplification is randomly exchanged by C n and C m The equal area areas of the same tissue type in the image are respectively obtained to obtain the amplified images C' with the disease category unchanged. n and C' m ; The skin clinical image augmentation based on the transformation of common skin components includes: Based on the distribution relationship between age, gender and skin color, normal skin color strips for different age segments of different genders are constructed; and the shape elements of common normal components on the skin surface are collected; According to the original clinical image C n The gender and age information in the corresponding medical history data is used to refer to the normal skin color bars of different genders, and the original clinical image C is adjusted within the corresponding age group. n The skin color of normal skin in the original clinical image C n Common normal components of the skin surface are randomly added to obtain an augmented image with the disease type remaining unchanged.

2. The skin multimodal data processing method according to claim 1, characterized in that: The S22 includes: Build a random forest model, calculate the information gain of each attribute value when dividing nodes in all decision trees, average these values ​​​​in all decision trees, and obtain the weight of each attribute value.

3. The skin multimodal data processing method according to claim 1, characterized in that: The distance set in S23 satisfies the following relationship: in, Where w n represents the weight, s i and s n Represents medical history data x i and x n The missing value of the attribute in d n Represents medical history data x i and x n The distance between them, ∞ represents infinity, N represents the total number of medical history data, and n represents the nth data.

4. The skin multimodal data processing method according to claim 1, characterized in that: The types of attribute values ​​of the medical history data include discrete and continuous types, and S24 includes: For discrete attribute values Assume that it has B values, recorded as Follows multinomial distribution in Represents attribute value The probability of occurrence is based on K nearest neighbor sample data and attribute values. Calculate the probability value based on the frequency of occurrence The multinomial distribution satisfies the following relationship: in, Represents the attribute value, Represents a calculated property value exist Number of occurrences; For continuous attribute values Assume it follows a Gaussian distribution where μ d and σ d Represents the mean and variance respectively, and μ is calculated based on K nearest neighbor sample data d and σ d The value of the score satisfies the following relationship:

5. The skin multimodal data processing method according to claim 4, characterized in that: When calculating the missing attribute value based on the m sampling results, for discrete attribute values, the mode of the m sampling values ​​is used as the missing attribute value; for continuous attribute values, the average of the m sampling values ​​is used as the missing attribute value.

6. A skin multimodal data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images