Skin state classification system based on multi-modal feature fusion

Abstract

The embodiment of the invention discloses a skin state classification system based on multi-modal feature fusion. A specific implementation mode of the system comprises a patient state text information input device, a skin image acquisition device, a main control chip and a classification result display, wherein the patient state text information input device inputs a patient state structured text information sequence and a patient state unstructured text information sequence; the skin image acquisition device acquires a skin lesion image of a patient; the main control chip is configured to obtain an initial first state text feature vector and an initial second state text feature vector sequence; performing image feature extraction processing on the skin lesion image of the patient to obtain an initial skin lesion region feature vector; inputting the initial first state text feature vector, the initial second state text feature vector sequence and the initial skin lesion region feature vector into a skin state classification model; and the classification result display displays the skin state classification result. According to the embodiment, the classification accuracy of the skin state can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a skin condition classification system based on multimodal feature fusion. Background Art

[0002] Since there are many types of skin diseases and their features are easily confused, doctors' visual examinations are inefficient and prone to misdiagnosis. Therefore, there is an urgent need for the assistance of a skin condition classification system based on AI technology to improve diagnostic efficiency and quality. At present, when identifying and classifying skin diseases, the usual method is: the skin condition classification system obtains the patient's skin lesion image and simple structured oral text information (such as age, skin type), and uses the skin lesion classification model to classify the patient's skin condition based on the fusion features of the skin lesion image and structured text.

[0003] However, when using the above method to identify and classify skin diseases, the following technical problems often occur:

[0004] Structured oral texts usually have fewer dimensions and limited features, making it difficult to characterize the characteristics of various skin diseases. This makes it difficult for the fusion features of skin lesion images and structured texts to accurately and comprehensively characterize the patient's skin lesion status, resulting in low classification accuracy of skin conditions.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a skin condition classification system based on multimodal feature fusion to solve one or more of the technical problems mentioned in the above background technology section.

[0008] Some embodiments of the present disclosure provide a skin state classification system based on multimodal feature fusion, and the skin state classification system based on multimodal feature fusion includes: a patient state text information input device, a skin image acquisition device, a main control chip and a classification result display, wherein the patient state text information input device is configured to input a patient state structured text information sequence and a patient state unstructured text information sequence, and send the patient state structured text information sequence and the patient state unstructured text information sequence to the main control chip; the skin image acquisition device is configured to acquire a patient skin lesion image, and send the patient skin lesion image to the main control chip; the main control chip is configured to receive the patient The state structured text information and the patient state unstructured text information sequence are subjected to text vectorization processing to obtain an initial first state text feature vector and an initial second state text feature vector sequence; the above-mentioned main control chip is configured to perform image feature extraction processing on the received patient skin lesion image to obtain an initial skin lesion area feature vector; the above-mentioned main control chip is configured to input the above-mentioned initial first state text feature vector, the above-mentioned initial second state text feature vector sequence and the above-mentioned initial skin lesion area feature vector into a pre-trained skin state classification model to obtain a skin state classification result, and send the above-mentioned skin state classification result to the above-mentioned classification result display; the above-mentioned classification result display is configured to display the received skin state classification result.

[0009] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the skin state classification system based on multimodal feature fusion of some embodiments of the present disclosure, the classification accuracy of skin state can be improved. Specifically, the reason for the low classification accuracy of skin state is that structured oral text usually has fewer dimensions and limited features, and it is difficult to characterize the characteristics of various skin diseases, which makes it difficult for the fusion features of skin lesion images and structured texts to accurately and comprehensively characterize the skin lesion status of patients, thereby resulting in low classification accuracy of skin state. Based on this, some embodiments of the present disclosure include a skin state classification system based on multimodal feature fusion, including: a patient state text information entry device, a skin image acquisition device, a main control chip and a classification result display, wherein the patient state text information entry device is configured to enter a patient state structured text information sequence and a patient state unstructured text information sequence, and send the patient state structured text information sequence and the patient state unstructured text information sequence to the main control chip; the skin image acquisition device is configured to acquire patient skin lesion images, and send the patient skin lesion images to the main control chip; the main control chip is configured to process the received patient state structured text information and The patient's unstructured text information sequence is subjected to text vectorization processing to obtain an initial first-state text feature vector and an initial second-state text feature vector sequence; the main control chip is configured to perform image feature extraction processing on the received patient's skin lesion image to obtain an initial skin lesion area feature vector; the main control chip is configured to input the initial first-state text feature vector, the initial second-state text feature vector sequence and the initial skin lesion area feature vector into a pre-trained skin state classification model to obtain a skin state classification result, and send the skin state classification result to the classification result display; the classification result display is configured to display the received skin state classification result. Therefore, in some embodiments of the present disclosure, the skin state classification system based on multimodal feature fusion can improve the feature dimension of the skin lesion feature by performing multimodal feature fusion on the patient's skin lesion image, the structured text corresponding to the skin state and the unstructured text through the main control chip, so as to more accurately and comprehensively characterize various skin diseases, thereby improving the classification accuracy of the skin state. Furthermore, the skin condition classification results can be displayed through the classification result display, thereby assisting doctors in diagnosing skin diseases and improving the efficiency of skin disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0011] Figure 1 is an exemplary structural diagram of some embodiments of a skin state classification system based on multimodal feature fusion according to the present disclosure;

[0012] Figure 2 is an exemplary structural diagram of some other embodiments of the skin state classification system based on multimodal feature fusion according to the present disclosure;

[0013] Figure 3 is a data flow diagram of a skin lesion image feature extraction step in some embodiments of the skin condition classification system based on multimodal feature fusion according to the present disclosure;

[0014] Figure 4 It is a schematic diagram of experimental data on skin lesion classification performance according to some embodiments of the skin condition classification system based on multimodal feature fusion of the present disclosure. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0017] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0018] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0019] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0020] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0021] First, see Figure 1 , Figure 1 The following is an exemplary structural diagram of some embodiments of the skin state classification system based on multimodal feature fusion disclosed in the present invention. The skin state classification system based on multimodal feature fusion includes: a patient state text information input device 101, a skin image acquisition device 102, a main control chip 103 and a classification result display 104.

[0022] In some embodiments, the patient status text information input device 101 may be configured to input a patient status structured text information sequence and a patient status unstructured text information sequence, and send the patient status structured text information sequence and the patient status unstructured text information sequence to the main control chip 103. Among them, the patient status text information input device may be a terminal with a touch screen for users to input data by typing and interactive selection. The patient status structured text information in the patient status structured text information sequence may correspond to the patient structured attributes one by one. The patient structured attributes may be attributes with a fixed encoding format for the corresponding attribute values ​​related to the patient. The patient status structured text information in the patient status structured text information sequence may be information of structured text describing the patient status. The structured text may be attribute value text corresponding to the patient structured attribute. Each patient status unstructured text information in the patient status unstructured text information sequence may be a natural language text describing the patient's condition, which is different from the structured text. The main control chip may be a chip for diagnosing and classifying the patient's skin lesions according to the text describing the patient's status and the skin image. The above main control chip can be deployed separately on a server terminal.

[0023] As an example, the above-mentioned patient structured attributes may be, but are not limited to, one of the following: age, gender, skin type, and whether or not a scarring physique. The patient's age is usually expressed as a numerical value. Gender is usually represented by a single character to indicate "male" or "female". The above-mentioned skin type may be a skin type. The skin type may be one of the following: neutral skin, dry skin, oily skin, combination skin, and sensitive skin. Whether or not a scarring physique is present may be represented by "yes" and "no" to indicate "scarring physique" and "not scarring physique" respectively. When the patient structured attribute corresponding to the patient status structured text information is "skin type", the patient status structured text information may be "sensitive skin". The patient status unstructured text information may be "more obvious brown patches on the left cheekbone".

[0024] In practice, the above-mentioned patient status text information entry device can display each patient structured attribute corresponding to the structured text and each corresponding attribute value option, a text input box for filling in the unstructured text describing the user's skin state, and a submit button in the touch display screen. First, according to the patient's response to each patient structured attribute corresponding to the structured text, the user can select an attribute value for each attribute on the touch display screen included in the patient status text information entry device through a finger or a stylus. Then, according to the description of the patient's main complaint, type the unstructured text in the displayed text input box through the stylus input or the soft keyboard. Finally, the user clicks the submit button to send the above-mentioned patient status structured text information sequence and the above-mentioned patient status unstructured text information sequence to the above-mentioned main control chip through a wired connection or a wireless connection. It should be pointed out that the above-mentioned wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0025] In some embodiments, the skin image acquisition device 102 may be configured to acquire skin lesion images of patients, and send the skin lesion images of patients to the main control chip 103. The skin image acquisition device may be a device equipped with a camera. The skin lesion images of patients may be skin images taken including the skin lesion site of the patient and the patient's body area around the skin lesion site. The skin image acquisition device may acquire skin lesion images of patients through a camera, and send the skin lesion images of patients to the main control chip through a wired connection or a wireless connection. It should be noted that the patient's body area around the skin lesion site may be a skin area not covered with clothes, or a skin area covered with clothes, which is not specifically limited here.

[0026] In some embodiments, the main control chip 103 can be configured to perform text vectorization processing on the received patient status structured text information sequence and the patient status unstructured text information sequence to obtain an initial first state text feature vector and an initial second state text feature vector sequence. Among them, the initial first state text feature vector can be an embedded representation of the patient status structured text information sequence. The initial second state text feature vector in the initial second state text feature vector sequence can correspond one-to-one to the patient status unstructured text information in the patient status unstructured text information sequence. The initial second state text feature vector in the initial second state text feature vector sequence can be an embedded representation of the corresponding patient status unstructured text information.

[0027] In some optional implementations of some embodiments, each patient status structured text information in the above patient status structured text information sequence may include a status attribute value. Among them, the above status attribute value may be an attribute value of a corresponding patient structured attribute. The above main control chip 103 may be further configured to perform the following steps to perform text vectorization processing on the received patient status structured text information and patient status unstructured text information sequence to obtain an initial first state text feature vector and an initial second state text feature vector sequence:

[0028] In the first step, for each patient status structured text information in the above patient status structured text information sequence, the following steps are performed:

[0029] In a first sub-step, in response to determining that the patient status structured text information satisfies a preset attribute condition, the status attribute value included in the patient status structured text information is normalized to obtain a numerical attribute characteristic value. The preset attribute condition may be that the attribute value of the patient structured attribute corresponding to the patient status structured text information is a numerical type. The numerical attribute characteristic value may represent the status attribute value of the numerical attribute. For example, the numerical attribute may be, but is not limited to, age.

[0030] As an example, the main control chip can normalize the state attribute values ​​included in the patient state structured text information by a preset numerical normalization method to obtain numerical attribute feature values. For example, the numerical normalization method can be a Min-Max normalization method.

[0031] In the second sub-step, in response to determining that the patient status structured text information does not meet the preset attribute condition, the status attribute value included in the patient status structured text information is subjected to word embedding processing to obtain a category attribute feature vector. The category attribute feature vector may represent the status attribute value of the category attribute. For example, the category attribute may be, but is not limited to, one of the following: skin type, gender, and whether the patient has a scar physique.

[0032] As an example, the main control chip can perform word embedding processing on the state attribute value included in the patient state structured text information by a preset first word embedding processing method to obtain a category attribute feature vector. For example, the first word embedding processing method can be a one-hot encoding method.

[0033] In the second step, each obtained category attribute feature vector and each numerical attribute feature value are concatenated to obtain an initial first state text feature vector.

[0034] As an example, the main control chip may perform horizontal splicing on the obtained category attribute feature vectors and numerical attribute feature values ​​according to the order of the patient status structured text information sequence to obtain an initial first status text feature vector.

[0035] The third step is to perform word embedding processing on each patient status unstructured text information in the patient status unstructured text information sequence to obtain an initial second state text feature vector.

[0036] As an example, the main control chip can perform word embedding processing on the patient's state unstructured text information through a preset second word embedding processing method to obtain an initial second state text feature vector. For example, the second word embedding processing method can be, but is not limited to, one of: a bidirectional encoder representation model, a general language model for conversational applications.

[0037] In some embodiments, the main control chip 103 may be configured to perform image feature extraction processing on the received patient skin lesion image to obtain an initial skin lesion region feature vector, wherein the initial skin lesion region feature vector may represent the features of the local image of the skin lesion region.

[0038] In some optional implementations of some embodiments, the main control chip 103 may be configured to perform the following steps to perform image feature extraction processing on the received patient skin lesion image to obtain an initial skin lesion area feature vector:

[0039] In the first step, target detection processing is performed on the received patient skin lesion image to obtain skin lesion area bounding box information. The skin lesion area bounding box information may include the coordinates of the upper left corner of the bounding box, the width of the bounding box, and the height of the bounding box. The coordinates of the upper left corner of the bounding box may be the coordinates of the upper left corner of the skin lesion area bounding box in the image coordinate system. The skin lesion area bounding box may be a bounding box for locating the skin injury area. The width of the bounding box may be the width of the skin lesion area bounding box. The height of the bounding box may be the height of the skin lesion area bounding box.

[0040] As an example, the main control chip can perform target detection processing on the received patient skin lesion image through a preset target detection model to obtain skin lesion area boundary box information. The target detection model can be but is not limited to one of the following: Faster R-CNN (Faster Region-based Convolutional Neural Network) model, YOLO (You Only Look Once, single-stage target detection) model.

[0041] In practice, the main control chip can input the patient's skin lesion image into the target detection model, first extract the features of the patient's skin lesion image through the convolution layer of the target detection model, and then predict the bounding box of the skin lesion area based on the extracted features through the target detection model, and then filter out the overlapping bounding boxes through the non-maximum suppression method, and retain the bounding box with the highest confidence as the bounding box of the skin lesion area. In this way, through target detection, the interference of background information such as clothing, hair and normal skin can be effectively removed, ensuring that subsequent feature extraction is more accurate, and further improving the accuracy of skin lesion classification, reducing misdiagnosis by doctors.

[0042] The second step is to perform image cropping on the patient's skin lesion image based on the skin lesion area boundary box information to obtain a skin lesion area image, wherein the skin lesion area image may be an image of the skin lesion area framed by the skin lesion area boundary box.

[0043] As an example, the main control chip may crop a local image where the lesion area is located from the patient's skin lesion image according to the upper left corner coordinates of the bounding box, the bounding box width and the bounding box height included in the lesion area bounding box information to obtain a skin lesion area image.

[0044] The third step is to perform normalization processing on the skin lesion area image to obtain a normalized skin lesion area image, wherein the normalized skin lesion area image may be a skin lesion area image after the size and color channel are adjusted.

[0045] As an example, the main control chip can be configured to, first, scale the skin lesion area image according to preset image size information to obtain an updated skin lesion area image. The image size information can be the preset width and height information of the image. For example, the image size information can be: {"height": 224px, "width": 224px}. Then, in response to determining that the updated skin lesion area image is not in RGB format, perform color channel conversion on the updated skin lesion area image to obtain a normalized skin lesion area image. It should be noted that when the updated skin lesion area image is in RGB format, the updated skin lesion area image can be used as a normalized skin lesion area image.

[0046] In the fourth step, the normalized skin lesion area image is input into a pre-trained image feature extraction model to obtain an initial skin lesion area feature vector. The image feature extraction model can be a ResNet18 network with the fully connected layer removed.

[0047] Optionally, the above-mentioned skin condition classification system based on multimodal feature fusion may also include a first model training server and a second model training server.

[0048] As an example, Figure 22 shows exemplary structural diagrams of other embodiments of the skin state classification system based on multimodal feature fusion of the present disclosure. Figure 2 As shown, the skin state classification system based on multimodal feature fusion includes a patient state text information input device 101, a skin image acquisition device 102, a main control chip 103, a classification result display 104, a first model training server 105 and a second model training server 106. The first model training server 105 and the second model training server 106 can both be connected to the main control chip 103 for communication. The first model training server can be used to train the skin state classification model. The second model training server can be used to train the image feature extraction model.

[0049] Optionally, the second model training server 106 may be configured to perform the following steps:

[0050] The first step is to obtain a positive sample skin image information set, a negative sample skin image set and an initial image feature extraction model. Among them, each positive sample skin image information in the above positive sample skin image information set may include a positive sample skin image and a positive sample skin category label. The above positive sample skin image may be a patient skin lesion image corresponding to the target skin lesion category. The above target skin lesion category may be a category of skin damage to be identified. The above target skin lesion category may be one of the following: pigmented nevus, nevus of Ota, freckles, chloasma, coffee spots, brown-blue nevus, seborrheic keratosis, freckle-like nevus, pigmented hair epidermal nevus and melanosis, etc. Each negative sample skin image in the above negative sample skin image set may be an image of clothing, hair, normal skin and skin with other skin diseases that is different from the positive sample skin image. The above initial image feature extraction model may be a residual network including a fully connected layer.

[0051] As an example, the second model training server may obtain a positive sample skin image information set, a negative sample skin image set, and an initial image feature extraction model from a database.

[0052] In the second step, each negative sample skin image in the above-mentioned negative sample skin image set is annotated to obtain a negative sample annotated skin image information set. Each negative sample annotated skin image information in the above-mentioned negative sample annotated skin image information set may include a negative sample annotated image and a negative sample skin category label. The above-mentioned negative sample annotated image may be an image after annotating the non-skin lesion area. The above-mentioned non-skin lesion area may be a skin area where no skin lesions corresponding to the above-mentioned target skin lesion category occur. The above-mentioned negative sample skin category label may be a text used to characterize the skin category of the non-skin lesion area.

[0053] As an example, the non-lesion area may be an area of ​​normal skin. The negative sample skin category label may be normal skin or other skin disease types. The main control chip uses a manual labeling method for each negative sample skin image in the negative sample skin image set. First, the non-lesion area in the negative sample skin image is labeled and the non-lesion area is labeled with a skin category through a polygonal box or other image box. Then, the image with the labeled box is used as a negative sample labeled image, and the labeled skin category text is determined as a negative sample skin category label.

[0054] In the third step, for each positive sample skin image information in the positive sample skin image information set, the positive sample skin image included in the positive sample skin image information is preprocessed to obtain a positive sample normalized skin lesion area image, and the positive sample skin category label included in the positive sample skin image information and the positive sample normalized skin lesion area image are determined as positive samples. The positive sample normalized skin lesion area image may be a normalized skin lesion area image corresponding to the positive sample skin image.

[0055] As an example, the main control chip can pre-process the positive sample skin image included in the positive sample skin image information through operations such as target detection, image cropping and normalization processing to obtain a positive sample normalized skin lesion area image.

[0056] In the fourth step, for each negative sample annotated skin image information in the negative sample annotated skin image information set, the negative sample annotated image included in the negative sample annotated skin image information is preprocessed to obtain a negative sample normalized skin lesion area image, and the negative sample skin category label included in the negative sample annotated skin image information and the negative sample normalized skin lesion area image are determined as negative samples. The negative sample normalized skin lesion area image may be a normalized skin lesion area image corresponding to the negative sample annotated skin image.

[0057] As an example, the main control chip may pre-process the negative sample annotated image included in the negative sample annotated skin image information through operations such as image cropping and normalization processing to obtain a negative sample normalized skin lesion area image.

[0058] In the fifth step, based on the determined positive samples and negative samples, the initial image feature extraction model is trained, and the fully connected layer in the trained initial image feature extraction model is deleted to generate an image feature extraction model.

[0059] As an example, the above-mentioned main control chip can train the initial image feature extraction model based on each positive sample and each negative sample determined by a conventional training method. After the initial image feature extraction model is trained, the fully connected layer in the initial image feature extraction model is deleted, and the initial image feature extraction model that does not include the fully connected layer is determined as the image feature extraction model.

[0060] It is necessary to add that Figure 3 A data flow diagram of the skin image feature extraction step in some embodiments of the skin condition classification system based on multimodal feature fusion of the present disclosure is shown. Figure 3 Taking a nevus as an example, first, the original image of the nevus is subjected to target detection, image cropping and normalization to obtain a normalized skin lesion area image. Then, the above-mentioned image feature extraction model is used to extract features from the normalized skin lesion area image to obtain a skin lesion area image feature map. Figure 3 Based on the steps shown, the above image feature extraction model can also be used through global average pooling to Figure 3 The skin lesion area image feature map shown in is converted into an initial skin lesion area feature vector.

[0061] In some embodiments, the main control chip 103 may be configured to input the initial first state text feature vector, the initial second state text feature vector sequence and the initial skin lesion area feature vector into a pre-trained skin state classification model to obtain a skin state classification result, and send the skin state classification result to the classification result display 104. The skin state classification model may be obtained by training the initial skin state classification model. The initial skin state classification model may include a feature dimension alignment module, a text fusion module, a text and image fusion module, a classification module and a multimodal feature adaptive contrast enhancement module. The feature dimension alignment module may be used to map different feature vectors to a preset feature space through a linear transformation to obtain vectors of the same dimension. The preset feature space may be a feature space pre-set for image and text shared representation. The text fusion module may be used to fuse structured text features and unstructured text features. The text and image fusion module may be used to fuse text features and image features. The classification module may be composed of a neural network for predicting skin lesion categories. For example, the classification module may be a multi-layer perceptron or a CNN model. The multimodal feature adaptive contrast enhancement module can be used to introduce the contrast loss of consistency within the same category and difference between different categories to enhance the ability to distinguish patient characteristics. The multimodal feature adaptive contrast enhancement module is only used in the model training stage and is not used in the model application reasoning stage. The skin state classification result can be the corresponding prediction probabilities when the patient's skin belongs to different skin lesion categories. The classification result display can be a terminal with a display screen.

[0062] Optionally, the first model training server 105 may be configured to perform the following steps to train a skin condition classification model:

[0063] The first step is to obtain a patient sample set and an initial skin state classification model. Each patient sample in the patient sample set may include a sample patient identifier, a sample initial skin lesion area feature vector, a sample initial first state text feature vector, a sample initial second state text feature vector sequence, and a sample skin lesion category label. The sample patient identifier may be a unique identifier of the sample patient. The sample patient may be a patient whose corresponding patient data is used to train the model. The sample initial skin lesion area feature vector may be an initial skin lesion area feature vector corresponding to the skin lesion area image of the sample patient. The sample initial first state text feature vector may be an initial first state text feature vector corresponding to the sample patient. The sample initial second state text feature vector sequence may be an initial second state text feature vector sequence corresponding to the sample patient. The sample skin lesion category label may characterize the real skin lesion category to which the skin of the sample patient belongs.

[0064] As an example, the first model training server may obtain a patient sample set and an initial skin condition classification model from a database.

[0065] Continuing, in the process of solving the technical problems in the background technology, the present disclosure is also accompanied by the following technical problem 2: how to fuse image features and text features to improve the classification accuracy of skin conditions. In response to the above technical problem 2, the conventional solution is generally to set different fixed weights for features of different modalities. However, the above conventional solution has the following problem: since congenital skin pigmentation diseases such as brown-blue nevus are not closely related to the structured text information of the patient's skin, when the skin lesion image and structured text are feature-fused, if the two features are weightedly fused in a fixed weight allocation manner, the feature information will be redundant, resulting in overfitting, further reducing the difference between categories, and thus easily leading to low classification accuracy of skin conditions. Therefore, in the face of the above technical problem 2, combined with the technical advantages of the solution development team itself in the field of deep learning, the present disclosure decided to adopt the following solution.

[0066] In the second step, at least one patient sample is selected from the patient sample set, and the following training steps are performed:

[0067] In the first sub-step, the sample initial skin lesion area feature vector included in each patient sample in the at least one patient sample is mapped to a preset feature space through a feature dimension alignment module included in the initial skin state classification model to obtain at least one sample skin lesion area feature vector. The sample skin lesion area feature vector can be generated by the following formula:

[0068] I′=W 图像 I+b 图像 .

[0069] Where I represents the initial lesion area feature vector of the sample. 图像 Represents the weight matrix corresponding to the image features. b 图像 represents the bias term corresponding to the image feature. I′ represents the feature vector of the sample skin lesion area. It should be noted that when the model starts training, W 图像 It is initialized by the Xavier initialization method, which can solve the problem of gradient disappearance or gradient explosion during model training.

[0070] In the second sub-step, the sample initial first state text feature vector and the sample initial second state text feature vector sequence included in each patient sample in the at least one patient sample are respectively mapped to the above feature space through the feature dimension alignment module included in the initial skin state classification model to obtain at least one sample first state text feature vector and at least one sample second state text feature vector sequence. The sample first state text feature vector and the sample second state text feature vector sequence can be generated by the following formula group:

[0071]

[0072] Where S represents the sample initial first state text feature vector. s Represents the weight matrix corresponding to the structured text features. s Represents the bias term corresponding to the structured text feature. S′ represents the sample first state text feature vector. T represents the sample initial second state text feature vector. W t The weight matrix representing the unstructured text features. t represents the bias term. T′ represents the second state text feature vector of the sample. It should be noted that when the model starts training, W s and W t It is initialized by the Xavier initialization method.

[0073] In the third sub-step, the text fusion module included in the initial skin state classification model determines the sample patient text feature vector corresponding to each patient sample in the at least one patient sample according to the at least one sample first state text feature vector and the at least one sample second state text feature vector sequence, and obtains at least one sample patient text feature vector. The sample patient text feature vector in the at least one sample patient text feature vector can characterize the features of the text describing the skin lesion state of the sample patient.

[0074] In some optional implementations of some embodiments, the first model training server 105 may be further configured to perform the following steps to generate at least one sample patient text feature vector through a text fusion module included in the initial skin state classification model:

[0075] Step 1, normalize the preset first learnable weight parameter and the second learnable weight parameter to obtain the first normalized weight parameter and the second normalized weight parameter. Among them, the above-mentioned first learnable weight parameter can be the initial weight parameter of the structured text feature. The above-mentioned second learnable weight parameter can be the initial weight parameter of the unstructured text feature. The above-mentioned first normalized weight parameter can be a weight parameter between 0 and 1, which is applicable to the structured text feature. The above-mentioned second normalized weight parameter can be a weight between 0 and 1, which is applicable to the unstructured text feature. It should be noted that the above-mentioned first learnable weight parameter and the second learnable weight parameter can be learned and updated through the back propagation of the model during the training stage, and the sum of the above-mentioned first normalized weight parameter and the above-mentioned second normalized weight parameter can be 1.

[0076] As an example, the first model training server may normalize the first learnable weight parameter and the second learnable weight parameter through a Softmax function to obtain a first normalized weight parameter and a second normalized weight parameter.

[0077] Step 2: for each patient sample in the at least one patient sample, perform the following steps:

[0078] Sub-step 1: performing feature aggregation processing on each sample second state text feature vector in the sample second state text feature vector sequence corresponding to the patient sample to obtain a sample second state fused feature vector. The sample second state fused feature vector can represent the fusion result of each unstructured text describing the patient's skin lesion state.

[0079] As an example, the first model training server may perform feature aggregation processing on each sample second state text feature vector in the sample second state text feature vector sequence corresponding to the patient sample through a preset feature aggregation processing method to obtain a sample second state fusion feature vector. The feature aggregation processing method may include but is not limited to at least one of the following: average pooling, maximum pooling or attention mechanism method.

[0080] Sub-step 2, based on the preset first learnable gating factor, the second learnable gating factor, the above-mentioned first normalization weight parameter and the above-mentioned second normalization weight parameter, the sample first state text feature vector and the sample second state fusion feature vector corresponding to the above-mentioned patient sample are subjected to feature weighted fusion processing to obtain the sample patient text feature vector. Among them, the above-mentioned first learnable gating factor can be a learnable gating factor for adjusting the first normalization weight parameter. The above-mentioned second learnable gating factor can be a learnable gating factor for adjusting the second normalization weight parameter. The sample patient text feature vector can be generated by the following formula:

[0081] F 文本 =g t α t ·T′+g s α s ·S′.

[0082] Among them, F 文本 g represents the sample patient text feature vector. s represents the first learnable gating factor. g t Represents the second learnable gating factor. α s Represents the first normalized weight parameter. α t Represents the second normalized weight parameter.

[0083] The fourth sub-step is to determine the sample multimodal fusion feature vector corresponding to each patient sample in the at least one patient sample according to the at least one sample patient text feature vector and the at least one sample skin lesion area feature vector through the text and image fusion module included in the initial skin state classification model, and obtain at least one sample multimodal fusion feature vector. Among them, the sample multimodal fusion feature vector in the at least one sample multimodal fusion feature vector can represent the fusion features of the text and image describing the patient's skin lesion state.

[0084] In some optional implementations of some embodiments, the first model training server 105 may be further configured to perform the following steps to generate at least one sample multimodal fusion feature vector through a text and image fusion module included in the initial skin state classification model:

[0085] Step 1, normalize the preset third learnable weight parameter and the fourth learnable weight parameter to obtain the third normalized weight parameter and the fourth normalized weight parameter. Among them, the above-mentioned third learnable weight parameter can be the initial weight parameter of the text feature. The above-mentioned fourth learnable weight parameter can be the initial weight parameter of the image feature. The above-mentioned third normalized weight parameter can be a weight parameter between 0 and 1, which is applicable to text features. The above-mentioned fourth normalized weight parameter can be a weight between 0 and 1, which is applicable to image features. It should be noted that the above-mentioned third learnable weight parameter and the fourth learnable weight parameter can be learned and updated through the back propagation of the model during the training stage, and the sum of the above-mentioned third normalized weight parameter and the above-mentioned fourth normalized weight parameter can be 1.

[0086] As an example, the first model training server may normalize the third learnable weight parameter and the fourth learnable weight parameter through a Softmax function to obtain a third normalized weight parameter and a fourth normalized weight parameter.

[0087] Step 2, for each patient sample in the at least one patient sample, based on the preset third learnable gating factor, the fourth learnable gating factor, the third normalization weight parameter and the fourth normalization weight parameter, the sample patient text feature vector and the sample skin lesion area feature vector corresponding to the patient sample are subjected to feature weighted fusion processing to obtain a sample multimodal fusion feature vector. Among them, the third learnable gating factor can be a learnable gating factor for adjusting the third normalization weight parameter. The fourth learnable gating factor can be a learnable gating factor for adjusting the fourth normalization weight parameter. The sample patient text feature vector can be generated by the following formula:

[0088] F 多模态 =g 图像 α 图像 ·I′+g 文本 α 文本 ·F 文本 .

[0089] Among them, F 多模态 Represents the sample multimodal fusion feature vector. g 文本 Represents the third learnable gating factor. α 文本 Represents the third normalized weight parameter. g 图像 Represents the fourth learnable gating factor. α 图像 Represents the fourth normalization weight parameter.

[0090] The fifth sub-step is to determine the skin condition classification result corresponding to each patient sample in the at least one patient sample according to the at least one sample multimodal fusion feature vector through the classification module included in the initial skin condition classification model.

[0091] Continuing, in the process of solving the technical problems in the background technology and the above-mentioned technical problem 2, the present disclosure is also accompanied by the following technical problem 3: different types of skin diseases often have similar image features or description features. How to train the model based on limited samples to improve the classification performance of the model. For the above-mentioned technical problem 3, the conventional solution is generally: based on the existing positive sample images and positive sample texts, new sample images and sample texts are generated through image generation models and text generation models. However, the above-mentioned conventional solution has the following problems: the images and texts generated by the neural network model depend on the performance of the neural network model, which are greatly different from the real environment, and the generation quality is also greatly different, which leads to poor classification performance of the trained skin state classification model, and then leads to low classification accuracy of the skin state. Therefore, in the face of the above-mentioned technical problem 3, combined with the technical advantages of the solution development team itself in the field of deep learning, the present disclosure decided to adopt the following solution.

[0092] The sixth sub-step is to determine a sample feature distance information set according to the sample skin lesion area feature vector and the sample patient text feature vector corresponding to each patient sample in the at least one patient sample through the multimodal feature adaptive contrast enhancement module included in the initial skin state classification model. The sample feature distance information in the sample feature distance information set can characterize the degree of similarity or difference between the sample skin lesion area feature vector and the sample patient text feature vector.

[0093] In some optional implementations of some embodiments, the first model training server 105 may be further configured to perform the following steps to generate a sample feature distance information set through a multimodal feature adaptive contrast enhancement module included in the initial skin state classification model:

[0094] Step 1: For each patient sample in the at least one patient sample, a preset positive sample label, a sample skin lesion area feature vector corresponding to the patient sample, and a sample patient text feature vector are determined as positive training samples. The preset positive sample label may be a preset text representing the positive sample. For example, the preset positive sample label may be represented by "1".

[0095] Step 2: for each skin lesion category identifier in the preset skin lesion category identifier group, perform the following steps:

[0096] Sub-step 1, according to the above-mentioned skin lesion category identification, the skin lesion area feature vectors of each sample and the text feature vectors of each sample patient corresponding to the above-mentioned at least one patient sample are classified and processed to obtain a skin lesion area feature vector set of the same sample, a text feature vector set of the same sample patient, a skin lesion area feature vector set of different sample and a text feature vector set of different sample patients. Among them, the skin lesion category identification in the above-mentioned skin lesion category identification group can be a unique identification of a pre-set skin lesion category. The skin lesion area feature vector set of the same sample can be a collection of the skin lesion area feature vectors of each sample corresponding to the skin lesion category identification. The text feature vector set of the same sample patient can be a collection of the text feature vectors of each sample patient corresponding to the skin lesion category identification. The skin lesion area feature vector set of different sample can be a collection of the skin lesion area feature vectors of each sample that does not correspond to the skin lesion category identification. The text feature vector set of different sample patients can be a collection of the text feature vectors of each sample patient that does not correspond to the skin lesion category identification.

[0097] As an example, the first model training server is configured to, first, select the sample lesion area feature vectors that match the lesion category identifier from the sample lesion area feature vectors corresponding to the at least one patient sample as the same sample lesion area feature vectors, and obtain the same sample lesion area feature vector set. Wherein, matching the lesion category identifier may be that the sample lesion category label corresponding to the sample lesion area feature vector is the same as the lesion category identifier. Then, select the sample patient text feature vectors that match the lesion category identifier from the sample patient text feature vectors corresponding to the at least one patient sample as the same sample patient text feature vectors, and obtain the same sample patient text feature vector set. Wherein, matching the lesion category identifier may be that the sample lesion category label corresponding to the sample patient text feature vector is the same as the lesion category identifier. Afterwards, the sample lesion area feature vectors that do not match the lesion category identifier in the sample lesion area feature vectors corresponding to the at least one patient sample are determined as the heterogeneous sample lesion area feature vector set. Finally, each sample patient text feature vector that does not match the skin lesion category identifier among each sample patient text feature vector corresponding to the at least one patient sample is determined as a heterogeneous sample patient text feature vector set.

[0098] Sub-step 2: Generate a first negative training sample set based on the above-mentioned skin lesion area feature vector set of the same type sample and the above-mentioned patient text feature vector set of the heterogeneous sample. The first negative training sample in the above-mentioned first negative training sample set can be a negative sample consisting of a skin lesion area feature vector of the same type sample and a patient text feature vector of the heterogeneous sample.

[0099] As an example, the first model training server may randomly select a vector element from the skin lesion area feature vector set of the same type of samples and the text feature vector set of the different type of samples of patients for multiple times to form a first negative training sample.

[0100] Sub-step three: generating a second negative training sample set based on the above-mentioned patient text feature vector set of the same sample and the above-mentioned skin lesion area feature vector set of the heterogeneous sample. The second negative training sample in the above-mentioned second negative training sample set may be a negative sample consisting of a patient text feature vector of the same sample and a skin lesion area feature vector of the heterogeneous sample.

[0101] As an example, the first model training server may randomly select a vector element from the same type sample patient text feature vector set and the heterogeneous sample skin lesion area feature vector set multiple times in a row to form a first negative training sample.

[0102] Step three: generating a comparison sample set based on the determined positive training samples, the first negative training sample set and the second negative training sample set, wherein the comparison sample set may be a set of non-repetitive comparison samples.

[0103] As an example, the first model training server first determines each positive training sample, each first negative training sample, and each second negative training sample as a comparison sample to be de-redundanted to obtain a comparison sample set to be de-redundanted. Then, the comparison sample set to be de-redundanted is de-duplicated to obtain a comparison sample set.

[0104] Step 4: Determine the sample feature distance information corresponding to each comparison sample in the comparison sample set to obtain a sample feature distance information set.

[0105] As an example, the first model training server may determine the sample feature distance information corresponding to each comparison sample by the following formula:

[0106]

[0107] Among them, d(I′, F 文本 ) represents the distance between the text features and image features included in the comparison sample.

[0108] The seventh sub-step is to determine the total loss value based on the sample feature distance information set, the sample skin lesion category label corresponding to each patient sample in the at least one patient sample, and the skin condition classification result.

[0109] In some optional implementations of some embodiments, the first model training server 105 may be further configured to perform the following steps to determine the total loss value:

[0110] Step 1: Determine the classification loss value according to the sample skin lesion category label and skin condition classification result corresponding to each patient sample in the at least one patient sample through a preset classification loss function. The classification loss function may be:

[0111]

[0112] Among them, L cls represents the classification loss value. N represents the total number of patient samples in the at least one patient sample. C represents the total number of skin lesion categories. i represents the sequence number of the sample. j represents the sequence number of the skin lesion category. y j，i Represents the sample skin lesion category label corresponding to the j-th patient sample. It represents the predicted probability that the jth sample belongs to the i-th class.

[0113] Step 2: Determine the regularization loss value according to the first learnable gating factor, the second learnable gating factor, the third learnable gating factor and the fourth learnable gating factor by using a preset regularization loss function. The regularization loss function may be:

[0114] L reg =λ·(|g 图像 |+|g 文本 |+|g t |+|g s |).

[0115] Among them, L reg represents the regularization loss value. λ represents the hyperparameter.

[0116] The steps for generating the multimodal fusion feature vector and regularization loss value of the above-mentioned samples and their related contents, as an inventive point of an embodiment of the present disclosure, solve the above-mentioned technical problem 2 "how to fuse image features and text features to improve the classification accuracy of skin conditions". The reason why the classification accuracy of skin conditions in conventional solutions is low is that congenital skin pigmentation diseases such as brown-blue nevus are not closely related to the structured text information of the patient's skin. When the skin lesion image and structured text are feature-fused, if the two features are weightedly fused in a fixed weight distribution manner, the feature information will be redundant, resulting in overfitting, and further reducing the difference between categories. If the above problems are solved, the effect of improving the classification accuracy of skin conditions can be achieved. In order to achieve this effect, the present disclosure first maps the skin lesion image features, structured text features, and unstructured text features to the same feature space, thereby ensuring the dimensional consistency of the image features and text features during the feature fusion process. Then, the weights in the multimodal feature fusion process are dynamically adjusted through a gating mechanism. Finally, the gating factor is also sparsely controlled through the subsequent L1 regularization loss to ensure that the weight of features that are not closely related to skin lesions is 0. In this way, redundant feature information can be reduced, the difference between skin lesion categories can be increased, and overfitting can be avoided. As a result, the robustness and accuracy of the skin condition classification model can be improved. In turn, the classification accuracy of skin conditions can be improved.

[0117] Step 3: Determine the contrast loss value corresponding to the contrast sample set according to the sample feature distance information set by using a preset contrast loss function. The contrast loss function can be:

[0118]

[0119] Among them, L contrastive Represents the contrast loss value. I′ i F represents the feature vector of the sample skin lesion area in the i-th comparison sample.文本，i represents the sample patient text feature vector in the ith comparison sample. M represents the total number of comparison samples in the comparison sample set. i Represents the indicator variable. When the i-th contrast sample is a positive training sample, y i is 1, when the i-th comparison sample is a negative training sample, y i is 0. margin is a set hyperparameter, which indicates the minimum distance threshold between two feature vectors in negative training samples.

[0120] The above-mentioned contrast loss value generation step and its related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem three mentioned in the background technology, "how to train the model based on limited samples to improve the classification performance of the model". The reason why the classification performance of the trained skin state classification model and the low classification accuracy of the skin state in the conventional solution is that the images and texts generated by the neural network model depend on the performance of the model neural network model, which are greatly different from the real environment, and the generation quality difference is also large. Therefore, if the above-mentioned problems are solved, the effect of improving the accuracy of skin state classification can be achieved. In order to achieve this effect, the present disclosure first constructs a contrast sample set including positive training samples and negative training samples according to the feature vectors of each sample skin lesion area and each sample patient text feature vector corresponding to each patient sample through a multimodal feature adaptive contrast enhancement module during the training process of the skin state classification model. In this way, it can be ensured that each generated contrast sample is consistent with the image and text in the real environment. Then, for each contrast sample in the contrast sample set, the feature distance between the skin lesion image features and the text features included in the contrast sample is determined. Afterwards, the above-mentioned contrast loss function can be used to increase the similarity between the skin lesion image features and text features included in the positive training samples, and reduce the similarity between the skin lesion image features and text features included in the negative training samples. Finally, the contrast loss value is integrated into the back propagation process of the model, and the embedding distribution of the fusion features of the skin lesion image and text is continuously optimized, so that the image features and text features corresponding to the same skin lesion category are similar, and the image features and text features corresponding to different skin lesion categories are different. The fused feature representation has stronger separability between skin lesion categories. In this way, a more discriminative input can be provided to the classification module, thereby improving the classification performance of the skin state classification model. Furthermore, the classification accuracy of the skin state can be improved.

[0121] Step 4: Determine the total loss value as the sum of the classification loss value, the regularization loss value and the contrast loss value.

[0122] In an eighth sub-step, in response to determining that the above-mentioned total loss value is less than a preset loss threshold, the trained initial skin condition classification model is determined as the skin condition classification model.

[0123] Optionally, the first model training server 105 may be further configured to adjust the network parameters of the initial skin state classification model in response to determining that the total loss value is greater than or equal to the preset loss threshold, and to form a patient sample set with each unused patient sample, and to perform the training step again using the adjusted initial skin state classification model. The first model training server 105 may optimize the network parameters of the initial skin state classification model according to the total loss value through back propagation.

[0124] In some embodiments, the classification result display 104 may be configured to display the received skin condition classification result, so that the user can view the skin condition classification result and assist the user in making a diagnosis.

[0125] As an example, Figure 4 The experimental data of skin lesion classification performance of some embodiments of the skin condition classification system based on multimodal feature fusion disclosed in the present invention are shown. Figure 4 The table in the figure lists different skin lesion types and various performance indicators for measuring the effect of skin lesion classification. Different skin lesion types include pigmented nevus, nevus of Ota, freckles, melasma, café-au-lait spots, brown-blue nevus, seborrheic keratosis, freckle-like nevus, pigmented hair epidermal nevus, melanosis, normal skin and others. The various performance indicators include accuracy, sensitivity, specificity, precision and F1 value. The F1 value is the harmonic mean of accuracy and sensitivity. The values ​​in the table can characterize the performance of the model on the corresponding performance indicators when identifying the corresponding skin lesion type.

[0126] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the skin state classification system based on multimodal feature fusion of some embodiments of the present disclosure, the classification accuracy of skin state can be improved. Specifically, the reason for the low classification accuracy of skin state is that structured oral text usually has fewer dimensions and limited features, and it is difficult to characterize the characteristics of various skin diseases, which makes it difficult for the fusion features of skin lesion images and structured texts to accurately and comprehensively characterize the skin lesion status of patients, thereby resulting in low classification accuracy of skin state. Based on this, some embodiments of the present disclosure include a skin state classification system based on multimodal feature fusion, including: a patient state text information entry device, a skin image acquisition device, a main control chip and a classification result display, wherein the patient state text information entry device is configured to enter a patient state structured text information sequence and a patient state unstructured text information sequence, and send the patient state structured text information sequence and the patient state unstructured text information sequence to the main control chip; the skin image acquisition device is configured to acquire patient skin lesion images, and send the patient skin lesion images to the main control chip; the main control chip is configured to process the received patient state structured text information and The patient's unstructured text information sequence is subjected to text vectorization processing to obtain an initial first-state text feature vector and an initial second-state text feature vector sequence; the main control chip is configured to perform image feature extraction processing on the received patient's skin lesion image to obtain an initial skin lesion area feature vector; the main control chip is configured to input the initial first-state text feature vector, the initial second-state text feature vector sequence and the initial skin lesion area feature vector into a pre-trained skin state classification model to obtain a skin state classification result, and send the skin state classification result to the classification result display; the classification result display is configured to display the received skin state classification result. Therefore, in some embodiments of the present disclosure, the skin state classification system based on multimodal feature fusion can improve the feature dimension of the skin lesion feature by performing multimodal feature fusion on the patient's skin lesion image, the structured text corresponding to the skin state and the unstructured text through the main control chip, so as to more accurately and comprehensively characterize various skin diseases, thereby improving the classification accuracy of the skin state. Furthermore, the skin condition classification results can be displayed through the classification result display, thereby assisting doctors in diagnosing skin diseases and improving the efficiency of skin disease diagnosis.

[0127] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A skin condition classification system based on multimodal feature fusion, the skin condition classification system based on multimodal feature fusion comprising: Patient status text information input device, skin image acquisition device, main control chip and classification result display, wherein, The patient status text information input device is configured to input a patient status structured text information sequence and a patient status unstructured text information sequence, and send the patient status structured text information sequence and the patient status unstructured text information sequence to the main control chip; The skin image acquisition device is configured to acquire a patient's skin lesion image and send the patient's skin lesion image to the main control chip; The main control chip is configured to perform text vectorization processing on the received patient status structured text information and patient status unstructured text information sequence to obtain an initial first state text feature vector and an initial second state text feature vector sequence; The main control chip is configured to perform image feature extraction processing on the received patient skin lesion image to obtain an initial skin lesion area feature vector; The main control chip is configured to input the initial first state text feature vector, the initial second state text feature vector sequence and the initial skin lesion area feature vector into a pre-trained skin state classification model to obtain a skin state classification result, and send the skin state classification result to the classification result display; The classification result display is configured to display the received skin condition classification result.

2. The skin condition classification system based on multimodal feature fusion according to claim 1, wherein: Each patient status structured text information in the patient status structured text information sequence includes a status attribute value; and the main control chip is further configured to: For each patient status structured text information in the patient status structured text information sequence, perform the following steps: In response to determining that the patient status structured text information satisfies a preset attribute condition, normalizing the status attribute value included in the patient status structured text information to obtain a numerical attribute feature value; In response to determining that the patient status structured text information does not satisfy the preset attribute condition, performing word embedding processing on the status attribute value included in the patient status structured text information to obtain a category attribute feature vector; The obtained feature vectors of each category attribute and the feature values ​​of each numerical attribute are concatenated to obtain an initial first state text feature vector; For each piece of patient status unstructured text information in the patient status unstructured text information sequence, word embedding processing is performed on the patient status unstructured text information to obtain an initial second state text feature vector.

3. The skin condition classification system based on multimodal feature fusion according to claim 1, wherein: The main control chip is further configured as: Performing target detection processing on the received patient skin lesion image to obtain skin lesion area boundary box information; Based on the skin lesion area boundary box information, performing image cropping processing on the patient's skin lesion image to obtain a skin lesion area image; Normalizing the skin lesion area image to obtain a normalized skin lesion area image; The normalized skin lesion area image is input into a pre-trained image feature extraction model to obtain an initial skin lesion area feature vector.

4. The skin condition classification system based on multimodal feature fusion according to claim 1, wherein: The skin condition classification system based on multimodal feature fusion also includes a first model training server; and the first model training server is configured to: Acquire a patient sample set and an initial skin state classification model, wherein each patient sample in the patient sample set includes a sample patient identifier, a sample initial skin lesion area feature vector, a sample initial first state text feature vector, a sample initial second state text feature vector sequence, and a sample skin lesion category label, and the initial skin state classification model includes a feature dimension alignment module, a text fusion module, a text and image fusion module, a classification module, and a multimodal feature adaptive contrast enhancement module; Select at least one patient sample from the patient sample set and perform the following training steps: By using a feature dimension alignment module included in the initial skin state classification model, the sample initial skin lesion area feature vector included in each patient sample of the at least one patient sample is mapped to a preset feature space to obtain at least one sample skin lesion area feature vector; By using a feature dimension alignment module included in the initial skin state classification model, the sample initial first state text feature vector and the sample initial second state text feature vector sequence included in each patient sample of the at least one patient sample are respectively mapped to the feature space to obtain at least one sample first state text feature vector and at least one sample second state text feature vector sequence; Determine, by means of a text fusion module included in the initial skin state classification model, a sample patient text feature vector corresponding to each patient sample in the at least one patient sample according to the at least one sample first state text feature vector and the at least one sample second state text feature vector sequence, and obtain at least one sample patient text feature vector; Determining, by means of a text and image fusion module included in the initial skin state classification model, a sample multimodal fusion feature vector corresponding to each patient sample in the at least one patient sample according to the at least one sample patient text feature vector and the at least one sample skin lesion area feature vector, and obtaining at least one sample multimodal fusion feature vector; Determining, by a classification module included in the initial skin state classification model, a skin state classification result corresponding to each patient sample in the at least one patient sample according to the at least one sample multimodal fusion feature vector; Determine a sample feature distance information set according to a sample skin lesion area feature vector and a sample patient text feature vector corresponding to each patient sample in the at least one patient sample by a multimodal feature adaptive contrast enhancement module included in the initial skin state classification model; Determine a total loss value according to the sample feature distance information set, the sample skin lesion category label corresponding to each patient sample in the at least one patient sample, and the skin state classification result; In response to determining that the total loss value is less than a preset loss threshold, the trained initial skin condition classification model is determined as the skin condition classification model.

5. The skin condition classification system based on multimodal feature fusion according to claim 4, wherein: The first model training server is further configured to: In response to determining that the total loss value is greater than or equal to the preset loss threshold, the network parameters of the initial skin condition classification model are adjusted, and unused patient samples are combined into a patient sample set, and the training step is performed again using the adjusted initial skin condition classification model.

6. The skin condition classification system based on multimodal feature fusion according to claim 3, wherein: The skin condition classification system based on multimodal feature fusion also includes a second model training server; and the second model training server is configured to: Acquire a positive sample skin image information set, a negative sample skin image set and an initial image feature extraction model, wherein each positive sample skin image information in the positive sample skin image information set includes a positive sample skin image and a positive sample skin category label, and the initial image feature extraction model is a residual network including a fully connected layer; Performing labeling processing on each negative sample skin image information in the negative sample skin image set to obtain a negative sample labeled skin image information set, wherein each negative sample labeled skin image information in the negative sample labeled skin image information set includes a negative sample labeled image and a negative sample skin category label; For each positive sample skin image information in the positive sample skin image information set, preprocessing the positive sample skin image included in the positive sample skin image information to obtain a positive sample normalized skin lesion area image, and determining the positive sample skin category label included in the positive sample skin image information and the positive sample normalized skin lesion area image as a positive sample; For each negative sample annotated skin image information in the negative sample annotated skin image information set, preprocessing the negative sample annotated image included in the negative sample annotated skin image information to obtain a negative sample normalized skin lesion area image, and determining the negative sample skin category label included in the negative sample annotated skin image information and the negative sample normalized skin lesion area image as a negative sample; Based on the determined positive samples and negative samples, the initial image feature extraction model is trained, and the fully connected layer in the trained initial image feature extraction model is deleted to generate an image feature extraction model.

Images