Skin disease auxiliary evaluation system and device, storage medium and program product

Abstract

The invention provides a skin disease auxiliary evaluation system and device, a storage medium and a program product, and relates to the technical field of edge computation.The skin disease auxiliary evaluation system is deployed on a terminal device and comprises a feature extraction module, a feature extraction module and an evaluation module, obtaining semantic feature information corresponding to a disease description text in the skin disease information and image feature information corresponding to an affected part image in the skin disease information; the feature fusion module is used for associating the semantic feature information with the image feature information based on a cross attention mechanism to generate first multi-mode fusion feature information of the skin disease to be evaluated; and the large language model evaluation module is used for generating an evaluation result corresponding to the to-be-evaluated skin disease through a first large language model based on the first multi-mode fusion feature information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing, and in particular to a skin disease auxiliary evaluation system, device, storage medium and program product. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, at present, the visual analysis technology can be used to combine the description of the patient's own skin disease condition, and the skin disease condition of the patient is evaluated through the skin disease image, which effectively assists the medical staff to evaluate and analyze the patient's condition before diagnosis.

[0003] In the skin disease auxiliary evaluation scene based on visual analysis and patient description, the skin disease image and the skin disease description provided by the patient are often uploaded to the cloud server through the terminal device first, and then the powerful computing resources provided by the cloud server are used for analysis and evaluation, and finally the analysis result is fed back to the terminal device and is checked by the patient and / or relevant medical staff. However, considering that the terminal device and the cloud server need to be connected and data transmission through the network, a large amount of skin disease image data will occupy high network bandwidth resources, and it is easy to cause high network delay between the terminal device and the cloud server, affecting the patient's condition evaluation response. At the same time, since the skin disease related data (such as skin disease image, skin disease history and other content) of the patient contains a large amount of sensitive information related to user privacy, the storage of the skin disease related data in the cloud server and the data transmission between the cloud server and the terminal device have the risk of privacy data leakage.

[0004] Therefore, some embodiments of the present application provide a skin disease auxiliary evaluation system, device, storage medium and program product, which can realize the auxiliary evaluation of skin disease through the skin disease auxiliary evaluation system deployed in the terminal device and supporting edge computing, improve the response speed of skin disease auxiliary evaluation, and protect the privacy data of the patient from being leaked. SUMMARY

[0005] One or more embodiments of the present application provide a skin disease auxiliary evaluation system deployed in a terminal device, the skin disease auxiliary evaluation system comprising: a feature extraction module configured to obtain semantic feature information corresponding to a disease description text in skin disease information and image feature information corresponding to a lesion image in the skin disease information based on user input skin disease information; a feature fusion module configured to associate the semantic feature information and the image feature information based on a cross-attention mechanism to generate first multi-modal fusion feature information of a skin disease to be evaluated; and a large language model evaluation module configured to generate an evaluation result corresponding to the skin disease to be evaluated by a first large language model based on the first multi-modal fusion feature information.

[0006] In some embodiments, the feature extraction module, in obtaining the image feature information corresponding to the lesion image in the process of obtaining the skin disease information, performs the following steps: dividing the lesion image into a plurality of image blocks of the same shape and size, each image block corresponding to a mask lesion image, at least part of the image blocks in the mask lesion image being occluded; obtaining a vectorized representation corresponding to the image block; obtaining a self-attention weight corresponding to each image block based on the vectorized representation and the mask lesion image through a self-attention mechanism; and obtaining a self-attention feature map corresponding to the lesion image based on the self-attention weight corresponding to each image block, the image feature information including the self-attention feature map.

[0007] In some embodiments, the image block can be obtained based on division of the lesion image along a first direction and a second direction, the first direction being perpendicular to the second direction; in the mask lesion image, the occluded image block is adjacent to or spaced apart from the image block corresponding to the mask lesion image by an even number of pixel blocks in the first direction and / or the second direction.

[0008] In some embodiments, the feature fusion module, in obtaining the first multi-modal fusion feature information, performs the following steps: associating the image feature information and the semantic feature information through a cross-attention mechanism based on the self-attention feature map and the semantic feature information, and obtaining a cross-attention weight corresponding to each image block; and updating the self-attention feature map to generate a cross-attention feature map based on the cross-attention weight corresponding to each image block, the first multi-modal fusion feature information including the cross-attention feature map.

[0009] In some embodiments, in the process of associating the image feature information and the semantic feature information through the cross-attention mechanism based on the self-attention feature map and the semantic feature information, the occluded image block in the mask lesion image is reconstructed by forcing dependence on the semantic feature information.

[0010] In some embodiments, the first large language model is obtained based on knowledge distillation of a pre-trained second large language model, the second large language model being deployed on a cloud server, the cloud server being remotely connected to the terminal device: the model size of the first large language model is smaller than the model size of the second large language model.

[0011] In some embodiments, the first large language model is obtained based on knowledge distillation of the pre-trained second large language model, including: training the preset large language model based on a training set of the second large language model to obtain a third large language model, wherein the training set includes training disease information and corresponding skin disease evaluation labels; obtaining a hard label annotation loss of the third large language model for the skin disease evaluation labels; obtaining a skin disease evaluation probability label corresponding to the training disease information based on the second large language model, and obtaining a soft label annotation loss of the third large language model for the skin disease evaluation probability label; based on the second large language model and the third large language model, obtaining an alignment difference loss of an attention feature map corresponding to the same training disease information; obtaining a professional knowledge loss of the third large language model for a professional knowledge constraint condition; and based on the hard label annotation loss, the soft label annotation loss, the alignment difference loss, and the professional knowledge loss, optimizing the third large language model to generate the first large language model.

[0012] In some embodiments, the feature extraction module is further configured to obtain patient data information, and obtain corresponding patient feature information based on the patient data information, the patient feature information at least including historical skin disease records of the patient; the feature fusion module is further configured to associate the patient feature information and the first multi-modal fusion feature information to generate second multi-modal fusion feature information of the skin disease to be evaluated; and the large language model evaluation module is further configured to generate an evaluation result corresponding to the skin disease to be evaluated by the first large language model based on the second multi-modal fusion feature information.

[0013] Some embodiments of the present specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the functions of the skin disease auxiliary evaluation system of any one of the preceding embodiments of the present application can be realized.

[0014] Some embodiments of the present specification also provide a computer readable storage medium, which stores computer instructions, and when the computer instructions are executed by a processor, the functions of the skin disease auxiliary evaluation system of any one of the preceding embodiments of the present specification can be realized.

[0015] Some embodiments of the present specification also provide a computer program product, which includes a computer program, and when at least part of the computer program is executed by a processor, the functions of the skin disease auxiliary evaluation system of any one of the preceding embodiments of the present specification can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0016] The present specification will be further illustrated in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same reference numbers in the drawings represent the same structures or steps.

[0017] Figure 1 is a schematic diagram of an application scenario of a skin disease auxiliary evaluation system according to some embodiments of the present specification.

[0018] Figure 2 is an exemplary structural diagram of a skin disease auxiliary evaluation system according to some embodiments of the present specification.

[0019] Figure 3 is an exemplary flowchart of obtaining image feature information corresponding to a lesion image in skin disease information according to some embodiments of the present specification.

[0020] Figure 4 is a schematic diagram of dividing a lesion image and a corresponding mask lesion image according to some embodiments of the present specification.

[0021] Figure 5 is a schematic diagram of a self-attention feature map corresponding to a lesion image according to some embodiments of the present specification.

[0022] Figure 6 is an exemplary flowchart of obtaining first multi-modal fusion feature information according to some embodiments of the present specification.

[0023] Figure 7 is an interactive schematic diagram of knowledge distillation of a first large language model according to some embodiments of the present specification.

[0024] Figure 8 is an exemplary flowchart of obtaining a first large language model according to some embodiments of the present specification.

[0025] Figure 9 is a hardware architecture schematic diagram of a computer device according to some embodiments of the present specification. DETAILED DESCRIPTION

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the embodiments will be described in detail below with reference to the drawings. Obviously, the following description is some examples or embodiments of the present specification, and for those skilled in the art, without creative labor, the technical solutions or means disclosed in the present specification can also be applied to other scenarios.

[0027] It should be understood that the "system", "device", "unit" and / or "module" used in the present specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the above words can be replaced by other expressions.

[0028] Unless otherwise noted, technical terms, including the terms used in describing the components of the application, are intended to have their ordinary and accustomed meanings in the art. Generally, the nomenclature used for describing chemical, biological, and / or medical quantities, such as "include", "comprise", and the like, are used in their open-ended, non-limiting sense unless otherwise noted. Generally, the nomenclature used for describing methodological steps, such as "comprise", "include", and the like, are used in their open-ended, non-limiting sense unless otherwise noted. Generally, the nomenclature used for describing methodological steps, such as "comprise", "include", and the like, are used in their open-ended, non-limiting sense unless otherwise noted.

[0029] The flow diagrams herein are used to describe the operations performed by the apparatus or system of the relevant embodiments. Unless otherwise specified, the order of the steps described in these flow diagrams should not be construed as a limitation on the order of the steps performed. One of ordinary skill in the art can adjust the order of the steps based on the knowledge conveyed by the embodiments of the present disclosure, including but not limited to reversing the order, combining multiple steps into one, and splitting a step into multiple steps.

[0030] Figure 1 is a schematic diagram of an application scenario of a skin disease auxiliary evaluation system according to some embodiments of the present disclosure. In some embodiments, as shown in Figure 1 application scenario 100 can include server 110, network 120, and terminal device 130, wherein server 110 can connect terminal device 130 through network 120. In some embodiments, terminal device 130 can receive skin disease symptom information provided by a patient and send it to server 110, which is analyzed and processed by a skin disease auxiliary evaluation model deployed on the server to generate a corresponding auxiliary evaluation result. In some embodiments, terminal device 130 can deploy a skin disease auxiliary evaluation model, and use the locally deployed skin disease auxiliary evaluation model to perform auxiliary evaluation on the skin disease symptom information provided by the patient; server 110 can train and / or update the skin disease auxiliary evaluation model deployed by terminal device 130.

[0031] In some embodiments, the server 110 can be a computer device with high computing performance. In some embodiments, the server 110 can analyze and process the skin disease information provided by the terminal device 130 to generate a corresponding auxiliary evaluation result. In some embodiments, the server 110 can be used to train and / or update the skin disease auxiliary evaluation model deployed on the terminal device 130. In some embodiments, the server 110 can be a single computer device, or a computing cluster composed of multiple computer devices, thereby providing strong computing power and efficient response for skin disease auxiliary evaluation and training and / or updating of the skin disease auxiliary evaluation model. In some embodiments, the server 110 can be a server, which can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Distribute Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0032] In some embodiments, the network 120 can be any form of wired or wireless network, or any combination thereof. For example only, the network 120 can be one or more of a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), a wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a Public Switched Telephone Network (PSTN), a Bluetooth network, etc. The network 120 can have multiple access points, and the server 110 and the terminal device 130 can access the network 120 through the access points.

[0033] In some embodiments, the terminal device 130 can include, but is not limited to, a desktop computer, a smartphone, a notebook computer, a VR (Virtual Reality) device, a tablet computer, etc. In some embodiments, a user can provide skin disease information such as skin disease image and skin disease description through the terminal device 130, and check the preliminary evaluation result of the skin disease through the visual interface of the terminal device 130, which is not limited herein.

[0034] It should be noted that, Figure 1 The application scenario of the skin disease auxiliary evaluation system shown is only an example, and the application scenarios described in the embodiments of the present specification are used to more clearly illustrate the technical solutions of the embodiments of the present specification, and do not constitute a limitation on the technical solutions provided by the embodiments of the present specification. For example, Figure 1The number of the server 110 and the terminal device 130 in the system is only illustrative, and a person skilled in the art can set any number of the server 110 and the terminal device 130 according to needs, which does not limit the patent protection scope of the present specification. As known by a person skilled in the art, with the development of the field of skin disease auxiliary assessment and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present specification are also applicable to similar technical problems.

[0035] With the continuous development of artificial intelligence technology, at present, the visual analysis technology can be used to combine the description of the patient's own skin disease condition, and the skin disease condition of the patient is assisted to be evaluated through the skin disease image, which effectively assists medical personnel to perform pre-diagnosis condition evaluation and analysis on the patient. In some embodiments, in the skin disease auxiliary assessment scene based on visual analysis and patient description, the skin disease image and the skin disease description provided by the patient are often uploaded to the cloud server through the terminal device first, and then the powerful computing resources provided by the cloud server are used for analysis and evaluation, and finally the analysis result is fed back to the terminal device and is checked by the patient and / or the relevant medical personnel. However, considering that the terminal device and the cloud server need to be connected and data transmission through the network, a large amount of skin disease image data will occupy high network bandwidth resources, and is easy to cause high network delay between the terminal device and the cloud server, affecting the patient's condition evaluation response. At the same time, since the skin disease related data of the patient (such as skin disease image, skin disease history, etc.) contains a large amount of sensitive information related to user privacy, the storage of the skin disease related data in the cloud server and the data transmission between the cloud server and the terminal device have the risk of privacy data leakage.

[0036] In order to overcome the above problems, the present specification provides a skin disease auxiliary assessment system, device, storage medium and program product, which can realize the auxiliary assessment of skin disease through the skin disease auxiliary assessment system deployed in the terminal device and supporting edge computing, improve the response speed of the skin disease auxiliary assessment, and protect the privacy data of the patient from being leaked. Specifically, Figure 2 is an exemplary structural diagram of a skin disease auxiliary assessment system according to some embodiments of the present specification. In some embodiments, as shown in Figure 2 The skin disease auxiliary assessment system 200 shown in Figure 2 The skin disease auxiliary assessment system 200 shown in Figure 2 The skin disease auxiliary assessment system 200 shown in some embodiments, as shown in Figure 2The illustrated dermatology auxiliary assessment system 200 can be deployed on a terminal device, where the terminal device can be a device that directly interacts with the patient, such as a mobile phone, personal computer, etc. used by the patient, or a triage device or pre-diagnosis device with an interactive interface located in a hospital or other place, without limitation. In some embodiments, when the dermatology auxiliary assessment system is deployed on a terminal device, the user can implement an auxiliary assessment of the skin disease simply by interacting with the terminal device. The skin disease-related data (such as skin disease images, skin disease descriptions, skin disease history, etc.) provided to the terminal device is only stored in the terminal device and will not be transmitted or interacted with other service terminals over the network, which can effectively prevent the leakage of skin disease-related data containing private information. At the same time, because the auxiliary assessment of the skin disease is generated locally on the terminal device, it will not occupy high network bandwidth resources or generate network delays, greatly shortening the response time for generating the skin disease auxiliary assessment results and simultaneously improving the user experience.

[0037] In some embodiments, as Figure 2 As shown, the skin disease auxiliary assessment system 200 may include a feature extraction module 210, a feature fusion module 220, and a large language model evaluation module 230. The functional implementation of the feature extraction module 210 to the large language model evaluation module 230 will be described below in conjunction with specific embodiments.

[0038] In some embodiments, Figure 2 In the illustrated dermatology assisted assessment system 200, the feature extraction module 210 can be used to obtain semantic feature information corresponding to the symptom description text in the dermatology information and image feature information corresponding to the affected area image in the dermatology information based on the dermatology information input by the user. In some embodiments, the dermatology information input by the user can include symptom description text and affected area image, where the symptom description text refers to the user's description of the dermatology condition of themselves or the patient, and can specifically include the onset time, symptoms, current feelings of the affected area, previous medical history, etc., without limitation here; the affected area image refers to a photographic image of the user or the patient containing the dermatology area provided by the user. Considering that dermatology often manifests as redness, swelling, rashes, darkening of skin color, etc. on the patient's skin surface, which are different from normal skin conditions, the combination of the affected area image and symptom description text can more completely reflect the user's actual dermatology condition and serve as the evaluation basis for dermatology assisted assessment. In some embodiments, the user can input dermatology information through an interactive interface provided by a terminal device, such as uploading a dermatology image through the interactive interface of a specific application on a mobile phone, and providing the dermatology description in the form of text input or voice input, without limitation here.

[0039] In some embodiments, the semantic feature information can be a semantic vector obtained according to the disease description text, which can reflect the core content and information in the disease description text. In some embodiments, the semantic feature information can be obtained by a BERT (Bidirectional Encoder Representation from Transformer, bidirectional encoder representation from transformer) model. In some embodiments, those skilled in the art can also select other suitable semantic recognition models according to actual needs to obtain semantic feature information, which is not limited herein. In some embodiments, the semantic feature information can include global text features, and local text features (such as skin disease symptom features, skin disease occurrence site features, etc.) generated by extracting keywords, which are used for fusion with other feature information including image feature information in the subsequent feature fusion stage, which is not limited herein. In some embodiments, before obtaining the semantic feature information, the disease description text can be parsed by using the semantic recognition model, and the user-provided biased colloquial disease description is converted into a more standardized disease description, and then vectorized representation is performed. For example, taking the user-provided disease description text as "found several small patches with darker color, no pain or itching, but a little rough when touched, like a feeling of sticking to the skin" as an example, the corresponding standardized disease description can be "patient complains of local skin surface appearing patches, dark color, no obvious abnormality". In some embodiments, the semantic recognition model can be trained and / or optimized by using skin disease professional knowledge and skin pathology samples, so that the semantic recognition model can better extract professional description content required for skin disease auxiliary evaluation from the disease description text, which can further improve the embodiment of the semantic feature information for the core content in the disease description.

[0040] In some embodiments, the image feature information can be an image feature set or an image feature map obtained according to the affected area image, which can reflect the specific content of the possible skin disease area in the affected area image, such as the content of the skin surface appearing erythema in a part of the affected area image, etc. In some embodiments, the image feature information can be obtained by a VIT model (Vision Transformer, computer vision model based on Transformer architecture), and those skilled in the art can also select other suitable visual models according to actual needs to obtain image feature information, which is not limited herein. In some embodiments, the image feature information can be obtained based on an attention mechanism, specifically, the affected area image can be divided into a plurality of query blocks, and each query block is associated with all other query blocks in the affected area image to calculate the similarity between all query blocks, and the image feature information corresponding to the affected area image is obtained by global calculation.

[0041] In some embodiments, further, considering that in the process of acquiring image feature information for the affected area image, often rely on a single local feature, such as the presence of red patches or the presence of concentrated dense rash areas in the affected area image, and the single local feature is prone to feature extraction deviation due to local noise or local hair blockage, etc. In order to overcome the problems encountered in the above actual application process, adapt to the actual feature extraction needs of the affected area image, the image feature information can be acquired based on the self-attention mechanism of blind point reconstruction, and the mask window is added in the traditional self-attention module to force the visual model to reconstruct the shielded area using the visual features of the surrounding unshielded area, which can effectively improve the acquisition ability of image feature information. The specific implementation of image feature information acquisition will be further described below in conjunction with specific embodiments.

[0042] Figure 3 According to the embodiments of the present specification, an example flowchart for obtaining image feature information corresponding to the affected area image in the skin disease information is shown. In some embodiments, as shown in flow 300, it can be implemented based on feature extraction module 210. In some embodiments, as shown in flow 300, it can include the following steps. Figure 3 Figure 3

[0043] Step 310: divide the affected area image into a plurality of image blocks of the same shape and size, each image block corresponding to a mask affected area image. In some embodiments, the mask affected area image can be an image formed by randomly shielding at least part of the image blocks in the affected area image based on a preset mask matrix. In some embodiments, the affected area image is divided into a plurality of image blocks of the same shape and size, and each image block corresponds to a mask affected area image, which can enhance the sensitivity of acquiring key image features (such as erythema regions and other skin disease manifestation regions) in the affected area image based on the self-attention mechanism of blind point reconstruction, and can more effectively capture the local region features related to the skin disease in the affected area image.

[0044] Figure 4 According to the embodiments of the present specification, a schematic diagram of dividing the affected area image and the corresponding mask affected area image is shown, as shown in Figure 4 As can be seen from the above, the affected area image 400 contains a plurality of image blocks 410 of the same shape and size. In some embodiments, as shown in Figure 4

[0045] ​​​In some embodiments, for a selected image block 410a in the lesion image 400, its corresponding mask lesion image is shown as 420, wherein the mask lesion image 420 includes occluded image blocks 421 (i.e. image blocks filled with white color in the mask lesion image 420) and non-occluded image blocks 422 (i.e. image blocks filled with shade in the mask lesion image). In some embodiments, each image block in the lesion image corresponds to a mask lesion image to realize the blind spot reconstruction, and the occluded part in the mask lesion image can be randomly generated. In some embodiments, the mask lesion image corresponding to each image block in the lesion image can be generated based on a preset mask occlusion rule, so that each image block only focuses on the content of a specific position in the lesion image in the self-attention mechanism, which can indirectly enhance the sensitivity to key image feature content in the lesion image. In some embodiments, as shown in Figure 4 the mask lesion image 420 corresponding to the selected image block 410a, in the mask lesion image 420, the occluded image block 421 is adjacent to the image block 410a corresponding to the mask lesion image 420 in the first direction x and / or the second direction y, or is spaced apart from the image block 410a corresponding to the mask lesion image 420 by an even number of pixel blocks, for example Figure 4 the image block in the same row (i.e. having the same spatial coordinate in the second direction y) as the marked another occluded image block 421a, which is adjacent to the image block 410a corresponding to the mask lesion image 420 in the second direction y (or is spaced apart from the image block 410a corresponding to the mask lesion image 420 by zero image blocks); for example Figure 4 the image block in the same column (i.e. having the same spatial coordinate in the first direction x) as the marked another occluded image block 421a, which is spaced apart from the image block 410a corresponding to the mask lesion image 420 by two image blocks in the first direction x, without limitation. In some embodiments, those skilled in the art can also select other suitable mask occlusion rules according to actual needs, without limitation.

[0046] Step 320: Obtain the vectorized representation corresponding to the image block, and based on the vectorized representation and the masked affected part image, obtain the self-attention weight corresponding to each image block through the self-attention mechanism. In some embodiments, the self-attention mechanism is a technology widely used in deep learning, especially in natural language processing and computer vision processing. It can focus on the relationship between different parts of the input when processing input data, and extract feature information of the data accordingly. In some embodiments, by vectorizing the image block, the affected part image can be divided into multiple vectorized elements, and the self-attention mechanism is used to obtain the relationship between each element and other elements, thereby generating a weighted representation for each element. The above weighted representation corresponds to the self-attention weight of the image block in the affected part image. In some embodiments, the self-attention weight corresponding to each image block can be represented by the following mathematical expression.

[0047] .

[0048] in, Used to represent the query matrix, corresponding to the vectorized representation of the image block that currently needs to calculate the self-attention weight; Used to represent the key matrix, corresponding to the affected area image after image block division, is the transposed matrix of the key matrix, It can be used to characterize the similarity between each image block in the affected area image and the rest of the image blocks: Used to represent the dimension of the bond matrix, in order to prevent The value obtained when using the function is too large, resulting in The output of the function has the problem of vanishing gradient; The position encoding matrix used to represent the masked affected part image can introduce the position information of the occluded image block in the affected part image; Function is used to convert ( ) is converted into a probability distribution, which can characterize the importance of each key to the current query. The output of the function is a weight matrix, where all weights add up to 1 and each weight corresponds to each image block. Used to represent the value matrix, corresponding to the specific vector representation content of each image block in the affected area image after image block division, through The weight matrix output by the function is multiplied by a dot to obtain the self-attention weight corresponding to each image block. In some embodiments, those skilled in the art may also use other technical means to achieve the specific acquisition of self-attention weights, which is not limited here.

[0049] Step 330: Based on the self-attention weights corresponding to each image block, obtain a self-attention feature map corresponding to the affected part image. In some embodiments, the self-attention weights corresponding to each image block can be used to generate a self-attention feature map corresponding to the affected part image. Those skilled in the art can use appropriate technical means to achieve this, and are not limited here. In some embodiments, the image feature information corresponding to the affected part image may include the self-attention feature map obtained based on the above step 340. Figure 5 is a schematic diagram of a self-attention feature map corresponding to an affected part image according to an embodiment of this specification. In some embodiments, Figure 5 As can be seen, in Figure 5 The affected area image provided by the user on the left shows a darker plaque area. Figure 5 In the corresponding self-attention feature map on the right, the high-attention area is concentrated in the darker patches in the affected area, indicating that the image feature information obtained based on the self-attention mechanism can reflect the specific content of the areas where skin diseases may occur in the affected area.

[0050] In some embodiments, the skin condition information input by the user may include not only a description of the condition and an image of the affected area, but also patient data information. In some application scenarios, the user who inputs the skin condition information may be the patient with the skin condition himself, or a relative or friend who assists the patient in entering the information. In the process of entering the skin condition information, it is necessary to declare the patient's identity information. For example, the patient's identity information may be entered before the skin condition information is entered. The patient's identity information may include the patient's name, electronic medical record number, etc., which is not limited here. In some embodiments, the feature extraction module 210 can obtain corresponding patient feature information based on the patient data information input by the user. For example, based on the patient's identity information input by the user, the system database can be searched for patient-related information, and the information content of interest to the skin disease auxiliary assessment can be selected based on the search results to generate patient feature information. In some embodiments, the patient feature information includes at least the patient's historical skin disease record, which can provide information support for the current skin disease auxiliary assessment based on the patient's previous medical history information. In some embodiments, the patient feature information may also include the patient's medication history, allergy history, etc., which can provide reference auxiliary assessment opinions for subsequent skin disease diagnosis and treatment, which is not limited here.

[0051] In some embodiments, Figure 2In the illustrated skin disease auxiliary evaluation system 200, the feature fusion module 220 can be used to associate semantic feature information and image feature information based on a cross-attention mechanism to generate first multi-modal fusion feature information of the skin disease to be evaluated. In some embodiments, the cross-attention mechanism is an attention mechanism for processing multi-modal input or multiple information sources, which can further enhance the understanding of input information by focusing on the relationship between different inputs. For example, when performing cross-attention mechanism association of text features and image features, the textual description of a specific object in the text content and the image region containing the specific object in the image can be associated to better understand and extract features from the combination of the two different modal inputs of image and text. The function implementation of the feature fusion module 220 will be further described below in conjunction with specific embodiments.

[0052] Figure 6 is an example flowchart for obtaining first multi-modal fusion feature information according to the embodiments illustrated in the present specification. In some embodiments, as shown in Figure 6 The flow 600 can be implemented based on the feature fusion module 220. In some embodiments, as shown in Figure 6 The flow 600 can include the following steps.

[0053] Step 610: Based on the self-attention feature map and the semantic feature information, associate the image feature information and the semantic feature information through the cross-attention mechanism, and obtain the cross-attention weight corresponding to each image block. In some embodiments, the cross-attention mechanism can be used to associate the image feature information and the semantic feature information. For example, the semantic feature of "local skin surface appears plaque" in the semantic feature information can be associated with the region containing the plaque part in the image feature information. By giving the image block containing the plaque part a higher cross-attention weight through the semantic feature information, the extraction of key information in the user input skin disease information is further strengthened.

[0054] Step 620: Update the self-attention feature map based on the cross-attention weight corresponding to each image block to generate a cross-attention feature map. In some embodiments, after the association of the semantic feature information, the attention weight corresponding to each image block may change. The cross-attention feature map can be generated according to the updated attention weight, and the semantic feature information can be labeled in the generated cross-attention feature map. For example, the "plaque" label can be labeled in the image block containing the plaque part. In some embodiments, the first multi-modal fusion feature information can include the cross-attention feature map obtained based on the above step 620, which is not limited herein.

[0055] In some embodiments, in the process of associating image feature information and semantic feature information based on the self-attention feature map and the semantic feature information through the cross-attention mechanism, the occluded image blocks in the mask image can also be reconstructed by forcing the semantic feature information to depend on the mask image, and the reconstruction of the image blocks can be used as an auxiliary training task to effectively enhance the image feature extraction capability of the visual model for the affected image.

[0056] In some embodiments, based on the patient feature information obtained by the feature extraction module 210 synchronously, the feature fusion module 220 can be used to associate the patient feature information and the first multi-modal fusion feature information to generate the second multi-modal fusion feature information of the skin disease to be evaluated. In some embodiments, the feature fusion module 220 can splice the image feature information processed by the cross-modal attention mechanism, the semantic feature information, and the patient feature information in a vectorized manner to generate the corresponding second multi-modal fusion feature information, which contains the semantic feature information, the patient feature information, and the image feature information processed by the cross-modal attention mechanism. In some embodiments, those skilled in the art can also use other technical means to realize the multi-modal data fusion of the semantic feature information, the patient feature information, and the image feature information, which is not limited here.

[0057] In some embodiments, in the skin disease auxiliary evaluation system 200 as shown in Figure 2 In some embodiments, in the skin disease auxiliary evaluation system 200 as shown in

[0058] In some embodiments, in order to avoid the application problems caused by the data interaction between the terminal device and the cloud server in the actual application process of the traditional skin disease image analysis and evaluation, the first large language model can be deployed locally on the terminal device, and the skin disease data provided by the user can be processed locally on the terminal device, which can effectively shorten the analysis response time and improve the privacy data security of the user.

[0059] In some embodiments, the first large language model can be obtained based on knowledge distillation of a pre-trained second large language model, where the second large language model is deployed on a cloud server, the cloud server is remotely connected with the terminal device, and the model size of the first large language model is smaller than the model size of the second large language model. In some embodiments, knowledge distillation is a machine learning model compression method that aims to transfer the knowledge of a large and complex model to a small model to improve the processing and analysis performance and generalization ability of the small model. The core idea is to convert the knowledge of a large and complex model (which can be referred to as a teacher model) into a more concise and effective representation, so that the small model (which can be referred to as a student model) can maintain high performance while reducing computational complexity and resource requirements. This allows the small model to have better processing and analysis performance in resource-constrained environments such as terminal devices. Figure 7 is a knowledge distillation interaction diagram of a first large language model according to some embodiments of the present specification. In some embodiments, as shown in Figure 7 , the first large language model is deployed on a terminal device 710, and the second large language model is deployed on a cloud server 720. Both the first large language model and the second large language model are trained based on the same training data set. Meanwhile, the second large language model is transferred through knowledge distillation by remote connection between the cloud server 720 and the terminal device 710, so that the first large language model with smaller model size deployed on the terminal device can obtain similar skin disease auxiliary evaluation performance as the second large language model.

[0060] In some embodiments, the first large language model and the second large language model have similarities in overall structure. For example, in a specific example, the second large language model can be trained using the Qwen2.5-7B large language model as the base model, and the first large language model can be trained using the Qwen2.5-0.5B large language model as the base model. Qwen2.5 is a general-purpose thousand-question series open-source large language model developed by Alibaba Cloud, 7B represents that the second large language model has 7 billion parameters, and 0.5B represents that the first large language model has 50 million parameters. That is, the model size of the first large language model is smaller than the model size of the second large language model. The acquisition of the first large language model will be further described below in combination with specific embodiments.

[0061] In some embodiments, it can be understood that, since the first large language model needs to rely on the knowledge distillation of the second large language model to improve the performance of the skin disease auxiliary assessment, in the process of obtaining the first large language model, the second large language model deployed on the cloud server needs to be trained and obtained first. In some embodiments, the training of the second large language model can include two stages, the first stage can be the pre-training of the feature extractor part, which can enhance the model's ability to capture multi-modal input forms and ensure that it can effectively process data from different modalities; the second stage can be the parameter adjustment of the large language model part, which realizes the alignment of input features by mapping the encoding output of the feature extractor to the feature space of the base model of the second large language model. In some embodiments, the training set of the second large language model can include image training data, text training data, and training metadata, wherein the image training data can include skin disease images, at least part of which correspond to preset skin disease assessment labels, and the specific skin disease symptoms of these skin disease assessment labels are the real skin disease symptoms corresponding to the skin disease images; the text training data can specifically include the patient's self-reported symptom characteristics, course evolution situation explanation, past medical history and / or past treatment history, etc., which are not limited here; the training metadata can specifically include the patient's demographic characteristics corresponding to a certain skin disease, and the key clinical variables related to the skin disease, etc., which are not limited here. In some embodiments, for the data content in the training set containing skin disease assessment labels, both blind spot reconstruction and skin disease assessment tasks can be performed, while for the data content in the training set not containing skin disease assessment labels, only self-supervised training for blind spot reconstruction can be performed to strengthen the model's local feature perception and extraction ability.

[0062] In some embodiments, in the first training stage of the second large language model, the training process can specifically include three parts of multi-modal feature extraction, feature fusion and iterative optimization, which can specifically load the pre-trained image feature extractor and text feature extractor, extract image features and text features based on the training set, then generate multi-modal fusion features based on the cross-modal attention mechanism, and perform skin disease assessment based on the multi-modal fusion features. In some embodiments, the iterative optimization of the second large language model can use gradient descent method to update the model parameters to minimize the training loss, wherein the training loss is obtained based on the skin disease assessment labels in the training set, and can also be called hard label annotation loss. In some embodiments, the training loss of the second large language model can be represented by the following mathematical expression.

[0063] .

[0064] wherein, is the training loss of the second large language model, a temperature-scaled cross-entropy loss for a skin condition evaluation classification task, a blind spot reconstruction loss for a feature reconstruction blind spot reconstruction task, and respectively, and corresponding loss weighting coefficients, and are positive numbers greater than 0 and sum to 1. A person skilled in the art can set the loss weighting coefficients according to actual needs, which are not limited herein.

[0065] In some embodiments, in the second training stage of the second large language model, the output results of the feature extractor in the first training stage can be combined with the base model of the second large language model to align the feature space and fine-tune the parameters. In some embodiments, the training set used in the second training stage can be the same as the training set used in the first training stage, and the corresponding skin condition evaluation labels can specifically include skin condition types, skin condition diagnosis opinions, etc., which are not limited herein. In some embodiments, the training method used in the second training stage can adopt a supervised LoRA (Low-Rank Adaptation, low-rank adaptation) fine-tuning form, that is, all model weights except the feature space alignment projection layer are frozen, and only the query matrix, key matrix and value matrix of the attention layer are injected with trainable low-rank adapters. A person skilled in the art can also choose other suitable technical means to implement the above-mentioned second training stage, which is not limited herein.

[0066] In some embodiments, the second large language model can also apply expert knowledge during training to further improve the accuracy of skin disease auxiliary evaluation, where expert knowledge refers to the professional knowledge accumulated by experienced doctors or medical researchers in long-term practice and research, which can specifically include classification standards for common skin diseases (such as eczema, psoriasis, acne, etc.), key features of skin images (such as erythema, papules, scales, pigmentation, etc.), and etiology analysis (such as skin diseases may be caused by allergies, infections, genetics, environmental factors, etc.). In some embodiments, expert knowledge can be stored in a third-party memory that is in communication with a cloud server and a terminal device, and expert knowledge can be used to further improve the accuracy of skin disease auxiliary evaluation during the training of the second large language model and the first large language model.

[0067] In some embodiments, based on the trained second large language model, knowledge distillation can be performed using the second large language model to train and obtain the first large language model. Figure 8 is an example flowchart for obtaining a first large language model according to some embodiments of the present specification. In some embodiments, as shown in Figure 8As shown, the flow 800 can include the following steps.

[0068] Step 810: training the preset large language model based on the training set of the second large language model to obtain a third large language model. In some embodiments, the preset large language model can be a base model of the first large language model and a training basis, and the third large language model can be an intermediate state model formed by the first large language model in the training optimization process. In some embodiments, the training set of the first large language model and the training set of the second large language model can be the same training set, which can include image training data, text training data, and training metadata. In some embodiments, the training set can include a plurality of skin disease information to be trained and corresponding skin disease evaluation labels, wherein the text training data and / or image training data corresponding to the skin disease evaluation labels have relevant feature information of the corresponding skin disease.

[0069] In some embodiments, considering the case that the training sample of the possible partial rare skin disease is insufficient, the data augmentation method can be used to expand the image training data and / or text training data corresponding to the rare skin disease. In some embodiments, traditional data augmentation methods such as rotation, reflection, cropping, scaling, and tone adjustment can be used to expand the image samples in the existing image training data corresponding to the rare skin disease. Those skilled in the art can use other feasible data augmentation methods to expand the image samples in the image training data, which are not limited herein. In some embodiments, the associated image data such as pathological sections, dermatoscopes, and clinical photos can also be combined, and the sample migration method can be used to migrate the skin disease image samples of one affected area to other potential affected areas, and the expansion of the rare skin disease image training data can be achieved by constructing multi-view image samples, which are not limited herein.

[0070] Step 820: obtaining the hard label annotation loss of the third large language model for the skin disease evaluation label. In some embodiments, during the training process of the third large language model, since the same training set as the second large language model is used, the hard label annotation loss related to the skin disease evaluation label can be obtained by referring to the training loss in the iterative optimization of the second large language model, that is, the hard label annotation loss can be obtained in the manner of the foregoing embodiments, which are not limited herein.

[0071] ​​Step 830: obtaining a skin disease evaluation probability label corresponding to the training disease information based on the second large language model, and obtaining a soft label annotation loss corresponding to the skin disease evaluation probability label based on the third large language model. In some embodiments, in the process of training the first large language model, the training of the first large language model is not simply imitating the output of the second large language model, but inherits the knowledge structure of the second large language model. Specifically, the second large language model can generate soft labels through high-temperature Softmax to pass the relative relationship between various skin disease diagnosis categories (for example, the similarity of two or more skin diseases in symptoms) to the first large language model. In some embodiments, unlike the hard labels corresponding to the skin disease evaluation labels in the training set, the essence of the soft labels is the quantitative expression of the uncertainty of the second large language model in the process of skin disease auxiliary evaluation. The third large language model, as an intermediate state model of the first large language model, can learn the soft label loss of such ambiguous but information-rich distribution through the KL (Kullback-Leibler) divergence loss, which can complement the accurate classification of the hard labels.

[0072] Step 840: obtaining an alignment difference loss of the attention feature map corresponding to the same training disease information based on the second large language model and the third large language model. In some embodiments, in the process of using the second large language model for knowledge distillation, an attention mode inheritance mechanism can also be added. As can be understood from the related description of the foregoing embodiments, the second large language model has attention response between the key areas of the image and the key words of the text, such as the plaque in the image and the word "plaque" in the text. The first large language model can be forced to imitate and inherit it through feature map alignment, which can help the first large language model focus on the local features that are crucial to skin disease auxiliary evaluation even if a simplified network structure is used. In some embodiments, the alignment difference loss can be represented by the following mathematical expression.

[0073] .

[0074] Wherein, is the alignment difference loss, and are the attention feature maps of the last attention layer of the second large language model and the first large language model for the same picture input, respectively, is used to represent the mean square error loss.

[0075] Step 850: obtaining a professional knowledge loss corresponding to the professional knowledge constraint condition based on the third large language model. In some embodiments, the above steps 820 to 850 can be executed synchronously or asynchronously in execution order, which is not limited here.

[0076] In some embodiments, the representation capability of the first large language model can be continuously enhanced by introducing rule constraints of professional knowledge, specifically, the diagnostic rules defined by experts in professional knowledge (e.g., "blister in children patients should be considered as hand-foot-mouth disease first") can be converted into a differentiable loss term and run through the whole process of knowledge distillation. In some embodiments, the first large language model is guided by the soft labels provided by the second large language model and corrected by the rule logic during the optimization process, ensuring that its predictions conform to both data rules and medical common sense. In some embodiments, the professional knowledge loss generated based on the rule constraints of professional knowledge can be represented by the following mathematical expression.

[0077] , .

[0078] wherein, is the professional knowledge loss; is used to represent the i-th rule; corresponding weight coefficient; is used to represent the importance of the i-th rule in the training process; is a trainable scalar parameter used to represent the importance of the i-th rule in the training process, which can be updated by the optimizer as part of the model weights; is a trainable scalar parameter used to represent the importance of all rules in the training process; is a differentiable loss used to represent the conditional triggering based on the training metadata; is used to represent the i-th rule; is a penalty coefficient used to control the penalty when the rule is not triggered (i.e., when tends to 0), making the model more robust in determining which rules should be followed by controlling the penalty strength when the rule is not activated; is used to represent the probability value of the model output for each type of skin disease auxiliary assessment result; is a minimum value.

[0079] In some embodiments, each expert-defined diagnostic rule in professional knowledge can be considered as a condition-triggered penalty item, for example, when the input sample satisfies the conditions of being less than 12 years old and the image detects the presence of a blister, the set rule is triggered, at this time the value of approaches 1, and during training, the skin disease symptom auxiliary assessment result when the penalty is determined for an adult can be represented in a differentiable form, which can be represented by the following mathematical expression.

[0080] .

[0081] wherein, ​sigmoid, which can convert the expert-defined diagnostic rules in the professional knowledge (e.g., when the patient is less than 12 years old and the image detects the presence of a blister, the detection probability of the skin disease auxiliary assessment result being a blister is greater than 0.5) into a continuous differentiable form so as to participate in back propagation and loss optimization in the deep learning of the model; for representing the age of the patient; for representing the detection probability of the skin disease auxiliary assessment result being a blister; for representing the scaling coefficient, which can be set to 10 here.

[0082] Step 860: based on the hard label annotation loss, the soft label annotation loss, the alignment difference loss, and the professional knowledge loss, the third large language model is optimized to generate the first large language model. In some embodiments, the optimization objective function for knowledge distillation of the third large language model can adopt the following mathematical expression.

[0083] .

[0084] wherein, for representing the optimization objective function of the third large language model, which needs to make the value of tend to be minimized in the optimization process, for representing the hard label annotation loss, for representing the soft label annotation loss, for representing the alignment difference loss, for representing the professional difference loss, to respectively for representing the loss weighting coefficients corresponding to the above losses, are all positive numbers greater than 0 and the sum is 1. Those skilled in the art can set the loss weighting coefficients according to actual needs, which are not limited here. In some embodiments, the optimization of the third large language model by making the above optimization objective function take the minimum value can obtain the required first large language model. The obtained first large language model, due to the combination of soft labels, attention patterns, and expert rule constraints, can effectively improve the diagnostic logical rationality of skin disease assessment and the recognition ability of key pathological features, and significantly reduce the reasoning complexity of the model, which is suitable for deployment on terminal devices and realizes efficient auxiliary assessment response.

[0085] For more information about each module, please refer to the related description of Figures 2 to 8 which will not be repeated here. It should be understood that, Figure 2The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented by using special logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above method and system can be implemented by using computer executable instructions and / or control codes contained in a processor, such as a carrier medium, such as a disk, CD or DVD-ROM, or the memory of a programmable device. The system and its modules of the present specification can not only be implemented by hardware circuit, such as ultra large scale integrated circuit or gate array, semiconductor, such as logic chip, transistor, or programmable hardware device, such as field programmable gate array, programmable logic device, but also by software, such as executed by various types of processors, and also by a combination of the above hardware circuit and software (for example, firmware).

[0086] It should be noted that the above description of the system and its modules is for convenience of description only, and cannot limit the present specification within the scope of the embodiments. It can be understood that, for those skilled in the art, after understanding the principle of the system, the modules can be combined in any way to form a subsystem connected with other modules, or some modules can be split to obtain more modules or multiple units under the module. Such variations are within the scope disclosed in the present specification.

[0087] Figure 9 Fig. 9 is a schematic diagram of a hardware architecture of a computer device 900 according to some embodiments of the present specification. In some embodiments, the computer device 900 can include a memory, a processor, and a computer program stored in the memory and executable on the processor, which can realize the functions of the skin disease auxiliary evaluation system provided by any one of the preceding embodiments when the processor executes the computer program. In some embodiments, the computer device 900 can be a terminal device such as a smartphone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workstation, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 900 can be a rack-mounted server, a blade server, a tower server, or a cabinet server (including a standalone server, or a server cluster composed of multiple servers).

[0088] In some embodiments, as shown in Fig. 9, the computer device 900 can include a processor 901, a bus 902, a memory 903, an input device 904, an output device 905, and a communication device 906. Figure 9As shown, the computer device 900 includes, but is not limited to, a memory 910, a processor 920, and a network interface 930 which can be communicatively linked through a system bus. The memory 910 includes at least one type of computer-readable storage media, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD memory, DX memory, and the like), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 910 can be an internal memory module of the computer device 900, such as a hard disk or a memory of the computer device 900. In other embodiments, the memory 910 can also be an external storage device of the computer device 900, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 910 can include both an internal memory module and an external storage device of the computer device 900. In this embodiment, the memory 910 is generally used to store an operating system and various application software installed in the computer device 900, such as program codes related to the implementation of the functions of the skin disease auxiliary assessment system provided in any one of the foregoing embodiments, and the like. In addition, the memory 910 can also be used to temporarily store various data that have been output or will be output.

[0089] In some embodiments, the processor 920 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 920 is generally used to control the overall operation of the computer device 900, such as performing control and processing related to data interaction or communication of the computer device 900, and the like. In this embodiment, the processor 920 is used to run program codes or process data stored in the memory 910.

[0090] In some embodiments, the network interface 930 can include a wireless network interface or a wired network interface, which is generally used to establish a communication link between the computer device 900 and other computer devices. For example, the network interface 930 is used to connect the computer device 900 with an external terminal through a network, to establish a data transmission channel and a communication link between the computer device 900 and the external terminal, and the like. The network can be an Intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, a Bluetooth, a WiFi, and the like wireless or wired network.

[0091] It should be noted that, Figure 9 Only the computer device with the memory 910, the processor 920, and the network interface 930 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented. In some embodiments, the related program code stored in the memory 910 can also be divided into one or more program modules, and executed by one or more processors (such as the processor 920) to implement the embodiments of the present specification, which are not limited herein.

[0092] One or more embodiments of the present specification also provide a computer readable storage medium, having stored thereon a computer program, wherein the computer program is executed by a processor to implement the functions of the skin disease auxiliary assessment system provided by the foregoing embodiments. In some embodiments, the computer readable storage medium can include a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In the present embodiment, the computer readable storage medium is generally used to store the operating system and various application software installed on the computer device, such as the program code for extracting skin disease feature information, fusing feature information, and generating assessment results in the embodiments. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0093] One or more embodiments of the present specification also provide a computer program product, including a computer program, which is executed by a processor to implement the functions of the skin disease auxiliary assessment system provided by any one of the foregoing embodiments. In some embodiments, the computer program product can only involve the computer program, which can be carried by a storage medium or a processing device. In other embodiments, the computer program product can also be a storage medium or a processing device containing the foregoing computer program. The processing device can include one or more processors, and the storage medium.

[0094] The beneficial effects that the embodiments of the present specification can bring include but are not limited to: (1) by deploying the skin disease auxiliary assessment system to the terminal device, the user can realize the auxiliary assessment of the skin disease only by interacting with the terminal device, the skin disease related data provided to the terminal device only exists in the terminal device and will not be transmitted or interacted with other servers through the network, which can effectively avoid the leakage of skin disease related data containing private information; at the same time, since the auxiliary assessment of the skin disease is generated locally in the terminal device, it will not occupy high network bandwidth resources or cause network delay, which greatly shortens the response time of generating the auxiliary assessment result of the skin disease and synchronously improves the user experience. (2) the skin disease auxiliary assessment system adopts the blind spot occlusion reconstruction method in the process of image feature information acquisition, and multi-modal fusion of image feature information, text feature information and patient feature information, which improves the analysis ability of the system for skin disease information and the accuracy of auxiliary assessment. (3) the model deployed in the terminal device adopts the knowledge distillation method to inherit the knowledge of the large and complex model deployed in the cloud server, so that the model deployed in the terminal device can reduce the computational complexity and resource demand while maintaining high performance, and also has good processing and analysis performance and skin disease auxiliary assessment effect in the resource limited environment. It should be noted that different embodiments can have different beneficial effects, and in different embodiments, the beneficial effects that can be produced can be any one or a combination of the above, or any other beneficial effects that can be obtained.

[0095] The above detailed description has described the basic concepts, and it is obvious that the above detailed description is only used as an example and does not limit the present specification. Although it is not explicitly stated here, those skilled in the art can make various modifications, improvements and corrections to the present specification. Such modifications, improvements and corrections are taught in the present specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present specification.

Claims

1. A skin disease auxiliary assessment system, characterized in that: Deployed on a terminal device, the skin disease auxiliary assessment system includes: a feature extraction module for obtaining, based on the skin condition information input by the user, semantic feature information corresponding to the condition description text in the skin condition information and image feature information corresponding to the affected area image in the skin condition information; a feature fusion module, configured to associate the semantic feature information and the image feature information based on a cross-attention mechanism to generate first multimodal fusion feature information of the skin condition to be evaluated; A large language model evaluation module is used to generate an evaluation result corresponding to the skin condition to be evaluated through a first large language model based on the first multimodal fusion feature information.

2. The skin disease auxiliary assessment system according to claim 1, characterized in that: The feature extraction module performs the following steps in the process of obtaining the image feature information corresponding to the affected part image in the skin disease information: Dividing the affected part image into a plurality of image blocks of the same shape and size, each of the image blocks corresponds to a masked affected part image, and at least part of the image blocks in the masked affected part image is blocked; Obtaining a vectorized representation corresponding to the image block; Based on the vectorized representation and the masked affected area image, obtaining a self-attention weight corresponding to each image block through a self-attention mechanism; Based on the self-attention weight corresponding to each image block, a self-attention feature map corresponding to the affected part image is obtained, and the image feature information includes the self-attention feature map.

3. The skin disease auxiliary assessment system according to claim 2, characterized in that: The image blocks are obtained by dividing the affected part image along a first direction and a second direction, wherein the first direction is perpendicular to the second direction; In the masked affected part image, the blocked image block is adjacent to the image block corresponding to the masked affected part image in the first direction and / or the second direction, or is spaced apart from the image block corresponding to the masked affected part image by an even number of pixel blocks.

4. The skin disease auxiliary assessment system according to claim 2, characterized in that: The feature fusion module performs the following steps in the process of obtaining the first multimodal fusion feature information: Based on the self-attention feature map and the semantic feature information, associating the image feature information and the semantic feature information through a cross-attention mechanism, and obtaining a cross-attention weight corresponding to each image block; Based on the cross-attention weight corresponding to each of the image blocks, the self-attention feature map is updated to generate a cross-attention feature map, and the first multimodal fusion feature information includes the cross-attention feature map.

5. The skin disease auxiliary assessment system according to claim 4, characterized in that: In the process of associating the image feature information and the semantic feature information through a cross-attention mechanism based on the self-attention feature map and the semantic feature information, the image block occluded in the masked affected part image is reconstructed by forcibly relying on the semantic feature information.

6. The skin disease auxiliary assessment system according to claim 1, characterized in that: The first language model is obtained based on knowledge distillation of a pre-trained second language model, and the second language model is deployed on a cloud server, which is remotely connected to the terminal device. The model scale of the first largest language model is smaller than the model scale of the second largest language model.

7. The skin disease auxiliary assessment system according to claim 6, characterized in that: The first language model is obtained based on knowledge distillation of the pre-trained second language model, including: Training the preset large language model based on the training set of the second large language model to obtain a third large language model, wherein the training set includes training disease information and corresponding skin disease assessment labels; Obtaining a hard label annotation loss of the third language model for the skin disease assessment label; Obtaining a skin disease assessment probability label corresponding to the training condition information based on the second language model, and obtaining a soft label annotation loss corresponding to the skin disease assessment probability label by the third language model; Obtaining, based on the second language model and the third language model, an alignment difference loss of attention feature maps corresponding to the same training disease information; Obtaining the professional knowledge loss corresponding to the professional knowledge constraint condition of the third language model; The third largest language model is optimized based on the hard label annotation loss, the soft label annotation loss, the alignment difference loss, and the expertise loss to generate the first largest language model.

8. The skin disease auxiliary assessment system according to claim 1, characterized in that: The feature extraction module is further configured to obtain patient data information, and based on the patient data information, obtain corresponding patient feature information, wherein the patient feature information at least includes the patient's historical skin disease record; The feature fusion module is further configured to associate the patient feature information with the first multimodal fusion feature information to generate second multimodal fusion feature information of the skin condition to be evaluated; The large language model evaluation module is also used to generate an evaluation result corresponding to the skin condition to be evaluated through the first large language model based on the second multimodal fusion feature information.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the functions of the system according to any one of claims 1 to 8 are realized.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the function of the system according to any one of claims 1 to 8 is realized.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the function of the system according to any one of claims 1 to 8 is realized.

Images