Automatic skin disease diagnosis method and system based on multi-modal large language model

Abstract

The invention belongs to the technical field of medical image and big data analysis, and provides an automatic skin disease diagnosis method and system based on a multi-modal big language model, and the method comprises the steps: obtaining a dermatoscope diagnosis report of a historical patient, and extracting an original image and a diagnosis text in the dermatoscope diagnosis report; constructing a multi-modal large language model, wherein the multi-modal large language model comprises a visual encoder, a linear projection layer and a large language model; training the multi-modal large language model; obtaining an external medical knowledge base, extracting text information in the external medical knowledge base, and constructing a vector database based on the text information; in response to an input image and a query text of a user, N vectors with the highest similarity are retrieved in the vector database according to the query text, the retrieved vectors are decoded into a text, and the text and the query text jointly construct complete prompt word input; and based on the input image and cue word input, obtaining a final dermatoscope diagnosis report by using the trained multi-modal large language model. The diagnosis report is automatically generated.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging and big data analysis technology, specifically relating to an automatic diagnostic method and system for skin diseases based on a multimodal large language model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Skin diseases, as a general term for diseases occurring on the skin and its appendages, have a multi-dimensional impact on patients. From appearance to health, from physiological to psychological aspects, different types of skin diseases trouble patients with their own unique characteristics.

[0004] Currently, dermoscopy is the most commonly used non-invasive diagnostic technique for skin diseases, as it can efficiently and non-invasively expose the characteristics of lesions. Despite its significant advantages, the diverse distribution patterns of skin diseases under a dermoscope and the reliance on highly experienced physicians for differential diagnosis of various subtypes make it difficult to widely implement standardized dermoscopic diagnostic criteria. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an automatic diagnostic method and system for skin diseases based on a multimodal large language model. This method uses an AI system to learn the feature distribution patterns of dermoscopic images to assist in generating diagnostic results.

[0006] According to some embodiments, the present invention adopts the following technical solution: An automatic diagnosis method for skin diseases based on a multimodal large language model includes the following steps: Obtain historical dermoscopy reports from patients and extract the original images and diagnostic text. Build prompt word templates; Construct a multimodal large language model, which includes a visual encoder, a linear projection layer, and a large language model; Construct an image dataset and perform self-supervised visual representation pre-training on the visual encoder of a multimodal large language model in the image dataset; Using the original image and diagnostic text, the visual features generated by the visual encoder based on the original image are aligned with the corresponding prompt word templates to train the linear projection layer; The large language model is trained using aligned visual features and cue word templates; Obtain an external medical knowledge base, extract its text information, and construct a vector database based on the text information using a pre-trained language model; In response to the user's input image and query text, the system uses a pre-trained language model to retrieve the N most similar vectors from the vector database based on the query text, decodes the retrieved vectors into text, and constructs a complete prompt input together with the query text according to the prompt word template. Based on the input image and prompt words, the system uses a trained multimodal large language model to process the data and obtain the final dermoscopy diagnosis report.

[0007] As an alternative implementation method, the process of performing self-supervised visual representation pre-training on the visual encoder of a multimodal large language model in an image dataset includes: generating multiple data-augmented images from any image using a data augmentation strategy, unifying the perspective of the generated images, and normalizing them. Define a teacher model and a student model. The two models have completely identical architectures. The teacher model is initialized with random parameters that follow a Gaussian distribution, and the student model is initialized with the same parameters as the teacher model. The global perspective from the set is input into the teacher model, and the remaining local perspectives are input into the student model. The network parameters of the student model are updated only using stochastic gradient descent in the form of image pairs to optimize the objective function. After one round of training, the weights of the teacher model are updated using the exponential moving average algorithm.

[0008] As a further defined implementation, both the teacher model and the student model are ViT models, with ViT-B / 16 as the backbone network, and the number of Transformer blocks is 12, with 12 self-attention heads in each block.

[0009] As a further defined implementation, the global perspective in the set randomly selects a region in the original image and scales it to a resolution of a specified number of pixels, while the local perspective randomly selects a region in the original image and scales it to a resolution of a set number of pixels. After cropping, all perspectives are randomly flipped horizontally with a 50% probability. The first global view is given a Gaussian blur with a 100% probability, the second global view is given a Gaussian blur with a 10% probability, and regions with pixel values ​​higher than the set value are inverted with a 20% probability; for local views, a Gaussian blur is given with a 50% probability.

[0010] As a further defined implementation, the objective function is:

[0011] Among them, H( || ) is the cross-entropy. For the teacher model, For student models, 、 This provides a global perspective within the set.

[0012] As an alternative implementation, the process of training the linear projection layer by aligning the visual features generated by the visual encoder based on the original image with the corresponding cue word template, using the original image and diagnostic text, includes: aligning the visual features with the corresponding cue word template of the original image. After serialization and segmentation, a flattened two-dimensional image patch sequence is formed, denoted as ,in, Indicates the resolution of the original image. Indicates the number of channels. This indicates the resolution of each image patch. This represents the number of image patches, which are flattened and mapped to a D-dimensional vector space through a linear layer of ViT; Location coding Embedded into the corresponding image blocks to preserve positional information; Image representation of ViT output After passing through a linear projection layer.

[0013] As an alternative implementation, the process of training the large language model using aligned visual features and cue word templates includes: semantic embedding of visual features combined with cue word templates, with all embeddings having the same dimensional shape, and concatenating them from the second dimension and inputting them into the encoder of the large language model to generate the desired text output.

[0014] As an alternative implementation, the process of using a pre-trained language model to retrieve the N most similar vectors in the vector database based on the query text is as follows: the DPR algorithm is used to retrieve the top N most similar vectors between the query vector and the vector database. The number of N is set according to the requirements, and N is a positive integer.

[0015] An automated skin disease diagnosis system based on a multimodal large language model includes: The historical data acquisition module is configured to acquire dermoscopy diagnostic reports of historical patients and extract the original images and diagnostic text from them; The prompt word template building module is configured to build prompt word templates; A multimodal large language model construction module is configured to construct a multimodal large language model, which includes a visual encoder, a linear projection layer, and a large language model. The multimodal large language model training module is configured to construct an image dataset, perform self-supervised visual representation pre-training on the visual encoder of the multimodal large language model in the image dataset; align the visual features generated by the visual encoder based on the original image with the corresponding prompt word template using the original image and diagnostic text, and train the linear projection layer; train the large language model using the aligned visual features and prompt word template. The medical knowledge base injection module is configured to acquire an external medical knowledge base, extract text information from it, and construct a vector database based on the text information using a pre-trained language model. The automatic diagnosis module is configured to respond to the user's input image and query text. It uses a pre-trained language model to retrieve the N most similar vectors in the vector database based on the query text, decodes the retrieved vectors into text, and constructs a complete prompt word input together with the query text according to the prompt word template. Based on the input image and prompt word input, it uses a trained multimodal large language model for processing to obtain the final dermoscopy diagnosis report.

[0016] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method described above.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention utilizes large-scale dermoscopy image data to perform self-supervised pre-training on the visual encoder, thereby improving the visual representation capability of ViT in the vertical domain of dermoscopy. (2) This invention makes full use of image and text data in clinical diagnosis and combines multimodal feature alignment technology to build an AI assistant that can be used for automatic diagnosis of skin diseases; (3) This invention enhances the diagnostic reliability and scientific nature of the model by injecting an authoritative medical knowledge base from outside, and greatly reduces the hallucination output of the large language model.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is a flowchart of an automatic skin disease diagnosis method based on a multimodal large language model, as described in an embodiment of the present invention. Figure 2This is a self-supervised pre-training framework for the visual encoder in this embodiment of the invention; Figure 3 This serves as a prompt word template for training in this embodiment of the invention; Figure 4 This serves as a prompt word template for reasoning in the embodiments of the present invention; Figure 5 This is a flowchart illustrating the workflow of an automated skin disease diagnosis system based on a multimodal large language model, as described in an embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0025] Example 1 In one or more embodiments, an automatic diagnosis method for skin diseases based on a multimodal large language model is disclosed, combining... Figure 1 Specifically, it includes the following processes: S101: Obtain historical dermoscopy diagnostic reports from patients and extract raw images and diagnostic text from them; S102: Implement self-supervised visual representation pre-training on a large-scale dermoscopy image dataset for the visual encoder of a multimodal large language model; The self-supervised pre-training method specifically includes: For any image A multi-crop strategy is used to generate 10 data-augmented images to form a set V. The global viewpoint uses the torchvision library function RandomResizedCrop() to randomly select a region in the original image and scale it to a resolution of 224×224 pixels. The local viewpoint uses a region in the original image to randomly select a region and scale it to a resolution of 96×96 pixels. After cropping, set the function RandomHorizontalFlip(p=0.5) to randomly flip all views horizontally with a 50% probability; Define the GaussianBlur() function and set the parameter p=1 so that the first global viewpoint is subject to Gaussian blur with a 100% probability. The second global view applies a Gaussian blur with a 10% probability and further performs a solarization operation with a 20% probability, i.e., inverting regions with pixel values ​​higher than 128. For the local view, a Gaussian blur is applied with a 50% probability. All generated image views are converted to torch.tensor format and then normalized. The teacher and student models are defined as `VisionTransformer()` classes. Both models have identical architectures, using the ViT-B / 16 standard backbone. The patch size is 16×16, the latent space dimension is 768, and the number of Transformer blocks is 12, with 12 self-attention heads in each block. The model follows a Gaussian distribution. Teacher model initialization with random parameters Student Model The initialization parameters are the same as those of the teacher model; The global perspective in set V ， Input the teacher model, and input the student model from other local perspectives, and combine them. Figure 2 The objective function is optimized by using stochastic gradient descent to update only the network parameters of the student model in the form of image pairs.

[0026] Among them, H( || ) is the cross-entropy; After one round of training, the teacher model weights are updated using the exponential moving average algorithm. :

[0027] in, It is a momentum coefficient with a value range of [0.995, 1]. For student model weights; S103: Training a multimodal large language model using dermoscopy images and diagnostic text; The training process, in detail, is as follows: Input image denoted as After serialization and segmentation, a sequence of flattened two-dimensional image patches is formed, denoted as... ,in, Indicates the resolution of the original image. Indicates the number of channels. This indicates the resolution of each patch. This represents the number of patches. Patches are flattened and mapped to a D-dimensional vector space through a linear layer of ViT. Additionally, positional encoding... According to the formula:

[0028] in, Add it to the patches embedding to preserve positional information. Formula: ，

[0029] The description is that each Transformer encoder layer contains a multi-headed self-attention (MSA) and MLP module. Each module is preceded by a normalization layer to improve data stability, and each module is followed by a residual connection. Image representation of ViT output After passing through a linear projection layer, which is defined using the torch.nn.Linear class, the input size of this layer is... The dimensions are the same, and the output size is the same as the hidden layer size of the large language model DeepSeek-13b-chat; Combination Figure 3 The prompt template shown has the following first part: "###User: The last part asks, "Based on the dermoscopy image, please make a definitive diagnosis of which skin condition is most likely?" The AI ​​assistant then provides semantic embedding using DeepSeek-13b-chat's tokenizer and embedding. The middle part... <imghere>"Replaced by the output of a trainable linear projection layer. All embeddings have a dimensional shape of..." They are then concatenated from the second dimension and fed into the DeepSeek-13b-chat encoder to generate the desired text output; During training, the visual encoder is frozen, and only the linear projection layer and the large language model are trained. The maximum number of training epochs is 15, the batch size is 32, the number of iterations per epoch is set to 2000, the number of warmup steps is set to 2000, and the initial learning rate is set to 1e-5.

[0030] S104: Construct a reasoning framework and use RAG technology to inject an external medical knowledge base into the trained model; Combination Figure 4 , Figure 5 The DPR (Dense passage retrieval) algorithm was used to retrieve the three text segments in the knowledge base that had the highest relevance to the visual-text instructions. Specifically: like Figure 5 As shown, text from an external knowledge base is input into a pre-trained Sentence BERT model to construct a vector database; Input the user's query text into the same pre-trained Sentence BERT model, and use the DPR algorithm to retrieve the top 3 texts with the highest similarity between the query vector and the vector database; Decode the retrieved vectors into text, and then... Figure 4 The inference prompt template shown, together with the user's query text, constructs a complete prompt input; Finally, after concatenating the input prompt with the embedded image, input DeepSeek-13b-chat to generate a diagnostic report.

[0031] In this embodiment, the training prompt word template for the large language model is specifically as follows: "###user: <imghere>Based on the dermoscopy image, please make a definitive diagnosis of which skin condition is most likely present. ###AI Assistant:”; As an alternative implementation, the process of training a multimodal large language model using dermoscopy images and diagnostic text includes: Collect a dermoscopy image-diagnostic text pair dataset, where the images are the visual signals to be parsed and the diagnostic text is the target text signal; Image Input pre-trained visual encoder Then through the projection layer To obtain visual representation :

[0032] in, ; The first part of the prompt template is "###User: "And the last part: Based on the dermoscopy image, please make a definitive diagnosis of which skin condition is most likely?" ###AI Assistant: "Semantic embedding was performed using DeepSeek-13b-chat's tokenizer and embedding to obtain the text representation." At the same time <imghere>Placeholders replaced with ; After concatenating the visual and textual representations, input them into DeepSeek-13b-chat to obtain the diagnostic results: , in, This represents vector concatenation. This refers to the large language model DeepSeek-13b-chat. This represents the text output by the model. During training, the visual encoder is frozen, and the parameters of the linear projection layer and the large language model DeepSeek-13b-chat are optimized using the cross-entropy loss function. .

[0033] The latest expert consensus and diagnostic guidelines for various skin diseases at home and abroad serve as an external medical knowledge base; The DPR (Dense passage retrieval) algorithm is used to retrieve visual-text instructions from the knowledge base. The three most relevant text segments; according to Figure 4 The inference template shown is input into the large language model to obtain the final diagnostic result.

[0034] Example 2 In one or more embodiments, an automated skin disease diagnosis system based on a multimodal large language model includes: The historical data acquisition module is configured to acquire dermoscopy diagnostic reports of historical patients and extract the original images and diagnostic text from them; The prompt word template building module is configured to build prompt word templates; A multimodal large language model construction module is configured to construct a multimodal large language model, which includes a visual encoder, a linear projection layer, and a large language model. The multimodal large language model training module is configured to construct an image dataset, perform self-supervised visual representation pre-training on the visual encoder of the multimodal large language model in the image dataset; align the visual features generated by the visual encoder based on the original image with the corresponding prompt word template using the original image and diagnostic text, and train the linear projection layer; train the large language model using the aligned visual features and prompt word template. The medical knowledge base injection module is configured to acquire an external medical knowledge base, extract text information from it, and construct a vector database based on the text information using a pre-trained language model. The automatic diagnosis module is configured to respond to the user's input image and query text. It uses a pre-trained language model to retrieve the N most similar vectors in the vector database based on the query text, decodes the retrieved vectors into text, and constructs a complete prompt word input together with the query text according to the prompt word template. Based on the input image and prompt word input, it uses a trained multimodal large language model for processing to obtain the final dermoscopy diagnosis report.

[0035] Multimodal large language models include: Visual encoder: Processes input standard dermoscopic images, extracts high-dimensional visual feature representations, and compresses the original image into vectors that can be understood by computers.

[0036] Linear projection layer: Aligns the visual features generated by the visual encoder with canonical diagnostic descriptors, ensuring that the model can understand and associate features in the image with medical definitions or diagnostic criteria for skin diseases.

[0037] Large Language Model: Generates microscopic manifestations and diagnostic results of skin diseases based on aligned visual representations and textual prompts.

[0038] Some embodiments also include: combining external medical knowledge bases such as diagnostic guidelines for various skin diseases under dermoscopy to construct a retrieval augmented generation (RAG) module, where visual representations are processed by a visual encoder and projection layer and input prompts are fed into a text decoder according to a fixed template to generate standardized diagnostic text.

[0039] The specific implementation methods of the above modules are the same as those in Example 1, and will not be described in detail again.

[0040] Example 3 An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps provided in Embodiment 1 above.

[0041] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0042] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0044] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / imghere> < / imghere> < / imghere>

Claims

1. A method for automatic diagnosis of skin diseases based on a multi-modal large language model, characterized in that, The method comprises the following steps: obtaining a dermoscopy diagnosis report of a historical patient, extracting original images and diagnosis texts therefrom; constructing a prompt word template; constructing a multi-modal large language model, the multi-modal large language model comprising a visual encoder, a linear projection layer and a large language model; constructing an image dataset, and performing self-supervised visual representation pre-training on the visual encoder of the multi-modal large language model in the image dataset; aligning visual features generated by the visual encoder based on the original images with corresponding prompt word templates, and training the linear projection layer; training the large language model using the aligned visual features and prompt word templates; obtaining an external medical knowledge base, extracting text information therefrom, and constructing a vector database based on the text information using a pre-trained language model; in response to an input image and query text of a user, retrieving N vectors with the highest similarity in the vector database according to the query text using the pre-trained language model, decoding the retrieved vectors into texts, and constructing a complete prompt word input according to the prompt word template and the query text; processing the input image and the prompt word input using the trained multi-modal large language model to obtain a final dermoscopy diagnosis report.

2. The automatic skin disease diagnosis method based on a multi-modal large language model according to claim 1, characterized in that, The process of performing self-supervised visual representation pre-training on the visual encoder of the multi-modal large language model in the image dataset comprises: generating a set of multiple data-enhanced pictures by adopting a data enhancement strategy on any one image, unifying the generated image perspectives, and performing normalization processing; defining a teacher model and a student model, the two model architectures being completely consistent, initializing the teacher model with random parameters following a Gaussian distribution, and initializing the student model with the same parameters as the teacher model; inputting global perspectives in the set into the teacher model, and inputting the remaining local perspectives into the student model, using random gradient descent to update only the network parameters of the student model to optimize the objective function in the form of an image pair; after one round of training, updating the teacher model weight using an exponential moving average algorithm. 3.The automatic skin disease diagnosis method based on a multi-modal large language model of claim 2, wherein, Both the teacher model and the student model are ViT models, the ViT-B / 16 is used as the backbone network, and the number of layers of the Transformer block is 12, and there are 12 self-attention heads in each block. 4.The automatic skin disease diagnosis method based on a multi-modal large language model of claim 2, wherein, The global perspective in the set randomly selects a region in the original image and scales it to a resolution of a specified pixel, and the local perspective randomly selects a region in the original image and scales it to a resolution of a specified pixel; after cropping, all perspectives are randomly horizontally flipped with a probability of 50%; the first global perspective is applied with a Gaussian blur with a probability of 100%, the second global perspective is applied with a Gaussian blur with a probability of 10%, and further applied with a pixel value inversion on a region with a pixel value higher than a specified value with a probability of 20%; and for the local perspective, a Gaussian blur is applied with a probability of 50%.

5. The automatic skin disease diagnosis method based on a multi-modal large language model according to claim 2, characterized in that, the objective function is where H( || ) is the cross-entropy, is the teacher model, is the student model, 、 is the global perspective in the set.

6. The automatic skin disease diagnosis method based on a multi-modal large language model of claim 1, wherein the multi-modal large language model is a large language model based on a multi-modal dataset, and the multi-modal dataset is a dataset including at least one of a text, an image, a video, and an audio. The process of training the linear projection layer includes: performing serialization segmentation on the original image and the diagnostic text to form a flattened two-dimensional image block sequence, denoted as After the serialization segmentation, a flattened two-dimensional image block sequence is formed, denoted as , wherein, represents the resolution of the original image, represents the number of channels, represents the resolution of each image block, represents the number of image blocks, and the image blocks are flattened and mapped to a D-dimensional vector space through the linear layer of the ViT. Position encoding are embedded into the respective image patch to preserve position information; Image representation output by ViT through a linear projection layer.

7. The automatic skin disease diagnosis method based on a multi-modal large language model of claim 1, wherein the multi-modal large language model is a large language model based on a multi-modal dataset, and the multi-modal dataset is a dataset including at least one of a text, an image, a video, and an audio. The process of training the large language model with the aligned visual features and the prompt template includes: semantic embedding of the visual features combined with the prompt template, consistent dimension shape of all the embedded, splicing them from the second dimension into the encoder of the large language model to generate the required text output. 8.The automatic skin disease diagnosis method based on a multi-modal large language model of claim 1, wherein, The process of retrieving the N vectors with the highest similarity in the vector database according to the query text by using the pre-trained language model is: using the DPR algorithm to retrieve the query vector and the first N vectors with the highest similarity in the vector database, the number of N is set according to the demand, and N is a positive integer. 9.A skin disease automatic diagnosis system based on a multi-modal large language model, characterized in that, The method comprises the following steps: The historical data acquisition module is configured to acquire the dermoscopy diagnosis report of the historical patient, and extract the original image and the diagnosis text therein; The prompt template construction module is configured to construct the prompt template; The multi-modal large language model construction module is configured to construct a multi-modal large language model, which comprises a visual encoder, a linear projection layer and a large language model; The multi-modal large language model training module is configured to construct an image data set, implement self-supervised visual representation pre-training on the visual encoder of the multi-modal large language model in the image data set, align the visual features generated by the visual encoder based on the original image with the corresponding prompt template by using the original image and the diagnosis text, and train the linear projection layer; and train the large language model by using the aligned visual features and the prompt template; The medical knowledge base injection module is configured to acquire an external medical knowledge base, extract text information therein, and construct a vector database by using a pre-trained language model based on the text information. The automatic diagnosis module is configured to respond to the input image and the query text of a user, retrieve N vectors with the highest similarity in the vector database according to the query text by using a pre-trained language model, decode the retrieved vectors into text, and construct a complete prompt input together with the query text according to the prompt template. Based on the input image and the prompt input, the trained multi-modal large language model is used for processing to obtain a final dermoscopy diagnosis report.

10. An electronic device, characterized by comprising: The computer program product comprises a memory and a processor, and computer instructions stored in the memory and running on the processor, and when the computer instructions are run by the processor, the steps in the method of any one of claims 1-8 are completed.

Images