Intelligent question and answer method, device and equipment for skin diseases and storage medium

Abstract

The invention discloses an intelligent question and answer method, device and equipment for skin diseases and a storage medium, and the method comprises the steps: obtaining a skin lesion image and query request information inputted by a user, carrying out the feature extraction of the skin lesion image and the query request information, and obtaining an image feature and a query request feature; retrieving related knowledge from a preset knowledge base based on the query request information, and encoding the related knowledge to obtain knowledge base features; performing multi-modal alignment on the image features, the query request features and the knowledge base features to generate multi-modal features; and performing chain reasoning on the multi-modal features by using a multi-step chain reasoning mechanism to obtain a reasoning result, and generating response information based on the reasoning result. According to the method, the image features, the query request features and the knowledge base features are subjected to multi-modal alignment, and finally a multi-step chain reasoning mechanism is introduced to generate the response information, so that the accuracy and reliability of an output result when the system processes skin disease related tasks are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, and in particular to an intelligent question-and-answer method, device, equipment, and storage medium for skin diseases. Background Technology

[0002] Skin diseases are prevalent, and their diagnosis relies heavily on visual observation and specialized medical knowledge. In recent years, artificial intelligence (AI) technology has been increasingly applied in the medical field, particularly intelligent question-and-answer systems based on natural language processing, which have demonstrated great potential in providing patients with initial disease consultations and health guidance.

[0003] However, when applying general-purpose intelligent question-answering systems to the specific field of dermatology, the accuracy and reliability of their output results still face serious challenges, mainly in the following aspects: 1. Limited information modality and lack of deep understanding of visual details. Most existing systems only interact based on text dialogue or can only perform simple classification of uploaded images, lacking deep feature extraction of lesion images and correlation analysis with text descriptions. 2. Weak reasoning process and lack of logical chains. Many systems adopt an end-to-end direct generation model, lacking a transparent, multi-step logical reasoning process. The differential diagnosis of dermatology often requires a complex reasoning chain (e.g., eliminating possibilities one by one based on the characteristics of skin lesions). This "black box" decision-making process not only reduces the reliability of the results but also makes the results difficult to verify and trust. 3. Simple multimodal information fusion and failure to achieve effective alignment. Even when some systems attempt to combine image and text information, they often remain at the level of simple feature splicing, failing to achieve deep interaction and alignment between visual information, textual intent, and professional knowledge. This results in the model's inability to accurately understand user intent, thus affecting the accuracy of subsequent analysis and responses.

[0004] Therefore, there is an urgent need for an intelligent question-answering method for skin diseases that can improve the accuracy and reliability of the output results when the system is handling skin disease-related tasks. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for intelligent question answering of skin diseases, aiming to solve the technical problem that existing intelligent question answering systems lack accuracy and reliability in outputting results when processing skin disease-related tasks.

[0006] To achieve the above objectives, the present invention provides an intelligent question-and-answer method for skin diseases, the method comprising the following steps: The system acquires a skin lesion image and a query request information input by the user, and performs feature extraction on the skin lesion image and the query request information respectively to obtain image features and query request features; Based on the query request information, relevant knowledge is retrieved from a preset knowledge base, and the relevant knowledge is encoded to obtain knowledge base features; The image features, the query request features, and the knowledge base features are aligned in a multimodal manner to generate multimodal features; A multi-step chain reasoning mechanism is used to perform chain reasoning on the multimodal features to obtain reasoning results, and response information is generated based on the reasoning results.

[0007] Optionally, the step of acquiring the skin lesion image and query request information input by the user, and extracting features from the skin lesion image and the query request information respectively to obtain image features and query request features includes: The system acquires a user-input image of a skin lesion and a query request, the query request including text query information and visual cues. The skin lesion images are used to extract features using a visual encoder to obtain image features; The text query information is encoded using a text encoder to obtain text features; The visual cue information is encoded using a visual cue encoder to obtain visual cue features; The text features and the visual cue features are used as query request features.

[0008] Optionally, the step of encoding the visual cue information using a visual cue encoder to obtain visual cue features includes: When the visual cue information is a point cue information, coordinate mapping is performed on the point cue information and region type embedding is added to generate visual cue features; When the visual cue information is a bounding box cue information, coordinate mapping is performed on the bounding box cue information and corner point type embedding is added to generate visual cue features.

[0009] Optionally, the step of performing multimodal alignment of the image features, the query request features, and the knowledge base features to generate multimodal features includes: The image features, the query request features, and the knowledge base features are concatenated to obtain a multimodal input sequence; The multimodal input sequence and the learnable query vector are input into a cross-attention mechanism for multimodal alignment to generate instruction-guided query embedding; The instruction-guided query is embedded and mapped through a projection layer to generate multimodal features.

[0010] Optionally, the step of using a multi-step chain reasoning mechanism to perform chain reasoning on the multimodal features, obtaining reasoning results, and generating response information based on the reasoning results includes: The inference state is initialized based on the aforementioned multimodal features; Iterative reasoning is performed based on the current reasoning state and the multimodal features to obtain the updated reasoning state; After a preset number of iterations, the updated inference state is taken as the inference result; The reasoning results are input into a preset large language model to generate response information.

[0011] Optionally, the step of inputting the reasoning result into a preset large language model to generate response information includes: The reasoning results are fused with historical dialogue memories to obtain context-enhanced features; The context-enhanced features are input into a preset large language model to generate a natural language response sequence. Based on the natural language response sequence, determine the skin disease concept set and inference chain information corresponding to the natural language response sequence; The natural language response sequence, the set of skin disease concepts, and the inference chain information are used as response information.

[0012] Optionally, the method further includes: Obtain a training dataset, which includes sample images, sample query request information, and corresponding expected response information; The sample image and the sample query request information are input into the model to be trained to obtain the predicted response information; The multi-task joint loss is determined based on the predicted response information and the expected response information. The multi-task joint loss includes text generation loss, ethical boundary loss, dialogue coherence loss, and regularization term loss. The parameters of the model to be trained are optimized based on the multi-task joint loss to obtain a skin disease intelligent question answering model.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes a smart question-and-answer device for skin diseases, the device comprising: The information processing module is used to acquire the skin lesion image and query request information input by the user, and to extract features from the skin lesion image and the query request information respectively to obtain image features and query request features; The knowledge retrieval module is used to retrieve relevant knowledge from a preset knowledge base based on the query request information, and to encode the relevant knowledge to obtain knowledge base features; The feature alignment module is used to perform multimodal alignment of the image features, the query request features, and the knowledge base features to generate multimodal features; The result output module is used to perform chain reasoning on the multimodal features using a multi-step chain reasoning mechanism, obtain reasoning results, and generate response information based on the reasoning results.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a skin disease intelligent question-and-answer device, the device comprising: a memory, a processor, and a skin disease intelligent question-and-answer program stored in the memory and executable on the processor, the skin disease intelligent question-and-answer program being configured to implement the steps of the skin disease intelligent question-and-answer method as described above.

[0015] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a skin disease intelligent question-and-answer program, wherein when the skin disease intelligent question-and-answer program is executed by a processor, it implements the steps of the skin disease intelligent question-and-answer method as described above.

[0016] This invention discloses a method for acquiring user-input images of skin lesions and query request information, extracting features from both the images and the query request information to obtain image features and query request features, respectively; retrieving relevant knowledge from a preset knowledge base based on the query request information and encoding the relevant knowledge to obtain knowledge base features; performing multimodal alignment of the image features, query request features, and knowledge base features to generate multimodal features; and using a multi-step chain reasoning mechanism to perform chain reasoning on the multimodal features to obtain reasoning results, and generating response information based on the reasoning results. Because this invention acquires images of skin lesions and query request information and extracts features from them respectively, then retrieves and encodes knowledge base features based on the query request information, performs multimodal alignment of the image features, query request features, and knowledge base features, and finally introduces a multi-step chain reasoning mechanism to generate response information, compared with existing technologies, this invention effectively improves the accuracy and reliability of the output results when the system processes skin disease-related tasks. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the intelligent question-and-answer method for skin diseases of the present invention; Figure 2 This is a flowchart illustrating the second embodiment of the intelligent question-and-answer method for skin diseases of the present invention; Figure 3 This is a flowchart illustrating the third embodiment of the intelligent question-and-answer method for skin diseases of the present invention; Figure 4 This is a structural block diagram of the first embodiment of the intelligent question-and-answer device for skin diseases of the present invention; Figure 5This is a schematic diagram of the structure of the intelligent question-and-answer device for skin diseases in the hardware operating environment involved in the embodiments of the present invention.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] This invention provides an intelligent question-and-answer method for skin diseases, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the intelligent question-and-answer method for skin diseases of the present invention.

[0021] In this embodiment, the intelligent question-and-answer method for skin diseases includes steps S10 to S40: Step S10: Obtain the skin lesion image and query request information input by the user, and extract features from the skin lesion image and the query request information respectively to obtain image features and query request features.

[0022] It should be noted that the executing entity in this embodiment can be a computer server device with data processing, network communication, and program execution functions applied in a dermatology question-and-answer scenario, such as a server, tablet computer, or personal computer, or an electronic device capable of performing the above functions (such as a smart dermatology question-and-answer device). The following uses a system including a smart dermatology question-and-answer device (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.

[0023] Understandably, the above images of skin lesions This can refer to a two-dimensional digital image of a skin lesion area captured by a user using an image acquisition device (such as a mobile phone, tablet, dermatoscope, or digital camera). The image must contain at least visual information relevant to the diagnosis of skin diseases, including color, texture, shape, edges, size, and distribution. (The above query request information...) This can refer to text content entered by the user in natural language to request conclusions or suggestions related to skin diseases.

[0024] It should be noted that the aforementioned image features can refer to those obtained after feature extraction from the skin lesion image using a visual encoder. For example, in this embodiment, a Vision Transformer (ViT) can be used to capture the global contextual features of the skin lesion image through a self-attention mechanism to obtain image features, as shown in the following formula: ; In the formula, Let I represent the image features, and ViT(·) represent the visual encoder. The visual encoder uses the Vision Transformer architecture to perform forward operations on I, outputting a high-dimensional visual representation vector. (·) denotes the projection layer, used to transform high-dimensional visual representation vectors into image feature vectors of the target dimension (i.e., image features). ).

[0025] It should be noted that the above query request features can refer to the text features obtained by segmenting, embedding, and projecting the text content in the query request information through a text encoder, as shown in the following formula: ; In the formula, Representing text features, This represents the text content in the query request information. (·) indicates a text segmenter, used to segment text into words. The segmentation is divided into sub-words or word sequences and mapped to corresponding word index sequences. (·) denotes the projection layer, used to convert the segmented word index sequence into a text feature vector of the target dimension (i.e., text features). ). Step S20: Based on the query request information, retrieve relevant knowledge from the preset knowledge base, encode the relevant knowledge, and obtain knowledge base features.

[0026] It should be noted that, in order to ensure the professionalism and traceability of subsequent response information, the aforementioned preset knowledge base may refer to a pre-constructed knowledge base for skin diseases. and a daily knowledge base of ethics .

[0027] It is understandable that the aforementioned relevant knowledge may refer to a subset of entries in a preset knowledge base that are related to the current skin lesion image and the user's question intent, obtained by filtering through similarity or semantic matching algorithms based on the query request information.

[0028] Furthermore, the aforementioned knowledge base features can refer to the fixed-dimensional vector representation obtained after the relevant knowledge is vectorized by the knowledge base encoder.

[0029] In practical implementation, the most relevant knowledge to the query request can be selected through a retrieval enhancement module or an explicit memory module and encoded as knowledge base features. This enables the system to generate response information based on accurate professional medical knowledge and strictly comply with ethical standards. ; In the formula, This indicates the query request information. (·) indicates the knowledge base encoder. This represents the characteristics of the knowledge base.

[0030] Step S30: Perform multimodal alignment on the image features, the query request features, and the knowledge base features to generate multimodal features.

[0031] Understandably, skin lesion images, query request information, and knowledge base entries (i.e., related knowledge) exist in heterogeneous representation spaces, and direct concatenation or weighting can lead to semantic inconsistencies and information conflicts. Multimodal alignment maps the three types of features to the same latent space through a cross-attention mechanism, ensuring that visual, linguistic, and knowledge signals maintain consistency in dimensionality and distribution. This provides a unified and compatible input for subsequent chained reasoning, avoiding error accumulation caused by spatial misalignment.

[0032] It should be explained that multimodal features can refer to a single vector sequence obtained after processing with cross-attention and projection layers. Each dimension of the vector carries the visual attributes of skin lesions, user intent, and corresponding disease-specific knowledge. This is used to directly input a multi-step chain reasoning mechanism to achieve a unified representation after cross-modal information fusion.

[0033] It should be noted that the image features, the query request features, and the knowledge base features can be concatenated to obtain a multimodal input sequence; the multimodal input sequence and the learnable query vector are input into a cross-attention mechanism for multimodal alignment to generate an instruction-guided query embedding; the instruction-guided query embedding is mapped through a projection layer to generate multimodal features.

[0034] It should be understood that a multimodal input sequence can refer to a one-dimensional vector sequence formed by sequentially concatenating image features, query request features, and knowledge base features, which serves as the input for subsequent cross-attention mechanisms, carrying visual, linguistic, and knowledge information at once.

[0035] It should be explained that a learnable query vector can refer to a vector that is randomly initialized in the model parameters and continuously updated through backpropagation. It is used to act as a query in the cross-attention mechanism, guiding the attention weights to focus on cross-modal information related to the diagnosis of skin diseases.

[0036] Understandably, the cross-attention mechanism can refer to an attention computation unit that uses a learnable query vector as the Query and a multimodal input sequence as the Key and Value. By calculating the similarity between the Query and the Key to obtain the weight distribution, the Value is weighted and summed to achieve information alignment and fusion between different modalities.

[0037] The calculation of cross-attention can be expressed as: ; In the formula, It is a learnable linear transformation matrix. Let be the dimension of the key vector. These represent Query, Key, and Value, respectively.

[0038] The output is then processed through residual connections, RMSnorm normalization, and feedforward layers, and then stacked. The layer receives the final instruction to guide the query embedding: ; In the formula, Represents a multimodal input sequence. This represents the learnable query vector, and L represents the preset number of cross-attention layers. This indicates that the instruction-guided query embedding is generated at level L. express and It passes through the L-layer cross-attention module in sequence.

[0039] Then, through a projection layer, it is mapped to multimodal features that can be input into a pre-defined large language model: ; In the formula, Represents multimodal features, (·) indicates the projection layer.

[0040] Furthermore, the instruction-guided query embedding can refer to the vector representation output by the cross-attention mechanism, which integrates the visual features of skin lesions, the user's query intent, and the corresponding knowledge base information, and is guided by the learnable query vector to form an embedding for use in subsequent chained reasoning steps.

[0041] Step S40: Use a multi-step chain reasoning mechanism to perform chain reasoning on the multimodal features, obtain reasoning results, and generate response information based on the reasoning results.

[0042] Understandably, a multi-step chain reasoning mechanism can refer to taking multimodal features as initial input, executing several reasoning units sequentially and iteratively. At each step, the current reasoning state interacts with the multimodal features again and updates the state, forming a traceable reasoning chain that is "step by step" until the preset number of iterations is reached.

[0043] It should be understood that the diagnosis of skin diseases requires a multi-factor progressive analysis that integrates visual perception, symptoms, and knowledge. Single-step reasoning is insufficient to depict the clinical thinking from appearance to etiology and then to treatment. Chain reasoning, through layer-by-layer abstraction and verification, reduces the error caused by a one-time decision and improves the accuracy and interpretability of the answer.

[0044] It should be noted that, in order to enable models with a small number of parameters to perform complex inference, a multi-step chain-of-thought mechanism is introduced: Initialize inference state: ; Perform multi-step reasoning:

[0045] go through The result obtained after the second reasoning Input into the preset large language model.

[0046] It should be noted that, Represents multimodal features, Indicates the initial reasoning state. (·) indicates initialization; This represents the reasoning state obtained after i-1 inferences. (·) represents a single learnable reasoning unit, responsible for interacting with the previous state and multimodal features and outputting a new state.

[0047] Understandably, through multiple iterations, visual, linguistic, and knowledge evidence can be gradually integrated from the surface to the core, resulting in more accurate and interpretable reasoning results.

[0048] In a specific implementation, the inference state can be initialized based on the multimodal features; iterative inference can be performed based on the current inference state and the multimodal features to obtain the updated inference state; after a preset number of iterations, the updated inference state can be used as the inference result; the inference result can be input into a preset large language model to generate response information.

[0049] Understandably, the reasoning state can refer to the vector representation used to carry intermediate reasoning information in a multi-step chain reasoning mechanism.

[0050] Furthermore, the reasoning result can refer to the final reasoning state obtained after the last iteration, which condenses cross-modal evidence and intermediate logic and is used to directly drive the subsequent large language model to generate response information.

[0051] It should be added that, in order to ensure traceability and professionalism, the reasoning results can be fused with historical dialogue memory to obtain context-enhanced features; the context-enhanced features are input into a preset large language model to generate a natural language response sequence; based on the natural language response sequence, the skin disease concept set and reasoning chain information corresponding to the natural language response sequence are determined; and the natural language response sequence, the skin disease concept set, and the reasoning chain information are used as response information.

[0052] ; In the formula, R represents the response information. Represents a sequence of natural language responses. This represents a collection of concepts related to specific skin diseases. This represents inference chain information.

[0053] It should be noted that a natural language response sequence can refer to a continuous text string generated by a large language model based on context-enhanced feature decoding, which can be directly displayed to the user. The content may include diagnostic suggestions, explanations, or prompts for the next steps in treating skin lesions.

[0054] It should be understood that the skin disease concept set can refer to the set of medical entities (disease name, symptoms, location, drugs, examination items, etc.) related to the diagnosis and treatment of skin diseases extracted from the natural language response sequence. It is used to quantify the medical keyword coverage of the answer and support subsequent structured storage.

[0055] It should be noted that the reasoning chain information can refer to the sequence of key intermediate nodes recorded by the system during the multi-step chain reasoning process, including the knowledge base entries, visual / textual evidence indexes and logical relationships relied upon by each step, which is used to reproduce the complete path from evidence to conclusion, ensuring the interpretability and verifiability of the response information.

[0056] In the specific implementation, a pre-defined large language model (such as Qwen, Vicuna, or LLaMA) is used to generate a natural language response sequence. It can maintain the contextual memory of multiple rounds of dialogue simultaneously: ; It should be noted that, Indicates the index of the current dialogue round. This represents the dialogue memory vector saved after the (k-1)th round. This represents the query request information provided by the user in the kth round. This represents the response information generated by the system in the k-th round. (·) represents a learnable memory update function, such as a GRU, LSTM, or attention fusion layer, responsible for updating the new dialogue pair ( , Integrate into existing memory and output a new memory vector. The updated It will be used as historical dialogue memory in the next round and integrated with subsequent reasoning results to ensure semantic consistency and referential coherence across multiple rounds of question and answer.

[0057] When generating the next response, the current reasoning result is combined with the historical dialogue. This ensures that the pre-defined large language model maintains semantic consistency and logical self-consistency in multiple rounds of dialogue. .

[0058] It should be noted that, This indicates context-enhanced features. A learnable context fusion function is used to fuse the final inference results. Dialogue with History and Memories Merge into a unified vector.

[0059] This embodiment discloses a method for acquiring a user-input image of a skin lesion and query request information, and extracting features from both the image and the query request information to obtain image features and query request features, respectively. Based on the query request information, relevant knowledge is retrieved from a preset knowledge base and encoded to obtain knowledge base features. The image features, query request features, and knowledge base features are then multimodally aligned to generate multimodal features. A multi-step chain reasoning mechanism is used to perform chain reasoning on the multimodal features to obtain reasoning results, and response information is generated based on these results. Because this embodiment acquires the image of the skin lesion and query request information and extracts features from both, then retrieves and encodes knowledge base features based on the query request information, performs multimodal alignment on the image features, query request features, and knowledge base features, and finally introduces a multi-step chain reasoning mechanism to generate response information, compared to existing technologies, this embodiment effectively improves the accuracy and reliability of the system's output results when processing skin disease-related tasks.

[0060] refer to Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the intelligent question-and-answer method for skin diseases of the present invention.

[0061] Based on the first embodiment described above, in this embodiment, step S10 includes steps S101 to S105: Step S101: Obtain the skin lesion image and query request information input by the user, wherein the query request information includes text query information and visual prompt information.

[0062] Step S102: Extract features from the skin lesion image using a visual encoder to obtain image features.

[0063] Step S103: Encode the text query information using a text encoder to obtain text features.

[0064] Step S104: Encode the visual cue information using a visual cue encoder to obtain visual cue features.

[0065] Step S105: Use the text features and the visual cue features as query request features.

[0066] It should be understood that text query information This can refer to user queries submitted as natural language strings; or visual cues. This could refer to additional signals that users actively mark on images of skin lesions to indicate the areas of skin lesions that the clinic is currently focusing on.

[0067] The visual cue information is encoded using a visual cue encoder to obtain visual cue features, as shown below: ; In the formula, Indicates visual cue features, Indicates visual cues, This indicates a visual cue encoder.

[0068] It should be noted that the step of encoding the visual cue information through a visual cue encoder to obtain visual cue features includes: when the visual cue information is a point cue information, performing coordinate mapping on the point cue information and adding region type embedding to generate visual cue features; when the visual cue information is a bounding box cue information, performing coordinate mapping on the bounding box cue information and adding corner point type embedding to generate visual cue features.

[0069] It should be explained that dot hints refer to the two-dimensional coordinates (x, y) formed by a single click on a skin lesion image, used to mark the location of the lesion of interest. Bounding box hints refer to the coordinates of a bounding box formed by dragging on a skin lesion image, used to delineate the area of ​​skin lesion of interest.

[0070] It should be noted that region type embedding can refer to a learnable vector label used to distinguish between "foreground" and "background" roles; corner point type embedding can refer to another learnable vector label used to label "top left corner" and "bottom right corner" roles respectively.

[0071] In the specific implementation, for point-based tooltips, the coordinates are... Normalization is performed and mapped to the feature space via positional encoding, along with learnable embeddings that distinguish foreground / background. For bounding box cues, the top-left corner is extracted. and bottom right corner The coordinates are also normalized and mapped to the feature space, and learnable embeddings representing the "top left corner" and "bottom right corner" are added respectively.

[0072] This embodiment discloses a method for acquiring a user-input image of a skin lesion and query request information, wherein the query request information includes text query information and visual cues. Feature extraction is performed on the skin lesion image using a visual encoder to obtain image features; the text query information is encoded using a text encoder to obtain text features; the visual cues are encoded using a visual cues encoder to obtain visual cues features; and the text features and visual cues features are used as query request features. Because the query request information in this embodiment includes both text query information and visual cues, compared to existing technologies, this embodiment reduces background skin noise interference and lowers the probability of misdiagnosis.

[0073] refer to Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the intelligent question-and-answer method for skin diseases of the present invention.

[0074] Based on the above embodiments, in this embodiment, the method further includes steps S01 to S04: Step S01: Obtain the training dataset, which includes sample images, sample query request information, and corresponding expected response information.

[0075] Step S02: Input the sample image and the sample query request information into the model to be trained to obtain the predicted response information.

[0076] Step S03: Determine the multi-task joint loss based on the predicted response information and the expected response information. The multi-task joint loss includes text generation loss, ethical boundary loss, dialogue coherence loss, and regularization term loss.

[0077] Step S04: Optimize the parameters of the model to be trained based on the multi-task joint loss to obtain a skin disease intelligent question answering model.

[0078] It should be understood that the aforementioned intelligent question-and-answer model for skin diseases can be a model used by the system in the process of obtaining skin lesion images and query request information input by the user and generating response information. That is, the system's functions can be implemented based on the intelligent question-and-answer model for skin diseases.

[0079] Multi-task joint loss, including text generation loss Loss of ethical boundaries Loss of dialogue coherence And the regularization loss (i.e., the LoRA weight regularization term): ; In the formula, L represents the joint loss of multiple tasks. , , This represents the weighting coefficient, used to balance the contributions of each loss term. This represents the Frobenius norm of the LoRA low-rank fitness matrix, used to constrain the fine-tuning amplitude and prevent catastrophic forgetting.

[0080] LoRA performs low-rank updates on the pre-trained weight matrix: ; In the formula, It is a pre-trained weight matrix that is frozen during training and does not receive gradient updates. Let ΔW = BA, where B and A are trainable matrices; The scaling factor is fixed during training. Update only And by adjusting (If set to 16) Control the fine-tuning range.

[0081] This embodiment discloses the acquisition of a training dataset, which includes sample images, sample query request information, and corresponding expected response information. The sample images and sample query request information are input into the model to be trained to obtain predicted response information. A multi-task joint loss is determined based on the predicted response information and the expected response information. This multi-task joint loss includes text generation loss, ethical boundary loss, dialogue coherence loss, and regularization term loss. The parameters of the model to be trained are optimized according to the multi-task joint loss to obtain a dermatology intelligent question-answering model. Because this embodiment optimizes the parameters of the model to be trained based on the multi-task joint loss to obtain a dermatology intelligent question-answering model, compared to existing technologies, this embodiment improves the accuracy and coherence of the system output results and reduces the misdiagnosis rate.

[0082] Furthermore, this embodiment of the invention also proposes a storage medium storing a skin disease intelligent question-and-answer program, which, when executed by a processor, implements the steps of the skin disease intelligent question-and-answer method described above.

[0083] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the intelligent question-and-answer device for skin diseases of the present invention.

[0084] like Figure 4 As shown, the intelligent question-and-answer device for skin diseases proposed in this embodiment of the invention includes: an information processing module 501, a knowledge retrieval module 502, a feature alignment module 503, and a result output module 504.

[0085] The information processing module 501 is used to acquire the skin lesion image and query request information input by the user, and to extract features from the skin lesion image and the query request information respectively to obtain image features and query request features.

[0086] The knowledge retrieval module 502 is used to retrieve relevant knowledge from a preset knowledge base based on the query request information, and to encode the relevant knowledge to obtain knowledge base features.

[0087] The feature alignment module 503 is used to perform multimodal alignment of the image features, the query request features, and the knowledge base features to generate multimodal features.

[0088] The result output module 504 is used to perform chain reasoning on the multimodal features using a multi-step chain reasoning mechanism, obtain reasoning results, and generate response information based on the reasoning results.

[0089] The feature alignment module 503 is further configured to concatenate the image features, the query request features, and the knowledge base features to obtain a multimodal input sequence; input the multimodal input sequence and the learnable query vector into a cross-attention mechanism for multimodal alignment to generate an instruction-guided query embedding; and map the instruction-guided query embedding through a projection layer to generate multimodal features.

[0090] The result output module 504 is further configured to initialize the inference state based on the multimodal features; perform iterative inference based on the current inference state and the multimodal features to obtain an updated inference state; after a preset number of iterations, use the updated inference state as the inference result; and input the inference result into a preset large language model to generate response information.

[0091] The result output module 504 is further configured to fuse the reasoning result with historical dialogue memory to obtain context-enhanced features; input the context-enhanced features into a preset large language model to generate a natural language response sequence; determine the skin disease concept set and reasoning chain information corresponding to the natural language response sequence based on the natural language response sequence; and use the natural language response sequence, the skin disease concept set, and the reasoning chain information as response information.

[0092] This device embodiment discloses a method for acquiring a user-input image of a skin lesion and query request information, and extracting features from both the image and the query request information to obtain image features and query request features, respectively. Based on the query request information, relevant knowledge is retrieved from a preset knowledge base and encoded to obtain knowledge base features. The image features, query request features, and knowledge base features are then multimodally aligned to generate multimodal features. A multi-step chain reasoning mechanism is used to perform chain reasoning on the multimodal features to obtain a reasoning result, and response information is generated based on the reasoning result. Because this device embodiment acquires the image of the skin lesion and query request information and extracts features from both, then retrieves and encodes knowledge base features based on the query request information, performs multimodal alignment of the image features, query request features, and knowledge base features, and finally introduces a multi-step chain reasoning mechanism to generate response information, compared to existing technologies, this device embodiment effectively improves the accuracy and reliability of the output results when processing skin disease-related tasks.

[0093] Based on the first embodiment of the intelligent question-and-answer device for skin diseases described above, a second embodiment of the intelligent question-and-answer device for skin diseases of the present invention is proposed.

[0094] In this embodiment, the information processing module 501 is further configured to acquire a skin lesion image and query request information input by the user, wherein the query request information includes text query information and visual cue information; extract features from the skin lesion image using a visual encoder to obtain image features; encode the text query information using a text encoder to obtain text features; encode the visual cue information using a visual cue encoder to obtain visual cue features; and use the text features and the visual cue features as query request features.

[0095] The information processing module 501 is further configured to, when the visual prompt information is a point prompt information, perform coordinate mapping on the point prompt information and add region type embedding to generate visual prompt features; and when the visual prompt information is a bounding box prompt information, perform coordinate mapping on the bounding box prompt information and add corner point type embedding to generate visual prompt features.

[0096] Other embodiments or specific implementations of the intelligent question-and-answer device for skin diseases of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0097] This application provides a smart question-and-answer device for skin diseases, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the smart question-and-answer method for skin diseases in the first embodiment described above.

[0098] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the intelligent question-and-answer device for skin diseases in the embodiments of this application. The intelligent question-and-answer device for skin diseases in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The intelligent question-and-answer device for skin diseases shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0099] like Figure 5 As shown, the intelligent question-and-answer device for dermatology may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the intelligent question-and-answer device for dermatology. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the intelligent dermatology question-and-answer device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show intelligent dermatology question-and-answer devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0100] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0101] The intelligent question-and-answer device for skin diseases provided in this application, employing the intelligent question-and-answer method for skin diseases described in the above embodiments, can solve the technical problem that existing intelligent question-and-answer systems lack accuracy and reliability in outputting results when processing skin disease-related tasks. Compared with the prior art, the beneficial effects of the intelligent question-and-answer device for skin diseases provided in this application are the same as those of the intelligent question-and-answer method for skin diseases provided in the above embodiments, and other technical features in this intelligent question-and-answer device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0102] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0103] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0105] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0107] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the protection scope of the present invention.

Claims

1. A smart question-and-answer method for skin diseases, characterized in that, The method includes: The system acquires a skin lesion image and a query request information input by the user, and performs feature extraction on the skin lesion image and the query request information respectively to obtain image features and query request features; Based on the query request information, relevant knowledge is retrieved from a preset knowledge base, and the relevant knowledge is encoded to obtain knowledge base features; The image features, the query request features, and the knowledge base features are aligned in a multimodal manner to generate multimodal features; A multi-step chain reasoning mechanism is used to perform chain reasoning on the multimodal features to obtain reasoning results, and response information is generated based on the reasoning results.

2. The intelligent question-and-answer method for skin diseases as described in claim 1, characterized in that, The steps of acquiring the skin lesion image and query request information input by the user, and extracting features from the skin lesion image and the query request information respectively to obtain image features and query request features include: The system acquires a user-input image of a skin lesion and a query request, the query request including text query information and visual cues. The skin lesion images are used to extract features using a visual encoder to obtain image features; The text query information is encoded using a text encoder to obtain text features; The visual cue information is encoded using a visual cue encoder to obtain visual cue features; The text features and the visual cue features are used as query request features.

3. The intelligent question-and-answer method for skin diseases as described in claim 2, characterized in that, The step of encoding the visual cue information using a visual cue encoder to obtain visual cue features includes: When the visual cue information is a point cue information, coordinate mapping is performed on the point cue information and region type embedding is added to generate visual cue features; When the visual cue information is a bounding box cue information, coordinate mapping is performed on the bounding box cue information and corner point type embedding is added to generate visual cue features.

4. The intelligent question-and-answer method for skin diseases as described in claim 1, characterized in that, The step of performing multimodal alignment of the image features, the query request features, and the knowledge base features to generate multimodal features includes: The image features, the query request features, and the knowledge base features are concatenated to obtain a multimodal input sequence; The multimodal input sequence and the learnable query vector are input into a cross-attention mechanism for multimodal alignment to generate instruction-guided query embedding; The instruction-guided query is embedded and mapped through a projection layer to generate multimodal features.

5. The intelligent question-and-answer method for skin diseases as described in claim 1, characterized in that, The step of using a multi-step chained reasoning mechanism to perform chained reasoning on the multimodal features, obtaining reasoning results, and generating response information based on the reasoning results includes: The inference state is initialized based on the aforementioned multimodal features; Iterative reasoning is performed based on the current reasoning state and the multimodal features to obtain the updated reasoning state; After a preset number of iterations, the updated inference state is taken as the inference result; The reasoning results are input into a preset large language model to generate response information.

6. The intelligent question-and-answer method for skin diseases as described in claim 5, characterized in that, The step of inputting the reasoning result into a preset large language model to generate response information includes: The reasoning results are fused with historical dialogue memories to obtain context-enhanced features; The context-enhanced features are input into a preset large language model to generate a natural language response sequence. Based on the natural language response sequence, determine the skin disease concept set and inference chain information corresponding to the natural language response sequence; The natural language response sequence, the set of skin disease concepts, and the inference chain information are used as response information.

7. The intelligent question-and-answer method for skin diseases as described in claim 1, characterized in that, The method further includes: Obtain a training dataset, which includes sample images, sample query request information, and corresponding expected response information; The sample image and the sample query request information are input into the model to be trained to obtain the predicted response information; The multi-task joint loss is determined based on the predicted response information and the expected response information. The multi-task joint loss includes text generation loss, ethical boundary loss, dialogue coherence loss, and regularization term loss. The parameters of the model to be trained are optimized based on the multi-task joint loss to obtain a skin disease intelligent question answering model.

8. A smart question-and-answer device for skin diseases, characterized in that, The device includes: The information processing module is used to acquire the skin lesion image and query request information input by the user, and to extract features from the skin lesion image and the query request information respectively to obtain image features and query request features; The knowledge retrieval module is used to retrieve relevant knowledge from a preset knowledge base based on the query request information, and to encode the relevant knowledge to obtain knowledge base features; The feature alignment module is used to perform multimodal alignment of the image features, the query request features, and the knowledge base features to generate multimodal features; The result output module is used to perform chain reasoning on the multimodal features using a multi-step chain reasoning mechanism, obtain reasoning results, and generate response information based on the reasoning results.

9. A smart question-and-answer device for skin diseases, characterized in that, The device includes: a memory, a processor, and a skin disease intelligent question-and-answer program stored in the memory and executable on the processor, the skin disease intelligent question-and-answer program being configured to implement the steps of the skin disease intelligent question-and-answer method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a skin disease intelligent question-and-answer program, which, when executed by a processor, implements the steps of the skin disease intelligent question-and-answer method as described in any one of claims 1 to 7.

Images