A Medical Problem Consultation System and Method Based on a Large Model

Abstract

This invention discloses a medical problem consultation system and method based on a large model. It includes a dialogue consultation module that processes input text and inputs it into a finely tuned large model for inference, outputting medical advice; a professional knowledge base module that processes input text and documents and inputs them into a framework based on the large model for further processing, outputting medical advice and related professional knowledge; an image analysis module that receives medical images and inputs them into an image classification model to obtain the corresponding disease name, then inputs the disease name into the finely tuned large model, which generates corresponding medical knowledge based on the disease name; and an image classification model that receives the input medical images, performs feature extraction, fusion, weighting, and classification to obtain the corresponding disease name. This system can simultaneously address the problems of young people lacking time for in-person consultations and the elderly lacking awareness of early disease symptoms, enabling early prevention, early detection, early diagnosis, and early treatment of diseases.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, and in particular to a medical problem consultation system and method based on a large model. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In recent years, large-scale models tailored to specific domains have made significant progress across multiple industries. Fields such as medicine, industrial manufacturing, power, and transportation have begun leveraging large-scale models for data-driven optimization and innovation. Domain-specific large-scale models not only possess the reasoning and generation capabilities of general-purpose large-scale models but also integrate domain-specific knowledge and data, enabling them to provide more accurate predictions and more professional decision support.

[0004] On the other hand, many elderly people lack awareness of the early symptoms of diseases and easily overlook abnormal changes in their bodies. This blind spot in health leads to some diseases not being effectively diagnosed and treated in their early stages, thus missing the optimal intervention time. This trend has sparked widespread social attention to health issues and early disease prevention, especially among elderly people living alone and young people under heavy pressure, where early prevention and timely health management are particularly important.

[0005] To address these issues, measures have been implemented such as strengthening primary healthcare services, promoting tiered medical services, developing elderly health services, and extending medical services to homes and communities. However, a satisfactory solution remains lacking for enabling residents to conveniently and quickly access medical consultations and determine the cause of their illnesses, resulting in insufficient awareness among the elderly regarding early disease symptoms. While large-scale medical models exist, these models are generally designed to assist doctors or those with medical expertise in diagnosis and cannot accurately interpret descriptive texts from non-medical users. This leads to lower accuracy in disease diagnosis and a lack of understanding of the disease, which can seriously endanger lives. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, this invention provides a medical problem consultation system and method based on a large model, which can simultaneously solve the problems of young people not having time for offline consultations and the elderly lacking awareness of early symptoms of diseases, thereby achieving early prevention, early detection, early diagnosis and early treatment of diseases.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] In a first aspect, the present invention provides a medical problem consultation system based on a large model, comprising:

[0009] The data acquisition module is used to acquire user input text, documents, and medical images;

[0010] The dialogue consultation module is used to receive input text, process the input text, and then input it into a finely tuned large model for model inference, and output medical suggestions.

[0011] The professional knowledge base module is used to receive input text and documents, process the input text and documents, and then input them into a framework based on a large model for further processing, outputting medical advice and related professional knowledge;

[0012] The affected area image analysis module is used to receive medical images, input them into an image classification model to obtain the corresponding disease name, input the disease name into a fine-tuned large model, and the large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical images, performs feature extraction, fusion, weighting and classification on the medical images, and obtains the corresponding disease name.

[0013] A further technical solution involves processing the input text and then feeding it into a finely tuned large model for inference. Specifically:

[0014] The input text is tokenized and divided into multiple sub-units;

[0015] The segmented units are converted into numerical values, and each unit is converted into a word vector using an embedding matrix.

[0016] The input text is positionally encoded, and the positional encoding is combined with word vectors and then fed into a finely tuned large model to output text that the user can understand.

[0017] Further technical solutions, based on a large model framework, include the LangChain framework, which comprises a document loader, document indexing and retrieval, and chains.

[0018] A further technical solution is that the framework based on the large model outputs medical suggestions and related professional knowledge, specifically as follows:

[0019] Convert documents into a word vector database and convert input text into word vectors;

[0020] Calculate the spatial similarity between the word vectors and the sentences in the word vector database to obtain a similarity score;

[0021] The word vectors are combined with the paragraphs with the highest similarity scores and input into a large model to output medical advice and related professional knowledge.

[0022] In a further technical solution, the image classification model sequentially includes an input module, a feature extraction module, a feature fusion module, an SEBlock module, and a classification head.

[0023] A further technical solution is that the image processing process in the image classification module is as follows:

[0024] The input module receives the input medical image and feeds it into the feature extraction module. The feature extraction module includes two pre-trained models, VGG16 and ResNet50, which extract low-level and high-level features respectively. The extracted features are then fed into the feature fusion module. The feature fusion module receives the features extracted by VGG16 and ResNet50 respectively, concatenates them along the channel dimension to form a fused feature, and feeds the fused feature into the SEBlock module. The SEBlock module weights the fused feature to generate a channel weight vector, weights the features of each channel based on the channel weight vector to highlight important features, outputs the weighted fused feature, and feeds it into the classification head. The classification head obtains the disease name corresponding to the medical image based on the fused feature.

[0025] A further technical solution, the specific steps in the SEBlock module are as follows:

[0026] Receive the fused features and apply adaptive average pooling to compress their spatial dimensions;

[0027] The number of channels in the fused features is reduced by convolution, and non-linear activation is performed by an activation function.

[0028] The number of channels for the fused features is restored again through convolution, and a weight vector for each channel is generated through a function.

[0029] The fused features are multiplied by the weight vector channel by channel to obtain the weighted fused features.

[0030] Secondly, this invention provides a medical problem consultation method based on a large model, comprising:

[0031] Acquire user input text, documents, and medical images;

[0032] The system uses a dialogue consultation module to receive input text, processes the text, and then inputs it into a finely tuned large model for model inference, outputting medical suggestions.

[0033] The system uses a knowledge base module to receive input text and documents, processes the input text and documents, and then inputs them into a framework based on a large model for further processing, outputting medical advice and related professional knowledge.

[0034] The system uses an image analysis module to receive medical images, inputs them into an image classification model to obtain the corresponding disease name, and inputs the disease name into a fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name. The image classification model receives the input medical images, performs feature extraction, fusion, weighting, and classification on the medical images to obtain the corresponding disease name.

[0035] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a large-model-based medical problem consultation method as described in the second aspect.

[0036] Fourthly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a large-model-based medical problem consultation method as described in the second aspect.

[0037] The above one or more technical solutions have the following beneficial effects:

[0038] This invention focuses on residents' medical consultation needs, fine-tuning a recently released open-source large-scale model for the symptoms and treatments of common diseases, aiming to enable the application of this large-scale model in medical consultation scenarios. Through a dialogue consultation module, a professional knowledge base module, and a affected area image analysis module, it can provide patients with accurate and reliable treatment suggestions, meeting personalized medical needs. These modules are implemented based on the fine-tuned large-scale model, using a large amount of open-source medical datasets containing real patient-doctor dialogues and open-source Chinese medical dialogue datasets to refine the model. This makes the large-scale model more capable of understanding non-professional descriptions of symptoms, improving the accuracy and reliability of disease diagnosis.

[0039] In this invention, the dialogue consultation module is implemented through a fine-tuned large model. This large model, fine-tuned with a professional dataset, enables it to provide professional and accurate advice when conversing with patients, meeting the high standards required for medical consultation. The professional knowledge base module is implemented through a framework based on the large model. When answering user questions, the model not only relies on its own built-in knowledge base but also constructs an external knowledge base from user-uploaded files. After construction, the framework integrates internal and external information and provides it as output to the large model. The model will then provide more accurate and relevant answers to the user, along with original text materials for easy reading and comparison. The affected area image analysis module employs a dual-model analysis strategy. Based on the image classification results and the fine-tuned dataset, it comprehensively analyzes the condition, generates accurate judgment results, and proposes corresponding treatment suggestions, thereby helping patients better understand their condition and choose appropriate treatment plans.

[0040] Medical responses have a very low tolerance for large-scale model hallucinations. To address the hallucination problem caused by large-scale models, this invention employs a dual-model analysis strategy in the affected area image analysis module. It uses a concatenated approach of an image classification model and a finely tuned GLM4 model. The finely tuned GLM4 model outputs accurate knowledge based on the classification results of the image classification model, improving the accuracy of the output knowledge and preventing hallucination phenomena. Attached Figure Description

[0041] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0042] Figure 1 This is a flowchart of the medical problem consultation system according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of the professional knowledge base module in the medical problem consultation system of this invention.

[0044] Figure 3 This is a flowchart of the affected area image analysis module in the medical problem consultation system of this invention.

[0045] Figure 4 This is a flowchart of the dual-model analysis strategy in the affected area image analysis module of this invention.

[0046] Figure 5 This is an example diagram of the interface of the dialogue consultation module in the medical problem consultation system of this invention.

[0047] Figure 6 This is an example diagram of the interface of the professional knowledge base module in the medical problem consultation system of this invention.

[0048] Figure 7 This is an example diagram of the relevant context in the professional knowledge base module of the medical problem consultation system of this invention;

[0049] Figure 8 This is an example diagram of the interface of the affected area image analysis module in the medical problem consultation system of this invention. Detailed Implementation

[0050] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0051] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0053] Terminology Explanation: Lora: Fine-tuning method; LLM: Large Language Model; LangChain: LLM open-source framework.

[0054] This invention addresses the issues of young people lacking time for in-person medical consultations and the elderly's insufficient understanding of early disease symptoms by designing a medical consultation system to achieve early prevention, early detection, early diagnosis, and early treatment of diseases.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment discloses a medical problem consultation system based on a large model, including:

[0057] The data acquisition module is used to acquire user input text, documents, and medical images.

[0058] The dialogue consultation module receives input text, processes it, and then inputs it into a finely tuned large model for model inference, outputting medical suggestions.

[0059] In this embodiment, the dialogue consultation module is implemented by fine-tuning a large model, using an open-source medical dataset containing 1 million real patient-doctor dialogues, while incorporating professional drug function explanations. This module is based on the Zhipu open-source large model ChatGLM4, and after fine-tuning with a professional dataset, the large model can provide professional and accurate suggestions when conversing with patients, meeting the high standards required for medical consultation.

[0060] Specifically, the large-scale model used in the dialogue consultation module adopts the ChatGLM4-9B-Chat open-source large-scale model, and the training dataset used is the open-source Chinese medical dialogue dataset. This dataset covers multiple fields such as internal medicine, andrology, obstetrics and gynecology, oncology, pediatrics, and surgery, and contains rich professional medical knowledge, which can effectively support the medical dialogue and consultation needs of different departments.

[0061] Specifically, the fine-tuning process of the large model was optimized using the LLaMA-Factory framework. This framework provides a flexible and efficient fine-tuning mechanism that can accelerate model training and improve the model's performance in specific domain tasks while ensuring performance.

[0062] Specifically, the fine-tuning technique uses the Lora fine-tuning method. Compared with p-tuning, Lora fine-tuning shows a more significant advantage in accelerating training, achieving a 3.7-fold acceleration effect and significantly improving training efficiency and resource utilization.

[0063] The specific steps for the large model to perform model inference on the input text in the dialogue consultation module are as follows:

[0064] (1) Tokenize the input text and add some special tags to help the large model understand the text structure. Tokenization involves dividing the text into multiple sub-unit tokens.

[0065] (2) Convert the tokens of the segmented units into numerical values, and use the EmbeddingMatrix to convert each token into a word vector.

[0066] (3) To enable large models to understand the relative positions of text in a sequence, positional encoding is added.

[0067] (4) Combine word vectors and positional encodings and input them into the large model.

[0068] (5) The large model underwent LoRa fine-tuning, which changed the query weight matrix w in the transformer architecture. q The weight matrix w of the key k The weight matrix w of the sum v w q w k w v In the transformer architecture, weight matrices are used to map input vectors to query (Q), key (K), and value (V) spaces. They help the transformer compute weights and outputs in the self-attention mechanism. The model trains these weight matrices to capture the relationships between different elements in a sequence.

[0069] (6) After passing through multiple transformer layers, the self-attention mechanism extracts the merged word vectors and position-encoded contextual features. Finally, the large-scale dialogue diagnostic model outputs a result vector, which is the feature representation of each sub-unit token. Specifically, the self-attention mechanism encodes and understands each token (word or sub-unit) and its position in the input text through multiple stacked layers, thereby capturing contextual information. The transformer model calculates query (Q), key (K), and value (V) vectors for each token, and then calculates the similarity between them to obtain attention weights. Each token is weighted according to its similarity to other tokens, resulting in a new representation. This new representation is the contextual feature, which incorporates the relationship between the token and other tokens in the entire sequence.

[0070] (7) Finally, based on the feature representation of the sub-unit token, output text that the user can understand. The feature representation of the token is: after passing through the transformer, the model generates a high-dimensional feature vector for the input token. For example, "I" will become a feature vector of [0.12, -0.04, 0.22, 0.05, ..., 0.19]. These feature representations are numerical vectors and are not directly understandable to humans. In order for the model to generate an understandable answer, these feature representations must be decoded into natural language text, such as "I".

[0071] The process is as follows: First, the feature vector of each token passes through a linear layer, which transforms the feature vector into a score representing the vocabulary dimension. These scores indicate the probability that the token will generate each word in the vocabulary. The resulting scores are then transformed into a probability distribution using a softmax function, representing the probability of each word being the output of the current token. Finally, using a predefined sampling strategy, the word with the highest probability is selected as the output, resulting in natural language.

[0072] The professional knowledge base module receives input text and documents, processes them, and then inputs them into a framework based on a large model for further processing, outputting medical advice and related professional knowledge.

[0073] In this embodiment, the professional knowledge base module aims to support users who need a deeper understanding of their condition. Since the large model fine-tuning dataset mainly covers the disease consultation process, users who wish to gain a more comprehensive understanding of disease mechanisms and other professional knowledge can upload relevant medical books. This module adopts a framework based on the large model (i.e., the LangChain framework), supporting the large model in answering not only user questions but also professional knowledge questions, providing relevant professional knowledge for medical advice, extracting relevant explanations from professional books, and attaching original text materials for easy reading and comparison by users.

[0074] Specifically, the professional knowledge base module is built using the Langchain framework. First, the system loads medical-related files (i.e., user-input documents) locally and converts them into a processable text format using an unstructured loader. Second, a text segmenter breaks down longer medical texts into smaller text blocks for more efficient processing and analysis. Third, a vector embedding is generated for each text block and stored in a vector library to support fast retrieval and matching. Fourth, when a user submits a medical query, the system converts the query into a vector embedding and performs similarity matching with medical text blocks in the vector library to accurately extract relevant content. Fifth, the system inputs the extracted medical content into a large language model for reasoning using a prompt template. Sixth, the large model finally generates a professional medical answer and returns it to the user.

[0075] The LangChain framework consists of a document loader, document indexing and retrieval, and chains. The document loader loads document data from various data sources (such as text files, web pages, and databases), which typically contain domain-specific knowledge. Indexing and retrieving document content enables efficient extraction of relevant information from large-scale documents, supporting the model. This often involves using Retrieval Augmentation (RAG) techniques. In LangChain, chains are components that sequentially combine multiple tasks (such as document retrieval, data processing, and model inference). These chains enable complex workflows, such as extracting knowledge from documents and combining it with a large language model to generate answers. The large model, combined with LangChain framework technology, enables the reading and searching of user-uploaded files.

[0076] The input text (i.e., the professional knowledge question text) and the document are input into the Langchain framework for processing, specifically as follows:

[0077] (1) Using the LangChain framework, user-uploaded documents are converted into a word vector database, and professional knowledge question texts (input texts) are converted into word vectors. Subsequently, the LangChain framework judges the spatial similarity between the professional knowledge question texts and the statements in the word vector database, and calculates the score to obtain a similarity score. The spatial similarity adopts cosine similarity, which calculates the cosine value of the angle between two vectors. The closer the value is to 1, the more similar the two vectors are; the closer the value is to -1, the less similar the two vectors are.

[0078] (2) Subsequently, the LangChain framework combines the word vectors converted from the professional knowledge question text with the highest-scoring paragraphs in the word vector database to generate a new prompt as the input of the ChatGLM4 big model; the big model outputs medical advice and related professional knowledge based on this input.

[0079] Using the aforementioned technical solution, the system can read professional knowledge documents uploaded by patients and extract relevant medical information through analysis. Based on this, the system identifies the specific knowledge points needed by the user. This process, through precise model fitting and optimization, further improves the accuracy of the information, making the system's final output more professional and targeted than the original large model. This improvement enables users to quickly and accurately obtain the medical knowledge that meets their needs, providing stronger support for clinical consultation and self-diagnosis.

[0080] The affected area image analysis module is used to receive medical images, input them into an image classification model to obtain the corresponding disease name, input the disease name into a fine-tuned large model, and the large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical images, performs feature extraction, fusion, weighting and classification on the medical images, and obtains the corresponding disease name.

[0081] In this embodiment, if medical images are available, including images of affected areas (such as photos of skin lesions), CT images, etc., the user can identify and analyze them using the affected area image analysis module. This module first uses a combination of ResNet and VGG models to build and train an image classification model specifically designed to identify common medical image types, accurately distinguishing different lesion categories, i.e., disease names. Then, combining the capabilities of a large language model, based on the image classification results and a fine-tuned dataset, it comprehensively analyzes the condition, generating accurate judgment results and proposing corresponding treatment suggestions, thereby helping patients better understand their condition and choose appropriate treatment plans.

[0082] The affected area image analysis module employs a dual-model analysis strategy, combining the image classification model Fusion with a large model (a finely tuned ChatGLM4) to achieve efficient image content analysis and knowledge output. The Fusion image classification model identifies user-submitted medical images, parsing the image information into text descriptions. For example, when detecting skin tumor images, the Fusion model recognizes the image and outputs the corresponding tumor name, such as "melanoma." The ChatGLM4 large model receives the text descriptions output by the Fusion model as input, combines the input with a set prompt, and generates finely tuned professional knowledge output, providing users with more in-depth analysis results.

[0083] Fusion, an image classification model, is an improved version of ResNet. By combining ResNet with VGG and incorporating an attention mechanism, it significantly enhances image recognition capabilities. Training on the HAM skin cancer dataset, Fusion achieved 99% accuracy within 20 epochs. In contrast, the traditional ResNet model experienced accuracy fluctuations between epochs 11 and 13, leading to premature training termination. Fusion was trained on publicly available datasets and tested on the MNIST and HAM10000 datasets to ensure its accuracy in recognizing skin lesion images.

[0084] VGG and ResNet models each have their strengths in image feature extraction. VGG excels at extracting low-to-mid-level detail features, while ResNet focuses more on high-level semantic features. By combining the advantages of both, the image fusion model of this invention uses two modules to extract different features from the image and concatenates these features along the channel dimension to generate a fused feature map. Subsequently, the fused feature map is input into the SEBlock module, which generates weights for each channel through a channel weighting mechanism to measure its importance in the final classification task. The weighted and optimized fused features are then input into the classification head to progressively reduce the dimensionality of the high-dimensional features to the class space, ultimately outputting the confidence score for each class.

[0085] The image classification model comprises an input module, a feature extraction module, a feature fusion module, an SEBlock module, and a classification head. The input module receives the input feature map, i.e., the input medical image, and feeds it into the feature extraction module. The feature extraction module includes two pre-trained models, VGG16 and ResNet50, serving as low-level and high-level feature extractors respectively, and inputs the extracted features into the feature fusion module. The feature fusion module receives the features extracted by VGG16 and ResNet50 respectively, concatenates them along the channel dimension to form a fused feature, and inputs the fused feature into the SEBlock module. The SEBlock module weights the fused feature, generates a channel weight vector, and weights the features of each channel based on the channel weight vector, highlighting important features, outputting the weighted fused feature, and inputting it into the classification head. The classification head completes the final classification task based on the fused feature, obtaining the disease name corresponding to the input medical image.

[0086] The specific processing flow of the input image (i.e., the medical image submitted by the user) in the image classification model is as follows:

[0087] (1) Input image I is sequentially processed by the first 16 layers of VGG and ResNet for feature extraction, generating two feature maps F. VGG ∈R 512*H*W With F Resnet ∈R2048*H*W .

[0088] (2) The two feature maps are adjusted to have the same spatial dimension R through adaptive average pooling. C*1*1 The features are then concatenated along the dimensional lines to obtain the fused feature map X∈R. 2560*1*1 .

[0089] (3) The feature X is input into the SEBlock module to generate a weight vector W∈R. 2560 The enhanced features are obtained by weighting each channel. The enhanced features, i.e. the weighted fusion features, are input into the classification head. After adaptive average pooling, flattening, the first fully connected layer and the second fully connected layer, the final output is the category prediction result, i.e. the disease name corresponding to the input medical image.

[0090] Furthermore, the feature extraction module includes two pre-trained models: VGG16 and ResNet50, serving as low-level and high-level feature extractors, respectively. VGG16 extracts features from the first 16 layers, with an output of 516 channels; ResNet50 removes the last two layers to extract high-level features, with an output of 2048 channels. These two feature sets undergo adaptive average pooling to unify the spatial scale to 1*1, facilitating feature fusion.

[0091] Furthermore, the feature fusion module receives features extracted by VGG16 and ResNet50 respectively, and concatenates them along the channel dimension to form a fused feature. The number of channels in the fused feature is 2560.

[0092] Furthermore, the SEBlock module: The SEBlock module is used to weight the fused features, initially reducing the number of channels to 160, then restoring it to 2560, and generating a channel weight vector. This weight vector weights the features of each channel, highlighting important features.

[0093] The specific steps in the SEBlock module are as follows:

[0094] (1) Global Feature Convergence: The module first performs fusion on the feature map X∈R. C*H*W Applying adaptive average pooling, its spatial dimension is compressed to R. C*1*1 This yields global feature representations for each channel, helping the model obtain a more comprehensive image representation.

[0095] (2) Channel dimensionality reduction: The number of channels for the fused features is reduced from C to 1*1 through 1*1 convolution. (r stands for reductionradio, which defaults to 16), reducing parameters and computational load.

[0096] (3) Nonlinear activation: The ReLU activation function is introduced to introduce nonlinearity.

[0097] (4) Channel dimension upgrade: The number of channels of the fused features is restored to C again by 1*1 convolution, and the weight of each channel is generated by the sigmoid function.

[0098] (5) Feature recalibration: The input fused feature map is multiplied channel by channel with the weight vector to obtain the weighted fused feature map. The formula is expressed as:

[0099] X SE =X·σ(Conv2d(ReLU(Conv2d(Avgpool(X))))).

[0100] Furthermore, the classification head: The structure of the classification head includes adaptive average pooling, a flattening operation, a first fully connected layer, and a second fully connected layer. Adaptive average pooling further compresses the features to 1*1; the flattening operation flattens the features into a one-dimensional vector; the first fully connected layer maps the number of channels 2560 to 1024 and applies ReLU activation and Dropout (with a probability of 0.5); the second fully connected layer maps 1024 to the number of target categories.

[0101] Medical responses have a very low tolerance for hallucinations in large-scale models; existing large-scale models cannot completely eliminate hallucinations if they directly analyze images. To address this issue, the affected area image analysis module uses a concatenated approach of a Fusion model and a finely tuned GLM4 model. The Fusion model performs image recognition and classification, while the finely tuned GLM4 model outputs accurate knowledge based on the classification results from the Fusion model. The affected area image analysis module first requires training on datasets. System testing uses the MNIST and HAM1000 datasets, and after training, Fusion can maintain an accuracy of over 90% on these datasets. When fine-tuning the GLM4 model, information related to the classification results of the Fusion model needs to be prepared to enhance the large-scale model's understanding and generation of medical knowledge. For example, for skin diseases, information such as basal cell carcinoma, dermatofibroma, and melanoma needs to be prepared. After the Fusion model recognizes the user's medical image, it returns the classification result to the GLM4 model. The classification result input and prompt are then submitted together to the large-scale model. The large model can ultimately output accurate knowledge based on classification results, including a disease overview, causes, manifestations, and treatment methods, without revealing hallucinations.

[0102] The large-scale model receives the output information (disease name) from the image classification model Fusion, using it as input. Combined with a pre-set prompt, it generates finely tuned professional knowledge output, providing precise knowledge to the user. This knowledge includes a brief introduction to the disease, its causes, manifestations, and treatment methods. Through this collaborative approach, the image classification model handles the structured processing of image information, while the large-scale model further utilizes semantic understanding and professional knowledge reasoning to provide users with detailed and accurate interpretations. For example, in the scenario of detecting skin tumors, the large-scale model can use the name "melanoma" output by the image classification model, combined with finely tuned medical knowledge, to provide users with professional information such as tumor characteristics and treatment suggestions, thus achieving a seamless integration of image analysis and knowledge services.

[0103] The large-scale model generates medical knowledge corresponding to the disease, including detailed information such as the disease name, cause, progression, and corresponding treatment. Through this multi-model collaborative approach, the system can achieve accurate image recognition and professional medical interpretation, providing doctors and patients with more comprehensive diagnostic support and treatment suggestions.

[0104] In summary, the system processing flow in this embodiment is as follows: First, the system acquires various information submitted by the user, including the user's question text during the consultation process, relevant documents uploaded to the knowledge base, and medical images of the affected area; Second, based on the selected modules, the system automatically converts and generates the required input data, and then submits this processed and concatenated data to the large model for inference; Third, a distributed strategy is adopted to divide the model inference task across multiple GPUs for computation, effectively avoiding memory overflow on a single GPU and improving computational efficiency; Fourth, the system integrates the results of the model inference and outputs them through the user interface, ensuring that users can easily view relevant diagnostic information and treatment suggestions.

[0105] Example 2

[0106] This embodiment discloses a medical problem consultation method based on a large model, including:

[0107] Acquire user input text, documents, and medical images;

[0108] The system uses a dialogue consultation module to receive input text, processes the text, and then inputs it into a finely tuned large model for model inference, outputting medical suggestions.

[0109] The system uses a knowledge base module to receive input text and documents, processes the input text and documents, and then inputs them into a framework based on a large model for further processing, outputting medical advice and related professional knowledge.

[0110] The system uses an image analysis module to receive medical images, inputs them into an image classification model to obtain the corresponding disease name, and inputs the disease name into a fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name. The image classification model receives the input medical images, performs feature extraction, fusion, weighting, and classification on the medical images to obtain the corresponding disease name.

[0111] Example 3

[0112] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 2.

[0113] Example 4

[0114] The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 2.

[0115] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0116] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0118] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A medical problem consultation system based on a large model, characterized in that, include: The data acquisition module is used to acquire user input text, documents, and medical images; The dialogue consultation module is used to receive input text, process the input text, and then input it into a finely tuned large model for model inference, and output medical suggestions. The large model used in the dialogue consultation module adopts the ChatGLM4-9B-Chat open-source large model; The professional knowledge base module is used to receive input text and documents, process the input text and documents, and then input them into a large model-based framework for further processing, outputting medical advice and related professional knowledge; the large model-based framework is the LangChain framework, which includes a document loader, document indexing and retrieval, and chains; The affected area image analysis module receives medical images, inputs them into an image classification model to obtain the corresponding disease name, and inputs the disease name into a fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name. The image classification model receives the input medical images, performs feature extraction, fusion, weighting, and classification on the medical images to obtain the corresponding disease name. The fine-tuned large model in the affected area image analysis module is a fine-tuned ChatGLM4. The image classification model includes, in sequence, an input module, a feature extraction module, a feature fusion module, an SEBlock module, and a classification head. The image processing in the image classification model is as follows: the input module receives the input medical image and inputs it to the feature extraction module; the feature extraction module includes two pre-trained models, VGG16 and ResNet50, which extract low-level and high-level features respectively, and input the extracted features to the feature fusion module respectively. The feature fusion module receives features extracted by VGG16 and ResNet50 respectively, concatenates them along the channel dimension to form a fused feature, and inputs the fused feature into the SEBlock module. The SEBlock module weights the fused feature to generate a channel weight vector, and weights the features of each channel based on the channel weight vector to highlight important features, outputs the weighted fused feature, and inputs it into the classification head. The classification head obtains the disease name corresponding to the medical image based on the fused feature. The specific steps in the SEBlock module are as follows: Receive the fused features and apply adaptive average pooling to compress their spatial dimensions; The number of channels in the fused features is reduced by convolution, and non-linear activation is performed by an activation function. The number of channels for the fused features is restored again through convolution, and a weight vector for each channel is generated through a function. The fused features are multiplied by the weight vector channel by channel to obtain the weighted fused features.

2. The medical problem consultation system based on a large model as described in claim 1, characterized in that, After processing the input text, it is fed into a finely tuned large model for model inference, specifically: The input text is tokenized and divided into multiple sub-units; The segmented units are converted into numerical values, and each unit is converted into a word vector using an embedding matrix. The input text is positionally encoded, and the positional encoding is combined with word vectors and then fed into a finely tuned large model to output text that the user can understand.

3. The medical problem consultation system based on a large model as described in claim 1, characterized in that, The framework based on the large model outputs medical suggestions and related professional knowledge, specifically as follows: Convert documents into a word vector database and convert input text into word vectors; Calculate the spatial similarity between the word vectors and the sentences in the word vector database to obtain a similarity score; The word vectors are combined with the paragraphs with the highest similarity scores and input into a large model to output medical advice and related professional knowledge.

4. A medical problem consultation method based on a large model, characterized in that, include: Acquire user input text, documents, and medical images; The system uses a dialogue consultation module to receive input text, processes the text, and then inputs it into a finely tuned large model for model inference, outputting medical suggestions. The large model used in the dialogue consultation module adopts the ChatGLM4-9B-Chat open-source large model; The system uses a professional knowledge base module to receive input text and documents, processes the input text and documents, and then inputs them into a large model-based framework for further processing, outputting medical advice and related professional knowledge. The large model-based framework is the LangChain framework, which includes a document loader, document indexing and retrieval, and chains. The system uses an image analysis module to receive medical images, inputs them into an image classification model to obtain the corresponding disease names, and then inputs the disease names into a fine-tuned large model. Based on the disease names, the large model generates medical knowledge corresponding to the disease. The image classification model receives the input medical images, performs feature extraction, fusion, weighting, and classification on the medical images to obtain the corresponding disease names. The fine-tuned large model in the image analysis module is a fine-tuned ChatGLM4. The image classification model includes, in sequence, an input module, a feature extraction module, a feature fusion module, an SEBlock module, and a classification head. The image processing in the image classification model is as follows: the input module receives the input medical image and inputs it to the feature extraction module; the feature extraction module includes two pre-trained models, VGG16 and ResNet50, which extract low-level and high-level features respectively, and input the extracted features to the feature fusion module respectively. The feature fusion module receives features extracted by VGG16 and ResNet50 respectively, concatenates them along the channel dimension to form a fused feature, and inputs the fused feature into the SEBlock module. The SEBlock module weights the fused feature to generate a channel weight vector, and weights the features of each channel based on the channel weight vector to highlight important features, outputs the weighted fused feature, and inputs it into the classification head. The classification head obtains the disease name corresponding to the medical image based on the fused feature. The specific steps in the SEBlock module are as follows: Receive the fused features and apply adaptive average pooling to compress their spatial dimensions; The number of channels in the fused features is reduced by convolution, and non-linear activation is performed by an activation function. The number of channels for the fused features is restored again through convolution, and a weight vector for each channel is generated through a function. The fused features are multiplied by the weight vector channel by channel to obtain the weighted fused features.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the large-model-based medical problem consultation method as described in claim 4.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the medical problem consultation method based on a large model as described in claim 4.

Images