Large model-based medical problem consultation system and method

Abstract

The invention discloses a large model-based medical problem consultation system and method, and the method comprises the steps: a dialogue inquiry module processes an input text, inputs the input text to a fine-tuned large model for model reasoning, and outputs a medical suggestion; the professional knowledge base module processes input texts and documents, inputs the input texts and documents to the framework based on the large model for processing, and outputs medical suggestions and related professional knowledge; the affected part image analysis module receives the medical image and inputs the medical image to the image classification model to obtain a corresponding disease name, the disease name is input to the fine-tuned large model, and the large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives an input medical image, and performs feature extraction, fusion, weighting and classification on the medical image to obtain a corresponding disease name. The problems that young people have no time for offline inquiry and old people have insufficient understanding of early symptoms of diseases can be simultaneously solved, and early prevention, early discovery, early diagnosis and early treatment of the diseases are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of medical information processing, and in particular to a medical problem consultation system and method based on a large model. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In recent years, big models combined with specific fields have made significant progress in many industries. Fields such as medicine, industrial manufacturing, electricity, and transportation have begun to use big models for data-driven optimization and innovation. Domain big models not only have the reasoning and generation capabilities of general big models, but also integrate knowledge and data in specific fields, which can provide more accurate predictions and more professional decision support.

[0005] On the other hand, many elderly people do not have enough knowledge of the early symptoms of diseases and tend to ignore abnormal changes in their bodies. Such health blind spots lead to the failure to effectively diagnose and treat some diseases in the early stages, thus missing the best time for intervention. This trend has aroused widespread social concern about health issues and early prevention of diseases, especially among elderly people living alone and young people under great pressure. Early prevention and timely health management are particularly important.

[0006] In order to solve the above problems, measures have been taken to strengthen primary medical services, promote hierarchical diagnosis and treatment, develop elderly health services, and promote the extension of medical services to homes and communities. However, there is no good solution for how to make it convenient and quick for residents to consult and determine the cause of the disease, which makes the elderly lack understanding of the early symptoms of the disease. Although there are big models in the medical field, the existing big models in the medical field are generally for doctors or people with medical expertise to assist in diagnosis. They cannot accurately understand the description texts of non-professional users, resulting in low accuracy in disease judgment and inaccurate understanding of the disease, which can seriously endanger life safety. Summary of the invention

[0007] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a medical problem consultation system and method based on a large model, which can simultaneously solve the problems that young people have no time for offline consultations and the elderly have insufficient understanding of the early symptoms of the disease, thereby achieving early prevention, early detection, early diagnosis and early treatment of the disease.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] In a first aspect, the present invention provides a medical problem consultation system based on a large model, comprising:

[0010] A data acquisition module, used to acquire user input text, documents and medical images;

[0011] The dialogue consultation module is used to receive input text, process the input text, and then input it into the fine-tuned large model for model inference and output medical advice;

[0012] The professional knowledge base module is used to receive input text and documents, process the input text and documents, input them into the framework based on the big model for processing, and output medical advice and related professional knowledge;

[0013] The affected area image analysis module is used to receive a medical image, input it into an image classification model to obtain a corresponding disease name, and input the disease name into a fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical image, extracts, fuses, weights and classifies the medical image to obtain a corresponding disease name.

[0014] A further technical solution is to process the input text and then input it into the fine-tuned large model for model inference, specifically:

[0015] Tokenize the input text and split it into multiple sub-units;

[0016] Convert the sub-units after word segmentation into numerical values, and use the embedding matrix to convert each sub-unit into a word vector;

[0017] The input text is positionally encoded, and the positional encoding is merged with the word vector and input into the fine-tuned large model to output the text that the user understands.

[0018] Further technical solutions,the framework based on the large model is the LangChain framework,,including document loader, document indexing and retrieval,chain.

[0019] A further technical solution is that the framework based on the big model outputs medical advice and related professional knowledge, specifically:

[0020] Convert documents into word embedding databases and convert input text into word embeddings;

[0021] Calculate the spatial similarity between the word vector and the sentence in the word vector database to obtain a similarity score;

[0022] The word vector is combined with the paragraph with the highest similarity score and input into a large model to output medical advice and related expertise.

[0023] According to a further technical solution, the image classification model includes an input module, a feature extraction module, a feature fusion module, a SEBlock module and a classification head in sequence.

[0024] According to a further technical solution, the image processing process in the image classification module is specifically as follows:

[0025] The input module receives the input medical image and inputs it into the feature extraction module; the feature extraction module includes two pre-trained models, VGG16 and Resnet50, which extract low-level and high-level features respectively, and input the extracted features into the feature fusion module respectively; the feature fusion module receives the features extracted by VGG16 and Resnet50 respectively, and splices them in the channel dimension to form fusion features, and inputs the fusion features into the SEBlock module; the SEBlock module weights the fusion features to generate a channel weight vector, weights the features of each channel based on the channel weight vector, highlights important features, outputs the weighted fusion features, and inputs them into the classification head; the classification head obtains the disease name corresponding to the medical image according to the fusion features.

[0026] Further technical solution, the specific steps in the SEBlock module are:

[0027] Receive the fused features, apply adaptive average pooling to the fused features to compress their spatial dimensions;

[0028] The number of channels of the fused features is reduced through convolution, and nonlinear activation is performed through activation functions;

[0029] The number of channels of the fused features is restored again through convolution, and the weight vector of each channel is generated through the function;

[0030] Multiply the fused features by the weight vector channel by channel to obtain the weighted fused features.

[0031] In a second aspect, the present invention provides a medical problem consultation method based on a large model, comprising:

[0032] Obtain user input text, documents, and medical images;

[0033] The dialogue inquiry module receives input text, processes it, and then inputs it into the fine-tuned large model for model reasoning, and outputs medical advice.

[0034] Using the professional knowledge base module to receive input text and documents, process the input text and documents and then input them into the big model-based framework for processing, and output medical advice and related professional knowledge;

[0035] The affected area image analysis module is used to receive medical images, which are input into the image classification model to obtain the corresponding disease name, and the disease name is input into the fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical image, extracts, fuses, weights and classifies the medical image to obtain the corresponding disease name.

[0036] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a large model-based medical problem consultation method as described in the second aspect.

[0037] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a large model-based medical problem consultation method as described in the second aspect are implemented.

[0038] One or more of the above technical solutions have the following beneficial effects:

[0039] The present invention focuses on residents' consultation needs, and fine-tunes the latest open source big model for the symptoms and treatment methods of common diseases, aiming to realize the application of the big model in the consultation scenario. Through the dialogue consultation module, professional knowledge base module and affected area image analysis module, it can provide patients with accurate and reliable treatment suggestions to meet personalized medical needs; the dialogue consultation module, professional knowledge base module and affected area image analysis module are implemented based on the fine-tuned big model, and a large number of open source medical data sets of real patient-doctor dialogues, open source Chinese medical dialogue data sets and other data are used to fine-tune the big model, making it easier for the big model to understand non-professional condition description texts, thereby improving the accuracy and reliability of disease judgment.

[0040] The dialogue consultation module in the present invention is realized by a fine-tuned big model. The big model is fine-tuned by a professional data set so that the big model can provide professional and accurate suggestions when talking with patients, meeting the high standards of medical consultation; the professional knowledge base module is realized by a framework based on the big model. When answering user questions, the model will not only rely on its own built-in knowledge base to answer, but also build the files uploaded by the user into an external knowledge base. After the construction is completed, the framework will fuse the internal and external information and provide it as output to the big model. The model will provide more accurate and relevant answers to the user, and will be accompanied by original text materials for user reading and comparison; the affected area image analysis module adopts a dual-model analysis strategy, which can comprehensively analyze the condition based on the image classification results and the fine-tuned data set, generate accurate judgment results and put forward corresponding treatment suggestions, thereby helping patients better understand the condition and choose appropriate treatment plans.

[0041] Medical responses have little tolerance for large model hallucinations. In order to solve the hallucination problem of large models, the present invention adopts a dual-model analysis strategy in the affected area image analysis module, and uses an image classification model and a fine-tuned GLM4 model in series. The fine-tuned GLM4 model outputs accurate knowledge based on the classification results of the image classification model, which improves the accuracy of the output knowledge and does not cause hallucinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 is a flow chart of a medical problem consultation system according to an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of a professional knowledge base module in a medical problem consultation system according to an embodiment of the present invention;

[0045] Figure 3 This is a flow chart of an affected area image analysis module in a medical problem consultation system according to an embodiment of the present invention;

[0046] Figure 4 is a flow chart of a dual-model analysis strategy in an affected area image analysis module according to an embodiment of the present invention;

[0047] Figure 5 This is an example diagram of the interface of the dialogue consultation module in the medical problem consultation system according to an embodiment of the present invention;

[0048] Figure 6 This is an example diagram of the interface of the professional knowledge base module in the medical problem consultation system according to an embodiment of the present invention;

[0049] Figure 7 is an example diagram of relevant context in the professional knowledge base module in the medical problem consultation system of an embodiment of the present invention;

[0050] Figure 8 This is an example diagram of the interface of the affected area image analysis module in the medical problem consultation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0054] Terminology explanation: Lora: fine-tuning method; LLM: large language model; LangChain: LLM open source framework.

[0055] Aiming at the problems that young people have no time for offline consultation and the elderly have insufficient understanding of the early symptoms of diseases, the present invention designs a medical problem consultation system to achieve early prevention, early detection, early diagnosis and early treatment of diseases.

[0056] Embodiment 1

[0057] like Figure 1 As shown, this embodiment discloses a medical problem consultation system based on a large model, including:

[0058] The data acquisition module is used to acquire user input text, documents and medical images.

[0059] The dialogue consultation module is used to receive input text, process the input text, and then input it into the fine-tuned large model for model inference to output medical advice.

[0060] In this embodiment, the dialogue consultation module is implemented by fine-tuning the big model, using an open source medical data set containing 1 million real patient-doctor conversations, and incorporating professional drug function explanation information. This module is based on Zhipu's open source big model ChatGLM4. After fine-tuning the professional data set, the big model can provide professional and accurate suggestions when talking to patients, meeting the high standards of medical consultation.

[0061] Specifically, the large model in the dialogue consultation module uses the ChatGLM4-9B-Chat open source large model, and the training data set used is the open source Chinese medical dialogue data set. This data set covers multiple fields such as internal medicine, andrology, obstetrics and gynecology, oncology, pediatrics, and surgery, and contains rich professional medical knowledge, which can effectively support the medical dialogue and consultation needs of different departments.

[0062] Specifically, the fine-tuning process of the large model is optimized using the LLaMA-Factory framework. This framework provides a flexible and efficient fine-tuning mechanism that can accelerate model training while ensuring performance and improve the performance of the model in specific domain tasks.

[0063] Specifically, the fine-tuning technology adopts the Lora fine-tuning method. Compared with the p-tuning fine-tuning technology, Lora fine-tuning shows more significant advantages in accelerating the training process, achieving a 3.7-fold acceleration effect, significantly improving training efficiency and resource utilization.

[0064] The specific steps for the large model in the dialogue inquiry module to perform model reasoning on the input text are:

[0065] (1) Tokenize the input text and add some special tags to help the large model understand the text structure. Tokenization is to divide the text into multiple sub-units, tokens.

[0066] (2) Convert the sub-unit tokens after word segmentation into numerical values ​​and use the embedding matrix EmbeddingMatrix to convert each sub-unit token into a word vector.

[0067] (3) In order to enable the large model to understand the relative position of the text in the sequence, position encoding is added.

[0068] (4) Merge the word vector and positional encoding and input them into the large model.

[0069] (5) The large model was fine-tuned by Lora, changing the query weight matrix w in the transformer architecture q , the key weight matrix w k The weight matrix w of the sum value v .w q 、w k 、w v It is a weight matrix used in the transformer architecture to map the input vector to the query (Q), key (K), and value (V) space, helping the transformer calculate the weights and outputs in the self-attention mechanism. The model captures the relationship between different elements in the sequence by training these weight matrices.

[0070] (6) After passing through multiple layers of transformers, the self-attention mechanism extracts and combines the contextual features of word vectors and positional encodings. Finally, the dialogue interrogation large model outputs a result vector, which is the feature representation of each sub-unit token. Specifically, through multiple layers of stacking, the self-attention mechanism encodes and understands each token (word or sub-unit) and its position in the input text, thereby capturing contextual information. The transformer model calculates query (Q), key (K), and value (V) vectors for each token, then calculates the similarity between them to obtain attention weights; each token weights the value (V) vector according to its similarity to other tokens, thereby obtaining a new representation. This new representation is the contextual feature, which incorporates the relationship between this token and other tokens in the entire sequence.

[0071] (7) Finally, based on the feature representation of the sub-unit token, text that can be understood by the user is output. The feature representation of the token is: after passing through the transformer, the model generates a high-dimensional feature vector for the input token. For example, for the word "我", a feature vector such as [0.12, -0.04, 0.22, 0.05, …, 0.19] will be generated. These feature representations are digital vectors and are not directly understandable by humans. In order for the model to generate understandable answers, these feature representations must be decoded into natural language text, such as decoding to "我".

[0072] The process is as follows: First, the feature vector of each token passes through a linear layer, which converts the feature vector into scores in the dimension of the vocabulary. These scores represent the likelihood of this token generating each word in the vocabulary. The obtained scores are transformed into a probability distribution through the Softmax function, representing the probability of each word being the output of the current token. Finally, through the set sampling strategy, the word with the highest probability is selected as the output, becoming natural language.

[0073] The professional knowledge base module is used to receive input text and documents, process the input text and documents, and then input them into the large model-based framework for processing, and output medical advice and related professional knowledge.

[0074] In this embodiment, the professional knowledge base module aims to provide support for users who need to deeply understand the condition. Since the large model fine-tuning dataset mainly covers the disease interrogation process, if users hope to understand professional knowledge such as disease mechanisms more comprehensively, they can upload relevant medical books. This module adopts a large model-based framework (i.e., the LangChain framework), which supports the large model to answer not only the user's question statements but also the user's professional knowledge question statements, give relevant professional knowledge of medical advice, extract relevant explanations from professional books, and attach the original text materials for the convenience of users to read and compare.

[0075] Specifically, the professional knowledge base module is built using the Langchain framework. First, the system loads medical-related files (i.e., documents entered by users) from the local computer and converts them into a processable text format through an unstructured loader; second, a text segmenter is used to split longer medical texts into smaller text blocks for more efficient processing and analysis; third, vector embedding is generated for each text block and stored in the vector library to support fast retrieval and matching; fourth, when a user makes a medical query, the system converts the query into a vector embedding and performs similarity matching with the medical text blocks in the vector library to accurately extract relevant content; fifth, the system inputs the extracted medical content into the large language model through a prompt template for reasoning; sixth, the large model finally generates professional medical answers and returns them to the user.

[0076] The Langchain framework structure includes document loaders, document indexing and retrieval, and chains. The document loader is used to load document data from different data sources (such as text files, web pages, databases, etc.). These documents usually contain domain-related knowledge; by indexing and retrieving document content, relevant information can be efficiently extracted from large-scale documents and provide support for models. This usually involves the use of retrieval-augmented generation (RAG) technology; in LangChain, a chain is a component that combines multiple tasks (such as document retrieval, data processing, and model reasoning) in sequence. Through these chains, complex workflows can be implemented, such as extracting knowledge from documents and generating answers in combination with large language models. The large model combined with the Langchain framework technology can realize the reading and search of user-uploaded files.

[0077] Input the input text (i.e., professional knowledge question text) and documents into the Langchain framework for processing, specifically:

[0078] (1) Through the LangChain framework, the documents uploaded by the user are converted into a word vector database, and the professional knowledge question text (input text) is converted into a word vector; then the LangChain framework determines the spatial similarity between the professional knowledge question text and the sentence in the word vector database, and calculates the score to obtain the similarity score. The spatial similarity uses cosine similarity to calculate the cosine value of the angle between the two vectors. The closer the value is to 1, the more similar the two vectors are, and the closer the value is to -1, the less similar the two vectors are.

[0079] (2) Subsequently, the LangChain framework combines the word embeddings converted from the professional knowledge question text with the highest-scoring paragraph in the word embedding database to generate a new prompt as input for the ChatGLM4 large model; based on this input, the large model outputs medical advice and related professional knowledge.

[0080] Through the above technical solutions, the system can read the professional knowledge documents uploaded by patients and extract relevant medical information from them through analysis. On this basis, the system identifies the specific knowledge points required by users. This process further improves the accuracy of information through precise fitting and optimization of the model, making the final output of the system more professional and targeted than the original large model. This improvement enables users to quickly and accurately obtain medical knowledge that meets their needs, providing more powerful support for clinical consultation and self-diagnosis.

[0081] The affected area image analysis module is used to receive a medical image, input it into an image classification model to obtain a corresponding disease name, and input the disease name into a fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical image, extracts, fuses, weights and classifies the medical image to obtain a corresponding disease name.

[0082] In this embodiment, if there are medical images, including images of diseased parts (such as photos of skin lesions), CT images and other medical images, users can use the image analysis module to identify and analyze them. The module first uses the ResNet model combined with the VGG model to build and train an image classification model specifically for identifying common image types in medicine, which can accurately distinguish different lesion categories, i.e., disease names; then, combined with the capabilities of the large language model, based on the image classification results and fine-tuned data sets, a comprehensive analysis of the condition is performed, accurate judgment results are generated, and corresponding treatment recommendations are made, thereby helping patients better understand their condition and choose appropriate treatment plans.

[0083] The affected area image analysis module adopts a dual-model analysis strategy, combining the image classification model Fusion and the large model (fine-tuned ChatGLM4) to work together to achieve efficient image content analysis and knowledge output. The image classification model Fusion is used to identify medical images submitted by users and parse image information into text descriptions. For example, when detecting skin tumor images, the Fusion model will identify the image and output the corresponding tumor name, such as "melanoma"; the ChatGLM4 large model receives the text description output by the Fusion model as input, combines the input with the set prompt, and generates fine-tuned professional knowledge output to provide users with more in-depth analysis results.

[0084] The image classification model Fusion is an improved algorithm of the ResNet model. By combining the ResNet and VGG models and incorporating the attention mechanism, the image recognition ability is significantly improved. In the training on the HAM skin cancer dataset, the Fusion model can increase the accuracy to 99% within 20 epochs. In contrast, the accuracy of the traditional ResNet model fluctuated from the 11th to the 13th epoch, causing the training to end early. The image classification model Fusion was trained on public datasets and tested on the MNIST and HAM10000 datasets to ensure its recognition accuracy for skin lesion images.

[0085] The VGG and ResNet models have their own strengths in image feature extraction. VGG is good at extracting low- and medium-level detail features, while ResNet focuses more on high-level semantic features. By combining the advantages of both, the image fusion model of the present invention uses dual modules to extract different features of the image and splices these features in the channel dimension to generate a fused feature map. Subsequently, the fused feature map is input into the SEBlock module, and a channel weighting mechanism is used to generate weights for each channel to measure its importance in the final classification task. The weighted optimized fused features are then input into the classification head, and the high-dimensional features are gradually reduced to the category space, and the confidence of each category is finally output.

[0086] The image classification model includes an input module, a feature extraction module, a feature fusion module, a SEBlock module, and a classification head. The input module receives the input feature map, i.e., the input medical image, and inputs it into the feature extraction module; the feature extraction module includes two pre-trained models, VGG16 and Resnet50, which serve as low-level and high-level feature extractors, respectively, and input the extracted features into the feature fusion module; the feature fusion module receives the features extracted by VGG16 and Resnet50, respectively, and splices them in the channel dimension to form fusion features, and inputs the fusion features into the SEBlock module; the SEBlock module weights the fusion features, generates a channel weight vector, weights the features of each channel based on the channel weight vector, highlights important features, outputs the weighted fusion features, and inputs them into the classification head; the classification head completes the final classification task based on the fusion features, and obtains the disease name corresponding to the input medical image.

[0087] The processing flow of the input image (i.e. the medical image submitted by the user) in the image classification model is as follows:

[0088] (1) The input image I passes through the first 16 layers of VGG and the feature extraction of Resnet in turn to generate two feature maps F VGG ∈R 512*H*W With F Resnet ∈R2048*H*W .

[0089] (2) The two feature maps are adjusted to the same spatial dimension R through adaptive average pooling. C*1*1 , and concatenate them in dimension to get the fused feature map X∈R 2560*1*1 .

[0090] (3) The fused feature X is input into the SEBlock module to generate a weight vector W∈R 2560 , the enhanced features are obtained by weighting each channel, and the enhanced features, i.e. the weighted fusion features, are input into the classification head. After adaptive average pooling, flattening operation, the first fully connected layer and the second fully connected layer, the category prediction result is finally output, i.e. the disease name corresponding to the input medical image.

[0091] Furthermore, the feature extraction module includes two pre-trained models, VGG16 and Resnet50, as low-level and high-level feature extractors respectively. VGG16 extracts the features of the first 16 layers, and its output channels are 516; Resnet50 removes the last two layers and extracts high-level features, and its output channels are 2048; these two parts of features are adaptively averaged and pooled to unify the spatial scale to 1*1, which is convenient for feature fusion.

[0092] Furthermore, the feature fusion module receives the features extracted by VGG16 and Resnet50 respectively, and concatenates them in the channel dimension to form a fused feature. The number of channels of the fused feature is 2560.

[0093] Furthermore, SEBlock module: SEBlock module is used to weight the fusion features, the number of channels is first reduced to 160, then restored to 2560, and a channel weight vector is generated. The weight vector weights the features of each channel to highlight important features.

[0094] The specific steps in the SEBlock module are:

[0095] (1) Global feature aggregation: The module first fused the feature map X∈R C*H*W Apply adaptive average pooling to compress its spatial dimension to R C*1*1 , and obtain the global feature representation of each channel, helping the model to obtain a more comprehensive image representation.

[0096] (2) Channel dimension reduction: Through 1*1 convolution, the number of channels of fusion features is reduced from C to (r is reductionradio, the default is 16), reducing parameters and calculation amount.

[0097] (3) Nonlinear activation: The ReLU activation function is introduced to introduce nonlinearity.

[0098] (4) Channel dimension upgrade: The number of channels of the fused features is restored to C through 1*1 convolution again, and the weight of each channel is generated through the sigmoid function.

[0099] (5) Feature recalibration: The input fusion feature map is multiplied by the weight vector channel by channel to obtain the weighted fusion feature map. The formula is expressed as:

[0100] X SE =X·σ(Conv2d(ReLU(Conv2d(Avgpool(X))))).

[0101] Further, the classification head: The structure of the classification head includes adaptive average pooling, flattening operation, the first fully connected layer and the second fully connected layer. Adaptive average pooling further compresses the features to 1*1; the flattening operation flattens the features into a one-dimensional vector; the first fully connected layer maps the number of channels 2560 to 1024, and applies ReLU activation and Dropout (with a probability of 0.5); the second fully connected layer maps 1024 to the number of target categories.

[0102] Medical responses have little tolerance for hallucinations in large models. Existing large models cannot eliminate hallucinations if they directly analyze images. In order to solve the hallucination problem of large models, the affected area image analysis module uses the Fusion model and the fine-tuned GLM4 model in series. The Fusion model acts as an image recognition and classification model. The fine-tuned GLM4 model outputs accurate knowledge based on the classification results of the Fusion model. The affected area image analysis module first needs to be trained using a data set. The system test uses the MNIST and HAM1000 data sets. After training, Fusion can maintain an accuracy of more than 90% on this data set. When fine-tuning the GLM4 model, it is necessary to prepare information related to the classification results of the Fusion model to enhance the large model's ability to understand and generate medical knowledge. For example, for skin diseases, it is necessary to prepare relevant information such as basal cell carcinoma, skin fibroma, melanoma, etc. After the Fusion model recognizes the user's medical image, it returns the classification result to the GLM4 model, and the classification result input is combined with the prompt and submitted to the large model. The large model can finally output accurate knowledge based on the classification results, including a brief introduction to the disease, its causes, manifestations, and treatment methods, without any hallucinations.

[0103] The big model receives the output information after the image classification model Fusion analyzes, that is, the name of the disease, and uses it as the input of the big model. It combines the preset prompt to generate fine-tuned professional knowledge output and outputs precise knowledge for users, including disease introduction, causes, manifestations, treatment methods, etc. Through this collaborative approach, the image classification model is responsible for the structured processing of image information, while the big model further uses semantic understanding and professional knowledge reasoning to provide users with detailed and accurate interpretation results. For example, in the scenario of detecting skin tumors, the big model can provide users with professional information such as tumor characteristics and treatment recommendations based on the name "melanoma" output by the image classification model and fine-tuned medical knowledge, thereby achieving seamless integration of image analysis and knowledge services.

[0104] The large model generates medical knowledge corresponding to the disease, that is, detailed information about the disease, including the name of the disease, the cause, the progression of the disease, and the corresponding treatment plan. Through this multi-model collaboration, the system can achieve accurate image recognition and professional medical interpretation, providing doctors and patients with more comprehensive diagnostic support and treatment recommendations.

[0105] In summary, the system processing flow in this embodiment is as follows: First, the system obtains various information submitted by the user, including the user's question text during the consultation, the relevant document materials uploaded to the knowledge base, and the medical images of the diseased part; Second, according to the different modules selected, the system automatically converts and generates the required input data, and submits these processed and spliced ​​data to the large model for reasoning; Third, a distributed strategy is used to divide the model reasoning task to multiple GPUs for calculation, so as to effectively avoid the overflow of the video memory of a single GPU and improve the calculation efficiency. Fourth, the system integrates the results after model reasoning and outputs them through the user interface to ensure that users can easily view relevant diagnostic information and treatment recommendations.

[0106] Embodiment 2

[0107] This embodiment discloses a medical problem consultation method based on a large model, including:

[0108] Obtain user input text, documents, and medical images;

[0109] The dialogue inquiry module receives input text, processes it, and then inputs it into the fine-tuned large model for model reasoning, and outputs medical advice.

[0110] Using the professional knowledge base module to receive input text and documents, process the input text and documents and then input them into the big model-based framework for processing, and output medical advice and related professional knowledge;

[0111] The affected area image analysis module is used to receive medical images, which are input into the image classification model to obtain the corresponding disease name, and the disease name is input into the fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical image, extracts, fuses, weights and classifies the medical image to obtain the corresponding disease name.

[0112] Embodiment 3

[0113] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of Embodiment 2 when executing the program.

[0114] Embodiment 4

[0115] The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the method of embodiment 2 are performed.

[0116] The steps involved in the apparatus of the above embodiments 3 and 4 correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0117] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0119] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A medical problem consultation system based on a large model, characterized in that: include: A data acquisition module, used to acquire user input text, documents and medical images; The dialogue consultation module is used to receive input text, process the input text, and then input it into the fine-tuned large model for model inference and output medical advice; The professional knowledge base module is used to receive input text and documents, process the input text and documents, input them into the framework based on the big model for processing, and output medical advice and related professional knowledge; The affected area image analysis module is used to receive a medical image, input it into an image classification model to obtain a corresponding disease name, and input the disease name into a fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical image, extracts, fuses, weights and classifies the medical image to obtain a corresponding disease name.

2. A medical problem consultation method based on a large model as claimed in claim 1, characterized in that: After processing the input text, it is input into the fine-tuned large model for model inference, specifically: Tokenize the input text and split it into multiple sub-units; Convert the sub-units after word segmentation into numerical values, and use the embedding matrix to convert each sub-unit into a word vector; The input text is positionally encoded, and the positional encoding is merged with the word vector and input into the fine-tuned large model to output the text that the user understands.

3. A medical problem consultation method based on a large model as claimed in claim 1, characterized in that: The framework based on the big model is the LangChain framework, which includes document loader, document indexing and retrieval, and chain.

4. A medical problem consultation method based on a large model as claimed in claim 3, characterized in that: The large model-based framework outputs medical advice and related expertise, specifically: Convert documents into word embedding databases and convert input text into word embeddings; Calculate the spatial similarity between the word vector and the sentence in the word vector database to obtain a similarity score; The word vector is combined with the paragraph with the highest similarity score and input into a large model to output medical advice and related expertise.

5. A medical problem consultation method based on a large model as claimed in claim 1, characterized in that: The image classification model includes an input module, a feature extraction module, a feature fusion module, a SEBlock module and a classification head in sequence.

6. A medical problem consultation method based on a large model as claimed in claim 5, characterized in that: The image processing process in the image classification module is specifically as follows: The input module receives the input medical image and inputs it into the feature extraction module; the feature extraction module includes two pre-trained models, VGG16 and Resnet50, which extract low-level and high-level features respectively, and input the extracted features into the feature fusion module respectively; The feature fusion module receives the features extracted by VGG16 and Resnet50 respectively, and concatenates them in the channel dimension to form fused features, and inputs the fused features into the SEBlock module; the SEBlock module weights the fused features to generate a channel weight vector, and weights the features of each channel based on the channel weight vector to highlight important features, outputs the weighted fused features, and inputs them into the classification head; The classification head obtains the disease name corresponding to the medical image based on the fused features.

7. A medical problem consultation method based on a large model as claimed in claim 6, characterized in that: The specific steps in the SEBlock module are: Receive fused features, apply adaptive average pooling to the fused features to compress their spatial dimensions; The number of channels of the fused features is reduced through convolution, and nonlinear activation is performed through activation functions; The number of channels of the fused features is restored again through convolution, and the weight vector of each channel is generated through the function; Multiply the fused features by the weight vector channel by channel to obtain the weighted fused features.

8. A medical problem consultation method based on a large model, characterized in that: include: Obtain user input text, documents, and medical images; The dialogue inquiry module receives input text, processes it, and then inputs it into the fine-tuned large model for model reasoning, and outputs medical advice. Using the professional knowledge base module to receive input text and documents, process the input text and documents and then input them into the big model-based framework for processing, and output medical advice and related professional knowledge; The affected area image analysis module is used to receive medical images, which are input into the image classification model to obtain the corresponding disease name, and the disease name is input into the fine-tuned large model. The large model generates medical knowledge corresponding to the disease based on the disease name; the image classification model receives the input medical image, extracts, fuses, weights and classifies the medical image to obtain the corresponding disease name.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a medical problem consultation method based on a large model as described in claim 8 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the medical problem consultation method based on a large model as described in claim 8 are implemented.

Images