Application-level medical large model training method, application method, equipment, medium and product

By segmenting and data processing of medical scenarios, application-level medical big models for specific small worlds were trained, and combined with multimodal fusion and emotion recognition technology, the problem of insufficient sensitivity and accuracy of general large language models in clinical environments is solved. , achieving higher specific performance and security.

CN119066171APending Publication Date: 2024-12-03INSTITUTE OF BASIC MEDICAL SCIENCES CHINESE ACADEMY OF MEDICAL SCIENCES

Patent Information

Application Number
CN202411154795.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The general large language model cannot achieve sufficient sensitivity and accuracy in clinical settings and cannot meet the needs of specific medical scenarios.

Method used

By segmenting medical scenarios, obtaining and anonymizing the doctor-patient dialogue data, training data suitable for the base large language model is formed, and application-level medical models for specific small worlds are trained, combined with multimodal fusion. and emotion recognition technology to enhance the specific performance of the model.

Benefits of technology

It improves the sensitivity and accuracy of application-level medical models in clinical environments, ensures higher safety and applicability, reduces dependence on general models, and realizes limited rules and possibilities learning in specific time and space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066171A_ABST
    Figure CN119066171A_ABST
Patent Text Reader

Abstract

The invention discloses an application-level medical large model training and application method, device, medium and product, and relates to the technical field of artificial intelligence, the method comprises the following steps: segmenting a medical scene to obtain a plurality of small worlds; for each small world, acquiring doctor-patient dialogue data in the small world; performing anonymization processing on the doctor-patient dialogue data to obtain anonymized data; refining the anonymized data to obtain training data suitable for the base large language model; and training the base large language model by using the training data to obtain an application-level medical large model suitable for the small world. According to the method, a data source of a universal base large language model is divided into a plurality of small worlds, panoramic full-amount data acquisition is performed in a specific small world, a small world enhancement model is established, finite rules and possibilities under specific time and space are learned, and compared with the universal base large language model, the method has the advantages that the method is simple and convenient to implement. And the specific performance aiming at the clinical environment is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an application-level medical large model training, application method, equipment, medium and product. Background Art

[0002] Medical treatment is one of the scenarios that is most closely related to social economy and people's lives. In recent years, medical artificial intelligence has been the most popular track.

[0003] As the latest revolutionary technology in the field of medical AI, generative artificial intelligence / large model technology, also known as the core technology behind ChatGPT, is widely favored. A McKinsey research report believes that generative AI is expected to contribute about $7 trillion in value to the global economy and increase the overall benefits of AI by about 50%.

[0004] The current mainstream large language models, such as ChatGPT, Flan-PaLM, LLaMa, etc., include a very wide range of knowledge, including papers, academic reports, online chat conversations and other information, and can answer many types of knowledge, forming a general large language model. However, this type of general large language model is "general but not refined". In certain specific scenarios, such as clinical environments, they cannot achieve the sensitivity and accuracy that can be used in clinical environments. Summary of the invention

[0005] The purpose of this application is to provide an application-level medical large model training, application method, equipment, medium and product, which can learn limited laws and possibilities under specific time and space. Compared with the general base large language model, it has stronger specific performance for clinical environment, ensuring sensitivity, accuracy and better safety.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides an application-level medical large model training method, the application-level medical large model training method comprising:

[0008] Segment the medical scene into several small worlds;

[0009] For each of the small worlds:

[0010] Acquiring doctor-patient conversation data in the small world;

[0011] Anonymizing the doctor-patient conversation data to obtain anonymized data;

[0012] Refining the anonymized data to obtain training data suitable for the base large language model; the training data includes: a knowledge base and prompt words;

[0013] The training data is used to train a base large language model to obtain an application-level medical large model suitable for the small world.

[0014] Optionally, the anonymized data includes audio data and video data; refining the anonymized data to obtain training data suitable for the base large language model specifically includes:

[0015] transcribing the audio data into text data;

[0016] Extracting modal features of the audio data, the video data, and the text data respectively to obtain audio modal features, video modal features, and text modal features;

[0017] Fusing the audio modality feature, the video modality feature, and the text modality feature to obtain a fused feature;

[0018] Performing emotion recognition on the fusion features to obtain an emotion recognition result;

[0019] Extracting character body features, character facial features, and scene features from the video modality features;

[0020] Inputting the character's body features, character's facial features and scene features into an action detection model to obtain character actions;

[0021] The emotion recognition result, the character action and the text data are used as training data.

[0022] Optionally, emotion recognition is performed on the fusion feature to obtain an emotion recognition result, which specifically includes:

[0023] The fusion features are subjected to emotion recognition using a long short-term memory neural network or a self-attention model, and the result of the emotion recognition is converted into an emotion coordinate value form to obtain an emotion recognition result; the emotion coordinate value form is in the form of (V, A, L); wherein V represents the valence of the emotion, A represents the arousal of the emotion, and L represents the liking of the emotion.

[0024] Optionally, the base large language model is trained using the training data to obtain an application-level medical large model suitable for the small world, specifically including:

[0025] The base large language model is trained with the training data using fine-tuning and prompt tuning methods to obtain an application-level medical large model suitable for the small world.

[0026] Optionally, the base large language model is trained using the training data to obtain an application-level medical large model suitable for the small world, specifically including:

[0027] Performing independent language model evaluation, RAG pipeline evaluation and keyword evaluation on the output of the base large language model respectively;

[0028] When the independent language model evaluation, RAG pipeline evaluation and keyword evaluation are all passed, the output of the base large language model is output; otherwise, an early warning display is output and the output of the base large language model is re-acquired.

[0029] In a second aspect, the present application provides an application-level medical large model application method, the application-level medical large model application method comprising:

[0030] Obtain information on medical guidance needs;

[0031] The guidance demand information is used as a prompt word and input into the application-level medical big model to obtain a guidance output; the application-level medical big model is trained by any of the application-level medical big model training methods described above.

[0032] Optionally, after inputting the medical guidance requirement information into the application-level medical big model to obtain the medical guidance output, the process further includes:

[0033] Obtaining the medical guidance demand information and the processed medical guidance output;

[0034] The medical guidance demand information and the processed medical guidance output are used to perform prompt tuning on the application-level medical big model.

[0035] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described application-level medical big model training methods or application-level medical big model application methods.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned application-level medical big model training methods or application-level medical big model application methods.

[0037] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned application-level medical big model training methods or application-level medical big model application methods.

[0038] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0039] The present application provides an application-level medical big model training, application method, equipment, medium and product. First, the medical scene is segmented to obtain several small worlds; then, for each of the small worlds: the doctor-patient dialogue data in the small world is obtained; the doctor-patient dialogue data is anonymized to obtain anonymized data; the anonymized data is refined to obtain training data suitable for the base big language model; the training data includes: knowledge base and prompt words; the base big language model is trained using the training data to obtain an application-level medical big model suitable for the small world. The present application defines the data source of the general base big language model as the "big world", divides this "big world" into multiple "small worlds" according to the scene, collects panoramic and full data in a specific small world, establishes a small world enhanced model, and learns the limited laws and possibilities under specific time and space. Compared with the general base big language model, it has stronger specific performance for the clinical environment, ensuring sensitivity, accuracy and better safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is an application environment diagram of an application-level medical large model training method in one embodiment of the present application;

[0042] Figure 2 A flowchart of an application-level medical large model training method provided in one embodiment of the present application;

[0043] Figure 3 A schematic diagram of a module implementation of an application-level medical large model training method provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of a method for enhancing the perception capability of a large language model through multimodal fusion provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a process flow of a large language model deep meaning enhancement prompt method based on prompt engineering provided in one embodiment of the present application;

[0046] Figure 6 A schematic diagram of an application-level medical large model replacing manual diagnosis guidance provided in an embodiment of the present application;

[0047] Figure 7A schematic diagram of manual guidance for training an application-level medical large model provided in one embodiment of the present application.

[0048] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] The application-level medical large model training method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the segmented medical scene to the server 104. After the server 104 receives the segmented medical scene, for the obtained small world, the server 104 obtains the doctor-patient dialogue data in the small world; anonymizes the doctor-patient dialogue data to obtain anonymized data; refines the anonymized data to obtain training data suitable for the base large language model; the training data includes: a knowledge base and prompt words; use the training data to train the base large language model to obtain an application-level medical large model suitable for the small world. The server 104 can feedback the obtained application-level medical large model to the terminal 102. In addition, in some embodiments, the application-level medical large model training method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly process the segmented small world, or the server 104 can obtain the segmented small world from the data storage system for processing.

[0052] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0053] In an exemplary embodiment, Figure 2 As shown, an application-level medical large model training method is provided, which is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps A1 to A5. Among them:

[0054] A1. Segment the medical scene into several small worlds.

[0055] For each of the small worlds:

[0056] A2. Obtain doctor-patient conversation data in the small world.

[0057] A3. Anonymize the doctor-patient conversation data to obtain anonymized data.

[0058] A4. Refining the anonymized data to obtain training data suitable for the base large language model; the training data includes: a knowledge base and prompt words.

[0059] A5. Use the training data to train the base large language model to obtain an application-level medical large model suitable for the small world.

[0060] For details, please refer to Figure 3 In this embodiment, three modules, namely, the small world panoramic data acquisition module, the small world enhanced model module, and the small world application landing module, are used to realize the customized application of the medical big language model technology of the base big language model.

[0061] The small world panoramic data collection module is used to segment a wider medical scene and collect panoramic data in each small scene (i.e. small world) after segmentation. The collected data is mainly based on doctor-patient dialogue, and is collected from voice, text, video and other modalities.

[0062] The small world enhanced model module includes a small world knowledge refining submodule and a security warning system submodule. The small world knowledge refining submodule is used to anonymize the collected data after it is transcribed into text. The anonymization process is to use computer algorithms or manual identification of personally identifiable information (PHI) defined in the Health Insurance Portability and Accountability Act (HIPAA Act), including name, address, fingerprint, ID number, date of birth, medical insurance information, medical record number, telephone number, bill record, etc., and use virtual strings for replacement, for example: replace the name "Zhang San" with "[**NAME**]", replace the telephone number "138-1234-5678" with "[**TEL**]", and use computer programs or manual refinement after anonymization to form a knowledge base and prompt words that comply with the HIPAA Act and are applicable to a variety of base large language models (such as ChatGPT, Wenxin Yiyan, etc.). The security warning system submodule is used to evaluate the response generated by the large language model. If it passes the evaluation, it will be output normally. If it fails the evaluation, it will return to the large language model to reply again and output warning information. At the same time, the small-world enhancement model module can train the base large language model based on the knowledge base and prompt words output by the small-world knowledge refinement sub-module using fine-tuning and prompt tuning techniques, thereby obtaining a large language model dedicated to the small world. This model can solve most of the problems occurring in the small world.

[0063] Fine-tuning is the retraining of the base large language model. Keeping the parameters of the base large language model unchanged, the data set is changed to training data, and the parameters are slightly changed during the retraining process to form a preliminary medical training large language model, which can achieve a better effect. The specific training process is to fine-tune the model's hyperparameters (learning rate, batch size, number of network layers, activation function, regularization function) in continuous training with the goal of reducing the model's training loss as much as possible. The intuitive result is that when the model's training loss is reduced to a specific range, the values ​​of these hyperparameters are obtained, and the performance of the large language model with these hyperparameters will be more suitable for the scenarios covered by the corpus data.

[0064] Before applying a large language model in medical training, it is necessary to provide specific prompt instructions to the large language model to provide more information and promote the large language model to form a better response method. This process is called prompt tuning. Prompt tuning mainly refers to using standardized dialogues to improve the performance of the large language model in the deployment of the large language model. The result is that the response performance of the large language model using prompt tuning is better.

[0065] The process of using the training data to train the base large language model to obtain the application-level medical large model suitable for the small world also includes:

[0066] The output of the base large language model is respectively subjected to independent language model evaluation, RAG pipeline evaluation and keyword evaluation.

[0067] When the independent language model evaluation, RAG pipeline evaluation and keyword evaluation are all passed, the output of the base large language model is output; otherwise, an early warning display is output and the output of the base large language model is re-acquired.

[0068] Specifically, when the base language model generates a response text based on the questions in the training data, the response text is input into the early warning system module for evaluation:

[0069] The process of independent language model evaluation is: the reply text is input into the independent large language model for special evaluation through the Langchain.evaluator module, and the large language model is asked to score the reply text on helpfulness and harmlessness. The score of this evaluation is selected between 0 and 10 points. If the score of the reply text is lower than a certain score, it is considered to have failed the evaluation.

[0070] The process of RAG pipeline evaluation is: the text is automatically evaluated through the RAG pipeline, and the evaluation criteria are text fidelity (Faithfulness) and answer relevancy (Answer Relevancy). Text fidelity refers to the degree of match between the reply text and the text prompt given in the data pre-training stage, and the score is completed according to the degree of match. Answer relevance refers to generating several sentences that may be questions about the text based on the reply text, and then identifying the semantic similarity between the real question and these sentences that may be questions, and scoring them according to the average semantic similarity. Texts with scores below a certain value are considered to have failed the evaluation.

[0071] The process of keyword evaluation is as follows: During the internal verification of the intelligent medical assistant system, a number of low-quality replies were collated and stored as a "risk corpus". All low-quality replies stored in the "risk corpus" are tested for semantic similarity with the reply text. If the largest semantic similarity is greater than a certain value, it means that the reply text may be risky and is considered to fail the evaluation.

[0072] All three warning modules will output a pass / fail result. If any of them fails, the system will need to return to regenerate the text until all of them pass or the maximum number of attempts is reached. In the case of a failure, a warning will be generated and displayed on the user communication interface. If there is still a failure result after the maximum number of attempts, the system will automatically contact the doctor for manual processing.

[0073] The small world application landing module is used to deploy the trained small world large language model and system to a real clinical environment for application.

[0074] In clinical scenarios, patients and medical staff often have face-to-face conversations. This conversation covers video, audio, and text, and contains a large amount of information. It is difficult for a single text modality to fully express this information. Therefore, this embodiment also provides a technical method for enhancing the perception ability of a large language model through multimodal fusion, such as Figure 4 shown.

[0075] Wherein, B1, transcribing the audio data into text data.

[0076] B2. Extract modal features of the audio data, the video data, and the text data respectively to obtain audio modal features, video modal features, and text modal features.

[0077] In this embodiment, devices are deployed in each small world to complete the collection of audio data and video data.

[0078] After collecting the data, the data is preprocessed to filter out noise and blank segments, and the audio data is transcribed into text. Then the modal features of the audio data, video data, and text data are extracted respectively. The audio modal feature selection extracts audio embedding vector features, oGeMaps features, and MFCC features. The video modal feature uses the OpenPose model to extract facial, body, and scene features, and the ResNet model to extract context features. The text modal feature extracts the embedding vector.

[0079] B3. Fusing the audio modality feature, the video modality feature and the text modality feature to obtain a fused feature.

[0080] B4. Perform emotion recognition on the fusion features to obtain an emotion recognition result, wherein the emotion recognition result is in the form of emotion coordinate values.

[0081] In this embodiment, multimodal fusion is performed to perform continuous emotion recognition and obtain the emotion coordinate value:

[0082] After extracting the audio modality features, the video modality features and the text modality features, the features of each modality are merged and mapped using a cross-attention mechanism to complete the fusion of the features of each modality. The fused features are input into a long short-term memory neural network (LSTM) or a self-attention model (Self-Attention Transformer). In the long short-term memory neural network, the multi-layer LSTM unit can capture the long-term and short-term dependencies in the fused features, output a high-level representation of the emotional features, and then obtain the predicted emotional coordinate value through the fully connected layer; the self-attention model is divided into a decoder layer and an encoder layer. The decoder layer is responsible for extracting the high-level representation of the input fused features, and the encoder layer is responsible for converting these high-level representations into predicted emotional coordinate values ​​by capturing the connection between emotional features. The emotional coordinate value will be specifically expressed in the form of (V, A, L), where V represents the valence of the emotion, A represents the arousal of the emotion, and L represents the liking of the emotion.

[0083] B5. Extracting character body features, character facial features and scene features from the video modality features.

[0084] B6. Input the character's body features, character's facial features and scene features into an action detection model to obtain the character's actions.

[0085] In this embodiment, this step mainly performs video action detection: after the extracted body, face and context scene features are fused at the feature level, they are input into the trained action detection model. The action detection network is trained based on the facial, body and scene features extracted from the video modality. The body, face and scene features are spliced ​​or fused at the feature level to form a comprehensive feature. This comprehensive feature is input into the 3D convolutional network to capture the timing information. The output layer uses a fully connected layer to regress these features into various specific action categories that have been set, such as raising hands, clasping fists, etc. Each action category can obtain a regressed probability, and the specific character action is determined through the probability distribution of these actions.

[0086] B7. Use the emotion recognition result, the character action and the text data as training data.

[0087] The obtained emotion recognition results, the character actions and the text data are combined and input into the base large language model to obtain a response and perform the training process mentioned above.

[0088] Deep meaning enhancement hint technology for large language models based on hint engineering

[0089] In the application of the large language model perception enhancement technology based on multimodal fusion, some of the medical large models trained by the base large language model cannot well understand the input emotional coordinate values ​​and the meaning of character actions. This embodiment mainly addresses this problem and proposes a deep meaning enhancement prompt method to enable the large language model to better understand the deep meaning of the text content. Please refer to Figure 5 , Figure 5 Schematic diagram of the process of deep meaning enhancement prompting method for large language models based on prompt engineering.

[0090] After the emotion recognition results, the character actions and the text data are used as training data, the training data will be directly input into the medical large language model. However, some base large language models may not be able to understand or fully understand the perception enhancement information in the training data. Therefore, this embodiment chooses to further process the perception enhancement information in the training data.

[0091] First, a large language model dedicated to processing perception enhancement information is obtained from a general base large language model through methods such as prompt tuning. By providing specific prompt instructions to the large language model to provide more information, the large language model is encouraged to form a better response method. This process is called prompt tuning. The framework of the prompt tuning method used in this embodiment is: (1) input the purpose of the task, that is, "deep processing of perception enhancement information", (2) input the specific content of the perception enhancement information, and the meaning represented by the content (interpretation of the emotional coordinate value, such as: L represents the degree of liking, which can reflect the degree of preference of the subject's emotions), (3) input the object output by the task, which is the large language model for doctor-patient communication, and (4) input the format specification of the output content. Under the framework of this prompt tuning, the deep perception large language model can better understand the input perception enhancement information, and can complete the task that most base large language models cannot stably complete at present - understanding the deep meaning of the text, and outputting the perception enhancement information as text content that is easier for the large language model to understand, that is, deep meaning perception enhancement prompts. Then, the deep meaning perception enhancement prompts and perception enhancement information are input into the medical big language model together to obtain a better response output.

[0092] The method of prompt tuning is mainly to use a standardized sentence on the application side of the base large language model to improve the response effect of the large language model. Among them, step (1) in the above text is specifically to input to the large language model: "You need to process xx information." Step (2) is to provide the content and teaching of this information processing task: "xx information includes content A, content B, etc. Content A includes xxx and represents xxx." Step (3) is to control the output content: "Your output is for the large language model for doctor-patient communication, and this model needs to be able to better understand information A and information B." Step (4) is to control the format specification of the output: "Your output needs to be based on xxx and should be as concise as possible."

[0093] This embodiment divides the data source of the general large language model, namely the "big world", into multiple "small worlds", collects panoramic and full data in a specific small world, establishes a small world enhanced model, and learns the limited laws and possibilities under a specific time and space. Compared with the general base large language model, it has stronger specificity and better security.

[0094] Moreover, since this embodiment can be deployed on the base large language model, it has significantly lower costs and higher cost-effectiveness in input-output compared to projects that build the base large language model from scratch. Moreover, since this embodiment does not rely on a specific base large language model, it can be quickly deployed on base large language models such as ChatGPT, LLaMa, Claude, etc., and the corresponding base large language model can be selected according to needs, which has strong flexibility and adaptability.

[0095] In an exemplary embodiment, a method for applying an application-level medical large model is provided, comprising:

[0096] Obtain information on medical guidance needs.

[0097] The guidance demand information is input as a prompt word into the application-level medical big model to obtain a guidance output; the application-level medical big model is trained by the application-level medical big model training method described above.

[0098] And after getting the guidance output, it also includes:

[0099] The medical guidance requirement information and the medical guidance output are obtained. The medical guidance output is processed to obtain a medical guidance output that meets the medical guidance requirement as the processed medical guidance output.

[0100] The medical guidance demand information and the processed medical guidance output are used to perform prompt tuning on the application-level medical big model.

[0101] It should be noted that the application-level medical large model training method mentioned above focuses on how to train. In this embodiment, more attention is paid to the selection of training locations and the grasp of training details, so the following specific methods combining training and application are proposed.

[0102] The application mode of the medical big language model technology based on "small world enhancement" is divided into three sub-application modes according to the functions of the application-level medical big model:

[0103] (1) Replacement of manual consultation:

[0104] like Figure 6 As shown, to replace manual guidance, the model is deployed at a location within the hospital, and the model replaces the work of manual guidance.

[0105] As shown in the attached figure, this application model is divided into four steps, namely data collection, model development, model deployment, and maintenance and upgrade. This application model includes two responsible parties. The hospital is responsible for providing pre-training data, deployment locations, and manual supervisors for the medical big model, and the inventor is responsible for providing pre-training data processing, development of a small world-specific big language model, and regular system maintenance and upgrades.

[0106] Step S1: Data collection: The inventor arranges equipment collection points at the required locations of the hospital to collect sufficient corpus data. The collected data is the real conversation between the guide and the patient. For example, the patient asks: "What does this MRI look at? Does it have nuclear radiation? Is it okay not to do it?" The guide answers: "MRI does not have nuclear radiation, so you can do it without worry. If you don't do it, you can't find out the cause of the disease."

[0107] In step S2, model development, the inventor establishes a knowledge base and prompt words based on the collected corpus data, and completes fine-tuning and prompt tuning of the base large language model to form a large language model dedicated to the small world. This large language model is then connected to the security warning system for internal testing and verification. The internal testing and verification includes expert manual evaluation and machine automatic evaluation methods. Expert manual evaluation is an expert's manual evaluation of several replies of the large language model. The evaluation dimensions are: security, readability, completeness, factuality, and empathy of the reply content, and an evaluation of 1 to 5 points is performed in these six dimensions. The machine automatic evaluation uses the Langchain.evaluator module to call GPT-4 to conduct an overall evaluation of the helpfulness and security of the reply content of the large language model. The evaluation score ranges from 1 to 10 points.

[0108] Step S3: Model deployment. After the model passes internal test verification (internal test verification is the evaluation of the responses generated by the large language model mentioned above. Internal testing is a preliminary evaluation of the trained large language model, including manual scoring and machine scoring), the model can be deployed to the hospital's required location, and manual supervision personnel are introduced to record and process the warning information of the safety warning system. Among them, the warning information is the warning information issued without passing the "evaluation of the responses generated by the large language model" mentioned above.

[0109] Step S4 is maintenance and upgrade. The inventor regularly collects and records warning information, and after answering the questions reflected in the warning information, adds them to the knowledge base, and adds prompts for optimization to achieve regular update and optimization of the large language model dedicated to the small world.

[0110] Before applying the large language model for medical training, it is necessary to provide specific prompt instructions to the large language model to provide more information and promote the large language model to form a better response method. This process is called prompt tuning. The prompt tuning method framework used in this embodiment is: (1) input background information such as hospital departments, (2) input the content of the role to be played and the task information to be completed, and (3) input the standard framework and content tendency of the generated text. After prompt tuning, a complete large language model for medical training dedicated to the small world is formed, and the model can be directly put into use. For example, the model needs to be deployed at the main service desk on the first floor of the outpatient building of Hospital A. Then the input content of the prompt tuning can be: "You are now at the main service desk on the first floor of the outpatient building of Hospital A. You are a guide. You need to accurately answer their questions and give guidance based on the patients' questions. At the same time, your wording should be concise, clear and appropriate. There should be no negative content in the response, and the patient should not be misled."

[0111] This is the method for performing prompt tuning in the following section. In actual deployment, you only need to input fixed text into the large language model to complete the prompt tuning.

[0112] (2) Assist and enhance manual medical guidance.

[0113] Assisting and enhancing manual medical guidance means that it does not replace the work of manual medical guidance, but plays an auxiliary role in the work of manual medical guidance, such as: giving reminders when the manual medical guidance's emotions are perceived to be abnormal, correcting the manual medical guidance's answers, and helping the manual medical guidance to retrieve information.

[0114] The process of this sub-application mode is the same as the sub-application mode replacing manual guidance, and the steps can also be referred to above.

[0115] (3) Training of manual medical guidance.

[0116] Training manual medical guides means not participating in clinical medical guidance work, but participating in the training of manual medical guides. Standardized training is provided to manual medical guides in the form of simulated real-life scenario dialogues, so that they can accumulate knowledge and experience and be better qualified for medical guidance work.

[0117] like Figure 7 As shown, this application model is divided into four steps, namely data collection, model development, personnel training and regular. This application model includes two responsible parties. The hospital is responsible for providing pre-training data, data collection locations and training personnel for the medical big model, and the inventor is responsible for providing pre-training data processing, small world dedicated big language model development and personnel training, as well as regular knowledge base updates.

[0118] In step S1, data collection, the inventor arranges equipment collection points at the required locations of the hospital to collect sufficient data on doctor-patient dialogues. For example, the content of the dialogue is as follows: The patient asks: "What does this MRI look at? Does it have nuclear radiation? Is it okay not to do it?" The medical guide answers: "MRI does not have nuclear radiation, so you can do it without worry. There is no way to find out the cause of the disease without it."

[0119] Step S2: Model development. The inventor deploys a remote teaching platform, establishes a knowledge base and prompt words based on the collected corpus data, and completes fine-tuning and prompt tuning of the base large language model to form a medical training large language model dedicated to the small world. Fine-tuning is the retraining of the base large language model. Keep the parameters of the base large language model unchanged, change the data set to the corpus data collected in step S1, and make slight changes to the parameters during the retraining process to form a preliminary medical training large language model. This model can achieve a better effect. Subsequently, before the application of the medical training large language model, it is necessary to provide specific prompt instructions to the large language model to provide more information and promote the large language model to form a better response method. This process is called prompt tuning. The prompt tuning method framework used in this technology is: (1) input background information such as hospital departments, (2) input the role content to be played and the task information to be completed, and (3) input the standard framework and content tendency of the generated text. After prompt tuning, a complete medical training large language model dedicated to the small world is formed, and the model can be directly put into use. For example, if the model needs to be deployed at the main service desk on the first floor of the outpatient building of Hospital A, the input content for prompt tuning can be: "You are now at the main service desk on the first floor of the outpatient building of Hospital A. You are a guide. You need to accurately answer their questions and give guidance based on the patients' questions. At the same time, your wording should be concise, clear and appropriate. There should be no negative content in the response, and the patients should not be misled." The model can provide real-life scenario simulation, simulating patients who come for consultation to ask questions to trainers, evaluate the trainers' responses, and correct the trainers' mistakes.

[0120] Step S3: Personnel training: hospital training personnel interact with the model to conduct training.

[0121] Step S4 is regular updating, and the inventor regularly completes the updating of the knowledge base.

[0122] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store doctor-patient conversation data or guidance demand information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an application-level medical large model training method or an application-level medical large model application method is implemented.

[0123] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0125] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0126] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0128] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0129] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for training an application-level medical large model, characterized in that: The application-level medical large model training method comprises: Segment the medical scene into several small worlds; For each of the small worlds: Acquiring doctor-patient conversation data in the small world; Anonymizing the doctor-patient conversation data to obtain anonymized data; Refining the anonymized data to obtain training data suitable for the base large language model; the training data includes: a knowledge base and prompt words; The training data is used to train a base large language model to obtain an application-level medical large model suitable for the small world.

2. The application-level medical large model training method according to claim 1 is characterized in that: The anonymized data includes audio data and video data; the anonymized data is refined to obtain training data suitable for the base large language model, specifically including: transcribing the audio data into text data; Extracting modal features of the audio data, the video data, and the text data respectively to obtain audio modal features, video modal features, and text modal features; Fusing the audio modality feature, the video modality feature, and the text modality feature to obtain a fused feature; Performing emotion recognition on the fusion features to obtain an emotion recognition result; Extracting character body features, character facial features, and scene features from the video modality features; Inputting the character's body features, character's facial features and scene features into an action detection model to obtain character actions; The emotion recognition result, the character action and the text data are used as training data.

3. The application-level medical large model training method according to claim 2 is characterized in that: Performing emotion recognition on the fusion features to obtain emotion recognition results specifically includes: The fusion features are subjected to emotion recognition using a long short-term memory neural network or a self-attention model, and the result of the emotion recognition is converted into an emotion coordinate value form to obtain an emotion recognition result; the emotion coordinate value form is in the form of (V, A, L); wherein V represents the valence of the emotion, A represents the arousal of the emotion, and L represents the liking of the emotion.

4. The application-level medical large model training method according to claim 1 is characterized in that: The training data is used to train the base large language model to obtain an application-level medical large model suitable for the small world, specifically including: The base large language model is trained with the training data using fine-tuning and prompt tuning methods to obtain an application-level medical large model suitable for the small world.

5. The application-level medical large model training method according to claim 4 is characterized in that: The training data is used to train the base large language model to obtain an application-level medical large model suitable for the small world, specifically including: Performing independent language model evaluation, RAG pipeline evaluation and keyword evaluation on the output of the base large language model respectively; When the independent language model evaluation, RAG pipeline evaluation and keyword evaluation are all passed, the output of the base large language model is output; otherwise, an early warning display is output and the output of the base large language model is re-acquired.

6. An application-level medical large model application method, characterized in that: The application method of the application-level medical large model includes: Obtain information on medical guidance needs; The guidance demand information is used as a prompt word and input into the application-level medical big model to obtain the guidance output; the application-level medical big model is trained by the application-level medical big model training method described in any one of claims 1-5.

7. The application-level medical large model application method according to claim 6, characterized in that: After inputting the guidance requirement information into the application-level medical big model to obtain the guidance output, the method further includes: Obtaining the medical guidance demand information and the processed medical guidance output; The medical guidance demand information and the processed medical guidance output are used to perform prompt tuning on the application-level medical big model.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the application-level medical big model training method described in any one of claims 1 to 5 or the application-level medical big model application method described in claim 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the application-level medical big model training method described in any one of claims 1-5 or the application-level medical big model application method described in claim 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the application-level medical big model training method described in any one of claims 1-5 or the application-level medical big model application method described in claim 6.

Citation Information

Patent Citations

  • Large language model training method and device

    CN118349852A

  • Intelligent medical assistant system based on large language model and interaction method and product thereof

    CN118471550A

Cited By

  • Multi-memory large model enhancement method and system based on adaptive resonance theory

    CN121212364A

  • A multi-memory large model enhancement method and system based on adaptive resonance theory

    CN121212364B