Medical agent construction method and device and related product

Abstract

The invention discloses a medical agent construction method and device and a related product. Inputting the medical sample question set into a medical multi-modal pre-training model under the intelligent agent interaction framework, and performing prediction processing on the medical sample question set through the medical multi-modal pre-training model to obtain a medical prediction answer set; and according to the medical reference answer set and the medical prediction answer set, parameters of the medical multi-modal pre-training model are finely adjusted under an agent interaction framework until the finely-adjusted model meets a fine adjustment cut-off condition, and a medical agent group is obtained after fine adjustment is finished. Visibly, the agents with different medical capabilities can be constructed under the action of the agent interaction framework, and the agents can generate different medical answers based on different medical questions, so that compared with related technologies, the method has the advantages that the efficiency is improved, and the user experience is improved. The intelligent agent in the application can give corresponding interpretable medical answers under the action of medical questions, so that the credibility of medical processing results can be improved.

Description

Technical Field

[0001] The present application relates to the field of medical intelligent body technology, and in particular to a method, device and related products for constructing a medical intelligent body. Background Art

[0002] With the continuous development of artificial intelligence, its application in the medical field has become increasingly extensive and in-depth, providing certain technical support for medical personnel. At present, medical models are usually used in related technologies to process medical images to obtain processing results corresponding to medical images, so as to improve the service efficiency of medical personnel based on the processing results. However, most medical models are usually constructed using black box technology, that is, the interpretability of the medical model is not strong, and it is difficult for medical personnel to trust the processing results obtained using the medical model. Therefore, how to improve the trust in medical processing results is a key issue that technicians in this field are concerned about. Summary of the invention

[0003] Based on the above problems, the present application provides a method, device and related products for constructing a medical intelligent entity to improve the trust in medical treatment results. The embodiments of the present application disclose the following technical solutions:

[0004] In a first aspect, the present application discloses a method for constructing a medical intelligent agent, comprising:

[0005] Obtaining a medical sample question set and a medical benchmark answer set corresponding to the medical sample question set, wherein one medical benchmark answer corresponds to one medical sample question;

[0006] Inputting the medical sample question set into a medical multimodal pre-training model under an agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question, and the medical multimodal pre-training model is used to support obtaining medical answers according to medical questions;

[0007] According to the medical benchmark answer set and the medical prediction answer set, the parameters of the medical multimodal pre-trained model are fine-tuned under the agent interaction framework until the fine-tuned model meets the fine-tuning cutoff condition, and a medical agent group is obtained after the fine-tuning is completed, wherein the medical agent group includes medical agents with medical detection capabilities, medical agents with medical retrieval capabilities, medical agents with report generation capabilities, and medical agents with medical backtracking capabilities.

[0008] Optionally, after obtaining the medical sample question set and the medical reference answer set corresponding to the medical sample question set, the method further includes:

[0009] Acquire a medical sample image set corresponding to the medical sample question set, wherein one medical sample image corresponds to one medical sample question;

[0010] The step of inputting the medical sample question set into the medical multimodal pre-training model under the agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, comprises:

[0011] The medical sample question set and the medical sample image set are input into a medical multimodal pre-training model under an intelligent agent interaction framework, so as to perform prediction processing on the medical sample question set and the medical sample image set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question and one medical sample image.

[0012] Optionally, after inputting the medical sample question set into the medical multimodal pre-training model under the agent interaction framework, and before performing prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, the method further includes:

[0013] The medical sample question set is judged and processed under the intelligent agent interaction framework to obtain a medical question type set corresponding to the medical sample question set, wherein one medical question type corresponds to one medical sample question.

[0014] Optionally, the medical sample question set includes a first sample question, a second sample question, a third sample question and a fourth sample question, and the medical sample image set includes a first sample image, a second sample image, a third sample image and a fourth sample image, wherein the first sample question corresponds to the first sample image, the second sample question corresponds to the second sample image, the third sample question corresponds to the third sample image, and the fourth sample question corresponds to the fourth sample image;

[0015] The method of performing prediction processing on the medical sample question set and the medical sample image set by the medical multimodal pre-training model to obtain a medical prediction answer set includes:

[0016] If the medical question type corresponding to the first sample question is a medical detection type, performing a medical detection on the first sample image according to the first sample question by using the medical multimodal pre-trained model to obtain a first predicted answer;

[0017] If the medical question type corresponding to the second sample question is a medical retrieval type, performing a medical retrieval on the second sample image according to the second sample question by using the medical multimodal pre-trained model to obtain a second predicted answer;

[0018] If the medical question type corresponding to the third sample question is a report generation type, generating a report on the third sample image according to the third sample question by using the medical multimodal pre-trained model to obtain a third predicted answer;

[0019] If the medical question type corresponding to the fourth sample question is a medical retrospective type, performing medical retrospective on the fourth sample image according to the fourth sample question by using the medical multimodal pre-trained model to obtain a fourth predicted answer;

[0020] A medical prediction answer set is obtained according to the first prediction answer, the second prediction answer, the third prediction answer and the fourth prediction answer.

[0021] Optionally, the intelligent agent interaction framework includes an AutoGen framework, the medical detection type is used to indicate sample image detection for the first sample image, the medical retrieval type is used to indicate retrieval of similar images for the second sample image, the report generation type is used to indicate generation of a medical report based on the third sample image, and the medical backtracking type is used to indicate backtracking of historical images for the fourth sample image.

[0022] Optionally, also include:

[0023] Acquire a first medical question submitted by a target subject and a first medical image corresponding to the first medical question;

[0024] If the medical question type corresponding to the first medical question is a medical detection type, calling a medical agent with medical detection capability in the medical agent group to perform image detection on the first medical image according to the first medical question to obtain a first medical answer;

[0025] The first medical answer is displayed to the target subject.

[0026] Optionally, also include:

[0027] Obtaining a second medical question submitted by the target subject;

[0028] If the medical question type corresponding to the second medical question is a medical retrieval type, calling a medical agent with medical retrieval capability in the medical agent group to perform image retrieval on the first medical image according to the second medical question to obtain a second medical answer;

[0029] Alternatively, if the medical question type corresponding to the second medical question is a report generation type, calling a medical agent with report generation capability in the medical agent group to generate a report on the first medical image according to the second medical question to obtain a second medical answer;

[0030] Alternatively, if the medical question type corresponding to the second medical question is a medical backtracking type, calling a medical agent with medical backtracking capability in the medical agent group to perform image backtracking on the first medical image according to the second medical question to obtain a second medical answer;

[0031] The second medical answer is displayed to the target subject.

[0032] In a second aspect, the present application discloses a device for constructing a medical intelligent agent, comprising:

[0033] A question answer set acquisition unit, used to acquire a medical sample question set and a medical reference answer set corresponding to the medical sample question set, wherein one medical reference answer corresponds to one medical sample question;

[0034] A prediction answer set obtaining unit, used for inputting the medical sample question set into the medical multimodal pre-training model under the agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question, and the medical multimodal pre-training model is used to support obtaining medical answers according to medical questions;

[0035] The intelligent agent group obtaining unit is used to fine-tune the parameters of the medical multimodal pre-training model under the intelligent agent interaction framework according to the medical benchmark answer set and the medical prediction answer set, until the fine-tuned model meets the fine-tuning cutoff condition, and obtain a medical intelligent agent group after the fine-tuning, wherein the medical intelligent agent group includes a medical intelligent agent with medical detection capability, a medical intelligent agent with medical retrieval capability, a medical intelligent agent with report generation capability, and a medical intelligent agent with medical backtracking capability.

[0036] Optionally, the device further comprises:

[0037] A sample image set acquisition unit, used to acquire a medical sample image set corresponding to the medical sample question set, wherein one medical sample image corresponds to one medical sample question;

[0038] The predicted answer set obtaining unit is specifically used for:

[0039] The medical sample question set and the medical sample image set are input into a medical multimodal pre-training model under an intelligent agent interaction framework, so as to perform prediction processing on the medical sample question set and the medical sample image set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question and one medical sample image.

[0040] Optionally, the device further comprises:

[0041] A question type set obtaining unit is used to judge and process the medical sample question set under the intelligent agent interaction framework to obtain a medical question type set corresponding to the medical sample question set, wherein one medical question type corresponds to one medical sample question.

[0042] Optionally, the predicted answer set obtaining unit is specifically used to:

[0043] If the medical question type corresponding to the first sample question is a medical detection type, performing a medical detection on the first sample image according to the first sample question by using the medical multimodal pre-trained model to obtain a first predicted answer;

[0044] If the medical question type corresponding to the second sample question is a medical retrieval type, performing a medical retrieval on the second sample image according to the second sample question by using the medical multimodal pre-trained model to obtain a second predicted answer;

[0045] If the medical question type corresponding to the third sample question is a report generation type, generating a report on the third sample image according to the third sample question by using the medical multimodal pre-trained model to obtain a third predicted answer;

[0046] If the medical question type corresponding to the fourth sample question is a medical retrospective type, performing medical retrospective on the fourth sample image according to the fourth sample question by using the medical multimodal pre-trained model to obtain a fourth predicted answer;

[0047] A medical prediction answer set is obtained according to the first prediction answer, the second prediction answer, the third prediction answer and the fourth prediction answer.

[0048] Optionally, the intelligent agent interaction framework includes an AutoGen framework, the medical detection type is used to indicate sample image detection for the first sample image, the medical retrieval type is used to indicate retrieval of similar images for the second sample image, the report generation type is used to indicate generation of a medical report based on the third sample image, and the medical backtracking type is used to indicate backtracking of historical images for the fourth sample image.

[0049] Optionally, the device further comprises:

[0050] A question image acquisition unit, configured to acquire a first medical question submitted by a target subject and a first medical image corresponding to the first medical question;

[0051] a medical image detection unit, configured to, if the medical question type corresponding to the first medical question is a medical detection type, call a medical agent with medical detection capability in the medical agent group to perform image detection on the first medical image according to the first medical question to obtain a first medical answer;

[0052] The first medical answer presenting unit is used to present the first medical answer to the target object.

[0053] Optionally, the device further comprises:

[0054] A medical question obtaining unit, configured to obtain a second medical question submitted by the target subject;

[0055] a medical image retrieval unit, configured to, if the medical question type corresponding to the second medical question is a medical retrieval type, call a medical agent with medical retrieval capability in the medical agent group to perform image retrieval on the first medical image according to the second medical question to obtain a second medical answer;

[0056] Alternatively, a medical report generating unit is configured to, if the medical question type corresponding to the second medical question is a report generating type, call a medical agent with report generating capability in the medical agent group to generate a report on the first medical image according to the second medical question, so as to obtain a second medical answer;

[0057] Alternatively, a medical image backtracking unit is configured to, if the medical question type corresponding to the second medical question is a medical backtracking type, call a medical agent with medical backtracking capability in the medical agent group to perform image backtracking on the first medical image according to the second medical question to obtain a second medical answer;

[0058] The second medical answer presenting unit is used to present the second medical answer to the target object.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0060] Memory for storing computer programs;

[0061] A processor is used to implement the steps of the above-mentioned method for constructing a medical intelligent agent when executing the computer program.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned medical agent-based construction method are implemented.

[0063] Compared with the prior art, this application has the following beneficial effects:

[0064] In this application, we first obtain a medical sample question set and a medical benchmark answer set corresponding to the medical sample question set; then, under the agent interaction framework, we input the medical sample question set into the medical multimodal pre-training model, so as to predict and process the medical sample question set through the medical multimodal pre-training model, and obtain a medical prediction answer set; finally, according to the medical benchmark answer set and the medical prediction answer set, we fine-tune the parameters of the medical multimodal pre-training model under the agent interaction framework until the fine-tuned model meets the fine-tuning cutoff condition, and the fine-tuning is completed to obtain a medical agent group. It should be noted that the medical agent group includes medical agents with medical detection capabilities, medical agents with medical retrieval capabilities, medical agents with report generation capabilities, and medical agents with medical backtracking capabilities.

[0065] It can be seen that in the present application, intelligent agents with different medical capabilities can be constructed under the action of the intelligent agent interaction framework. These intelligent agents can generate different medical answers based on different medical problems. Compared with the models in related technologies that can only give image results, the intelligent agents in the present application can give corresponding explainable medical answers under the action of medical problems, thereby improving the trust in the medical treatment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0067] Figure 1 A flowchart of a method for constructing a medical intelligent agent provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of constructing a medical intelligent agent in a method for constructing a medical intelligent agent provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of the output of a medical agent with medical detection capabilities in a method for constructing a medical agent provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of the output of a medical agent with medical retrieval capability in a method for constructing a medical agent provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of the output of a medical agent with report generation capability in a method for constructing a medical agent provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of the structure of a medical intelligent body construction device provided in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] It should be noted that the medical intelligent body construction method, device and related products provided in this application are used in the field of medical intelligent body technology. The above is only an example and does not limit the application field of the method and device name provided in this application.

[0076] As described above, with the continuous development of artificial intelligence, its application in the medical field has become increasingly extensive and in-depth, providing certain technical support for medical personnel. At present, in related technologies, medical models are usually used to process medical images to obtain processing results corresponding to medical images, including image segmentation results for medical images, so that under the effect of the processing results, the service efficiency of medical personnel can be improved to a certain extent.

[0077] However, most medical models are usually constructed using black box technology. In this case, the processing results predicted by the medical model are only image results for medical images, which lack a certain degree of predictive interpretability, making it difficult for medical personnel to trust the processing results obtained using the medical model. Therefore, how to improve the trust in medical processing results is a key issue that technicians in this field are concerned about.

[0078] Therefore, the inventors proposed the technical solution of this application, aiming to improve the trust in the medical treatment results. In the technical solution of this application, a medical sample question set and a medical benchmark answer set corresponding to the medical sample question set can be first obtained. After that, the medical sample question set is input into the medical multimodal pre-training model under the intelligent agent interaction framework, so as to predict and process the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set. Finally, according to the medical benchmark answer set and the medical prediction answer set, the parameters of the medical multimodal pre-training model are fine-tuned under the intelligent agent interaction framework until the fine-tuned model meets the fine-tuning cutoff condition, and the medical agent group is obtained after the fine-tuning. It should be noted that the medical agent group includes medical agents with medical detection capabilities, medical agents with medical retrieval capabilities, medical agents with report generation capabilities, and medical agents with medical backtracking capabilities.

[0079] It can be seen that in the present application, intelligent agents with different medical capabilities can be constructed under the action of the intelligent agent interaction framework. These intelligent agents can generate different medical answers based on different medical problems. Compared with the models in related technologies that can only give image results, the intelligent agents in the present application can give corresponding explainable medical answers under the action of medical problems, thereby improving the trust in the medical treatment results.

[0080] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0081] Next, the technical terms that may appear in this application are explained.

[0082] Medical agent: The medical agent in this application can output certain medical answers based on the medical input of the object, and can provide medical explanation information to the object in an interactive dialogue mode.

[0083] AutoGen: An Open-Source Programming Framework for AgenticAI that supports the construction of intelligent agents.

[0084] The following is an example of an embodiment to illustrate a method for constructing a medical intelligent agent provided by the present application. Figure 1 , which is a flow chart of a method for constructing a medical intelligent agent provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0085] S101: Obtain a medical sample question set and a medical benchmark answer set corresponding to the medical sample question set.

[0086] In this application, a medical benchmark answer corresponds to a medical sample question, and the medical benchmark answer set can be used as a training label for subsequent training of medical agents. For example, if the medical sample question is question A, and the medical benchmark answer corresponding to the medical sample question is answer A, then it is necessary to train the medical multimodal pre-training model in the subsequent process to generate answer A, so as to construct a medical agent.

[0087] S102: Inputting the medical sample question set into the medical multimodal pre-training model under the intelligent agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set.

[0088] In this step, a medical prediction answer corresponds to a medical sample question, that is, each medical sample question has a corresponding medical prediction answer and a medical benchmark answer, so that the model parameters can be adjusted based on the medical prediction answer and the medical benchmark answer in the subsequent process. And the medical multimodal pre-training model is used to support obtaining medical answers based on medical questions. The medical multimodal pre-training model is trained based on medical multimodal data (medical sample images and medical description text sets corresponding to the medical sample images). During the training process, the image visual features, image local features, and image global features corresponding to the medical sample images can be predicted in stages to obtain the medical description text for each stage. At this time, in this application, a medical agent is constructed under the agent interaction framework based on the medical multimodal pre-training model, so that the constructed agent can have an interpretable output.

[0089] It should be noted that after obtaining the medical sample problem set and the medical benchmark answer set corresponding to the medical sample problem set, the present application can also obtain the medical sample image set corresponding to the medical sample problem set, where one medical sample image corresponds to one medical sample problem, that is, there is a corresponding medical sample problem for each medical sample image. For example, if the medical sample image is image A, then the medical sample problem corresponding to the medical sample image may be question A (which abnormal areas exist in image A).

[0090] Specifically, in the present application, under the intelligent agent interaction framework, a medical sample question set and a medical sample image set can be input into a medical multimodal pre-training model, so as to predict and process the medical sample question set and the medical sample image set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to a medical sample question and a medical sample image, that is, the medical multimodal pre-training model can predict the medical sample image under the prompt of the medical sample question to obtain a medical prediction answer. In this way, the output of the model output can be explained to a certain extent, thereby improving the trust in the medical processing results.

[0091] It should also be noted that the medical sample image can be a medical image taken for a patient, and the image type of the medical sample image can be a CT type or an MRI type, which is not specifically limited here. The output of interpretability can be understood as an interpretation process or interpretation text only for medical images or medical problems, that is, an intermediate interpretation process corresponding to the conclusion of the medical problem is given, which can facilitate the subsequent diagnosis of patients to a certain extent.

[0092] It is understandable that, since the medical multimodal pre-trained model can predict the medical description text of each stage of obtaining medical images, further, the present application can train the medical multimodal pre-trained model to output medical answers to medical images more specifically under the action of medical questions. And the intelligent agent interaction framework includes the AutoGen framework. Since the intelligent agent interaction framework has the capabilities of dialogue interaction, dialogue programming, autonomous planning and role customization, the present application can construct medical agents with different capabilities (i.e., these medical agents play different roles) under the action of the intelligent agent interaction framework.

[0093] It should be further explained that a medical prediction answer can also correspond to multiple medical sample questions, that is, when it comes to interactive scenarios, medical personnel can ask multiple medical sample questions for a medical sample image. At this time, the medical multimodal pre-training model can make predictions based on different medical sample questions to obtain medical prediction answers corresponding to different medical sample questions.

[0094] Furthermore, after the medical sample problem set is input into the medical multimodal pre-training model under the intelligent agent interaction framework, and before the medical sample problem set is predicted and processed by the medical multimodal pre-training model to obtain the medical prediction answer set, the medical sample problem set can also be judged and processed under the intelligent agent interaction framework to obtain a set of medical problem types corresponding to the medical sample problem set, where one medical problem type corresponds to one medical sample problem. In this way, a clear framework can be provided for the prediction of subsequent models, which facilitates the model to understand the nature of the problem more quickly, thereby making more accurate judgments and decisions, and can also improve the prediction efficiency to a certain extent.

[0095] In an achievable implementation, the medical sample problem set includes a first sample problem, a second sample problem, a third sample problem, and a fourth sample problem, and the medical sample image set includes a first sample image, a second sample image, a third sample image, and a fourth sample image, wherein the first sample problem corresponds to the first sample image, the second sample problem corresponds to the second sample image, the third sample problem corresponds to the third sample image, and the fourth sample problem corresponds to the fourth sample image. It should be noted that the number of sample problems and sample images listed here are only some examples, and the specific number can also be determined in the actual model fine-tuning process, and is not specifically limited here.

[0096] Specifically, if the medical question type corresponding to the first sample question is a medical detection type, a medical detection is performed on the first sample image according to the first sample question through the medical multimodal pre-trained model to obtain a first predicted answer, wherein the medical detection type is used to indicate that a sample image detection is performed on the first sample image. That is, at this time, the medical multimodal pre-trained model can realize the recognition and extraction of abnormal areas of the sample image, so as to provide medical personnel with an interpretation of the abnormal areas of the sample image through an interactive dialogue mode.

[0097] If the medical question type corresponding to the second sample question is a medical retrieval type, a medical retrieval is performed on the second sample image according to the second sample question through the medical multimodal pre-training model to obtain a second predicted answer, wherein the medical retrieval type is used to indicate the retrieval of similar images for the second sample image. That is, at this time, the medical multimodal pre-training model can retrieve similar medical images similar to the second sample image based on the medical database, which is convenient for subsequent similarity comparison between images. And the medical database includes medical images and medical texts for medical images. At this time, the present application can retrieve medical texts in the medical database based on the second sample image, so as to achieve efficient image and text retrieval efficiency.

[0098] If the medical question type corresponding to the third sample question is a report generation type, a report is generated for the third sample image according to the third sample question through the medical multimodal pre-trained model to obtain a third predicted answer, wherein the report generation type is used to indicate the generation operation of a medical report according to the third sample image. That is, at this time, the medical multimodal pre-trained model can use natural language to generate an understandable, accurate interpretation and complete description of the third sample image, thereby realizing automatic generation of medical reports for medical personnel, facilitating report standardization, and thus saving time.

[0099] If the medical question type corresponding to the fourth sample question is a medical retrospective type, the fourth sample image is medically retrospected according to the fourth sample question through the medical multimodal pre-trained model to obtain a fourth predicted answer, wherein the medical retrospective type is used to indicate retrospection of historical images of the fourth sample image. That is, at this time, the medical multimodal pre-trained model can parse and obtain the medical patient corresponding to the fourth sample image, so as to retrospect the historical images of the medical patient based on the medical database, so that the retrospectively obtained historical images and the fourth sample image can be compared (comparing the change trend between the images), so as to generate a fourth predicted answer according to the change trend, so as to provide certain diagnostic tips for medical personnel.

[0100] Finally, a medical prediction answer set can be determined based on the first prediction answer, the second prediction answer, the third prediction answer and the fourth prediction answer, so that in the subsequent process, the medical multimodal pre-training model can be fine-tuned based on the medical benchmark answer set and the medical prediction answer set to obtain a medical agent group.

[0101] S103: According to the medical benchmark answer set and the medical prediction answer set, fine-tune the parameters of the medical multimodal pre-trained model under the agent interaction framework until the fine-tuned model meets the fine-tuning cutoff condition, and the medical agent group is obtained after the fine-tuning is completed.

[0102] In this step, the medical agent group includes medical agents with medical detection capabilities, medical agents with medical retrieval capabilities, medical agents with report generation capabilities, and medical agents with medical backtracking capabilities. It can be understood that the present application can generate medical agents with different interaction capabilities under the agent interaction framework, and these medical agents can work together in subsequent applications to improve the accuracy and convenience of interaction.

[0103] It should be noted that in this application, the parameters of the medical multimodal pre-training model can be fine-tuned according to the difference between the medical benchmark answer set and the medical prediction answer set, wherein a medical benchmark answer in the medical benchmark answer set corresponds to a medical prediction answer in the medical prediction answer set. And the fine-tuning cutoff conditions include the cutoff conditions for constructing a medical intelligent agent with medical detection capability, the cutoff conditions for constructing a medical intelligent agent with medical retrieval capability, the cutoff conditions for constructing a medical intelligent agent with report generation capability, and the cutoff conditions for constructing a medical intelligent agent with medical backtracking capability.

[0104] It should also be noted that before obtaining a medical intelligent agent with medical detection capabilities, a medical intelligent agent with medical retrieval capabilities, a medical intelligent agent with report generation capabilities, and a medical intelligent agent with medical backtracking capabilities based on a medical multimodal pre-training model, the present application may also conduct a survey of medical personnel to determine what kind of explainable processes the medical personnel need the model to provide in the course of their work, as well as the medical questions that medical personnel are most likely to raise, so that the medical multimodal pre-training model can be fine-tuned based on this survey information to construct a medical intelligent agent with medical detection capabilities, a medical intelligent agent with medical retrieval capabilities, a medical intelligent agent with report generation capabilities, and a medical intelligent agent with medical backtracking capabilities.

[0105] like Figure 2 As shown, Figure 2 A schematic diagram of constructing a medical intelligent agent in a method for constructing a medical intelligent agent provided in an embodiment of the present application. Figure 2 In the system, a medical agent with medical detection capability, a medical agent with medical retrieval capability, a medical agent with report generation capability, and a medical agent with medical backtracking capability can be constructed through a medical multimodal pre-training model under the action of the agent interaction framework. Figure 2 (a) can be a medical intelligent agent with medical detection capabilities, Figure 2 (b) can be a medical agent with medical search capabilities, Figure 2 (c) can be a medical agent with report generation capability, Figure 2 (d) It can be a medical intelligent agent with medical backtracking capabilities.

[0106] Next, the application process of medical agents with medical detection capabilities, medical agents with medical retrieval capabilities, medical agents with report generation capabilities, and medical agents with medical backtracking capabilities is introduced. Specifically, first, the first medical question submitted by the target object and the first medical image corresponding to the first medical question can be obtained, where the target object can be understood as a medical personnel. After that, the type of medical question corresponding to the first medical question can be determined. If the type of medical question corresponding to the first medical question is a medical detection type, a medical agent with medical detection capabilities in the medical agent group is called to perform image detection on the first medical image according to the first medical question, obtain a first medical answer, and show the first medical answer to the target object.

[0107] like Figure 3 As shown, Figure 3 This is a schematic diagram of the output of a medical intelligent agent with medical detection capabilities in a method for constructing a medical intelligent agent provided in an embodiment of the present application. Figure 3Four situations are shown in the figure. In the first situation, the medical intelligent agent with medical detection capability can detect the existence of occipital lobe cyst in the first medical image, and output the first medical answer based on the occipital lobe cyst (a cystic mass-like mass can be seen in the right occipital lobe); in the second situation, the medical intelligent agent with medical detection capability can detect the existence of T1WI low signal in the first medical image, and output the first medical answer based on the T1WI low signal (T1WI shows a slightly low signal, and T2WI shows a slightly high signal); in the third situation, the medical intelligent agent with medical detection capability can detect the existence of cyst solidity in the first medical image, and output the first medical answer based on the cyst solidity (enhanced scan shows moderate enhancement of solid components and cyst wall); in the fourth situation, the medical intelligent agent with medical detection capability can detect the existence of lateral ventricle in the first medical image, and output the first medical answer based on the lateral ventricle (the boundary of the lesion is unclear, and the compression of adjacent brain tissue and lateral ventricle is changed).

[0108] Furthermore, the target object can continue to input a second medical question, and the second medical question inputted at this time can be a further input question based on the medical image. After that, the medical question type corresponding to the second medical question can be further determined. If the medical question type corresponding to the second medical question is a medical retrieval type, a medical agent with medical retrieval capability in the medical agent group can be called to perform image retrieval on the first medical image according to the second medical question to obtain a second medical answer.

[0109] like Figure 4 As shown, Figure 4 This is a schematic diagram of the output of a medical agent with medical retrieval capability in a method for constructing a medical agent provided in an embodiment of the present application. Figure 4 In the present invention, a medical agent with medical detection capability can first detect the first medical image based on the first medical question to obtain the first medical answer corresponding to the first medical image (i.e., the image features corresponding to the first medical image and the image detection result, as shown in FIG. Figure 4 ). After that, the first medical answer can be retrieved based on the second medical question by a medical agent with medical retrieval capability to obtain a set of similar medical images corresponding to the first medical answer corresponding to the first medical image. Alternatively, the first medical image can be retrieved based on the first medical question by a medical agent with medical retrieval capability to obtain a set of similar medical images corresponding to the first medical image, and the set of similar medical images can be used as the second medical answer.

[0110] Alternatively, if the medical question type corresponding to the second medical question is a report generation type, a medical agent with report generation capability in the medical agent group can be called to generate a report on the first medical image based on the second medical question to obtain a second medical answer, wherein the medical report generated in this application can support medical personnel to read and edit the report content online.

[0111] like Figure 5 As shown, Figure 5 The output schematic diagram of a medical agent with report generation capability in a method for constructing a medical agent provided in an embodiment of the present application is shown in FIG. Figure 5 In the present application, a report can be generated for the first medical image based on the first medical answer obtained above to obtain a second medical answer (MRI shows a cystic solid mass-like mass in the right occipital lobe, T1WI shows a slightly low signal, T2WI shows a slightly high signal, enhanced scanning shows moderate enhancement of the solid component and cyst wall, unclear lesion boundaries, compression changes of adjacent brain tissue and lateral ventricle, no obvious surrounding edema, and left shift of the midline structure). The medical report generated in the present application can also give subsequent examination opinions, so as to enhance the target object's trust in the medical treatment results.

[0112] Alternatively, if the medical question type corresponding to the second medical question is a medical retrospective type, a medical agent with medical retrospective capability in the medical agent group can be called to perform image retrospection on the first medical image according to the second medical question to obtain a second medical answer (i.e., the historical medical image of the patient corresponding to the first medical image). Finally, the second medical answer can be shown to the target object to complete the interaction between the target object and the medical agent.

[0113] In summary, in this embodiment, explainable output can be obtained through the interactive method of multiple rounds of dialogue, and in this application, different medical agents can be called according to different inputs of the target object to output corresponding explainable outputs, which can output both the medical processing results corresponding to the medical data input by the object and the medical interpretation content of the medical processing results, so that the output medical processing results can have a certain degree of explainability, and also improve the target object's trust in the medical processing results.

[0114] The following is an introduction to a medical intelligent entity construction device provided in an embodiment of the present application. The medical intelligent entity construction device described below and the medical intelligent entity construction method described above can be referenced to each other. Figure 6 , which is a schematic diagram of the structure of a medical intelligent body construction device provided in an embodiment of the present application, such as Figure 6 As shown, the construction device of the medical intelligent body includes:

[0115] A question answer set acquisition unit 601 is used to acquire a medical sample question set and a medical reference answer set corresponding to the medical sample question set, wherein one medical reference answer corresponds to one medical sample question;

[0116] A prediction answer set obtaining unit 602 is used to input the medical sample question set into the medical multimodal pre-training model under the agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question, and the medical multimodal pre-training model is used to support obtaining medical answers according to medical questions;

[0117] The intelligent agent group obtaining unit 603 is used to fine-tune the parameters of the medical multimodal pre-training model under the intelligent agent interaction framework according to the medical benchmark answer set and the medical prediction answer set, until the fine-tuned model meets the fine-tuning cutoff condition, and a medical intelligent agent group is obtained after the fine-tuning is completed, wherein the medical intelligent agent group includes a medical intelligent agent with medical detection capability, a medical intelligent agent with medical retrieval capability, a medical intelligent agent with report generation capability, and a medical intelligent agent with medical backtracking capability.

[0118] Optionally, the device further comprises:

[0119] A sample image set acquisition unit, used to acquire a medical sample image set corresponding to the medical sample question set, wherein one medical sample image corresponds to one medical sample question;

[0120] The prediction answer set obtaining unit 602 is specifically used for:

[0121] The medical sample question set and the medical sample image set are input into a medical multimodal pre-training model under an intelligent agent interaction framework, so as to perform prediction processing on the medical sample question set and the medical sample image set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question and one medical sample image.

[0122] Optionally, the device further comprises:

[0123] A question type set obtaining unit is used to judge and process the medical sample question set under the intelligent agent interaction framework to obtain a medical question type set corresponding to the medical sample question set, wherein one medical question type corresponds to one medical sample question.

[0124] Optionally, the predicted answer set obtaining unit 602 is specifically configured to:

[0125] If the medical question type corresponding to the first sample question is a medical detection type, performing a medical detection on the first sample image according to the first sample question by using the medical multimodal pre-trained model to obtain a first predicted answer;

[0126] If the medical question type corresponding to the second sample question is a medical retrieval type, performing a medical retrieval on the second sample image according to the second sample question by using the medical multimodal pre-trained model to obtain a second predicted answer;

[0127] If the medical question type corresponding to the third sample question is a report generation type, generating a report on the third sample image according to the third sample question by using the medical multimodal pre-trained model to obtain a third predicted answer;

[0128] If the medical question type corresponding to the fourth sample question is a medical retrospective type, performing medical retrospective on the fourth sample image according to the fourth sample question by using the medical multimodal pre-trained model to obtain a fourth predicted answer;

[0129] A medical prediction answer set is obtained according to the first prediction answer, the second prediction answer, the third prediction answer and the fourth prediction answer.

[0130] Optionally, the intelligent agent interaction framework includes an AutoGen framework, the medical detection type is used to indicate sample image detection for the first sample image, the medical retrieval type is used to indicate retrieval of similar images for the second sample image, the report generation type is used to indicate generation of a medical report based on the third sample image, and the medical backtracking type is used to indicate backtracking of historical images for the fourth sample image.

[0131] Optionally, the device further comprises:

[0132] A question image acquisition unit, configured to acquire a first medical question submitted by a target subject and a first medical image corresponding to the first medical question;

[0133] a medical image detection unit, configured to, if the medical question type corresponding to the first medical question is a medical detection type, call a medical agent with medical detection capability in the medical agent group to perform image detection on the first medical image according to the first medical question to obtain a first medical answer;

[0134] The first medical answer presenting unit is used to present the first medical answer to the target object.

[0135] Optionally, the device further comprises:

[0136] A medical question obtaining unit, configured to obtain a second medical question submitted by the target subject;

[0137] a medical image retrieval unit, configured to, if the medical question type corresponding to the second medical question is a medical retrieval type, call a medical agent with medical retrieval capability in the medical agent group to perform image retrieval on the first medical image according to the second medical question to obtain a second medical answer;

[0138] Alternatively, a medical report generating unit is configured to, if the medical question type corresponding to the second medical question is a report generating type, call a medical agent with report generating capability in the medical agent group to generate a report on the first medical image according to the second medical question, so as to obtain a second medical answer;

[0139] Alternatively, a medical image backtracking unit is configured to, if the medical question type corresponding to the second medical question is a medical backtracking type, call a medical agent with medical backtracking capability in the medical agent group to perform image backtracking on the first medical image according to the second medical question to obtain a second medical answer;

[0140] The second medical answer presenting unit is used to present the second medical answer to the target object.

[0141] The medical intelligent body construction device provided in the embodiment of the present application has the same beneficial effects as the medical intelligent body construction method provided in the above-mentioned embodiment, so it will not be repeated here.

[0142] Electronic device embodiment

[0143] See also Figure 7 , which is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, including:

[0144] A memory 11, used for storing computer programs;

[0145] The processor 12 is used to implement the steps of the method for constructing a medical intelligent agent described in any of the above method embodiments when executing the computer program.

[0146] In this embodiment, the device may be a vehicle-mounted computer, a PC (Personal Computer), or a terminal device such as a smart phone, a tablet computer, a PDA, or a portable computer.

[0147] The device may include a memory 11, a processor 12 and a bus 13. The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the device, such as a hard disk of the device. In other embodiments, the memory 11 may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed on the device, such as program codes for executing the construction method of the medical intelligent body, but also can be used to temporarily store data that has been output or is to be output.

[0148] In some embodiments, the processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 11 or process data, such as executing the program code of the method for constructing a medical intelligent body.

[0149] The bus 13 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0150] Furthermore, the device may also include a network interface 14, which may optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the device and other electronic devices.

[0151] Optionally, the device may further include a user interface 15, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 15 may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the device and to display a visual user interface.

[0152] Figure 7 Only the device with components 11-15 is shown, and it can be understood by those skilled in the art that Figure 7 The structure shown does not constitute a limitation of the device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently.

[0153] Readable storage medium embodiment

[0154] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for constructing a medical intelligent agent described in any of the above method embodiments are implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0155] It should also be noted that the "first" and "second" (if any) in the names such as "first" and "second" mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0156] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0157] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0158] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0159] The above is a detailed introduction to the construction method, device and related products of a medical intelligent body provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for constructing a medical intelligent agent, characterized in that: include: Obtaining a medical sample question set and a medical benchmark answer set corresponding to the medical sample question set, wherein one medical benchmark answer corresponds to one medical sample question; Inputting the medical sample question set into a medical multimodal pre-training model under an agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question, and the medical multimodal pre-training model is used to support obtaining medical answers according to medical questions; According to the medical benchmark answer set and the medical prediction answer set, the parameters of the medical multimodal pre-trained model are fine-tuned under the agent interaction framework until the fine-tuned model meets the fine-tuning cutoff condition, and a medical agent group is obtained after the fine-tuning is completed, wherein the medical agent group includes medical agents with medical detection capabilities, medical agents with medical retrieval capabilities, medical agents with report generation capabilities, and medical agents with medical backtracking capabilities.

2. The method according to claim 1, characterized in that After obtaining the medical sample question set and the medical reference answer set corresponding to the medical sample question set, the method further includes: Acquire a medical sample image set corresponding to the medical sample problem set, wherein one medical sample image corresponds to one medical sample problem; The step of inputting the medical sample question set into the medical multimodal pre-training model under the agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, comprises: The medical sample question set and the medical sample image set are input into a medical multimodal pre-training model under an intelligent agent interaction framework, so as to perform prediction processing on the medical sample question set and the medical sample image set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question and one medical sample image.

3. The method according to claim 2, characterized in that After inputting the medical sample question set into the medical multimodal pre-training model under the agent interaction framework, and before performing prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, the method further includes: The medical sample question set is judged and processed under the intelligent agent interaction framework to obtain a medical question type set corresponding to the medical sample question set, wherein one medical question type corresponds to one medical sample question.

4. The method according to claim 3, characterized in that The medical sample question set includes a first sample question, a second sample question, a third sample question and a fourth sample question, and the medical sample image set includes a first sample image, a second sample image, a third sample image and a fourth sample image, wherein the first sample question corresponds to the first sample image, the second sample question corresponds to the second sample image, the third sample question corresponds to the third sample image, and the fourth sample question corresponds to the fourth sample image; The method of performing prediction processing on the medical sample question set and the medical sample image set by the medical multimodal pre-training model to obtain a medical prediction answer set includes: If the medical question type corresponding to the first sample question is a medical detection type, performing a medical detection on the first sample image according to the first sample question by using the medical multimodal pre-trained model to obtain a first predicted answer; If the medical question type corresponding to the second sample question is a medical retrieval type, performing a medical retrieval on the second sample image according to the second sample question by using the medical multimodal pre-trained model to obtain a second predicted answer; If the medical question type corresponding to the third sample question is a report generation type, generating a report on the third sample image according to the third sample question by using the medical multimodal pre-trained model to obtain a third predicted answer; If the medical question type corresponding to the fourth sample question is a medical retrospective type, performing medical retrospective on the fourth sample image according to the fourth sample question by using the medical multimodal pre-trained model to obtain a fourth predicted answer; A medical prediction answer set is obtained according to the first prediction answer, the second prediction answer, the third prediction answer and the fourth prediction answer.

5. The method according to claim 4, characterized in that The intelligent agent interaction framework includes an AutoGen framework, the medical detection type is used to indicate sample image detection for the first sample image, the medical retrieval type is used to indicate retrieval of similar images for the second sample image, the report generation type is used to indicate generation of a medical report based on the third sample image, and the medical backtracking type is used to indicate backtracking of historical images for the fourth sample image.

6. The method according to claim 1, characterized in that Also includes: Acquire a first medical question submitted by a target subject and a first medical image corresponding to the first medical question; If the medical question type corresponding to the first medical question is a medical detection type, calling a medical agent with medical detection capability in the medical agent group to perform image detection on the first medical image according to the first medical question to obtain a first medical answer; The first medical answer is displayed to the target subject.

7. The method according to claim 6, characterized in that Also includes: Obtaining a second medical question submitted by the target subject; If the medical question type corresponding to the second medical question is a medical retrieval type, calling a medical agent with medical retrieval capability in the medical agent group to perform image retrieval on the first medical image according to the second medical question to obtain a second medical answer; Alternatively, if the medical question type corresponding to the second medical question is a report generation type, calling a medical agent with report generation capability in the medical agent group to generate a report on the first medical image according to the second medical question to obtain a second medical answer; Alternatively, if the medical question type corresponding to the second medical question is a medical backtracking type, calling a medical agent with medical backtracking capability in the medical agent group to perform image backtracking on the first medical image according to the second medical question to obtain a second medical answer; The second medical answer is displayed to the target subject.

8. A device for constructing a medical intelligent agent, characterized in that: include: A question answer set acquisition unit, used to acquire a medical sample question set and a medical reference answer set corresponding to the medical sample question set, wherein one medical reference answer corresponds to one medical sample question; A prediction answer set obtaining unit, used for inputting the medical sample question set into the medical multimodal pre-training model under the agent interaction framework, so as to perform prediction processing on the medical sample question set through the medical multimodal pre-training model to obtain a medical prediction answer set, wherein one medical prediction answer corresponds to one medical sample question, and the medical multimodal pre-training model is used to support obtaining medical answers according to medical questions; The intelligent agent group obtaining unit is used to fine-tune the parameters of the medical multimodal pre-training model under the intelligent agent interaction framework according to the medical benchmark answer set and the medical prediction answer set, until the fine-tuned model meets the fine-tuning cutoff condition, and obtain a medical intelligent agent group after fine-tuning, wherein the medical intelligent agent group includes a medical intelligent agent with medical detection capability, a medical intelligent agent with medical retrieval capability, a medical intelligent agent with report generation capability, and a medical intelligent agent with medical backtracking capability.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for constructing a medical intelligent agent as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for constructing a medical intelligent agent as described in any one of claims 1 to 7.

Images