Medical information processing apparatus, medical information processing system, information processing apparatus, method for processing medical information, method for processing information, and medical information processing program

Abstract

To improve accuracy of reliability of responses from a generative model in application of the generative model to a medical field.SOLUTION: A medical information processing apparatus according to the present embodiment includes a generation unit, an acquisition unit, and an output unit. The generation unit generates a prompt to be input to a generative model in response to an input from a user. The acquisition unit acquires, from the generative model, an answer including medical information relating to the user in response to input of the prompt to the generative model. The output unit outputs reliability of the answer on the basis of the prompt and the answer.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The embodiments disclosed in the present specification and drawings relate to a medical information processing device, a medical information processing system, an information processing device, a medical information processing method, an information processing method, and a medical information processing program. [Background technology]

[0002] There have been examples of generative models (hereafter referred to as LLMs (Large Language Models)) being applied to the medical field, such as automating questions to doctors using LLMs. However, answers from LLMs can sometimes contain hallucination, a phenomenon in which the LLM outputs information that is inaccurate or completely unrelated to the context.

[0003] Here, technical information containing technical ideas and patent information to be compared with the technical information are acquired, and instructions specifying the technical information and patent information are input into a generative model, and an evaluation result showing the relevance between the technical information and the patent information is output based on the results obtained from the generative model (for example, Patent Document 1). This does not indicate the reliability of the answer from the LLM, and there is room for improvement. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7421740 Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve the reliability and accuracy of answers from generative models when applying generative models to the medical field. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0006] The medical information processing device according to this embodiment has a generation unit, an acquisition unit, and an output unit. The generation unit generates a prompt to be input to a generative model in response to an input from a user. The acquisition unit acquires an answer including medical information about the user from the generative model in response to the input of the prompt to the generative model. The output unit outputs a reliability of the answer based on the prompt and the answer. The information processing device also has a generation unit, an acquisition unit, and an output unit. The generation unit generates a prompt to be input to a generative model in response to an input from a user. The acquisition unit acquires an answer including information about the user from the generative model in response to the input of the prompt to the generative model. The output unit outputs a reliability of the answer based on the prompt and the answer. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a medical information processing system including a medical information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a prompt, an LLM's response to the prompt, a reliability, and a basis for the reliability according to the embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of a procedure of a reliability determination process according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the reliability of an answer displayed by a fact check function according to a comparative example of the embodiment. [Figure 5]FIG. 5 is a diagram showing an example of a problem with the reliability of answers displayed by the fact check function according to a comparative example of the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of a procedure of a reliability determination process performed in a first modified example of the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure of a reliability determination process performed in a second modified example of the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a procedure of a reliability determination process performed in a third modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, with reference to the drawings, embodiments of a medical information processing device, a medical information processing system, an information processing device, a medical information processing method, an information processing method, and a medical information processing program will be described. In the following embodiments, parts with the same reference numerals perform similar operations, and duplicated descriptions will be omitted as appropriate.

[0009] (Embodiment) Fig. 1 is a block diagram showing an example of the configuration of a medical information processing system 1 including a medical information processing device 30 according to an embodiment. As shown in Fig. 1, the medical information processing system 1 according to the embodiment includes an information processing server 10, a database server 20, and the medical information processing device 30. The information processing server 10 and the database server 20 are connected to the medical information processing device 30 via a network such as the Internet and / or an intranet. Note that the connection destinations of the medical information processing device 30 are not limited to the information processing server 10 and the database server 20, and other servers and / or other databases may also be connected.

[0010] The information processing server 10 has a memory that stores the generative model 11 and a processor that executes the generative model 11. The generative model 11 is, for example, a large-scale language model (hereinafter referred to as an LLM (Large Language Model)) or a multimodal model (MMM (Multimodal Model)). Here, an example in which the generative model 11 is an LLM will be described. Note that the LLM 11 may be replaced with a model such as an MMM. The LLM 11 is an AI model that has been pre-trained using a large-scale corpus in the field of natural language processing, and is configured to, for example, generate and output a response sentence that conforms to the meaning of the input sentence when a question or instruction is input as a sentence (prompt).

[0011] The processor in the information processing server 10 reads the LLM 11 from memory and generates a response sentence by inputting the prompt output from the medical information processing device 30 to the LLM 11. The processor transmits the generated response sentence (output from the LLM 11) to the medical information processing device 30. The generated response sentence, i.e., the answer from the LLM 11 corresponding to the prompt, includes medical information about the user. The LLM 11 may be installed in the medical information processing device 30.

[0012] The database server 20 has a memory that stores a medical information database 21. The medical information database 21 contains various types of medical information, such as medical papers, columns and reports in medical journals, etc. If the medical information database 21 is stored in a hospital information system (hereinafter referred to as an HIS (Hospital Information System)), the medical information database 21 contains various types of medical information stored in the HIS. Furthermore, the medical information database 21 is not limited to various types of medical information, and may further contain information on examinations, diagnoses, treatments, and responses.

[0013] The medical information processing device 30 transmits a prompt to the information processing server 10 via the network. The medical information processing device 30 acquires the response sentence output from the LLM 11 from the information processing server 10 via the network. The medical information processing device 30 also accesses the database server 20 via the network to search the medical information database 21. Note that the search in the medical information database 21 may be executed by a processor in the database server 20 according to the search conditions output from the medical information processing device 30. The medical information processing device 30 acquires the search results from the database server 20.

[0014] The medical information processing device 30 is realized, for example, by various terminals used by a user. The various terminals include, for example, smartphones, various personal computers, and tablet terminals that can connect to a network. The medical information processing device 30 may also be realized by a terminal device in a hospital. As shown in FIG. 1 , the medical information processing device 30 has an input interface 31, a display 32, a memory 33, a processing circuit 34, and a speaker 35.

[0015] The input interface 31 is realized by a trackball, switches, buttons, a mouse, a keyboard, and other devices for inputting various instructions and settings, a touchpad for performing input operations by touching the operation surface, a touchscreen that integrates a display screen and a touchpad, a non-contact input circuit using an optical sensor, an audio input circuit (such as a microphone), etc. The input interface 31 converts input operations received from the user into electrical signals and outputs them to the processing circuit 34. For example, the input interface 31 inputs voice or text related to a question to the LLM 11 in response to a user instruction.

[0016] The input interface 31 is not limited to one equipped with physical operation components such as a mouse and a keyboard. For example, an example of the input interface 31 also includes an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the medical information processing device 30 and outputs the electrical signal to the processing circuit 34. The input interface 31 is an example of an input unit, and may also be referred to as an operation unit.

[0017] The display 32 displays various types of information under the control of the display control function 34f. For example, the display 32 displays a GUI (Graphical User Interface) for receiving user instructions, prompts to the LLM 11, answers from the LLM 11, the reliability of the answers, and the basis for the answers. For example, the display 32 is a liquid crystal display or a CRT (Cathode Ray Tube) display. The display 32 is an example of a display unit.

[0018] The memory 33 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, an optical disk, etc. For example, the memory 33 stores various data generated by various processes performed by the processing circuitry 34 described below. The memory 33 also stores a response sentence (answer) output from the LLM 11. The memory 33 also stores search results in the medical information database 21. The memory 33 also stores programs that enable each circuit included in the medical information processing device 30 to realize its function. The memory 33 is an example of a storage unit.

[0019] The processing circuitry 34 controls the operation of the entire medical information processing device 30 by executing the generation function 34a, the acquisition function 34b, the output function 34c, and the display control function 34d. The processing circuitry 34 that realizes the generation function 34a corresponds to the generation unit. The processing circuitry 34 that realizes the acquisition function 34b corresponds to the acquisition unit. The processing circuitry 34 that realizes the output function 34c corresponds to the output unit. The processing circuitry 34 that realizes the display control function 34d corresponds to the display control unit.

[0020] The processing circuit 34 reads out from the memory 33 a program corresponding to the generation function 34a and executes it. As a result, the generation function 34a generates a prompt to be input to the LLM 11 in response to input from the user via the input interface 31. A known method can be applied to generating a prompt in response to input from the input interface 31, so a description thereof will be omitted. The generation function 34a stores the generated prompt in the memory 33.

[0021] The processing circuitry 34 reads out and executes a program corresponding to the acquisition function 34b from the memory 33. This causes the acquisition function 34b to transmit the generated prompt to the information processing server 10. The acquisition function 34b acquires, from the LLM 11, an answer generated by inputting the prompt to the LLM 11.

[0022] Specifically, the processor in the information processing server 10 inputs the generated prompt to the LLM 11 and generates an answer corresponding to the output from the LLM 11. The acquisition function 34b acquires the answer output from the LLM 11 from the information processing server 10. The answer output from the LLM 11 corresponds to the prompt input to the LLM 11.

[0023] The processing circuitry 34 reads out from the memory 33 a program corresponding to the output function 34c and executes it. As a result, the output function 34c determines the reliability of the answer based on the prompt and the answer. Specifically, the output function 34c extracts a named entity from the prompt using natural language processing (hereinafter, the named entity extracted from the prompt is referred to as a first named entity). For example, if the prompt is a sentence such as "I am a male, 30 years old. I have a lump on my neck. Is it some kind of illness?", the first named entity is expressed as, for example, gender: male, age: 30 years old, location: neck, lesion: lump, etc.

[0024] The output function 34c also extracts named entities from the answer provided by the LLM 11 using natural language processing (hereinafter, named entities extracted from the answer provided by the LLM 11 are referred to as second named entities). For example, if the answer is a sentence such as "The lump in the neck is basically benign, so please monitor the progress," the second named entity is expressed as, for example, location: neck, lesion: lump, etc.

[0025] The process of extracting named entities can be performed using known natural language processing, and therefore a detailed description thereof will be omitted. The extracted named entities may be classified by various items, as described above. In this case, data for the extracted named entities may be structured, as described above. The process of extracting a first named entity from a prompt and a second named entity from a response may be performed by processing circuitry 34 as a separate extraction function.

[0026] The processing circuitry 34 uses the output function 34c to search the medical information database 21 using the first named entity and the second named entity as search conditions. Through this search, the output function 34c identifies document data in the medical information database 21 based on the number of times the first named entity and the second named entity appear. For example, the output function 34c identifies document data in which the first named entity and the second named entity appear the most frequently. At this time, the output function 34c acquires the identified document data from the medical information database 21. The output function 34c stores the acquired document data in the memory 33 in association with the first named entity and the second named entity.

[0027] The processing circuit 34 vectorizes the identified document data using the output function 34c. That is, the output function 34c performs vectorization of natural language processing on the identified document data to generate a vector (hereinafter referred to as a first vector) corresponding to the document data. The first vector indicates the feature amount (text feature amount) of the identified document data.

[0028] The output function 34c also vectorizes the prompt and the answer. That is, the output function 34c performs vectorization of natural language processing on the prompt and the answer to generate a vector (hereinafter referred to as a second vector) corresponding to the prompt and the answer. The second vector indicates the feature quantity of the text summarizing the prompt and the answer.

[0029] The vectorization of natural language processing can be performed using known natural language processing or a known trained model, and therefore will not be described here. Furthermore, the process of vectorizing document data into a first vector and the process of vectorizing prompts and answers into a second vector may be performed by processing circuitry 34 as separate vectorization functions.

[0030] The processing circuit 34 uses the output function 34c to calculate the similarity of the document data for the prompt and answer based on the first vector and the second vector. Specifically, the output function 34c calculates the distance between the first vector and the second vector. This distance is, for example, the Mahalanobis distance in vector space. The shorter the Mahalanobis distance, the closer the first vector and the second vector are, so the output function 34c calculates a larger similarity. The output function 34c stores the calculated similarity in the memory 33.

[0031] The output function 34c may calculate the reciprocal of the Mahalanobis distance as the similarity, for example. The calculation of the similarity is not limited to the calculation of the Mahalanobis distance, and any method for calculating the similarity between vectors may be applied. The process of calculating the similarity between the first vector and the second vector may be executed by the processing circuitry 34 as a separate similarity calculation function.

[0032] The processing circuitry 34 determines the reliability of the answer based on the similarity using the output function 34c. For example, if reliability is classified into three categories, low, medium, and high, a first threshold indicating the boundary between low reliability and medium reliability and a second threshold indicating the boundary between medium reliability and high reliability are preset and stored in the memory 33. The first and second thresholds can be set arbitrarily and can be changed as appropriate. Note that the number of reliability categories is not limited to three, and any number of categories can be set. In this case, a number of thresholds corresponding to (the number of categories - 1) is set in advance. For the sake of concreteness, the following description will be given assuming that the number of categories is three. The threshold in this embodiment is a threshold (hereinafter referred to as a similarity determination threshold) that determines how similar the content of two document data, i.e., the identified document data and the document data based on the prompt and answer, is (i.e., how similar the content of two document data is).

[0033] The processing circuit 34 compares the similarity with a first similarity determination threshold using the output function 34c. If the similarity is less than the first similarity determination threshold, the output function 34c determines the answer as having a low reliability. If the similarity is equal to or greater than the first similarity determination threshold, the output function 34c compares the similarity with a second similarity determination threshold. If the similarity is less than the second similarity determination threshold, that is, if the similarity is equal to or greater than the first similarity determination threshold and less than the second similarity determination threshold, the output function 34c determines the answer as having a medium reliability. If the similarity is equal to or greater than the second similarity determination threshold, the output function 34c determines the answer as having a high reliability. The output function 34c associates the determined reliability with the answer and stores them in the memory 33.

[0034] The processing circuitry 34 reads out from the memory 33 a program corresponding to the display control function 34d and executes it. For example, the display control function 34d causes the display 32 to display the answer and its reliability together with the basis for the reliability. As a result, the display 32 displays the answer and its reliability together with the basis for the reliability. At this time, if the reliability is low, the display control function 34d also causes the display 32 to display a predetermined warning.

[0035] FIG. 2 is a diagram showing an example of a prompt, an answer given by the LLM 11 to the prompt, a reliability level, and the basis for the reliability level. As shown in FIG. 2, the answer given by the LLM 11 to the prompt includes medical information about the user (in FIG. 2, benign, follow-up observation, etc.). As shown in FIG. 2, the basis for the reliability level is displayed, for example, as the site name in the medical information database 21 and the text that serves as the basis. As shown in FIG. 2, if the reliability of the answer is low, a predetermined warning (an exclamation mark) is displayed with the basis attached. This allows the user to easily understand the basis for the reliability when the reliability of the answer is low.

[0036] The overall configuration of the medical information processing system 1 according to the embodiment has been described above. The following describes the process of determining the reliability of the answer output from the LLM 11 (hereinafter referred to as the reliability determination process). Figure 3 is a flowchart showing an example of the procedure for the reliability determination process.

[0037] (Reliability determination process) (Step S301) In response to a user's instruction via the input interface 31, the user inputs questions to the LLM 11. The questions are medical questions for the user.

[0038] (Step S302) The processing circuit 34 uses the generation function 34a to generate a prompt to be input to the LLM 11 in response to an input from the user via the input interface 31. The generation function 34a stores the generated prompt in the memory 33.

[0039] (Step S303) The processing circuit 34 transmits the prompt to the information processing server 10 by the acquisition function 34b. Then, the information processing server 10 inputs the prompt to the LLM 11.

[0040] (Step S304) The information processing server 10 generates an answer corresponding to the input prompt based on the output from the LLM 11. The processing circuit 34 uses the acquisition function 34b to acquire the answer to the prompt output from the LLM 11 from the information processing server 10. The acquisition function 34b stores the acquired answer in the memory 33 in association with the generated prompt.

[0041] (Step S305) The processing circuitry 34 extracts a first named entity from the prompt and a second named entity from the response using the output function 34c. The output function 34c associates the first named entity with the prompt and stores them in the memory 33. The output function 34c also associates the second named entity with the response from the LLM 11 and stores them in the memory 33.

[0042] (Step S306) The processing circuitry 34 uses the output function 34c to search the medical information database 21 using the first named entity and the second named entity as search conditions. Note that the search may be performed by a search function in the processing circuitry 34. Furthermore, the scope of the search is not limited to the medical information database 21, and may be on the Internet.

[0043] (Step S307) The processing circuitry 34 uses the output function 34c to identify document data in the medical information database 21 based on the frequency of appearance of the first named entity and the second named entity. Here, the output function 34c identifies document data in which the first named entity and the second named entity appear the most frequently. The output function 34c acquires the identified document data from the medical information database 21. Note that the acquisition of the identified document data may be performed by the acquisition function 34b.

[0044] (Step S308) The processing circuitry 34 uses the output function 34c to convert the document data into a first vector using vectorization of natural language processing. The output function 34c associates the first vector with the document data and stores the first vector in the memory 33. The output function 34c also converts the prompt and the answer together into a second vector using vectorization of natural language processing. The output function 34c associates the second vector with the prompt and the answer and stores the second vector in the memory 33.

[0045] (Step S309) The processing circuitry 34 uses the output function 34c to calculate the similarity of the document data for the prompt and the answer based on the first vector and the second vector. For example, the output function 34c calculates the distance between the first vector and the second vector in the vector space of the feature. The output function 34c calculates the similarity using the calculated distance.

[0046] The output function 34c calculates the similarity so that the closer the calculated distance is, the greater the similarity becomes. Also, the output function 34c calculates the similarity so that the farther the calculated distance is, the smaller the similarity becomes. Note that the similarity is not limited to being calculated from the distance, but may be calculated according to the degree of separation between the first vector and the second vector (for example, the value of the dot product).

[0047] (Step S310) The processing circuit 34 determines the reliability of the answer based on the similarity using the output function 34c. Specifically, the output function 34c determines the reliability of the answer by comparing the similarity with a preset similarity determination threshold.

[0048] (Step S311) The processing circuit 34 causes the display control function 34d to display the answer and the reliability together with the basis for the reliability on the display 32. As a result, the display 32 displays the reliability together with the basis for the reliability, for example, as shown in FIG. 2. The display 32 may also display the prompt and the answer together with the reliability. At this time, the display control function 34d may cause the display 32 to display a predetermined warning, as shown in FIG. 2. The predetermined warning may be, for example, in bold or highlighted. The display control function 34d may also cause the speaker 35 to output the predetermined warning as sound.

[0049] Fig. 4 is a diagram showing an example of the reliability of an answer displayed by the fact check function in a comparative example of this embodiment. As shown in Fig. 7, the fact check function displays the results of a fact check (fact check result) on an answer from an LLM to a prompt based on a user's input, along with the basis for the fact check. In Fig. 4, a website page on the Internet is shown as the basis for the fact check.

[0050] In the fact-checking function of the comparative example, the information that "information similar to the LLM's answer exists on the Internet" is insufficient to indicate the reliability of the answer from the LLM of the comparative example. For example, even if information similar to the answer from the LLM exists on the Internet, the fact-checking function may present to the user an answer that is not intended by the user and / or that does not reflect the conditions and context set in the input to the LLM as a reliable answer.

[0051] FIG. 5 is a diagram illustrating an example of a reliability issue with answers displayed by the fact-checking function in a comparative example of this embodiment. As shown in FIG. 5, when the page of the website in the fact-checking results is scrolled, the message "...however, young men should immediately go to the hospital..." is displayed. Meanwhile, the response from the LLM is "The lump in the neck is generally benign, so please monitor the condition," and the prompt states, "I am a man, 30 years old." From these facts, as shown in FIG. 5, the response from the LLM is factual, but it overlooks the necessary condition, "...however, young men should immediately go to the hospital...," resulting in hallucination.

[0052] As shown in Figure 5, even if the fact-checking function determines that an LLM's answer is appropriate, the reliability of the LLM's answer may not be sufficient. In particular, when applying LLM to the medical field, depending on the answer given by the LLM, users may suffer health disadvantages.

[0053] Meanwhile, the medical information processing apparatus 30 according to the embodiment generates a prompt to be input to the LLM 11 in response to an input from the user, acquires an answer including medical information about the user from the LLM 11 in response to the input of the prompt to the LLM 11, and determines the reliability of the answer based on the prompt and the answer. At this time, the medical information processing apparatus 30 according to the embodiment displays the answer and the reliability together with the basis of the reliability.

[0054] Specifically, the medical information processing device 30 according to the embodiment extracts a first named entity from the prompt, extracts a second named entity from the answer, searches the medical information database 21 using the first named entity and the second named entity as search conditions, identifies document data in which the first named entity and the second named entity appear the most frequently, calculates the similarity of the document data to the prompt and the answer based on a first vector obtained by vectorizing the document data and a second vector obtained by vectorizing the prompt and the answer, and determines the reliability based on the similarity.

[0055] As a result, the medical information processing apparatus 30 according to the embodiment can perform fact-checking that takes into account the context and condition settings of the prompt based on the user's input by determining how similar the content of two pieces of document data—identified document data and document data based on a prompt and an answer—is, unlike the comparative example. That is, the medical information processing apparatus 30 according to the embodiment performs fact-checking of identified document data using a prompt and an answer corresponding to the prompt, thereby determining whether the answer is relevant to the prompt's context. Therefore, when applying the LLM 11 to the medical field, the medical information processing apparatus 30 according to the embodiment can improve the reliability and accuracy of answers from the LLM 11 and reduce health risks to users.

[0056] (First Modification) In this modification, a search is performed in the medical information database 21 and / or the Internet using the first named entity and the second named entity as search conditions, and the reliability of the answer is determined according to the number of hits in the search, such as literature. The reliability determination process in this modification will be described below with reference to FIG. 6.

[0057] Fig. 6 is a flowchart showing an example of the procedure of reliability determination processing performed in Modification 1. In the flowchart in Fig. 6, the processing of steps S601 to S606 and step S609 is similar to steps S301 to S306 and step S311 in Fig. 3, and therefore description thereof will be omitted.

[0058] (Reliability determination process) (Step S607) The processing circuitry 34 uses the output function 34c to calculate the number of hits based on the search criteria using the first named entity and the second named entity. For example, the output function 34c counts (calculates) the number of hits of document data that includes both the first named entity and the second named entity in the search results of the medical information database 21, etc., in step S406. The output function 34c stores the counted number of hits in the memory 33.

[0059] (Step S608) The processing circuit 34 determines the reliability based on the number of hits using the output function 34c. Specifically, the output function 34c reads a threshold value (hereinafter referred to as the hit threshold value) for distinguishing reliability based on the number of hits from the memory 33. The hit threshold value is set in advance and stored in the memory 33. For the sake of concreteness, the following description will be given assuming that there are three categories based on the number of hits. In this case, the hit threshold values ​​read from the memory 33 are a first hit threshold value and a second hit threshold value that is larger than the first hit threshold value.

[0060] The output function 34c compares the number of hits with a first hit threshold. If the number of hits is less than the first hit threshold, the output function 34c determines the answer as having low reliability. If the number of hits is equal to or greater than the first hit threshold, the output function 34c compares the number of hits with a second hit threshold. If the number of hits is less than the second hit threshold, i.e., if the number of hits is equal to or greater than the first hit threshold and less than the second hit threshold, the output function 34c determines the answer as having medium reliability. If the number of hits is equal to or greater than the second hit threshold, the output function 34c determines the answer as having high reliability. The output function 34c associates the determined reliability with the answer and stores them in the memory 33.

[0061] The medical information processing device 30 according to the first modification of the embodiment described above extracts a first named entity from the prompt, extracts a second named entity from the answer, searches the medical information database 21 using the first named entity and the second named entity as search conditions, calculates the number of hits for the search conditions, and determines the reliability based on the number of hits. As a result, the medical information processing device 30 according to the first modification of the embodiment can determine the reliability of the answer according to the number of hits in the search results obtained by searching the medical information database 21 using the first named entity and the second named entity as search conditions. Other effects are similar to those of the embodiment, and therefore will not be described again.

[0062] (Second Modification) In this modification, the reliability of an answer is determined by comparing the number of attributes included in a first named entity with the number of attributes included in a second named entity. Attributes are, for example, words corresponding to adjectives, nouns, etc., and may also be referred to as elements. The reliability determination process in this modification will be described below with reference to FIG. 7.

[0063] Fig. 7 is a flowchart showing an example of the procedure of reliability determination processing performed in Modification 2. In the flowchart in Fig. 7, the processing of steps S701 to S705 and step S708 is similar to steps S301 to S305 and step S311 in Fig. 3, and therefore description thereof will be omitted.

[0064] (Reliability determination process) (Step S706) The processing circuitry 34 uses the output function 34c to calculate the number of corresponding named entities between the first named entity and the second named entity. Specifically, the output function 34c calculates (counts) the number of attributes included in the first named entity (hereinafter referred to as the first number of attributes). The first number of attributes may also be referred to as the first number of elements. The output function 34c associates the first number of attributes with a prompt and stores it in the memory 33.

[0065] Furthermore, the processing circuitry 34 uses the output function 34c to calculate (count) the number of attributes included in the second named entity (hereinafter referred to as the second number of attributes). The second number of attributes may also be referred to as the second number of elements. The output function 34c stores the second number of attributes in the memory 33 in association with the answer.

[0066] (Step S707) The processing circuit 34 determines the reliability of the answer using the output function 34c based on the number of corresponding named entities between the first named entity and the second named entity. Specifically, the output function 34c compares the number of first attributes with the number of second attributes. If the number of first attributes is close to the number of second attributes, the output function 34c determines that the reliability of the answer is high. On the other hand, if the number of first attributes is far from the number of second attributes, the output function 34c determines that the reliability of the answer is low. In other words, the output function 34c determines the reliability of the answer based on the difference between the number of first attributes and the number of second attributes (hereinafter referred to as the attribute difference).

[0067] Specifically, the output function 34c reads out thresholds for distinguishing reliability based on attribute differences (hereinafter referred to as attribute thresholds) from the memory 33. The attribute thresholds are set in advance and stored in the memory 33. For the sake of concreteness, the following description will be given assuming that there are three categories based on attribute differences. In this case, the attribute thresholds read out from the memory 33 are two: a first attribute threshold and a second attribute threshold that is greater than the first attribute threshold.

[0068] The output function 34c compares the attribute difference with a first attribute threshold. If the number of hits is less than the first hit threshold, the output function 34c determines the answer as having a high reliability. If the attribute difference is equal to or greater than the first attribute threshold, the output function 34c compares the attribute difference with a second attribute threshold. If the attribute difference is less than the second attribute threshold, that is, if the attribute difference is equal to or greater than the first attribute threshold and less than the second attribute threshold, the output function 34c determines the answer as having a medium reliability. If the attribute difference is equal to or greater than the second attribute threshold, the output function 34c determines the answer as having a low reliability. The output function 34c associates the determined reliability with the answer and stores them in the memory 33.

[0069] The medical information processing device 30 according to the second modification of the embodiment described above extracts a first named entity from the prompt, extracts a second named entity from the answer, calculates the number of corresponding named entities between the first named entity and the second named entity, and determines the reliability based on the number of corresponding named entities. As a result, the medical information processing device 30 according to the second modification of the embodiment can determine the reliability of the answer based on the difference between the number of first attributes in the first named entity and the number of second attributes in the second named entity. The medical information processing device 30 according to the second modification of the embodiment does not require a search using the first named entity and the second named entity, making it possible to more easily determine the reliability of the answer. Other effects are similar to those of the embodiment, and therefore will not be described further.

[0070] (Third Modification) This modified example generates a prompt (hereinafter referred to as a query prompt) that queries the reliability of an answer based on a prompt based on user input (hereinafter referred to as a user input prompt) and an answer, inputs the query prompt into LLM11 or a generative model different from LLM11 (hereinafter referred to as a differential LLM), and outputs the reliability of the answer depending on the output from LLM11 or the differential LLM.

[0071] The differential LLM is stored in the memory of the information processing server 10. The differential LLM may be stored in the memory of a cloud server different from the information processing server 10, or may be stored in the medical information processing device 30. The differential LLM corresponds to a generation AI different from the LLM 11. The differential LLM is an AI model that has been pre-trained using a large-scale corpus in the field of natural language processing, and is configured to function to, for example, when a question or instruction is input as a sentence (prompt), generate and output a response sentence that is in line with the meaning of the input sentence.

[0072] The processor in the information processing server 10 reads the differential LLM from memory and generates a response sentence by inputting the query prompt output from the medical information processing device 30 into the differential LLM. The processor transmits the generated response sentence (output from the differential LLM) to the medical information processing device 30. The generated response sentence, i.e., the answer from the differential LLM corresponding to the query prompt, includes medical information about the user. The LLM 11 may be installed in the medical information processing device 30.

[0073] The reliability determination process in this modification will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the procedure of the reliability determination process performed in the third modification. In the flowchart in Fig. 8, the processes of steps S801 to S804 and step S809 are similar to steps S301 to S304 and step S311 in Fig. 3, and therefore description thereof will be omitted.

[0074] (Reliability determination process) (Step S805) The processing circuit 34 uses the generation function 34a to generate a prompt (hereinafter referred to as a query prompt) that inquires about the reliability of the answer based on the user input prompt and the answer to the user input prompt. That is, the generation function 34a generates a query prompt to be input to the LLM 11 or a different LLM. For example, if the user input prompt is "I'm a 30-year-old man. I have a lump on my neck. Is it some kind of disease?" and the answer from the LLM 11 is "The lump on my neck is basically benign, so please monitor it," the query prompt includes the user input prompt, the answer, and the question, "In the above conversation, what is the reliability of the LLM?" Since known techniques can be applied to generating the query prompt, a description thereof will be omitted. The generation function 34a stores the generated query prompt in the memory 33.

[0075] (Step S806) The processing circuit 34 transmits the inquiry prompt to the information processing server 10 via the acquisition function 34b. Next, the information processing server 10 inputs the inquiry prompt to the LLM 11 or the differential LLM.

[0076] (Step S807) The information processing server 10 generates an answer (hereinafter, a reliability answer) corresponding to the input query prompt based on the output from the LLM 11 or the difference LLM. The processing circuit 34 uses the acquisition function 34b to acquire the output (reliability answer) from the LLM 11 or the difference LLM from the information processing server 10. The acquisition function 34b stores the acquired output in the memory 33 in association with the generated query prompt.

[0077] (Step S808) The processing circuit 34 determines the reliability of the answer to the user input prompt by the output function 34c in response to the output (reliability answer) from the LLM 11 or the differential LLM. If the reliability answer includes distinguishing information (such as high, low, medium, or high) that distinguishes the reliability, the output function 34c outputs the reliability of the answer to the user input prompt using the distinguishing information in the reliability answer.

[0078] The medical information processing device 30 according to the third variant of the embodiment described above generates a query prompt to inquire about the reliability of the user input answer based on the user input prompt and the answer to the user input prompt, obtains output from the LLM11 or the difference LLM in response to input of the query prompt to the LLM11 or the difference LLM, and determines the reliability in response to obtaining output from the LLM11 or the difference LLM.

[0079] As a result, the medical information processing device 30 according to the third modification of the embodiment can determine the reliability of an answer using the LLM11 or the differential LLM based on the user input prompt and the answer to the user input prompt. The medical information processing device 30 according to the second modification of the embodiment does not require extraction of the first named entity and the second named entity, so the reliability of the answer can be determined even more simply. Other effects are similar to those of the embodiment, and therefore will not be described.

[0080] (Application example) This application example is an arbitrary combination of the embodiment, the first modified example of the embodiment, the second modified example of the embodiment, and the third modified example of the embodiment. In this application example, the reliability of the answer determined by combining at least two of the embodiment, the first modified example of the embodiment, the second modified example of the embodiment, and the third modified example of the embodiment is separately displayed on the display 32 as the basis of the answer.

[0081] The processing circuitry 34 may use the output function 34c to determine the overall reliability of the answer by performing a predetermined calculation (such as a known weighted addition) on multiple reliabilities of the answer. At this time, the processing circuitry 34 uses the display control function 34d to display the overall reliability of the answer together with the user input prompt and the answer to the user input prompt on the display 32. The effects of this application example are similar to those of the embodiment, and therefore will not be described further.

[0082] When the technical idea of ​​this embodiment is realized in a medical information processing system 1, the medical information processing system 1 has a generation function 34a that generates a prompt to be input to the LLM 11 in response to a user input, the LLM 11 that outputs an answer including medical information related to the user in response to the input of the prompt, an acquisition function 34b that acquires the answer, and an output function 34c that outputs the reliability of the answer based on the prompt and the answer. The processing procedure of the reliability determination processing realized by the medical information processing system 1 conforms to the embodiment, etc. Furthermore, the effects of the medical information processing system 1 are similar to those of the embodiment. For these reasons, a description of the processing procedure and effects of the reliability determination processing in the medical information processing system 1 will be omitted.

[0083] When the technical idea of ​​this embodiment is realized in an information processing device, the configuration of the information processing device is the same as the medical information processing device 30 shown in FIG. 1. The information processing device has a generation function 34a that generates a prompt to be input to the generation model 11 in response to an input from a user, an acquisition function 34b that acquires an answer including information about the user from the generation model 11 in response to the input of the prompt to the generation model 11, and an output function 34c that outputs the reliability of the answer based on the prompt and the answer. The processing procedure of the reliability determination processing realized by the information processing device conforms to the embodiment, etc. Furthermore, the effects of the information processing device are the same as those of the embodiment. For these reasons, a description of the processing procedure and effects of the reliability determination processing in the information processing device will be omitted.

[0084] When the technical idea of ​​this embodiment is realized in a medical information processing method, the medical information processing method generates a prompt to be input to the LLM 11 in response to a user's input, inputs the prompt to the LLM 11, obtains an answer including medical information about the user from the LLM 11, and outputs a reliability of the answer based on the prompt and the answer. The processing procedure of the reliability determination processing realized by the medical information processing method conforms to the embodiment, etc. Furthermore, the effects of the medical information processing method are similar to those of the embodiment. For these reasons, a description of the processing procedure and effects of the reliability determination processing in the medical information processing method will be omitted. When the technical idea of ​​this embodiment is realized in an information processing method, the information processing method generates a prompt to be input to the LLM 11 in response to a user's input, inputs the prompt to the LLM 11, obtains an answer including information about the user from the LLM 11, and outputs a reliability of the answer based on the prompt and the answer.

[0085] When the technical idea of ​​this embodiment is realized by a medical information processing program, the medical information processing program causes a computer to generate a prompt to be input to the LLM 11 in response to a user's input, input the prompt to the LLM 11, obtain an answer including medical information about the user from the LLM 11, and output a reliability of the answer based on the prompt and the answer. For example, the reliability determination process can be realized by installing the medical information processing program in a computer such as the medical information processing device 30 shown in FIG. 1 and expanding the program in memory. In this case, the program that can cause a computer to execute the process can also be stored in a storage medium such as a magnetic disk (e.g., a hard disk), an optical disk (e.g., a CD-ROM or DVD), or a semiconductor memory and distributed.

[0086] Furthermore, distribution of the medical information processing program is not limited to the above-mentioned media, and may be distributed using an electronic communication function, such as downloading via the Internet. The processing procedure of the medical information processing program conforms to the reliability determination process. The effects of the medical information processing program are the same as those of the embodiment. For these reasons, a description of the processing procedure and effects of the reliability determination process of the medical information processing program will be omitted.

[0087] According to at least the embodiments, modifications, application examples, etc. described above, when applying the generative model 11 to the medical field, it is possible to improve the accuracy of the reliability of the answers from the generative model 11.

[0088] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0089] 1 Medical information processing system 10 Information Processing Server 11 Generative Model (LLM) 20 Database Server 21 Medical Information Database 30 Medical information processing device 31 Input Interface 32 Display 33 Memory 34 Processing circuit 34a generation function 34b Acquisition function 34c Output Function 34d Display control function

Claims

1. a generation unit that generates prompts to be input to the generative model in response to input from a user; an acquisition unit that acquires an answer including medical information about the user from the generative model in response to input of the prompt to the generative model; an output unit that outputs a reliability of the answer based on the prompt and the answer; A medical information processing device comprising:

2. The output unit extracting a first named entity from the prompt; extracting a second named entity from the response; searching a medical information database using the first named entity and the second named entity as search conditions, and identifying document data based on the number of times the first named entity and the second named entity appear; calculating a similarity of the document data to the prompt and the answer based on a first vector obtained by vectorizing the document data and a second vector obtained by vectorizing the prompt and the answer; determining the reliability based on the similarity; The medical information processing device according to claim 1 .

3. the output unit identifies document data in which the first named entity and the second named entity appear the most frequently; The medical information processing device according to claim 2 .

4. The output unit extracting a first named entity from the prompt; extracting a second named entity from the response; searching a medical information database using the first named entity and the second named entity as search criteria, and calculating the number of hits for the search criteria; determining the confidence level based on the number of hits; The medical information processing device according to claim 1 .

5. The output unit extracting a first named entity from the prompt; extracting a second named entity from the response; calculating the number of corresponding named entities between the first named entity and the second named entity; determining the confidence level based on the number of corresponding named entities; The medical information processing device according to claim 1 .

6. the generating unit generates an inquiry prompt inquiring about a reliability of the answer based on the prompt and the answer; the acquisition unit acquires an output from the generative model or a generative model different from the generative model in response to input of the query prompt to the generative model or a generative model different from the generative model; the output unit outputs the reliability in response to acquisition of an output from the generative model or a generative model different from the generative model. The medical information processing device according to claim 5 .

7. the generating unit generates an inquiry prompt inquiring about the reliability of the answer; the acquisition unit acquires an output from the generative model or a generative model different from the generative model in response to input of the inquiry prompt, the prompt, and the answer to the generative model or a generative model different from the generative model; the output unit outputs the reliability in response to an output from the generative model or a generative model different from the generative model. The medical information processing device according to claim 1 .

8. a display unit that displays the reliability together with a basis for the reliability; The medical information processing device according to claim 1 .

9. The medical image processing apparatus according to claim 1 , further comprising a display unit that displays the prompt and the answer together with the reliability.

10. a generation unit that generates prompts to be input to the generative model in response to user input; the generative model that outputs an answer including medical information about the user in response to the prompt; an acquisition unit that acquires the answer; an output unit that outputs a reliability of the answer based on the prompt and the answer; A medical information processing system comprising:

11. Generate prompts that are input to the generative model based on user input, inputting the prompt into the generative model; obtaining an answer from the generative model, the answer including medical information about the user; outputting a confidence level of the answer based on the prompt and the answer; A medical information processing method comprising:

12. To the computer Generate prompts that are input to the generative model based on user input, inputting the prompt into the generative model; obtaining an answer from the generative model, the answer including medical information about the user; outputting a confidence level of the answer based on the prompt and the answer; A medical information processing program that makes this possible.

13. a generation unit that generates prompts to be input to the generative model in response to input from a user; an acquisition unit that acquires an answer including information about the user from the generative model in response to input of the prompt to the generative model; an output unit that outputs a reliability of the answer based on the prompt and the answer; An information processing device comprising:

14. Generate prompts that are input to the generative model based on user input, inputting the prompt into the generative model; obtaining an answer from the generative model that includes information about the user; outputting a confidence level of the answer based on the prompt and the answer; An information processing method comprising:

Images