LLM model reasoning method based on medical rule base and related equipment

Abstract

The invention provides an LLM model reasoning method based on a medical rule base and related equipment. The rule base comprises a plurality of inference rules used for assisting the LLM model in executing an inference task, and the inference rules are structured data composed of a plurality of fields; wherein the plurality of fields comprise a plurality of condition fields used for describing reasoning conditions, and target fields used for describing reasoning targets matched with the reasoning conditions; the condition field comprises an entity serving as a reasoning condition. The method comprises the steps of obtaining a target dialogue text input by a user, and extracting a text entity contained in the target dialogue text; searching at least one target inference rule related to the text entity from a plurality of inference rules contained in a rule base; and on the basis of the at least one target reasoning rule, constructing a cue word, inputting the cue word into an LLM model, and executing logical reasoning related to the reasoning task by the LLM model on the basis of the cue word to generate a response text corresponding to the target dialogue text.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of large model technology, and in particular, to an LLM model reasoning method based on a medical rule base and related equipment. Background Art

[0002] Large Language Models (LLM), or pre-trained models, are language models built with deep neural networks containing more than tens of billions of parameters. They use self-supervised learning methods to master many language phenomena through pre-training on large and diverse public datasets.

[0003] However, the LLM model still has many defects when performing reasoning tasks in some specific application scenarios, such as lack of professional knowledge related to the application scenario, which may eventually lead to hallucinations in the dialogue content generated by the LLM model, that is, the content is unreliable and inaccurate, and cannot meet the actual usage needs of users. Even the erroneous inferences given by the LLM model may cause serious harm to users. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide an LLM model reasoning method and related devices based on a medical rule base.

[0005] In a first aspect, the present specification provides an LLM model reasoning method based on a rule base, wherein the rule base includes a plurality of reasoning rules for assisting the LLM model in performing reasoning tasks, and the reasoning rules are structured data consisting of a plurality of fields; wherein the plurality of fields include a plurality of condition fields for describing reasoning conditions, and a target field for describing a reasoning target matching the reasoning conditions; the condition field includes an entity as a reasoning condition; the method includes:

[0006] Acquire a target dialogue text input by a user, and extract text entities contained in the target dialogue text;

[0007] Finding at least one target reasoning rule related to the text entity from among the multiple reasoning rules included in the rule base;

[0008] A prompt word is constructed based on the at least one target reasoning rule, and the prompt word is input into the LLM model. The LLM model performs logical reasoning related to the reasoning task based on the prompt word, and generates a response text corresponding to the target dialogue text based on the reasoning result.

[0009] In a second aspect, the present specification provides an LLM model reasoning device based on a rule base, wherein the rule base includes a plurality of reasoning rules for assisting the LLM model in performing reasoning tasks, and the reasoning rules are structured data consisting of a plurality of fields; wherein the plurality of fields include a plurality of condition fields for describing reasoning conditions, and a target field for describing a reasoning target matching the reasoning conditions; the condition field includes an entity as a reasoning condition; the device includes:

[0010] An acquisition unit, used for acquiring a target dialogue text input by a user, and extracting text entities contained in the target dialogue text;

[0011] A search unit, used to search for at least one target reasoning rule related to the text entity from among the multiple reasoning rules included in the rule base;

[0012] An inference unit is used to construct a prompt word based on the at least one target inference rule, and input the prompt word into the LLM model, so that the LLM model performs logical reasoning related to the reasoning task based on the prompt word, and generates a response text corresponding to the target dialogue text based on the reasoning result.

[0013] Accordingly, this specification also provides a computing device, comprising: a memory and a processor; the memory stores a computer program / instruction that can be executed by the processor; when the processor executes the computer program / instruction, it executes the rule base-based LLM model reasoning method described in the first aspect above.

[0014] Accordingly, the present specification also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the LLM model reasoning method based on the rule base as described in the first aspect above is executed.

[0015] Accordingly, the present specification also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the rule-based LLM model reasoning method described in the first aspect above is executed.

[0016] In summary, after receiving the target dialogue text input by the user, the present application can first find out at least one target reasoning rule related to the text entity contained in the target dialogue text from the multiple reasoning rules contained in the rule library, and then use the at least one target reasoning rule found to assist the LLM model in performing logical reasoning related to the reasoning task, thereby making up for the defect of the LLM model's lack of professional knowledge, improving the reasoning performance of the LLM model, and ensuring the generation quality of the LLM model. In addition, each reasoning rule contained in the rule library is structured data composed of multiple fields, with a simple structure and clear semantics, and can be maintained using conventional relational data tables at a low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

[0018] Figure 2 It is a flowchart of an LLM model reasoning method based on a rule base provided by an exemplary embodiment;

[0019] Figure 3 It is a flowchart of an LLM model reasoning method based on medical guidance rules provided by an exemplary embodiment;

[0020] Figure 4 It is a structural schematic diagram of an LLM model reasoning device based on a rule base provided by an exemplary embodiment;

[0021] Figure 5 It is a structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0023] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0024] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0025] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] It should be understood that the pre-trained LLM model has excellent language understanding capabilities and can perform various reasoning tasks related to natural language, such as sentiment analysis tasks or recommendation tasks based on multi-round dialogues, etc. This specification does not make specific limitations on this. For example, taking the medical guidance scenario as an example, the recommendation task can be to recommend a suitable department to the user before the user sees the doctor (such as dentistry, neurology, cardiology, or respiratory medicine, etc.). For example, taking the shopping scenario as an example, the recommendation task can be to recommend various types of goods that meet the user's preferences (such as electronic products, home furnishings and clothing, etc.), etc. This specification does not make specific limitations on this.

[0027] However, since LLM models often lack professional knowledge in specific application scenarios, the text content generated by the LLM model is not professional and accurate enough, which seriously affects the user experience. For example, LLM models usually lack medical knowledge, which can easily lead to the recommended department finally output by the LLM model when performing medical guidance completely inconsistent with the user's actual situation, affecting the user's medical experience.

[0028] Based on this, this specification provides a technical solution, which uses multiple reasoning rules contained in the rule base to assist the LLM model in performing logical reasoning related to the reasoning task, thereby improving the generation quality of the LLM model.

[0029] In implementation, the rule base in the present application includes multiple reasoning rules for assisting the LLM model in performing reasoning tasks, and each reasoning rule can be structured data composed of multiple fields. Among them, these multiple fields can include several condition fields for describing reasoning conditions, and target fields for describing reasoning targets that match these reasoning conditions, and the condition fields can include entities as reasoning conditions. In an illustrated embodiment, the present application can first obtain the target dialogue text input by the user, and extract the text entities contained in the target dialogue text. Further, at least one target reasoning rule related to the text entity can be found among the multiple reasoning rules contained in the rule base. Further, a prompt word can be constructed based on the at least one target reasoning rule found, and the prompt word can be input into the LLM model, and the LLM model performs logical reasoning related to the above-mentioned reasoning task based on the prompt word, and generates a response text corresponding to the target dialogue text input by the user based on the reasoning result.

[0030] In the above technical solution, after receiving the target dialogue text input by the user, the present application can first find out at least one target reasoning rule related to the text entity contained in the target dialogue text from the multiple reasoning rules contained in the rule library, and then use the at least one target reasoning rule found to assist the LLM model in performing logical reasoning related to the reasoning task, thereby making up for the defect of the LLM model's lack of professional knowledge, improving the reasoning performance of the LLM model, and ensuring the generation quality of the LLM model. In addition, each reasoning rule contained in the rule library is structured data composed of multiple fields, with a simple structure and clear semantics, and can be maintained using conventional relational data tables at a low cost.

[0031] See also Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided by an exemplary embodiment. One or more embodiments provided in this specification may be Figure 1 The present invention is specifically implemented in the system architecture shown or a similar system architecture.

[0032] like Figure 1 As shown, the computing device is equipped with an LLM model, which can be used to perform reasoning tasks related to the target application scenario. Figure 1 As shown, the user can have multiple rounds of dialogue with the LLM model through the input device provided by the computing device (such as a keyboard or a touch screen, etc.) to consult the LLM model related to the target application scenario and obtain corresponding answers, etc.

[0033] In one illustrated embodiment, the target application scenario may be a medical consultation scenario, a shopping scenario, or any other possible scenario, and this specification does not specifically limit this.

[0034] In an illustrated implementation, the LLM model may be a pre-trained LLM basic model, and the LLM basic model often lacks professional knowledge related to a specific target application scenario.

[0035] Alternatively, in an illustrated embodiment, the LLM model may also be an LLM service model obtained by further fine-tuning the pre-trained LLM basic model based on a data set related to the target application scenario. It should be understood that although the LLM service model obtained after fine-tuning has mastered some professional knowledge related to the target application scenario, in application scenarios with strict requirements on professional knowledge, such as medical guidance scenarios, due to the vastness, rigor and complexity of medical knowledge, the LLM service model obtained after fine-tuning still cannot master complete and accurate medical knowledge, especially the long-tail knowledge in the medical field.

[0036] In order to solve the problem that the LLM model lacks professional knowledge of the target application scenario, in an illustrated embodiment, the retrieval-augmented generation (RAG) technology can be used to assist the LLM model in performing logical reasoning. RAG is a technology that combines information retrieval and text generation, and aims to use an external knowledge base to improve the generation quality of the LLM model.

[0037] In the case of RAG technology, if Figure 1 As shown, the present application utilizes a pre-built rule base related to the target application scenario, and the rule base contains multiple reasoning rules for assisting the LLM model to perform reasoning tasks related to the target application scenario. Among them, each reasoning rule can be structured data composed of multiple fields, and these multiple fields can include several condition fields for describing reasoning conditions, and target fields for describing reasoning targets that match these reasoning conditions. Among them, each condition field can contain an entity as a reasoning condition, and each target field can contain one reasoning target, or it can also contain a combination of multiple reasoning targets, which is not specifically limited in this specification.

[0038] In an illustrated implementation, the rule base may maintain the above-mentioned inference rules consisting of multiple fields through a relational data table.

[0039] Exemplarily, the above-mentioned reasoning task can be a recommendation task related to the target application scenario, and accordingly, the reasoning rule can be a recommendation rule composed of a recommendation condition and a recommendation target. Assuming that the target application scenario is a medical guidance scenario, the recommendation task can specifically be a recommendation task for a medical treatment department, the above-mentioned rule base can be a medical guidance rule base, and accordingly, the recommendation rule can be a medical guidance rule, and the recommendation target included in the recommendation rule can be a medical treatment department (such as dentistry, neurology, cardiology, or respiratory medicine, etc.), and the recommendation condition included in the recommendation rule can be various types such as diseases, symptoms, and parts, as shown in Table 1 below.

[0040] Table 1

[0041] disease symptom Location Department cold Runny nose, cough - Respiratory Medicine - Lower back pain, abdominal pain Umbilical Gastroenterology, Orthopedics jaundice Fever, rash, abdominal pain - Infectious Diseases

[0042] As shown in Table 1 above, in the medical guidance scenario, the condition fields used to describe the recommendation conditions may include a disease field, a symptom field, and a part field. Each condition field contains entities that serve as recommendation conditions (for example, "cold" contained in the disease field, "runny nose" and "cough" contained in the symptom field, etc.); the target field used to describe the recommendation target is the department field, which contains a combination of one or more recommended departments (for example, "respiratory medicine", and another example is a combination of "gastroenterology" and "orthopedics").

[0043] It should be noted that this specification does not specifically limit the specific implementation method of building the rule base.

[0044] In an illustrated embodiment, corresponding recommendation rules can be written based on clinical data of real patients or virtual patients to construct the above rule base. In an illustrated embodiment, corresponding recommendation rules can also be automatically generated based on data in the medical knowledge graph to construct the above rule base, and so on, which is not specifically limited in this specification.

[0045] Next, we will combine Figure 1 The system architecture shown illustrates the rule-base-based LLM model reasoning method provided in this application.

[0046] First, if Figure 1 As shown, the user can input the target dialogue text through the computing device, and accordingly, the computing device can obtain the target dialogue text input by the user. For example, taking the target application scenario as a medical consultation scenario, the target dialogue text can be "I have been having a headache recently, which department should I go to for treatment?", or "My wife is pregnant, but has always had high blood pressure, which department should I go to for treatment?", etc., and this specification does not make specific limitations on this.

[0047] Furthermore, if Figure 1As shown, the computing device may perform entity extraction on the acquired target conversation text to extract text entities (or entity mentions) contained in the target conversation text. The specific entity extraction process may refer to the following Figure 2 The description of the corresponding embodiment will not be elaborated here. For example, taking the above sentence “I have been having a headache recently, which department should I go to for treatment?” as an example, the extracted text entity may be “headache”.

[0048] Furthermore, if Figure 1 As shown, the computing device can find at least one target reasoning rule related to the extracted text entity from the multiple reasoning rules included in the rule base. For details, please refer to the following Figure 2 The description of the corresponding embodiments will not be elaborated in detail here.

[0049] Furthermore, if Figure 1 As shown, the computing device can construct a prompt word based on at least one target reasoning rule found, and input the prompt word into the LLM model, and the LLM model performs logical reasoning related to the reasoning task based on the prompt word, and generates a response text corresponding to the above target dialogue text based on the reasoning result. Figure 1 As shown, the computing device may output and display the response text to the user.

[0050] For example, still taking the medical guidance scenario as an example, the reasoning task can be a task of recommending medical treatment departments, that is, recommending a suitable treatment department to the user before the user's visit. Correspondingly, the response text generated by the LLM model can include the recommended department for the user's reference.

[0051] In one illustrated embodiment, Figure 1 The computing device shown may be, for example, a smart wearable device, a smart phone, a tablet computer, a laptop computer or a desktop computer, etc. having the above functions, and this specification does not make any specific limitation on this.

[0052] As mentioned above, in order to solve the problem that the LLM model lacks professional knowledge of the target application scenario, the present application uses RAG technology, and uses a pre-built rule base to assist the LLM model in performing reasoning tasks, thereby improving the generation quality of the LLM model. It should be noted that in conventional RAG technical solutions, knowledge graphs in specific application scenarios (or specific fields) are often used to enhance the professional knowledge of the LLM model. For example, in medical consultation scenarios, due to the rigor and complexity of clinical medicine, most LLM models need to rely on a huge medical knowledge graph for reasoning, and the graph constructs rich medical knowledge based on a large number of nodes and edges. However, in the medical guidance scenario before the visit, the LLM model only needs to give a suitable department recommendation, and does not require particularly complex and detailed medical knowledge. Based on this, the present application utilizes a pre-built medical guidance rule base, which can use a simple relational data table to maintain multiple guidance rules, each of which is structured data containing multiple fields, with a simple structure and clear semantics, and its data storage scale and cost are much smaller than the medical knowledge graph.

[0053] See also Figure 2 , Figure 2 is a flowchart of an LLM model reasoning method based on a rule base provided by an exemplary embodiment. The method can be applied to Figure 1 The computing devices in the system architecture shown. Figure 2 As shown, the method may specifically include the following steps S201 to S203.

[0054] Step S201, obtaining a target dialogue text input by a user, and extracting text entities contained in the target dialogue text.

[0055] As mentioned above, this application adopts RAG technology with the help of a pre-built rule base, which contains multiple inference rules for assisting the LLM model in performing inference tasks. Each inference rule can be structured data consisting of multiple fields. For details, please refer to the above Figure 1 The description in the corresponding embodiment will not be repeated here.

[0056] First, the computing device obtains a target dialogue text input by a user, and further extracts text entities contained in the target dialogue text.

[0057] It should be noted that this specification does not specifically limit the specific implementation method of entity extraction.

[0058] In one illustrated embodiment, an LLM model instruction tuning method may be used to input the target dialogue text input by the user and the preset entity extraction instruction into the LLM model, and the LLM model extracts the text entities contained in the target dialogue text according to the entity extraction instruction.

[0059] It should be understood that, considering that not every text entity contained in the target dialogue text input by the user has reference value for the reasoning task performed by the LLM model, it is necessary to specifically extract the text entities that have reference value.

[0060] Based on this, in an illustrated embodiment, the above-mentioned preset entity extraction instruction may include a preset entity type, and the preset entity type matches the entity type of the entity that can be used as an inference condition in the inference rule. Accordingly, when the LLM model extracts the text entities contained in the target dialogue text, it may specifically include: extracting text entities that match the preset entity type from the target dialogue text.

[0061] For example, taking the medical consultation scenario as an example, the entity types of entities used as reasoning conditions (i.e., recommendation conditions) may include the types of diseases, symptoms, and parts shown in Table 1. Correspondingly, the preset entity types included in the entity extraction instructions may also be types of diseases, symptoms, and parts. In other words, the LLM model only needs to extract text entities that match the types of diseases, symptoms, and parts from the target conversation text.

[0062] For example, if the target dialogue text input by the user is "I have been having a headache recently, which department should I go to for treatment?", the text entity extracted by the LLM model may be "headache". For example, if the target dialogue text input by the user is "My wife is pregnant, but has high blood pressure, which department should I go to for treatment?", the text entities extracted by the LLM model may be "pregnancy" and "high blood pressure".

[0063] In addition, in an illustrated embodiment, the LLM model can also extract user information contained in the target conversation text, such as age, gender, and medical history, etc., which is not specifically limited in this specification. The extracted user information is mainly used to determine whether the user is an adult, a child, and whether the user is pregnant, etc., so as to better recommend a suitable department for the user. For example, if the user is a child, pediatrics is recommended to him / her first. For example, if the user is a male, a series of departments such as gynecology and obstetrics can be excluded, and so on. This specification does not make specific limitations on this.

[0064] Step S202: searching for at least one target reasoning rule related to the text entity from among the multiple reasoning rules included in the rule base.

[0065] Furthermore, after extracting the text entity contained in the target dialogue text, at least one target reasoning rule related to the text entity can be found among the multiple reasoning rules contained in the rule base.

[0066] In an illustrated embodiment, considering that the text entities contained in the target dialogue text input by the user may contain non-standard situations such as abbreviations, omissions and aliases, it is necessary to first match the text entities to unified and standard entities in the rule library.

[0067] Specifically, a fuzzy matching method can be used to determine a target entity that matches the extracted text entity from among multiple entities included in the rule base as reasoning conditions. Further, at least one target reasoning rule related to the target entity is found from among multiple reasoning rules included in the rule base.

[0068] Exemplarily, assuming that the extracted text entity is "pregnancy", the target entity that matches it in the rule base may be "pregnancy". Accordingly, at least one target inference rule related to the target entity "pregnancy" may be an inference rule that includes "pregnancy" in the inference condition, such as "pregnancy-abdominal pain-obstetrics", etc. Exemplarily, assuming that the extracted text entity is "headache", the target entity that matches it in the rule base may be "headache". Accordingly, at least one target inference rule related to the target entity "headache" may be an inference rule that includes "headache" in the inference condition, such as "fever-headache-fever clinic", etc. This specification does not make specific limitations on this.

[0069] In an illustrated embodiment, the above-mentioned fuzzy matching method is used to determine the target entity that matches the extracted text entity from the multiple entities included in the rule base as reasoning conditions. Specifically, it can include: calculating the embedding vector corresponding to the extracted text entity; and, calculating the embedding vector corresponding to the multiple entities included in the rule base as reasoning conditions; then, calculating the similarity between the embedding vector of the text entity and the embedding vectors of multiple entities, and determining the target entity with the greatest similarity to the text entity from the multiple entities.

[0070] In addition, as described in step S201 above, the LLM model can also extract the user information of the user from the conversation text input by the user. Based on this, the present application can first obtain more comprehensive user information from the multiple rounds of historical conversation texts input by the user including the current target conversation text, and further, find out at least one target reasoning rule that is consistent with the user information and related to the text entity from the multiple reasoning rules contained in the rule base. In this way, it can be ensured that the target reasoning rule found is more consistent with the actual situation of the user, thereby improving the quality of subsequent LLM model generation and ensuring the user's experience.

[0071] Step S203, constructing prompt words based on the at least one target reasoning rule, and inputting the prompt words into the LLM model, the LLM model performing logical reasoning related to the reasoning task based on the prompt words, and generating a response text corresponding to the target dialogue text based on the reasoning result.

[0072] Furthermore, after finding at least one target reasoning rule, the at least one target reasoning rule can be used to assist the LLM model in performing the reasoning task. Specifically, a prompt word can be constructed based on the at least one target reasoning rule, and the prompt word can be input into the LLM model, and the LLM model can perform logical reasoning related to the reasoning task based on the prompt word, and generate a response text corresponding to the target dialogue text based on the reasoning result.

[0073] In an illustrated embodiment, during the rule search process of step S202, only one target reasoning rule related to the text entity may be found, and the computing device may construct a prompt word based on the target reasoning rule. Accordingly, the response text generated by the LLM model may include the reasoning target contained in the target reasoning rule.

[0074] In an illustrated embodiment, during the rule search process of step S202, multiple target inference rules related to the text entity may also be found. Based on this, it can be determined whether the multiple inference targets included in the multiple target inference rules are the same.

[0075] In an illustrated embodiment, if the multiple inference targets included in the multiple target inference rules are all the same, that is, the multiple target inference rules include the same inference target, it can be considered that the inference target can be clearly identified based on the currently acquired dialogue content, so the prompt word can be constructed based on all or any part of the inference rules in the multiple target inference rules. Accordingly, the response text generated by the LLM model can include the same inference target.

[0076] For example, still taking the medical guidance scenario as an example, the extracted text entities may include "cold", "runny nose" and "cough", and 20 target guidance rules related to "cold", "runny nose" and "cough" are found. The departments included in these 20 target guidance rules are all "respiratory medicine". Then, prompt words can be constructed based on all or any part of the guidance rules in these 20 target guidance rules. Accordingly, the response text generated by the LLM model can include the final recommended department - "respiratory medicine" for user reference.

[0077] In an illustrated embodiment, if the multiple inference targets included in the multiple target inference rules are all different or partially different, the number of inference rules corresponding to the multiple different inference targets and their proportion in the total number of the multiple target inference rules can be calculated. Further, based on the proportion calculation result, at least one target inference rule is selected from the multiple target inference rules, and a prompt word is constructed based on the selected at least one target inference rule.

[0078] In an illustrated embodiment, when at least one target inference rule is screened out from multiple target inference rules according to the proportion calculation result, if the number of inference rules corresponding to any inference target among multiple different inference targets accounts for a greater proportion of the total number of target inference rules than a preset threshold, at least one target inference rule can be screened out from the inference rules corresponding to the inference target. Subsequently, a prompt word can be constructed based on the at least one screened target inference rule, and so on.

[0079] Correspondingly, the response text generated by the LLM model can also include the reasoning target whose proportion is greater than the preset threshold. This is equivalent to that although there are still multiple different reasoning targets in the current search results, the vast majority of reasoning rules point to the same reasoning target, that is, the reasoning target is basically clear, and the reasoning target can be output to the user.

[0080] In an illustrated embodiment, a preset number P may be set to filter out P target inference rules from the inference rules corresponding to the inference targets whose proportion is greater than a preset threshold, where P is an integer greater than or equal to 1.

[0081] For example, still taking the medical guidance scenario as an example, and the preset threshold is 50%, P equals 5. The extracted text entities may include "headache" and "dizziness", and 5,000 target guidance rules related to "headache" and "dizziness" are found, of which 4,000 target guidance rules include departments that are a combination of "neurology" and "otolaryngology", 500 target guidance rules include departments that are "surgery", and 500 target guidance rules include departments that are "oncology".

[0082] Obviously, the number of guidance rules corresponding to the combination of "neurology" and "otolaryngology" (4000) accounts for 80% of the total number (5000), which is greater than the preset threshold of 50%, while the proportion of guidance rules for other departments is less than the preset threshold. Therefore, 5 target guidance rules can be screened out from the 4000 guidance rules corresponding to the combination of "neurology" and "otolaryngology". Subsequently, prompt words can be constructed based on the 5 screened target guidance rules, and so on. Accordingly, the response text generated by the LLM model can include the final recommended departments - "neurology" and "otolaryngology" for user reference.

[0083] In an illustrated embodiment, when at least one target reasoning rule is screened out from multiple target reasoning rules according to the proportion calculation result, if the number of reasoning rules corresponding to each reasoning target in multiple different reasoning targets accounts for less than a preset threshold in the total number of multiple target reasoning rules, at least one target reasoning rule can be screened out from the reasoning rules corresponding to at least one reasoning target with the largest proportion. Subsequently, a prompt word can be constructed based on the screened out at least one target reasoning rule, and so on.

[0084] Correspondingly, the response text generated by the LLM model may include inquiry text related to the reasoning conditions corresponding to at least one reasoning target with the largest proportion, which is equivalent to being unable to clarify the reasoning target based on the currently acquired conversation content. Therefore, the LLM model needs to ask the user for more valid information related to the reasoning conditions in order to clarify the reasoning target.

[0085] In an illustrated embodiment, preset numbers K and P may be set to select P target inference rules from the inference rules corresponding to the K inference targets with the largest proportions, respectively, where K and P are integers greater than or equal to 1.

[0086] For example, still taking the medical guidance scenario as an example, and the preset threshold is 50%, the preset number K is equal to 2, and P is equal to 4, the extracted text entity can only include "headache", and 10,000 target guidance rules related to "headache" are found, of which 2,500 target guidance rules include departments that are a combination of "neurology" and "neurosurgery", 1,000 target guidance rules include departments that are a combination of "neurology" and "cardiovascular medicine", 1,500 target guidance rules include departments that are "surgery", 2,000 target guidance rules include departments that are "oncology", and 3,000 target guidance rules include departments that are a combination of "neurology" and "otolaryngology".

[0087] Obviously, the number of guidance rules corresponding to each department or department combination accounts for less than 50% of the total number (10,000), among which the number of guidance rules corresponding to the combination of "neurology" and "otolaryngology" (3,000) accounts for 30% of the total number (10,000), and the number of guidance rules corresponding to the combination of "neurology" and "neurosurgery" (2,500) accounts for 25% of the total number (10,000), which are the two largest proportions among multiple departments or department combinations. Based on this, the computing device can filter out 4 target guidance rules from the 3,000 guidance rules corresponding to the combination of "neurology" and "otolaryngology", and filter out 4 target guidance rules from the 2,500 guidance rules corresponding to the combination of "neurology" and "neurosurgery". Subsequently, prompt words can be constructed based on the 8 filtered target guidance rules, and so on. Accordingly, the response text generated by the LLM model may include inquiry text related to the recommendation conditions corresponding to the combination of "neurology" and "otolaryngology", and the combination of "neurology" and "neurosurgery", such as "do you have other symptoms besides headache, such as dizziness or vomiting?", so as to further ask the user about other related symptoms, thereby obtaining more effective information that helps to clarify the department and ensure the accuracy of the final department recommendation.

[0088] In one illustrated embodiment, see Figure 3 , Figure 3 This is a flowchart of an LLM model reasoning method based on medical guidance rules provided by an exemplary embodiment. Figure 3 The LLM model reasoning method based on the rule base provided in this application is explained.

[0089] First, if Figure 3 As shown, after obtaining the dialogue text input by the user, the user information and text entities of the symptom type (for example, including symptom 1, symptom 2 and symptom 3) can be extracted therefrom.

[0090] Furthermore, if Figure 3 As shown, the embedding vectors corresponding to the extracted symptoms 1, 2, and 3 are calculated, and matching and retrieval (Seeker & Retriever) are performed in the rule base based on the embedding vectors and user information to find at least one guidance rule that matches the user information and is related to symptoms 1, 2, and 3, for example Figure 3 The specific matching and retrieval process can refer to the above Figure 2 The description in the corresponding embodiment will not be repeated here.

[0091] like Figure 3As shown, each round of dialogue can obtain all the extracted symptom information (which may also include disease information and location information, etc.) through the memory update module. Figure 3 not shown).

[0092] Furthermore, if Figure 3 As shown, prompt words can be constructed based on the structured instructions (Structured Instruction) of Diagnosis-RAG and diagnosis combinations (i.e., multiple guidance rules), and the prompt words can be input into the LLM model. The LLM model performs logical reasoning related to the department recommendation task based on the prompt words, and further asks the user about symptoms / locations based on the reasoning results, or outputs the recommended department to the user.

[0093] In summary, after receiving the target dialogue text input by the user, the present application can first find out at least one target reasoning rule related to the text entity contained in the target dialogue text from the multiple reasoning rules contained in the rule library, and then use the at least one target reasoning rule found to assist the LLM model in performing logical reasoning related to the reasoning task, thereby making up for the defect of the LLM model's lack of professional knowledge, improving the reasoning performance of the LLM model, and ensuring the generation quality of the LLM model. In addition, each reasoning rule contained in the rule library is structured data composed of multiple fields, with a simple structure and clear semantics, and can be maintained using conventional relational data tables at a low cost.

[0094] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides an LLM model reasoning device based on a rule base, which can be applied to Figure 1 In the computing device in the system architecture shown. The rule base contains multiple inference rules for assisting the LLM model in performing inference tasks. Each inference rule is structured data consisting of multiple fields, including several condition fields for describing inference conditions and target fields for describing inference targets matching the inference conditions. The condition fields contain entities as inference conditions. Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a structure of an LLM model reasoning device based on a rule base provided by an exemplary embodiment. Figure 4 As shown, the device 40 includes:

[0095] The acquisition unit 401 is used to acquire the target dialogue text input by the user and extract the text entities contained in the target dialogue text;

[0096] A search unit 402, configured to search for at least one target reasoning rule related to the text entity from among the multiple reasoning rules included in the rule base;

[0097] The reasoning unit 403 is used to construct a prompt word based on the at least one target reasoning rule, and input the prompt word into the LLM model, so that the LLM model performs logical reasoning related to the reasoning task based on the prompt word, and generates a response text corresponding to the target dialogue text based on the reasoning result.

[0098] In an illustrated embodiment, the acquisition unit 401 is specifically configured to:

[0099] The target dialogue text and a preset entity extraction instruction are input into the LLM model, and the LLM model extracts text entities contained in the target dialogue text according to the entity extraction instruction.

[0100] In an illustrated embodiment, the entity extraction instruction includes a preset entity type, and the preset entity type matches the entity type of the entity used as the reasoning condition in the reasoning rule;

[0101] The acquisition unit 401 is specifically configured to extract text entities matching the preset entity type from the target conversation text.

[0102] In an illustrated embodiment, the searching unit 402 is specifically configured to:

[0103] Determine a target entity that matches the extracted text entity among a plurality of entities included in the rule base as reasoning conditions;

[0104] At least one target inference rule related to the target entity is found out from the multiple inference rules included in the rule base.

[0105] In an illustrated embodiment, the searching unit 402 is specifically configured to:

[0106] Calculating an embedding vector corresponding to the extracted text entity; and,

[0107] Calculating embedding vectors corresponding to a plurality of entities included in the rule base as reasoning conditions;

[0108] The similarity between the embedding vector of the text entity and the embedding vectors of the multiple entities is calculated, and a target entity having the greatest similarity with the text entity is determined from the multiple entities.

[0109] In an illustrated embodiment, the searching unit 402 is specifically configured to:

[0110] Acquiring user information of the user from historical conversation texts input by the user that include the target conversation text;

[0111] At least one target inference rule related to the text entity that matches the user information is found among the multiple inference rules included in the rule base.

[0112] In an illustrated embodiment, at least one target reasoning rule associated with the text entity includes a plurality of target reasoning rules, wherein the plurality of target reasoning rules include a plurality of different reasoning targets;

[0113] The reasoning unit 403 is specifically used for:

[0114] Calculate the proportion of the number of inference rules corresponding to the multiple different inference targets in the total number of the multiple target inference rules;

[0115] According to the proportion calculation result, at least one target reasoning rule is screened out from the multiple target reasoning rules, and a prompt word is constructed based on the screened at least one target reasoning rule.

[0116] In an illustrated embodiment, the reasoning unit 403 is specifically configured to:

[0117] If the number of inference rules corresponding to any one of the multiple inference targets accounts for more than a preset threshold in the total number of the multiple target inference rules, then at least one target inference rule is selected from the inference rules corresponding to the inference target;

[0118] If the number of inference rules corresponding to each of the multiple inference targets accounts for less than the preset threshold in the total number of the multiple target inference rules, at least one target inference rule is selected from the inference rules corresponding to at least one inference target with the largest proportion.

[0119] In an illustrated embodiment, if the number of inference rules corresponding to any one of the multiple inference targets accounts for a greater proportion of the total number of the multiple target inference rules than a preset threshold, the generated response text includes the inference target;

[0120] If the number of inference rules corresponding to each of the multiple inference targets accounts for less than the preset threshold in the total number of the multiple target inference rules, the generated response text includes an inquiry text related to the inference condition corresponding to at least one inference target with the largest proportion.

[0121] In an illustrated embodiment, at least one target reasoning rule associated with the text entity includes a plurality of target reasoning rules, wherein the plurality of target reasoning rules include the same reasoning target;

[0122] The reasoning unit 403 is specifically used for:

[0123] The prompt words are constructed based on all or any part of the multiple target inference rules.

[0124] In an illustrated embodiment, the generated response text includes the same inference target included in the multiple target inference rules.

[0125] In an illustrated embodiment, the reasoning task includes a recommendation task related to a target application scenario, and the reasoning rule includes a recommendation rule consisting of a recommendation condition and a recommendation target.

[0126] In one illustrated embodiment, the target application scenario includes a medical guidance scenario, the recommended tasks include recommended tasks for medical treatment departments, the rule base includes a medical guidance rule base, the recommended rules include medical guidance rules, the recommended targets included in the recommended rules include medical treatment departments, and the entity types of entities included in the recommended rules as recommendation conditions include at least one type of disease, symptom and part.

[0127] The implementation process of the functions and effects of each unit in the above-mentioned device 40 is specifically described in the above-mentioned embodiment, and will not be repeated here. It should be understood that the above-mentioned device 40 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0129] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, a vehicle-mounted computer, or a combination of any of these devices.

[0130] Corresponding to the above method embodiment, the embodiment of this specification also provides a computing device. Figure 5 , Figure 5 is a schematic diagram of a computing device provided by an exemplary embodiment. The computing device may be Figure 1 The computing device in the system architecture shown, wherein the rule base contains multiple reasoning rules for assisting the LLM model in performing reasoning tasks, each reasoning rule is structured data consisting of multiple fields, the multiple fields including several condition fields for describing reasoning conditions, and a target field for describing a reasoning target matching the reasoning condition, the condition field containing an entity as a reasoning condition. Figure 5 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Figure 5 As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the LLM model reasoning method based on the rule base in the embodiment of this specification, including: obtaining the target dialogue text input by the user, and extracting the text entity contained in the target dialogue text; finding at least one target reasoning rule related to the text entity in the multiple reasoning rules contained in the rule base; constructing a prompt word based on the at least one target reasoning rule, and inputting the prompt word into the LLM model, the LLM model performs logical reasoning related to the reasoning task based on the prompt word, and generates a response text corresponding to the target dialogue text based on the reasoning result, and so on.

[0131] For a detailed description of each step of the above-mentioned rule-based LLM model reasoning method, please refer to the previous content and will not be repeated here.

[0132] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored. When these computer programs are executed by a processor, they execute the various steps of the LLM model reasoning method based on the rule base in the embodiment of this specification. Please refer to the description of the above embodiment for details, which will not be repeated here.

[0133] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0134] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0135] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0136] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0137] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0138] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0139] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A LLM model reasoning method based on a rule base, characterized in that: The rule base includes a plurality of inference rules for assisting the LLM model in performing inference tasks, wherein the inference rules are structured data consisting of a plurality of fields; wherein the plurality of fields include a plurality of condition fields for describing inference conditions, and a target field for describing an inference target matching the inference conditions; the condition field includes an entity as an inference condition; and the method includes: Acquire a target dialogue text input by a user, and extract text entities contained in the target dialogue text; Finding at least one target reasoning rule related to the text entity from among the multiple reasoning rules included in the rule base; A prompt word is constructed based on the at least one target reasoning rule, and the prompt word is input into the LLM model. The LLM model performs logical reasoning related to the reasoning task based on the prompt word, and generates a response text corresponding to the target dialogue text based on the reasoning result.

2. The method according to claim 1, characterized in that The extracting text entities contained in the target dialogue text includes: The target dialogue text and a preset entity extraction instruction are input into the LLM model, and the LLM model extracts text entities contained in the target dialogue text according to the entity extraction instruction.

3. The method according to claim 2, characterized in that The entity extraction instruction includes a preset entity type, and the preset entity type matches the entity type of the entity used as the reasoning condition in the reasoning rule; The extracting of text entities contained in the target dialogue text includes: extracting text entities matching the preset entity type from the target dialogue text.

4. The method according to claim 1, characterized in that: The step of searching for at least one target inference rule related to the text entity from among the multiple inference rules included in the rule base includes: Determine a target entity that matches the extracted text entity among a plurality of entities included in the rule base as reasoning conditions; At least one target inference rule related to the target entity is found out from the multiple inference rules included in the rule base.

5. The method according to claim 4, characterized in that The step of determining a target entity that matches the extracted text entity from among the multiple entities included in the rule base as reasoning conditions includes: Calculating an embedding vector corresponding to the extracted text entity; and, Calculating embedding vectors corresponding to a plurality of entities included in the rule base as reasoning conditions; The similarity between the embedding vector of the text entity and the embedding vectors of the multiple entities is calculated, and a target entity having the greatest similarity with the text entity is determined from the multiple entities.

6. The method according to claim 4, characterized in that The step of searching for at least one target inference rule related to the target entity from among the multiple inference rules included in the rule base includes: Acquiring user information of the user from historical conversation texts input by the user that include the target conversation text; At least one target inference rule related to the text entity that matches the user information is found among the multiple inference rules included in the rule base.

7. The method according to claim 1, characterized in that The at least one target reasoning rule related to the text entity includes a plurality of target reasoning rules, wherein the plurality of target reasoning rules include a plurality of different reasoning targets; The step of constructing a prompt word based on the at least one target reasoning rule includes: Calculate the proportion of the number of inference rules corresponding to the multiple different inference targets in the total number of the multiple target inference rules; According to the proportion calculation result, at least one target reasoning rule is screened out from the multiple target reasoning rules, and a prompt word is constructed based on the screened at least one target reasoning rule.

8. The method according to claim 7, characterized in that The step of selecting at least one target reasoning rule from the plurality of target reasoning rules according to the proportion calculation result includes: If the number of inference rules corresponding to any one of the multiple inference targets accounts for more than a preset threshold in the total number of the multiple target inference rules, then at least one target inference rule is selected from the inference rules corresponding to the inference target; If the number of inference rules corresponding to each of the multiple inference targets accounts for less than the preset threshold in the total number of the multiple target inference rules, at least one target inference rule is selected from the inference rules corresponding to at least one inference target with the largest proportion.

9. The method according to claim 8, characterized in that If the number of inference rules corresponding to any one of the multiple inference targets accounts for more than a preset threshold in the total number of the multiple target inference rules, the generated response text includes the inference target; If the number of inference rules corresponding to each of the multiple inference targets accounts for less than the preset threshold in the total number of the multiple target inference rules, the generated response text includes an inquiry text related to the inference condition corresponding to at least one inference target with the largest proportion.

10. The method according to claim 1, characterized in that At least one target reasoning rule related to the text entity includes multiple target reasoning rules, and the multiple target reasoning rules contain the same reasoning target; The step of constructing a prompt word based on the at least one target reasoning rule includes: The prompt words are constructed based on all or any part of the multiple target inference rules.

11. The method according to claim 10, characterized in that The generated response text includes the same inference target included in the multiple target inference rules.

12. The method according to any one of claims 1 to 11, characterized in that: The reasoning tasks include recommendation tasks related to target application scenarios, and the reasoning rules include recommendation rules consisting of recommendation conditions and recommendation targets.

13. The method according to claim 12, characterized in that The target application scenario includes a medical guidance scenario, the recommended tasks include recommended tasks for medical treatment departments, the rule base includes a medical guidance rule base, the recommended rules include medical guidance rules, the recommended targets included in the recommended rules include medical treatment departments, and the entity types of entities included in the recommended rules as recommendation conditions include at least one type of disease, symptom and part.

14. A LLM model reasoning device based on a rule base, characterized in that: The rule base includes a plurality of inference rules for assisting the LLM model in performing inference tasks, wherein the inference rules are structured data consisting of a plurality of fields; wherein the plurality of fields include a plurality of condition fields for describing inference conditions, and a target field for describing an inference target matching the inference conditions; the condition field includes an entity as an inference condition; and the device includes: An acquisition unit, used for acquiring a target dialogue text input by a user, and extracting text entities contained in the target dialogue text; A search unit, used to search for at least one target reasoning rule related to the text entity from among the multiple reasoning rules included in the rule base; An inference unit is used to construct a prompt word based on the at least one target inference rule, and input the prompt word into the LLM model, so that the LLM model performs logical reasoning related to the reasoning task based on the prompt word, and generates a response text corresponding to the target dialogue text based on the reasoning result.

15. A computing device, characterized in that: include: Memory and processor; The memory has stored thereon computer programs / instructions executable by the processor; When the processor runs the computer program / instructions, it performs the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

17. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images