Intelligent question and answer method, device and equipment based on large language model and medium

Abstract

The application discloses an intelligent question answering method and device based on a large language model, equipment and a medium, and belongs to the field of artificial intelligence. The method comprises the following steps: receiving a user question, obtaining permission information related to the user question, and the permission information comprising permission context information and permission level information; screening at least one piece of context knowledge in a knowledge base according to the permission context information and the user question, wherein the context knowledge comprises knowledge in the knowledge base that matches the permission context information and the user question, and the knowledge in the knowledge base has permission level tags and permission context tags; generating a preliminary answer based on the context knowledge and the user question through a large language model; determining reference knowledge of the preliminary answer in the context knowledge based on the preliminary answer and the context knowledge; generating a target answer according to the preliminary answer, the permission level information and the permission level tags of the reference knowledge, and feeding back the target answer to the user. According to the embodiment of the application, the quality of the answer output by the large language model can be improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and in particular relates to an intelligent question-answering method, device, equipment and medium based on a large language model. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is an artificial intelligence technique that combines information retrieval with the generation capabilities of Large Language Models (LLMs). It has wide applications in fields such as intelligent question answering and knowledge management. In practical knowledge base deployments, knowledge bases often contain knowledge with different access levels. To ensure data security, the RAG process employs metadata-based filtering mechanisms for access control before semantic matching. However, this approach can lead to a lack of crucial context when the LLM generates responses, thus reducing the quality of the responses output by the LLM. Summary of the Invention

[0003] This application provides an intelligent question-answering method, apparatus, device, and medium based on a large language model, which can improve the quality of the answers output by the large language model.

[0004] In a first aspect, embodiments of this application provide an intelligent question-answering method based on a large language model, comprising: receiving a user question and obtaining permission information related to the user question, the permission information including permission context information and permission level information; filtering at least one piece of contextual knowledge in a knowledge base based on the permission context information and the user question, the contextual knowledge including knowledge in the knowledge base that matches the permission context information and the user question, the knowledge in the knowledge base having permission level tags and permission context tags; generating a preliminary answer based on the contextual knowledge and the user question using a large language model; determining the referenced knowledge of the preliminary answer in the contextual knowledge based on the preliminary answer and the contextual knowledge; generating a target answer based on the preliminary answer, the permission level information, and the permission level tags of the referenced knowledge, and providing feedback to the user.

[0005] Secondly, embodiments of this application provide an intelligent question-answering device based on a large language model, comprising: a permission information acquisition module, used to receive user questions and acquire permission information related to the user questions, the permission information including permission context information and permission level information; a retrieval module, used to filter at least one piece of contextual knowledge in a knowledge base based on the permission context information and the user questions, the contextual knowledge including knowledge in the knowledge base that matches the permission context information and the user questions, the knowledge in the knowledge base having permission level tags and permission context tags; a first generation module, used to generate a preliminary answer based on the contextual knowledge and the user questions using a large language model; a reference knowledge determination module, used to determine the referenced knowledge of the preliminary answer in the contextual knowledge based on the preliminary answer and the contextual knowledge; and a second generation module, used to generate a target answer based on the preliminary answer, permission level information, and permission level tags of the referenced knowledge, and provide feedback to the user.

[0006] Thirdly, embodiments of this application provide an intelligent question-answering device based on a large language model, including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the intelligent question-answering method based on a large language model as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the intelligent question-answering method based on a large language model as described in the first aspect.

[0008] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the intelligent question-answering method based on a large language model as described in the first aspect.

[0009] This application provides an intelligent question-answering method, apparatus, device, and medium based on a large language model. It can acquire user-related permission context information and permission level information, search a knowledge base based on the permission context information and the user's question, filter out contextual knowledge matching the permission context information and the user's question, use this contextual knowledge as the context of the user's question, generate a preliminary answer corresponding to the user's question through the large language model, determine the referenced knowledge from the contextual knowledge based on the preliminary answer, and generate a target answer to be fed back to the user based on the comparison results of the permission level tags of the referenced knowledge and the permission level information related to the user's question, as well as the preliminary answer. In the retrieval stage of enhanced retrieval, rigid permission context information is used to narrow the scope of knowledge recall, but flexible permission level information is not involved in the retrieval, ensuring that the large language model can freely reference contextual knowledge of all permission levels to generate high-quality preliminary answers, thereby improving the quality of the answers output by the large language model. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating an intelligent question-answering method based on a large language model provided in an embodiment of this application; Figure 2 A schematic diagram of the system architecture for intelligent question answering based on a large language model provided in the embodiments of this application; Figure 3 A schematic diagram of the structure of an intelligent question-answering device based on a large language model provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an intelligent question-answering device based on a large language model, provided as an embodiment of this application. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the purpose, technical solution, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of the details in these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of this application are all authorized by users or relevant organizations and comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or necessarily used such a solution.

[0013] Retrieval-enhanced generation (REGG) is an artificial intelligence technology that combines information retrieval with the generation capabilities of large language models. It can be widely applied in fields such as intelligent question answering and knowledge management. In actual knowledge base deployments, knowledge bases often contain knowledge with different access levels. To ensure data security, REGG employs metadata-based filtering mechanisms for access control during the process, followed by semantic matching. Only the knowledge that the user has the right to access is provided to the large language model. However, many high-quality answers require comprehensive references to knowledge at different access levels. Pre-filtering can lead to a lack of crucial context when the large language model generates answers, resulting in incomplete or substandard responses, thus reducing the overall quality of the output. For example, a user asks a question about the merchant onboarding process. The answer to this question requires both a low-access overview and a high-access merchant onboarding fee information. However, the large language model cannot obtain the high-access merchant onboarding fee information, leading to a low-quality output. Furthermore, the access control dimension in REGG is too singular to meet the complex, multi-dimensional, and multi-layered access requirements of intelligent question answering scenarios. Furthermore, in retrieval-enhanced generation techniques, after generating answers using a large language model, it's impossible to trace the source of the knowledge referenced in the answer or the knowledge access permissions. In intelligent question-answering scenarios, when a user lacks permission to access a piece of knowledge, the system will either refuse to answer or not return a result, lacking guidance for the user and degrading the user experience.

[0014] This application provides an intelligent question-answering method, apparatus, device, and medium based on a large language model. It can annotate knowledge in a knowledge base with permission tags of different dimensions, comprehensively covering permission constraints and solving the problem of single-dimensional permission control. During the enhanced generation process, rigid permission context information can be used for pre-retrieval in the retrieval stage to narrow the scope of knowledge recall, while flexible permission level information is not involved in pre-retrieval, ensuring that the large language model can freely reference knowledge at all permission levels to generate high-quality answers. This application can also accurately trace the knowledge referenced in the answer and determine the final answer fed back to the user based on the permission level of the referenced knowledge. For content in the answer generated by the large language model that exceeds the user's permissions, a simple rejection strategy is not adopted. Instead, guidance is provided, allowing the user to know the existence of the referenced knowledge corresponding to the content exceeding their permissions and the operation path for permission escalation of the referenced knowledge, without disclosing the content exceeding the user's permissions or the referenced knowledge, thus maximizing information display while strictly protecting the content exceeding the user's permissions and the referenced knowledge.

[0015] The following describes the intelligent question-answering method, apparatus, device, medium, and program products based on a large language model provided in this application.

[0016] This application provides an intelligent question-answering method based on a large language model, which can be applied to the field of intelligent question answering or other fields that require large language models to generate answers. This intelligent question-answering method based on a large language model can be executed by intelligent question-answering devices, equipment, etc., based on a large language model. Figure 1 A flowchart of an intelligent question-answering method based on a large language model provided in an embodiment of this application is shown below. Figure 1 As shown, the intelligent question-answering method based on a large language model may include steps S101 to S105.

[0017] In step S101, the user's question is received, and the permission information related to the user's question is obtained.

[0018] User questions can include questions entered by the user. Permission information related to the user question can be obtained from the process of the user initiating the question. Permission information represents the user's permissions. Permission information includes permission context information and permission level information. Permission context information represents the contextual permissions of the knowledge the user can access, and can represent the external usage conditions applicable to the knowledge the user can access at the objective level; this is rigid information. In some examples, permission context information may include, but is not limited to, one or more of the following: user role, request time, access channel type, and user's location. User role represents the type of user role and can be obtained from user identity information. For example, user roles may include, but are not limited to, organizations, merchants, service providers, head offices, branch offices, tourists, etc. Request time can be the time when the user initiates the user question. Access channel type can be used to represent the type of channel the user accesses and can be obtained from the source terminal of the user question. For example, access channel types may include, but are not limited to, personal computers (PCs), mobile devices, mini-programs, and cross-platforms. User's location refers to the geographical area where the user is located. There is no limitation on the level of geographical area; the user's location can be obtained based on the user's location information and the organization to which the user belongs. Permission level information represents a user's permission level, which in turn represents the security management level of permissions. This is flexible information. Permission level information can be obtained from a user's login status. The method of setting permissions represented by permission level information is not limited here. For example, permission level information may include, but is not limited to, Level 1 permission (not logged in), Level 2 permission (logged in), and Level 3 permission (certificate verification). Level 1 permission is lower than Level 2 permission, and Level 2 permission is lower than Level 3 permission. Permission level information and permission context information can cover constraints such as level, role classification, time, channel, and region, thereby precisely controlling the knowledge visibility range of different users under different conditions. In some examples, when permission information related to a user's question is obtained, the permission information can be converted into a permission vector. For example, a permission vector could be U_perm={level, role, platform, region, timestamp}, where level represents the permission level information, role represents the user role, platform represents the access channel type, region represents the user's region, and timestamp represents the request time.

[0019] In step S102, at least one piece of contextual knowledge is obtained by filtering from the knowledge base based on the permission context information and the user question.

[0020] The knowledge base comprises multiple pieces of knowledge, which can originate from, but are not limited to, documents, tables, knowledge graphs, historical Q&A, and web page content within the corresponding field. Each piece of knowledge possesses permission level tags and permission context tags. Each piece of knowledge can be uniquely labeled with a multi-dimensional permission tag, including permission level tags and permission context tags. These tags describe the permissions granted to the knowledge from multiple dimensions. Permission level tags characterize the permission level of the knowledge and are flexible information. For example, permission level tags may include, but are not limited to, Level 1 permission (not logged in), Level 2 permission (logged in), and Level 3 permission (certificate verification), with Level 1 being lower than Level 2, and Level 2 lower than Level 3. Permission context tags characterize the external usage conditions applicable to the knowledge at an objective level and are rigid information. In some examples, permission context tags may include, but are not limited to, one or more of the following: role permission tags, effective time tags, channel permission tags, and region permission tags. Role permission tags characterize the type of user role that can access the knowledge. For example, role permission tags may include, but are not limited to, institutions, merchants, service providers, head offices, branch offices, and visitors. The effective time tag represents the validity period of the knowledge. For example, the effective time tag may include, but is not limited to, permanently valid or valid for a limited time period. The channel permission tag represents the access channels supported by the knowledge. For example, the channel permission tag may include, but is not limited to, personal computers (PCs), mobile devices, mini-programs, and all platforms. The region permission tag represents the geographical area supported by the knowledge; the level of the geographical area represented by the region permission tag is not limited here. Permission level tags and permission context tags are used to label knowledge with permissions, covering multi-dimensional permissions and achieving multi-dimensional access control. Permission level tags and permission context tags can be automatically pre-labeled for each piece of knowledge using a rule engine combined with a semantic classification model. The rule engine can execute keyword matching algorithms; the type of semantic classification model is not limited here. For example, the semantic classification model may include, but is not limited to, the Bidirectional Encoder Representations from Transformers (BERT) model. After the rule engine automatically labels each piece of knowledge using the semantic classification model, manual review and confirmation can be performed to ensure the accuracy of the permission level tags and permission context tags.

[0021] In the retrieval phase of the enhanced retrieval generation process, knowledge is retrieved from the knowledge base based on permission context information and user questions to obtain at least one piece of contextual knowledge. Contextual knowledge includes knowledge in the knowledge base that matches the permission context information and user questions. Matching knowledge with permission context information can refer to the permission context label of the knowledge matching the permission context information; that is, the external usage conditions applicable to the objective level represented by the permission context information are among the external applicable conditions applicable to the objective level represented by the permission context label of the knowledge. Matching knowledge with user questions can refer to the similarity between the semantic vector of the knowledge and the semantic vector of the user question satisfying the matching conditions. Matching conditions may include, but are not limited to, a similarity higher than a preset similarity threshold or a similarity ranking within the top few. It should be noted that contextual knowledge is not limited by permission level information to ensure that the large language model can obtain the full range of permission level contextual knowledge and avoid context loss caused by filtering.

[0022] In step S103, a preliminary answer is generated based on contextual knowledge and the user's question using a large language model.

[0023] Contextual knowledge can be input into the large language model along with the user's question. Leveraging the model's semantic understanding and reasoning capabilities, a preliminary answer corresponding to the user's question can be generated. It's important to note that this preliminary answer is not visible to the user.

[0024] In step S104, based on the preliminary answer and contextual knowledge, the reference knowledge of the preliminary answer in the contextual knowledge is determined.

[0025] Based on the initial answer and contextual knowledge, the contribution of each piece of contextual knowledge to the initial answer can be determined. The referenced knowledge of the initial answer is then determined by the degree of contribution of each piece of contextual knowledge. Referenced knowledge of the initial answer refers to contextual knowledge that has made a high contribution to the initial answer; at least part of the content in the initial answer can be referenced by this knowledge. In some examples, where the initial answer is long, it can be broken down into multiple segments, and the referenced knowledge of each segment can be determined. Specifically, based on each segment of the initial answer and contextual knowledge, the contribution of each piece of contextual knowledge to that segment can be determined, and the referenced knowledge of that segment can be determined by the degree of contribution.

[0026] In some examples, the weights of the attention heads of the transformer layer for generating the initial answer can be based on the large language model, thus obtaining the contribution of each piece of contextual knowledge to the initial answer. When the initial answer is broken down into multiple segments, the weights of the attention heads of the transformer layers for each segment in the initial answer can be generated based on the large language model, thus obtaining the contribution of each piece of contextual knowledge to the segment in the initial answer. In some examples, when the weights of the attention heads of the transformer layer of the large language model are not available, prompt words can be constructed to guide the large language model to annotate the referenced contextual knowledge (i.e., cited knowledge) in the initial answer. That is, prompt words guide the large language model to determine the contribution of contextual knowledge to the initial answer and annotate cited knowledge in the initial answer. Similarly, when the initial answer is broken down into multiple segments, prompt words can be constructed to guide the large language model to annotate the referenced contextual knowledge (i.e., cited knowledge) in the segments of the initial answer. That is, prompt words guide the large language model to determine the contribution of contextual knowledge to the segments in the initial answer. In some examples, when the weights of the attention heads of the transformer layer of the large language model are unavailable, the contribution of contextual knowledge to the initial answer can be reflected by the similarity between the initial answer and the contextual knowledge, thus determining the referenced knowledge of the initial answer. When the initial answer is broken down into multiple segments, the contribution of contextual knowledge to the segments of the initial answer can be reflected by the similarity between the segments of the initial answer and the contextual knowledge, thus determining the referenced knowledge of the segments of the initial answer. In some examples, when the weights of the attention heads of the transformer layer of the large language model are unavailable, the large language model can be guided by prompt words to label the referenced knowledge of the initial answer. Then, the referenced knowledge is verified by comparing the labeled referenced knowledge with the initial answer; only referenced knowledge whose similarity meets the verification criteria can be considered genuine referenced knowledge. Similarly, when the initial answer is broken down into multiple segments, the large language model can be guided by prompt words to label the segments of the initial answer with referenced knowledge. Then, the referenced knowledge is verified by comparing the labeled referenced knowledge with the segments of the initial answer; only referenced knowledge whose similarity meets the verification criteria can be considered genuine referenced knowledge.

[0027] In step S105, a target answer is generated based on the preliminary answer, permission level information, and permission level tags of the referenced knowledge, and then fed back to the user.

[0028] The permission level represented by the permission level information and the permission level represented by the permission level tag of the cited knowledge can be compared to obtain a comparison result. Based on the comparison result and the preliminary answer, a target answer is generated. The comparison result may include a first comparison result and a second comparison result. The first comparison result is that the permission level represented by the permission level information is lower than or equal to the permission level represented by the permission level tag of the cited knowledge. The second comparison result is that the permission level represented by the permission level information is higher than the permission level represented by the permission level tag of the cited knowledge. The target answer is the response given to the user. If the comparison result is the first comparison result, the target answer may include the original text of the preliminary answer. If the comparison result is the second comparison result, the target answer will not include the original text portion of the cited knowledge with a permission level higher than the user's permission level that corresponds to the initial answer. In some examples, in this case, guidance prompts may be generated and placed in the target answer. The guidance prompts can be used to guide the user to know about the existence of protected knowledge and the operation path to obtain the protected knowledge. The protected knowledge is cited knowledge with a permission level higher than the user's permission level. The user's permission level is represented by the permission level information in the embodiments of this application.

[0029] In this embodiment, user-related permission context information and permission level information can be obtained. Based on the permission context information and the user's question, a search is performed in the knowledge base to filter out contextual knowledge matching the permission context information and the user's question. This contextual knowledge is used as the context for the user's question. A preliminary answer corresponding to the user's question is generated through a large language model. Based on the preliminary answer and the contextual knowledge, the referenced knowledge cited in the preliminary answer is determined from the contextual knowledge. Based on the comparison results of the permission level tags of the referenced knowledge and the permission level information related to the user's question, as well as the preliminary answer, a target answer is generated to be fed back to the user. In the retrieval stage of the enhanced retrieval generation, rigid permission context information is used to narrow the scope of knowledge recall, but flexible permission level information is not involved in the retrieval. This ensures that the large language model can freely reference contextual knowledge of all permission levels to generate high-quality preliminary answers, improving the quality of the answers output by the large language model. Through permission level information, permission context information, permission level tags, and permission context tags, permission constraints such as level, role classification, time, channel, and region can be covered, thereby accurately controlling the knowledge visibility range of different users under different conditions. This can meet the complex permission requirements of multi-dimensional and multi-level scenarios in intelligent question answering. This application's embodiments can also trace the contextual knowledge referenced in the initial answer, i.e., the referenced knowledge, and the permission level of the referenced knowledge, meeting data security and compliance audit requirements. The final target answer fed back to the user includes the original text of the initial answer within the user's permission level. The original text of the initial answer exceeding the user's permission level is not included in the target answer, but is replaced by a semantic guidance prompt instead of a rejection response, maximizing information display while strictly protecting sensitive knowledge and improving user experience.

[0030] In some embodiments, during the retrieval stage of enhanced retrieval generation, retrieval filtering conditions can be generated based on permission context information; preliminary knowledge matching the retrieval filtering conditions can be filtered in the knowledge base based on permission context tags; and contextual knowledge matching the user's question can be retrieved from the preliminary knowledge. The retrieval filtering conditions correspond to the permission context information. For example, if the permission context information includes user role A1, request time A2, access channel type A3, and user's location A4, then the generated retrieval filtering conditions may include: role permission tag A1, request time A2 being within the valid time represented by the valid time tag, channel permission tag A3, and location permission tag A4. The retrieval process retrieves knowledge matching the user role's role permission tag using the retrieval filtering conditions corresponding to the user role; retrieves knowledge matching the request time's valid time tag using the retrieval filtering conditions corresponding to the request time; retrieves knowledge matching the access channel type's channel permission tag using the retrieval filtering conditions corresponding to the access channel type; and retrieves knowledge matching the user's location's location using the retrieval filtering conditions corresponding to the user's location's location. Preliminary knowledge includes knowledge in the knowledge base that matches the retrieval filtering conditions. Retrieving from the knowledge base using retrieval filtering conditions can employ an inverted index to achieve efficient multidimensional intersection matching. An inverted index for the knowledge base can be preset, where the keys are permission context labels and the values ​​are knowledge items. Using the inverted index to filter preliminary knowledge can complete the filtering of a large-scale knowledge base in milliseconds. Each piece of preliminary knowledge can be converted into a semantic vector, and the user question can also be converted into a semantic vector. Contextual knowledge is determined by the similarity between the semantic vectors converted from preliminary knowledge and the semantic vectors converted from the user question. In some examples, preliminary knowledge items with a similarity higher than a preset threshold can be identified as contextual knowledge for that user question; or, the top p pieces of preliminary knowledge, ranked from highest to lowest similarity, can be identified as contextual knowledge for that user question, where p is a positive integer.

[0031] In some embodiments, the contribution of contextual knowledge to the initial answer can be determined by the average attention weight of contextual knowledge on the initial answer, thereby identifying referenced knowledge within the contextual knowledge. Specifically, a first average attention weight of each piece of contextual knowledge on the initial answer can be obtained from at least one knowledge set; for each piece of contextual knowledge, the first average attention weight corresponding to each knowledge set is aggregated and averaged to obtain the contribution of each piece of contextual knowledge to the initial answer; if the contribution of contextual knowledge meets a preset reference condition, the contextual knowledge is identified as referenced knowledge.

[0032] The knowledge set includes at least a portion of the at least one piece of contextual knowledge obtained in the above embodiments. That is, the knowledge set may include all contextual knowledge, or the knowledge set may include a portion of all contextual knowledge. In the case where the knowledge set includes all contextual knowledge, this application embodiment has one knowledge set; in the case where the knowledge set includes a portion of all contextual knowledge, this application embodiment may have one knowledge set, or more than two knowledge sets, and this is not limited here. In some examples, when the knowledge set includes a portion of all contextual knowledge, before obtaining the first average attention weight of each piece of contextual knowledge for generating an initial answer in at least one knowledge set, B random samplings can be performed from at least one piece of contextual knowledge. Each time, M pieces of contextual knowledge are randomly sampled to form a knowledge set, where B and M are both positive integers, and B can specifically be a positive integer greater than 1. Suppose there are N pieces of contextual knowledge, forming a total contextual knowledge set C. We can randomly select contextual knowledge from C B times, with each selection involving fewer than N pieces. For example, we can select N×ρ pieces of contextual knowledge each time, forming a knowledge set, thus obtaining B knowledge sets. The B knowledge sets corresponding to one contextual knowledge set C can be represented as C1, C2, ..., C... B When multiple pieces of contextual knowledge jointly influence the generation of an initial answer, the attention of the large language model will be scattered among these pieces of contextual knowledge, resulting in the contribution of each piece of contextual knowledge being lower than the actual situation. By randomly sampling to form multiple knowledge sets, the contribution of the attention weights of the contextual knowledge in the knowledge sets to the generation of the initial answer can be obtained more accurately.

[0033] For each knowledge set, the first average attention weight of each context knowledge pair in generating the initial answer can be obtained. The first average attention weight of a context knowledge pair in generating the initial answer reflects the degree of dependence of the initial answer on the core features of that context knowledge. Furthermore, the first average attention weight can be the average weight of the attention heads of the core features of that context knowledge in generating the initial answer at the transformer layer.

[0034] In some examples, for each piece of contextual knowledge, the second average attention weight of each token in the contextual knowledge for each attention head in each transformer layer that generates the initial answer can be obtained. For each piece of contextual knowledge, the K tokens with the highest second average attention weights are selected, and the average of the second average attention weights of the K tokens is determined as the first average attention weight, where K is a positive integer. A piece of contextual knowledge may include multiple tokens, and the initial answer may include multiple tokens. The second average attention weight of a token in a piece of contextual knowledge is the average of the attention weights of that token for each token in the initial answer in each transformer layer, which is the average of the attention weights across transformer layers and across attention heads. For example, a knowledge set C includes N pieces of contextual knowledge, C={P1, P2, ..., P...} N}, P i Let P be the i-th contextual knowledge in the knowledge set C. i Including Q tokens, P i ={X1, X2, ..., X Q}, X i For contextual knowledge P i The i-th token in the sequence represents the initial answer Y, which includes S tokens, Y = {Y1, Y2, ..., Y...}. S Y2 is the j-th token in the initial response Y, and X is a token in the context knowledge. i The second average attention weight for the initial response can be obtained according to the following formula (1): (1) in, The second average attention weight of the i-th token in the contextual knowledge for the initial answer; For the attention weight of the i-th token in the context knowledge to the j-th token in the initial answer, the large language model provides the attention weight of the h-th attention head in the l-th transformer layer. The function allows inputting [user question][contextual knowledge P1][contextual knowledge P2]...[contextual knowledge P] into a large language model. N [Initial Response Y], the large language model can output the attention weights of the i-th token in each contextual knowledge and the j-th token in the initial response in the h-th attention head of the l-th converter layer of the large language model. This can be output in matrix form, but is not limited to this; L is the number of converter layers, and H is the number of attention heads. The number of tokens for the initial response.

[0035] Among the multiple tokens of contextual knowledge, some tokens are core tokens of contextual knowledge. Core tokens can reflect the semantics of contextual knowledge, while the attention weights of other tokens in contextual knowledge may cause the semantic features to be dispersed. The K tokens with the highest second average attention weights can be selected as core tokens that can reflect the semantics of contextual knowledge. The average of the second average attention weights of these K tokens can be determined as the first average attention weight. For example, the first average attention weight can be obtained according to the following formula (2): (2) in, Let be the first average attention weight of the i-th contextual knowledge point in generating the initial answer; It is the set of the K tokens with the highest average attention weights in the second category; Let be the second average attention weight of the i-th token in the context knowledge for the initial answer.

[0036] The contribution of each piece of contextual knowledge to the initial answer is obtained by aggregating and averaging its corresponding first average attention weight within the knowledge set. Aggregation and averaging can include aggregation and averaging. Aggregation can be performed based on whether the contextual knowledge is in the knowledge set and its corresponding first average attention weight, resulting in an aggregated result for the knowledge set. The average value of this aggregated result across the knowledge set is then calculated to determine the contribution of the contextual knowledge to the initial answer. For example, given B knowledge sets, the i-th piece of contextual knowledge P... i The contribution of the initial answer Y can be obtained according to the following formula (3): (3) in, Let i be the contribution of the i-th piece of contextual knowledge out of all contextual knowledge to the initial answer Y; This is the b-th knowledge set; The number of knowledge sets; It is an exponential function, if it satisfies ,but =1, otherwise 0; Let be the first average attention weight of the i-th contextual knowledge for generating the initial answer.

[0037] The contribution of contextual knowledge to the initial answer characterizes the degree to which contextual knowledge contributes to the generation of the initial answer. In other words, the contribution of contextual knowledge to the initial answer characterizes the degree to which the initial answer references or depends on that contextual knowledge. The preset citation criteria are used to filter referenced knowledge. If the contribution of contextual knowledge meets the citation criteria, then that contextual knowledge can be identified as referenced knowledge in the initial answer; if the contribution of contextual knowledge does not meet the citation criteria, then that contextual knowledge will not be identified as referenced knowledge. In some examples, the citation criteria may include, but are not limited to, one of the following: contribution exceeding a contribution threshold, contribution ranking within the top E from highest to lowest, or normalized contribution exceeding a contribution threshold, and normalized contribution ranking within the top E from highest to lowest. The contribution threshold can be set according to the scenario, needs, experience, etc., and is not limited here.

[0038] In some embodiments, the initial answer may be too long. In such cases, this application supports more refined access control for segmented tracing of the initial answer. Specifically, if the length of the initial answer exceeds a length threshold, it can be semantically divided into at least two answer segments. In at least one knowledge set, the first average attention weight of each contextual knowledge item on the answer segment is obtained. The length threshold is a length limit value used to determine whether the initial answer needs to be segmented. It can be set according to the scenario, requirements, experience, etc., and is not limited here. If the length of the initial answer exceeds the length threshold, it is segmented; if the length of the initial answer is less than or equal to the length threshold, it is not segmented, and the referenced knowledge of the entire initial answer is determined. The initial answer needs to be semantically segmented to obtain at least two answer segments. For each answer segment, the first average attention weight of each contextual knowledge item in the knowledge set on that answer segment is obtained. The first average attention weights corresponding to the knowledge set are aggregated and averaged to obtain the contribution of each contextual knowledge item to the answer segment. If the contribution of the contextual knowledge meets the reference condition, the contextual knowledge is determined as the referenced knowledge of the answer segment. For each piece of contextual knowledge, obtain the second average attention weight of each attention head in each transformer layer of the generated answer fragment for each token in the contextual knowledge; for each piece of contextual knowledge, select the K tokens with the highest second average attention weight, and determine the average of the second average attention weights of these K tokens as the first average attention weight. The calculation of the first average attention weight of contextual knowledge to the answer fragment and the contribution of contextual knowledge to the answer fragment are similar to the calculation of the first average attention weight of contextual knowledge to the initial answer and the contribution of contextual knowledge to the initial answer in the above embodiments, and can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0039] In some embodiments, if the internal state of the large language model is not visible—that is, if the attention weights of the tokens in the context knowledge relative to the tokens in the initial answer cannot be obtained—the large language model can be guided to output an initial answer with identifiers of cited knowledge through constructed prompts. The output format of the large language model can be constrained in the prompts, requiring the large language model to label the cited knowledge for each key argument when outputting the initial answer. The large language model can label the entire initial answer with cited knowledge identifiers, or it can divide the initial answer into at least two answer segments, with each answer segment in the initial answer output by the large language model through the prompts labeled with cited knowledge identifiers.

[0040] In some embodiments, if the internal state of the large language model is not visible—that is, if the attention weights of the tokens in the context knowledge and the tokens in the initial answer cannot be obtained—in the transformer layer of the large language model, the referenced knowledge can be determined based on the similarity parameter between the initial answer and the context knowledge. The initial answer and the context knowledge can be converted into semantic vectors. The similarity parameter between the semantic vectors converted from the initial answer and the semantic vectors converted from the context knowledge can be calculated. Context knowledge with a similarity higher than or equal to a similarity threshold can be selected as referenced knowledge; alternatively, the top F context knowledge items with the highest to lowest similarity can be selected as referenced knowledge, but this is not limited to these methods. The similarity parameter may include, but is not limited to, Euclidean distance, cosine similarity, etc.

[0041] In some embodiments, if the internal state of the large language model is not visible—that is, if the attention weight of the token in the context knowledge relative to the token in the initial answer cannot be obtained—the large language model can be guided by prompt words to output an initial answer with a reference knowledge identifier. The reference knowledge is then converted into a semantic vector, and the initial answer is also converted into a semantic vector. The reference knowledge is verified by the similarity parameter between the semantic vector converted from the reference knowledge and the semantic vector converted from the initial answer, determining whether the reference knowledge is genuine. For example, if the similarity parameter between the semantic vector converted from the reference knowledge and the semantic vector converted from the initial answer is higher than or equal to a similarity threshold, then the reference knowledge is genuine; if the similarity parameter is lower than the similarity threshold, then the reference knowledge cannot be used as the final reference knowledge.

[0042] In some embodiments, a target answer can be fed back to the user based on the permission level of the referenced knowledge and the user's permission level, implementing a hierarchical adaptation strategy. This ensures that the target answer includes the original text of the content corresponding to the referenced knowledge with a permission level lower than or equal to the user's permission level in the initial answer. If there is referenced knowledge with a permission level higher than the user's permission level, the target answer includes guidance prompts instead of a simple rejection, balancing information security and user experience. The initial answer may include at least one content fragment; if the initial answer includes one content fragment, the content fragment is the initial answer; if the initial answer includes two or more content fragments, the content fragments are answer fragments as described in the above embodiments, segmented semantically. Specifically, if the permission level represented by the permission level tag of the referenced knowledge corresponding to the content fragment is lower than or equal to the permission level represented by the permission level information, the target answer includes the content fragment; if the permission level represented by the permission level tag of the referenced knowledge corresponding to the content fragment is higher than the permission level represented by the permission level information, guidance prompts are generated based on the referenced knowledge and its permission level tag, and the target answer includes the guidance prompts. If the permission level tag of the cited knowledge corresponding to a content fragment represents a permission level lower than or equal to the permission level represented by the permission level information, it indicates that the content fragment is accessible to the user at their current permission level, and the target response provided to the user includes the original text of that content fragment. If the permission level tag of the cited knowledge corresponding to a content fragment represents a permission level higher than the permission level represented by the permission level information, it indicates that the content fragment is inaccessible to the user at their current permission level. In this case, the target response provided to the user includes guidance information. This guidance information informs the user of the existence of content related to their question that exceeds their permission level, and provides the operational path to access this content. The guidance information includes topic category information and operational prompts, which can be generated based on the semantic summary of the content fragment. The topic category information indicates the topic of the content fragment. The operational prompts indicate the operations required to upgrade to the permission level corresponding to the content fragment.

[0043] To facilitate understanding, the following example illustrates the intelligent method based on a large language model in this application embodiment. An unlogged-in user, i.e., a guest user, inputs the user question "How to join the merchant network platform?" via PC. The permission information related to the user question is obtained, and a permission vector U_perm={level=L1_Guest, role=Unknown, platform=PC, region=Unknown, timestamp=Current Time} can be constructed based on this information. Here, level represents the permission level, role represents the user role, platform represents the access channel type, region represents the user's location, and timestamp represents the request time. Based on the access channel type and request time in the permission vector, the knowledge base is filtered to obtain the intersection of knowledge that satisfies the channel permission tags for PC and the entire platform, and knowledge that matches the current time with a valid timestamp. It should be noted that the permission level information is not involved in the filtering. If there are three permission levels, L1, L2, and L3, even if the user's permission level is L1, the filtering still returns knowledge of L1, L2, and L3. The user question is then searched within the filtered knowledge, and the retrieved contextual knowledge includes knowledge P1, knowledge P2, and knowledge P3. Knowledge P1 states, "The merchant onboarding platform is UnionPay's merchant management platform...", with tags including permission level = L1, role permission = merchant, and channel permission = all platforms. Knowledge P2 states, "Operation path: Merchant onboarding platform → Direct connection merchant management → Add...", with tags including permission level = L2, role permission = merchant, and channel permission = PC. Knowledge P3 states, "Onboarding requires uploading a business license, the review period is 3 working days, and the fee rate...", with tags including permission level = L3, role permission = merchant, and channel permission = PC. The large language model generates a preliminary answer based on knowledge P1, knowledge P2, knowledge P3, and the user's question: "The Merchant Onboarding Platform is a management portal provided by UnionPay for merchants, supporting services such as merchant onboarding applications. The process is as follows: Log in to the Merchant Onboarding Platform, go to the 'Direct Merchant Management → Add' menu to initiate an application. A scanned copy of the business license is required during the application process. The review period is approximately 3 business days, and the onboarding fee rate is determined based on the merchant type." This preliminary answer can be semantically segmented into three answer fragments: Answer Fragment 1: "The Merchant Onboarding Platform is a management portal provided by UnionPay for merchants, supporting services such as merchant onboarding applications"; Answer Fragment 2: "The process is as follows: Log in to the Merchant Onboarding Platform, go to the 'Direct Merchant Management → Add' menu to initiate an application"; Answer Fragment 3: "A scanned copy of the business license is required during the application process. The review period is approximately 3 business days, and the onboarding fee rate is determined based on the merchant type."Using the knowledge tracing method based on a large language model in the intelligent question-answering method of this application, the knowledge referenced in answer fragment 1 can be determined as knowledge P1, the knowledge referenced in answer fragment 2 as knowledge P2, and the knowledge referenced in answer fragment 3 as knowledge P3. The permission level of the knowledge referenced in answer fragment 1 is L1, which is equal to the user's permission level L1. Therefore, the final target answer returned to the user can contain the original text of answer fragment 1. The permission level of the knowledge referenced in answer fragment 2 is L2, and the permission level of the knowledge referenced in answer fragment 3 is L3, both higher than the user's permission level. Based on answer fragment 2, a guiding prompt message "

After logging into the system

After logging into the system

After completing certificate verification

After completing certificate verification

[0044] The intelligent question answering method based on a large language model in this application embodiment can be implemented through a system architecture based on a large language model for intelligent question answering. Figure 2 A schematic diagram of the system architecture for intelligent question answering based on a large language model provided in the embodiments of this application is shown below. Figure 2 As shown, the system architecture of intelligent question answering based on a large language model may include a knowledge tag annotation layer 21, a system access layer 22, a user permission parsing layer 23, a multi-dimensional retrieval filtering layer 24, a large model generation and tracing layer 25, and a hierarchical adaptation output layer 26.

[0045] The knowledge tagging layer 21 can automatically or manually tag facts in the knowledge base with permission tags from multiple dimensions. Permission tags may include permission level tags, role permission tags, effective time tags, channel permission tags, and region permission tags.

[0046] The system access layer 22 is used for interaction with users. Users can interact with the system through personal computers, mobile devices, mini-programs, etc.

[0047] User permission parsing layer 23 can parse user permission information and construct user permission vectors. Permission information can be obtained through methods such as user login status detection, role resolution, terminal channel identification, and region resolution.

[0048] The multidimensional retrieval filtering layer 24 can perform pre-retrieval filtering based on the permission context information in the user's permission vector, and then retrieve contextual knowledge based on the semantic similarity between the user's question and the filtered knowledge. The multidimensional retrieval filtering layer 24 has semantic vector retrieval, time index filtering, role index filtering, channel index filtering, and region index filtering functions. The semantic vector retrieval function is used to retrieve knowledge based on the semantic similarity between the user's question and the filtered knowledge; the time index filtering function is used to filter knowledge based on the request time; the role index filtering function is used to filter knowledge based on the user's role; the channel index filtering function is used to filter knowledge based on the access channel type; and the region index filtering function is used to filter knowledge based on the user's location.

[0049] The large-scale model generation and tracing layer 25 can generate answers based on contextual knowledge obtained through filtering and retrieval. It also determines cited knowledge through attention weights, constructs prompts to guide the large-scale model in parsing and labeling cited knowledge, and determines cited knowledge through semantic similarity matching. The large-scale model generation and tracing layer 25 includes a large language model generation function and a knowledge security level tracing function. Specifically, the large language model generation function generates an initial answer while outputting attention weights. The knowledge security level tracing function is used to determine cited knowledge based on these attention weights and obtain the access level information of the cited knowledge.

[0050] The hierarchical adaptation output layer 26 can perform hierarchical adaptation of the content of the initial answer and generate guidance prompts based on the user's permission level and the permission level of the initial answer, thus obtaining the target answer. Specifically, if the user's permission level is higher than or equal to the permission level of the cited knowledge, the content fragments in the initial answer corresponding to the cited knowledge are retained and displayed; if the user's permission level is lower than the permission level of the cited knowledge, guidance prompts are generated.

[0051] This application also provides an intelligent question-answering device based on a large language model, which corresponds to the intelligent question-answering method based on a large language model in the above embodiments. Figure 3 A schematic diagram of the structure of an intelligent question-answering device based on a large language model provided in an embodiment of this application is shown below. Figure 3 As shown, the intelligent question-answering device 300 based on a large language model may include an authorization information acquisition module 301, a retrieval module 302, a first generation module 303, a reference knowledge determination module 304, and a second generation module 305.

[0052] The permission information acquisition module 301 can be used to receive user questions and obtain permission information related to the user questions. The permission information includes permission context information and permission level information.

[0053] The retrieval module 302 can be used to filter at least one piece of contextual knowledge from the knowledge base based on the permission context information and the user question. The contextual knowledge includes knowledge in the knowledge base that matches the permission context information and the user question. The knowledge in the knowledge base has permission level tags and permission context tags.

[0054] The first generation module 303 can be used to generate preliminary answers based on contextual knowledge and user questions using a large language model.

[0055] The reference knowledge determination module 304 can be used to determine the reference knowledge of the preliminary answer in the context knowledge based on the preliminary answer and the context knowledge.

[0056] The second generation module 305 can be used to generate a target answer based on the preliminary answer, permission level information, and permission level tags of the referenced knowledge, and then provide feedback to the user.

[0057] In some embodiments, the retrieval module 302 can be used to: generate retrieval filtering conditions based on permission context information; filter preliminary knowledge in the knowledge base that meets the retrieval filtering conditions based on permission context tags; and retrieve contextual knowledge that matches the user's question from the preliminary knowledge.

[0058] In some embodiments, the reference knowledge determination module 304 may be used to: obtain a first average attention weight for each piece of context knowledge in generating an initial answer in at least one knowledge set, wherein the knowledge set includes at least a portion of at least one piece of context knowledge; for each piece of context knowledge, perform aggregate averaging on the first average attention weight corresponding to the knowledge set to obtain the contribution of each piece of context knowledge to the initial answer; and determine the context knowledge as reference knowledge if the contribution of the context knowledge meets a preset reference condition.

[0059] In some examples, the reference knowledge determination module 304 can be used to: for each piece of context knowledge, obtain the second average attention weight of each attention head in each transformer layer that generates the initial answer for each token pair in the context knowledge; for each piece of context knowledge, select the K tokens with the highest second average attention weight, and determine the average of the second average attention weights of the K tokens as the first average attention weight, where K is a positive integer.

[0060] In some examples, the reference knowledge determination module 304 can also be used to: perform B random samplings from at least one piece of context knowledge, where each random sampling of M pieces of context knowledge forms a knowledge set, and B and M are both positive integers.

[0061] In some examples, the citation criteria include: the normalized contribution is higher than the contribution threshold.

[0062] In some examples, the reference knowledge determination module 304 can be used to: divide the initial answer into at least two answer segments according to semantics when the length of the initial answer is greater than a length threshold; and obtain the first average attention weight of each context knowledge segment for the answer segment in at least one knowledge set.

[0063] In some embodiments, the reference knowledge determination module 304 may be used to: guide the large language model to output a preliminary answer with an identifier of reference knowledge by constructing prompt words; and / or, determine reference knowledge based on a similarity parameter between the preliminary answer and contextual knowledge.

[0064] In some embodiments, the initial answer includes at least one content fragment. The second generation module 305 can be used to: when the permission level represented by the permission level tag of the referenced knowledge corresponding to the content fragment is lower than or equal to the permission level represented by the permission level information, the target answer includes the content fragment; when the permission level represented by the permission level tag of the referenced knowledge corresponding to the content fragment is higher than the permission level represented by the permission level information, based on the referenced knowledge and the permission level tag of the referenced knowledge, generate guidance prompt information, the target answer includes guidance prompt information, the guidance prompt information includes topic category information and operation prompt information, the topic category information is used to indicate the topic of the content fragment, and the operation prompt information is used to indicate the operation required to upgrade to the permission level corresponding to the content fragment.

[0065] In some embodiments, the permission context information includes one or more of the following: user role, request time, access channel type, and user's location. The permission context label includes one or more of the following: role permission label, effective time label, channel permission label, and region permission label.

[0066] It should be noted that the intelligent question-answering device 300 based on a large language model is a device corresponding to the intelligent question-answering method based on a large language model described above. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0067] This application also provides an intelligent question-answering device based on a large language model. Figure 4 A schematic diagram of the structure of an intelligent question-answering device based on a large language model provided in an embodiment of this application is shown below. Figure 4 As shown, the intelligent question-answering device 400 based on a large language model includes a memory 401, a processor 402, and a computer program stored on the memory 401 and capable of running on the processor 402.

[0068] In some examples, the processor 402 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that may be configured to implement the embodiments of this application.

[0069] Memory 401 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the intelligent question-answering method based on a large language model according to embodiments of this application.

[0070] The processor 402 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 401, so as to implement the intelligent question answering method based on a large language model in the above embodiments.

[0071] In some examples, the intelligent question-answering device 400 based on a large language model may also include a communication interface 403 and a bus 404. For example, Figure 4 As shown, the memory 401, processor 402, and communication interface 403 are connected through bus 404 and complete communication with each other.

[0072] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application. Input devices and / or output devices can also be connected through the communication interface 403.

[0073] Bus 404 includes hardware, software, or both, that couples the components of the large language model-based intelligent question-answering device 400 together. For example, and not limitingly, bus 404 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 404 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0074] This application also provides a computer-readable storage medium storing computer program instructions. When these computer program instructions are executed by a processor, they can implement the intelligent question-answering method based on a large language model as described in the above embodiments, and achieve the same technical effect. To avoid repetition, further details are omitted here. The aforementioned computer-readable storage medium may include non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc., and is not limited thereto.

[0075] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the intelligent question-answering method based on a large language model in the above embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0076] It should be clarified that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the device embodiments, equipment embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can be referred to the description section of the method embodiments. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0077] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0078] Those skilled in the art will understand that the above embodiments are exemplary and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, specification, and claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other means or steps; the quantifier "a" does not exclude a plurality; the terms "first" and "second" are used to identify names and not to indicate any particular order. No reference numerals in the claims should be construed as limiting the scope of protection. The functionality of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. An intelligent question-answering method based on a large language model, characterized in that, include: Receive user questions and obtain permission information related to the user questions, including permission context information and permission level information; Based on the permission context information and the user question, at least one piece of contextual knowledge is obtained by filtering in the knowledge base. The contextual knowledge includes knowledge in the knowledge base that matches the permission context information and the user question. The knowledge in the knowledge base has permission level tags and permission context tags. Based on the contextual knowledge and the user's question, a preliminary answer is generated using a large language model. Based on the preliminary answer and the contextual knowledge, determine the referenced knowledge of the preliminary answer in the contextual knowledge; Based on the preliminary answer, the permission level information, and the permission level tags of the referenced knowledge, a target answer is generated and provided to the user.

2. The method according to claim 1, characterized in that, The step of filtering at least one piece of contextual knowledge from the knowledge base based on the permission context information and the user question includes: Based on the permission context information, generate search filtering conditions; Based on permission context tags, preliminary knowledge that meets the search filtering conditions is filtered in the knowledge base; Retrieve the contextual knowledge that matches the user's question from the preliminary knowledge.

3. The method according to claim 1, characterized in that, The step of determining the referenced knowledge of the preliminary answer in the context knowledge based on the preliminary answer and the context knowledge includes: In at least one knowledge set, a first average attention weight is obtained for each of the contextual knowledge items in generating the preliminary answer, the knowledge set including at least a portion of the at least one contextual knowledge item; For each piece of contextual knowledge, the first average attention weight corresponding to the knowledge set is aggregated and averaged to obtain the contribution of each piece of contextual knowledge to the preliminary answer; If the contribution of the context knowledge meets the preset citation conditions, the context knowledge is determined as the citation knowledge.

4. The method according to claim 3, characterized in that, The step of obtaining the first average attention weight for each piece of contextual knowledge in generating the preliminary answer includes: For each piece of contextual knowledge, obtain the second average attention weight of each attention head in each converter layer that generates the initial answer for each token pair in the contextual knowledge; For each piece of contextual knowledge, select the K tokens with the highest second average attention weight, and determine the average of the second average attention weights of the K tokens as the first average attention weight, where K is a positive integer.

5. The method according to claim 3, characterized in that, Also includes: B random selections are made from the at least one piece of contextual knowledge, and each time M pieces of contextual knowledge are randomly selected to form a knowledge set, where B and M are both positive integers.

6. The method according to claim 3, characterized in that, The citation criteria include: the normalized contribution is higher than the contribution threshold.

7. The method according to claim 3, characterized in that, The step of obtaining the first average attention weight of each piece of contextual knowledge in at least one knowledge set for generating the preliminary answer includes: If the length of the preliminary response exceeds a length threshold, the preliminary response is divided into at least two response segments according to semantics. In at least one knowledge set, obtain the first average attention weight for each context knowledge pair of the answer fragment.

8. The method according to claim 1, characterized in that, The step of determining the referenced knowledge of the preliminary answer in the context knowledge based on the preliminary answer and the context knowledge includes: The constructed prompt words guide the large language model to output the initial answer with identifiers of referenced knowledge; And / or, The referenced knowledge is determined based on the similarity parameter between the preliminary answer and the contextual knowledge.

9. The method according to claim 1, characterized in that, The preliminary response includes at least one content segment; The step of generating a target answer based on the preliminary answer, the permission level information, and the permission level tags of the referenced knowledge includes: If the permission level represented by the permission level tag of the referenced knowledge corresponding to the content fragment is lower than or equal to the permission level represented by the permission level information, the target answer includes the content fragment; When the permission level represented by the permission level tag of the referenced knowledge corresponding to the content fragment is higher than the permission level represented by the permission level information, guidance prompts are generated based on the referenced knowledge and the permission level tag of the referenced knowledge. The target answer includes the guidance prompts, which include topic category information and operation prompts. The topic category information is used to indicate the topic of the content fragment, and the operation prompts are used to indicate the operation required to upgrade to the permission level corresponding to the content fragment.

10. The method according to claim 1, characterized in that, The permission context information includes one or more of the following: user role, request time, access channel type, and user's location. Permission context labels include one or more of the following: role permission label, effective time label, channel permission label, and region permission label.

11. An intelligent question-answering device based on a large language model, characterized in that, include: The permission information acquisition module is used to receive user questions and acquire permission information related to the user questions. The permission information includes permission context information and permission level information. The retrieval module is used to filter at least one piece of contextual knowledge from the knowledge base based on the permission context information and the user question. The contextual knowledge includes knowledge in the knowledge base that matches the permission context information and the user question. The knowledge in the knowledge base has permission level tags and permission context tags. The first generation module is used to generate a preliminary answer based on the contextual knowledge and the user question using a large language model; The reference knowledge determination module is used to determine the reference knowledge of the preliminary answer in the context knowledge based on the preliminary answer and the context knowledge. The second generation module is used to generate a target answer based on the preliminary answer, the permission level information, and the permission level tag of the referenced knowledge, and then provide feedback to the user.

12. An intelligent question-answering device based on a large language model, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the intelligent question-answering method based on a large language model as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the intelligent question-answering method based on a large language model as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the intelligent question-answering method based on a large language model as described in any one of claims 1 to 10.

Images