Method, device, equipment, medium and product for knowledge question answering

Abstract

Disclosed are methods, devices, equipment, media and products for knowledge question answering. The method includes determining the question intent and classification, question entities, and business lines related to the question by performing question understanding on user questions by a language model, and the user is identified based on the organizational structure and question and answer history. The method also includes determining the target business line based on the determined business line, organizational structure and question and answer history. The method also includes selecting knowledge to generate answers based on the question intent and classification and question entities among the target knowledge corresponding to the target business line in the knowledge base, and the knowledge base includes multiple sub-knowledge bases based on the knowledge type. The method also includes interactively providing the answer to the user in response to the answer meeting the accuracy threshold. In this way, the question itself can be analyzed, and the business knowledge highly related to the specific scenario can be customized and retrieved from the knowledge base managed differently in combination with user characteristics.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of computers, and in particular, relate to methods and apparatuses, devices, media, and products for knowledge question answering. Background Art

[0002] In the enterprise environment, the application of chatbots improves the response speed and efficiency of services. It can respond quickly when users need it, thereby improving the interactive experience. The various services provided by chatbots cover most common problems faced in the business, while reducing the manual burden and promoting the automation of solution formulation and improvement.

[0003] Chatbots provide new possibilities for business operations. Providing users with chatbot-based interactive functionality, such as responding to inquiries and providing business guidance without traditional time constraints, will save costs and improve efficiency for enterprises. Summary of the invention

[0004] The embodiments of the present disclosure provide a solution for knowledge question answering.

[0005] In a first aspect of the present disclosure, a method for knowledge question answering is provided, the method comprising determining, by a language model, the intent and classification of a question, entities in the question, and at least one business line associated with the question by performing question understanding on a user's question, wherein the user is identified based on an organizational structure and a question-answering history. The method also comprises determining, based on at least one business line associated with the question, and the organizational structure and question-answering history of the user, a target business line to be searched in a business knowledge base for the question. The method also comprises, among target business knowledge corresponding to the target business line in the business knowledge base, selecting business knowledge items based on the intent and classification of the question and entities in the question by a language model to generate an answer to the question, the business knowledge base comprising a plurality of sub-knowledge bases based on business knowledge types, the plurality of sub-knowledge bases comprising a first sub-knowledge base corresponding to structured business knowledge, a second sub-knowledge base corresponding to unstructured business knowledge, and a third sub-knowledge base corresponding to semi-structured business knowledge. The method also comprises providing the answer to the user in an interactive manner in response to the accuracy of the generated answer satisfying an accuracy threshold.

[0006] In a second aspect of the present disclosure, a device for knowledge question answering is provided, the device comprising a question understanding module, configured to determine the intent and classification of the question, the entity in the question, and at least one business line associated with the question by performing question understanding on the user's question using a language model, and the user is identified based on the organizational structure and the question and answer history. The device also includes a business line determination module, configured to determine the target business line to be searched in the business knowledge base for the question based on at least one business line associated with the question, the organizational structure of the user, and the question and answer history. The device also includes an answer acquisition module, configured to select business knowledge items from the target business knowledge corresponding to the target business line in the business knowledge base based on the intent and classification of the question and the entity in the question by the language model to generate an answer for the question, the business knowledge base includes a plurality of sub-knowledge bases based on the business knowledge type, the plurality of sub-knowledge bases include a first sub-knowledge base corresponding to structured business knowledge, a second sub-knowledge base corresponding to unstructured business knowledge, and a third sub-knowledge base corresponding to semi-structured business knowledge. The device also includes an answer providing module, configured to interactively provide the answer to the user in response to the accuracy of the generated answer satisfying the accuracy threshold.

[0007] According to a third aspect of the present disclosure, an electronic device is provided. The computing device includes a processor and a memory, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor executes a method or process according to an embodiment of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a machine-readable storage medium is provided, wherein machine-executable instructions are stored on the machine-readable storage medium, and when the machine-executable instructions are executed by a processor, the processor is enabled to execute a method or process according to an embodiment of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable storage medium and includes a computer program, which, when executed by a processor of a computer, causes the processor to perform a method or process according to an embodiment of the present disclosure.

[0010] Please note that the summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the present disclosure will become more clearly understood by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a diagram schematically illustrating an example environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0013] Figure 2 is a flowchart schematically illustrating a method for knowledge question answering according to an embodiment of the present disclosure;

[0014] Figure 3 is a diagram schematically illustrating a knowledge question answering architecture according to an embodiment of the present disclosure;

[0015] Figure 4 is a diagram schematically illustrating an arrangement of a knowledge base according to an embodiment of the present disclosure;

[0016] Figure 5 is a diagram schematically illustrating a progressive selection process according to an embodiment of the present disclosure;

[0017] Figure 6 is a diagram schematically illustrating a device for knowledge question answering according to an embodiment of the present disclosure;

[0018] Figure 7 is a schematic block diagram of an example device that may be used to implement embodiments according to the present disclosure.

[0019] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly indicated to be different.

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] As mentioned above, chatbots provide an almost seamless interactive experience for business operations in an enterprise environment. However, relevant chatbots only parse and understand questions by identifying keywords in the questions. For example, when faced with complex or ambiguous queries, due to their weak recognition ability, they fail to accurately understand the questions, often resulting in irrelevant answers.

[0027] In related solutions, chatbots provide feedback based on pre-configured specific questions and corresponding responses (e.g., question-answer pairs), which has certain limitations. For example, chatbots’ feedback is often mechanical and inflexible. Although they can answer questions to a certain extent, they usually only partially solve the problem, and it is difficult to propose practical and accurate solutions based on a deep and complete understanding of the user’s true intentions. In addition, chatbots have difficulty understanding data assets because data assets usually contain a large amount of information with complex and diverse structures and potential connections. It is difficult for general chatbots to accurately parse and effectively process these differently structured and intertwined information, thereby ignoring valuable clues, insights, connections, details, etc.

[0028] In an enterprise environment, each business line relies on its own unique knowledge system and professional skills. These knowledge systems are both interrelated and significantly different. For example, the business knowledge used by the video business line is obviously different from that used by the financial business line. If these differences are ignored and a one-size-fits-all management approach is adopted, the knowledge management of different business lines will not only weaken the pertinence and effectiveness of the management strategy, but also lead to the ambiguity and deviation of the focus of knowledge management.

[0029] In view of the above, chatbots may have limited knowledge reserves available and it is difficult to fully utilize this knowledge, which may result in the solutions provided being incomplete, lacking accuracy and comprehensiveness, and failing to meet user needs. In cases where the answers are biased and cannot fully solve the questions asked, or even when the answers are completely unavailable, manual intervention may be initiated. However, even if the manual answers meet user expectations, since these contents are usually scattered across multiple platforms such as emails and chat tools and in different formats, the arbitrary summarization methods (for example, manual recording and organization) may lead to information omissions and confusion, making it difficult to effectively use key content to supplement the knowledge base, making upgrading the knowledge base inefficient.

[0030] At least in order to solve at least some of the above and other potential problems, an embodiment of the present disclosure proposes a solution for knowledge question answering. The solution includes determining the intent and classification of the question, the entity in the question, and at least one business line associated with the question by a language model through performing question understanding on the user's question, and the user is identified based on the organizational structure and the question and answer history. The solution also includes determining the target business line to be searched in the business knowledge base for the question based on at least one business line associated with the question, and the organizational structure and question and answer history of the user. The solution also includes selecting business knowledge items based on the intent and classification of the question and the entity in the question among the target business knowledge corresponding to the target business line in the business knowledge base, and the business knowledge base includes multiple sub-knowledge bases based on the business knowledge type, and the multiple sub-knowledge bases include a first sub-knowledge base corresponding to structured business knowledge, a second sub-knowledge base corresponding to unstructured business knowledge, and a third sub-knowledge base corresponding to semi-structured business knowledge. The solution also includes providing the answer to the user in an interactive manner in response to the accuracy of the generated answer meeting the accuracy threshold. In this way, not only the problem itself is analyzed, but also business knowledge that is highly relevant to the user's specific business scenario is retrieved from the business knowledge base in a customized manner based on user characteristics. The business knowledge base performs refined and differentiated management based on business knowledge types, making the organization of knowledge clearer and the classification more specific, thereby ensuring the accuracy and pertinence of the retrieved results.

[0031] Reference below Figures 1 to 7 It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0032] Figure 11 is a schematic diagram schematically illustrating an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1 As shown in FIG. 1 , the computing system 100 includes a user 101, a cloud computing environment 110, and computing nodes 121, 122, and 123. Figure 1 The implementation environment of the embodiments of the present disclosure will be described using a distributed computing system as an example. It should be understood that this is only an exemplary and non-restrictive description, and other different types of computing systems are also feasible, such as a centralized computing system. A suitable computing environment configuration can be selected according to actual use needs.

[0033] The cloud computing environment 110 may indicate a computing and service model built based on cloud or distributed technology, coupling computing resources, storage resources, and applications through a network to form a highly adaptable and collaborative computing environment. The cloud computing environment 110 is not just a simple collection of computing resources, but is highly integrated, intelligently scheduled, and flexibly expandable. In such an environment, key elements such as computing resources, storage resources, and applications are tightly coupled, and users 101 can dynamically access or be allocated at least a portion of these resources as needed to complete corresponding computing without having to locally own or manage these physical resources.

[0034] The cloud computing environment 110 may include one or more computing nodes, or be coupled with one or more computing nodes. A computing node may be a device with computing capabilities. For example, a computing node may be provided with a processor and a memory, etc., or may be equipped with a dedicated accelerator. Accelerators such as a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. can significantly accelerate specific types of computing tasks. The various types of models involved in executing the method for knowledge question answering according to an embodiment of the present disclosure can be deployed in a distributed manner on each node.

[0035] like Figure 1 As shown in the example, the cloud computing environment 110 is coupled with computing nodes 121, 122, and 123, and the user 101 can use the computing capabilities of these computing nodes via the cloud computing environment 110. Multiple nodes in the cloud computing environment 110 can be interconnected with each other through a network for communication. Figure 1 For example, computing nodes 121, 122, and 123 can communicate via a network to achieve, for example, data synchronization or sharing between nodes. Multiple computing nodes in the cloud computing environment 110 can process computing tasks in parallel. Distributed computing is used to accelerate the computing process, and such parallel computing capabilities enable the cloud computing platform to handle more complex and large computing tasks.

[0036] In addition, multiple computing nodes in the cloud computing environment 110 can store and manage data. Computing nodes are usually equipped with high-performance storage devices for storing and managing data required for computing tasks. These data can be input data, intermediate computing results, or output data. By storing data on computing nodes, data access speed and efficiency can be improved. Multiple computing nodes in the cloud computing environment 110 can have redundancy and fault tolerance mechanisms to ensure reliable execution of computing tasks. When a node fails, the system can automatically migrate the job to other nodes that are working normally to ensure the continuity and availability of the task.

[0037] Examples of computing nodes of devices with computing capabilities may include supercomputers, personal computers, laptop computers, vehicle-mounted computing devices, mobile devices (such as smart phones, tablet computers, etc.), wearable electronic devices, multimedia devices, personal digital assistants (PDAs), smart home devices, consumer electronic products, or a combination of any one or more of the above devices, etc. It should be understood that the computing nodes described herein are only exemplary and non-restrictive, and for example, other different types of computing nodes may also be used.

[0038] It should be understood that for ease of explanation and illustration purposes, Figure 1 Only one user (ie, user 101) is shown in FIG. It should be understood that multiple users (enterprises or individuals) can access the cloud computing environment 110 at the same time to achieve parallel computing. Figure 1 Only one cloud computing environment (i.e., cloud computing environment 110) and three computing nodes (i.e., computing nodes 121, 122, and 123) are shown. However, this is merely exemplary, and there may be other cloud computing environments and computing nodes with different configurations and numbers, which may be customized according to specific usage scenarios and needs, as is also the case below.

[0039] Combined with the above Figure 1 Describes an example environment in which the methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 The method 200 for knowledge question answering according to an embodiment of the present disclosure is described. Through the method 200, the question itself is analyzed and the business knowledge highly relevant to the specific scenario is retrieved from the knowledge base managed differently in combination with user characteristics.

[0040] Figure 2It is a flowchart schematically illustrating a method 200 for knowledge question answering according to an embodiment of the present disclosure. At 210, the language model determines the intent and classification of the question, the entities in the question, and at least one business line associated with the question by performing question understanding on the user's question, and the user is identified based on the organizational structure and question and answer history. In some embodiments, the language model may include a large language model (LLM). LLM has powerful language understanding and contextual reasoning capabilities. In-depth parsing and intent recognition of user questions based on LLM can improve the accuracy of understanding. In addition, one or more business lines involved in the question are identified via business line discrimination to build an answer framework that is highly relevant to the user's specific business scenario.

[0041] At 220, the target business line to be searched in the business knowledge base for the question is determined based on at least one business line associated with the question, and the user's organizational structure and question-and-answer history. In addition to the (multiple) business lines that the question may involve, the characteristics or features of the user who asked the question are also considered, including but not limited to the organizational structure and question-and-answer history. In some embodiments, the organizational structure may include the user's position in the enterprise or organization, the department to which he belongs, the reporting line, and the relationship with other team members, and the question-and-answer history records the user's past interactions with the system or other resources within the enterprise, which may include query history, browsing behavior, feedback information, etc. It should be noted that the use of these user features requires the effective permission of the user himself.

[0042] According to the embodiments of the present disclosure, the target business line to be searched in the business knowledge base for the question can be accurately determined based not only on the one or more associated business lines identified from the question, but also on the specific business line potentially indicated by the user himself (i.e., the search direction and scope based on the business line are determined). In this way, this dual-dimensional consideration method not only improves the accuracy of the search, but also ensures the high relevance of the search results, and can directly respond to the actual needs of users.

[0043] At 230, among the target business knowledge corresponding to the target business line in the business knowledge base, business knowledge items are selected by a language model based on the intent and classification of the question and the entities in the question to generate an answer to the question, and the business knowledge base includes multiple sub-knowledge bases based on business knowledge types, and the multiple sub-knowledge bases include a first sub-knowledge base corresponding to structured business knowledge, a second sub-knowledge base corresponding to unstructured business knowledge, and a third sub-knowledge base corresponding to semi-structured business knowledge.

[0044] According to the embodiments of the present disclosure, the refined differentiated management of the business knowledge base enables the type of business knowledge and the structuring of data to be considered comprehensively to manage different business knowledge in different libraries. For example, help-type business knowledge and enumeration value-type business knowledge can be organized and stored differently. Similarly, structured business knowledge and unstructured knowledge are managed based on different processing mechanisms and indexing strategies. Such a management strategy enables different business knowledge to be captured and maintained according to their respective specifications and standards, which can improve the quality and consistency of data.

[0045] At 240, in response to the accuracy of the generated answer satisfying the accuracy threshold, the answer is provided to the user in an interactive manner. Before providing the answer, the quality of the answer can be checked for accuracy, thereby ensuring that the answer obtained by the user is of high quality, reducing the transmission of erroneous information, and improving the user experience. In addition, the user can interact with the provided answer, such as further asking questions, clarifying or supplementing questions, etc. to obtain improved answers. The user is no longer passively accepting the answer, but actively participating in the problem-solving process. In addition, in order to improve the efficiency of the user's answer retrieval, the fields related to the number of questions will be highlighted and recommended, and the highly relevant document blocks in the document will be highlighted.

[0046] According to the method 200 for knowledge question and answering of an embodiment of the present disclosure, not only the question itself is analyzed, but also business knowledge that is highly relevant to the user's specific business scenario is retrieved from the business knowledge base in a customized manner in combination with user characteristics, wherein the business knowledge base performs refined differentiated management based on business knowledge types, making the organization of knowledge clearer and the classification more explicit, thereby ensuring the accuracy and pertinence of the retrieved results.

[0047] Figure 3 It is a diagram schematically illustrating a knowledge question answering architecture 300 according to an embodiment of the present disclosure. It should be understood that the description of the knowledge question answering architecture 300 is to facilitate the understanding of the embodiment of the present disclosure, and is not intended to limit the scope of the embodiment of the present disclosure. Other different modifications or variations are obvious to those skilled in the art.

[0048] like Figure 3 As shown in FIG. 3 , a user 301 identified by a plurality of user characteristics 310 may initiate a consultation 320, wherein the plurality of user characteristics ( Figure 3The user features 1-3 schematically shown in the figure may include an organizational structure and a question and answer history, etc. In some embodiments, question recommendation / query association 330 may be performed based on the user's question to further clarify the user's needs. Question understanding 340 is performed on the user's question using, for example, LLM to determine the intent and classification of the question, the entities in the question, and at least one business line associated with the question, etc. In some embodiments, question understanding 340 may include at least one of the following: intent recognition, question classification, query enhancement, entity extraction, business line discrimination, term clarification, pattern recognition, sub-direction recognition, and question rewriting.

[0049] According to an embodiment of the present disclosure, in response to the intent of the question not satisfying the intent clarity threshold, the question can be clarified by facilitating the addition of context to the question. Question ambiguity, lack of context, etc. can be checked based on natural language processing (NLP) technology to determine whether the intent of the question is clear. In some embodiments, question clarification may include at least one of the following: short query / clarification, input source clarification, granularity / scope / table type clarification, indicator clarification, business line error correction, and other clarifications. Question understanding can be re-executed based on the clarified question.

[0050] According to an embodiment of the present disclosure, based on the business line information parsed from the question and combined with the business line potentially indicated by the user himself, the dialogue management 340 can retrieve business knowledge from the business knowledge base 370 in a customized manner, and generate targeted and accurate answers through various interfaces and tools, etc. by relying on the model capabilities in the knowledge question-answering architecture 300. The customized retrieval and answer generation according to an embodiment of the present disclosure are further described in detail below.

[0051] Among the target business knowledge corresponding to the determined target business line in the business knowledge base 370, business knowledge items can be searched 355 based on the analysis of the user's problem. For example, among the target business knowledge corresponding to the target business line, business knowledge items are searched 355 based on the fields, indicators, etc. associated with the problem in the analysis results. In some embodiments, at least one of the multiple searches 360 can be used depending on the result of the problem analysis, including: field search, indicator search, sub-direction search, table alias search, enumeration value search, and other searches. Business knowledge items can be unit knowledge in the business knowledge base 370, such as document blocks of knowledge documents, (multiple) columns and rows of tables, etc. The business knowledge base 370 can be finely differentiated and managed into multiple sub-knowledge bases, and the knowledge base organization and management according to the embodiments of the present disclosure will be further described in detail below.

[0052] As described above, the key information characterizing the problem, such as the intent and classification of the problem, the entities in the problem, and at least one business line associated with the problem, has been determined through user understanding 340. In some embodiments, the entity of the problem may include at least one of the following: keyword entity, general knowledge entity, help entity, query entity, large model scenario entity, and enumeration value entity. In an exemplary and non-restrictive manner, the business knowledge associated with precise number finding / keyword can be searched comprehensively via an application program interface (API) / tool ​​385, the business knowledge associated with general knowledge / chat can be processed via a large language model (LLM) 381, the help business knowledge can be obtained through a help query 382, ​​the business knowledge associated with query / number finding can be searched comprehensively via an API / tool ​​386, the business knowledge associated with the large model scenario can be represented and processed by a prompt / agent 383, and the business knowledge associated with the enumeration value in a specified range can be obtained through an enumeration value query 384, and so on. By further processing the searched target business knowledge (eg, understanding reasoning, organizing and sorting, filtering and removing redundancy, etc.), the answer generation 390 can be performed.

[0053] The answers generated by various interfaces and tools, etc. by means of the model capabilities in the knowledge question answering architecture 300 can be used to update the business knowledge base 370. According to an embodiment of the present disclosure, in response to the accuracy of the generated answer meeting the accuracy threshold, the corresponding sub-knowledge base in the business knowledge base 370 is updated based on the user's question and at least part of the generated answer. The question semantic similarity, contextual adaptability, etc. can be checked based on natural language processing (NLP) technology to determine whether the answer is adapted to the input question. After determining that the output answer meets the accuracy requirement, part or all of the input question and the output answer can be used. In an exemplary and non-restrictive manner, a part of the question is used as a column name or row name of a table, and the corresponding part of the answer is used as a row or column of a table, and both can be used to update the sub-knowledge base corresponding to the structured content. In addition, another part of the question and the answer may be a document title and a document block, which can be used to update the sub-knowledge base corresponding to the semi-structured content.

[0054] According to an embodiment of the present disclosure, in response to the accuracy of the generated answer not meeting the accuracy threshold, an artificial answer to the question can be obtained by initiating a manual response. In the case of a complex question, for example, the answer generated by the model may still not solve the problem, in which case manual intervention or manual consultation 387 can be initiated. Based on the accuracy of the artificial answer obtained and the score from the user for the artificial answer, it can be determined whether the artificial answer obtained solves the problem. After obtaining the artificial answer, in addition to the accuracy assessment given to the artificial answer by the knowledge question and answer architecture 300, the artificial answer can be scored by the user 301, and the combination of the two can accurately determine whether the artificial answer solves the problem. Answers that are allowed by the architecture and allowed by the user can be cached and stored for updating the business knowledge base 370.

[0055] The type of the artificial answer may be determined. For example, in response to determining that the artificial answer is unstructured business knowledge, data preprocessing (e.g., denoising, missing value and outlier processing, etc.) may be performed on the artificial answer, and the preprocessed artificial answer may be deposited 388 back into the business knowledge base 370. The preprocessed artificial answer may be used to update the unstructured business knowledge (also referred to as second knowledge) in the second sub-knowledge base in the business knowledge base 370. The processing of other types of artificial answers is similar.

[0056] According to an embodiment of the present disclosure, in response to the generated answer not meeting the answer clarity threshold, the question can be updated by clarifying the attributes of the target answer that are different from the attributes of the answer. For example, daily data for a certain year and month is required, but data for a certain year and month is returned, and the answer granularity does not meet expectations. At this point, the attributes of the target answer can be clarified to further update the question. In some embodiments, the answer clarification may include at least one of the following: scope clarification, table type clarification, field clarification, granularity clarification, time clarification, and asking him to clarify. The updated question including the attributes of the target answer will be used to initiate consultation 320 again.

[0057] Figure 4 3 is a diagram schematically illustrating the arrangement of a knowledge base according to an embodiment of the present disclosure. The business knowledge base 370 includes multiple sub-knowledge bases based on business knowledge type or business data type. For example, manual answers are mostly unstructured business knowledge such as dialogues, and the second sub-knowledge base 420 (a sub-knowledge base corresponding to unstructured business knowledge) can store and maintain such unstructured business knowledge.

[0058] like Figure 4As shown in , the first sub-knowledge base 410 (the sub-knowledge base corresponding to the structured knowledge) can store and maintain structured knowledge, such as hive tables, data sets, dashboards, indicator dimensions, etc. According to an embodiment of the present disclosure, structured extraction can be performed on the business knowledge determined as structured data to obtain one or more sub-domains of these structured business knowledge (i.e., the first business knowledge). A mapping between each of these sub-domains and the corresponding structured business knowledge can be further established. Examples of sub-domains may include fields, business lines, etc.

[0059] The second knowledge base 420 can store and maintain unstructured knowledge, such as oncall dialogues, chat interactions, forwarding robot corpora, etc. According to an embodiment of the present disclosure, data preprocessing and unstructured extraction can be performed on business knowledge determined as unstructured data, so that part of it may be converted into structured knowledge and the other part into semi-structured knowledge, such as business line (BP) documents.

[0060] The third knowledge base 430 can store and maintain semi-structured knowledge, such as BP documents and help documents. According to the embodiment of the present disclosure, data preprocessing can be performed on the business knowledge determined as semi-structured data, and then the preprocessed semi-structured business knowledge can be extracted and segmented for subsequent use (e.g., organizing answers, data analysis, etc.). The data in each sub-knowledge base will be indexed for easy identification.

[0061] Figure 5 Schematically illustrates a progressive selection process 500 according to an embodiment of the present disclosure. Figure 5 As shown in , in the process of selecting business knowledge items to organize answers, in addition to the business knowledge in the business knowledge base 370, the knowledge query API 502 and various models 504 (such as small model / large model LLM) in the question-answering architecture 300 can also be used. In addition, external knowledge can also be introduced through external APIs / tools for supplementation.

[0062] According to an embodiment of the present disclosure, at 510, a first number of business knowledge items can be recalled in multiple ways based on the entity in the question. At 520, a semantic model selects a second number of business knowledge items from the first number of business knowledge items based on heat and cumulative quantile regression, and the second number is less than the first number. In this way, after rough sorting, most of the irrelevant or low-quality items are quickly filtered out from a large number of business knowledge items, and the range of the candidate set is narrowed. At 530, a cross encoder selects a third number of business knowledge items from the second number of business knowledge items based on heat and click-through rate, and the third number is less than the second number. In this way, on the basis of rough sorting, further refined screening is performed to ensure that the knowledge items presented in the end are not only relevant, but also of high quality. In some embodiments, mixed sorting can be performed to mix multiple sorting strategies to achieve a better overall effect. In addition, at 540, a large model can select a fourth number of business knowledge items from the third number of business knowledge items based on the question and answer history, and the fourth number is less than the third number. In this way, the final business knowledge item selection is optimized by introducing a large model and considering the question and answer history data.

[0063] Figure 6 Schematically illustrates a device 600 for knowledge question answering according to an embodiment of the present disclosure. The device 600 may include multiple units or modules for performing the steps or actions in the method or process discussed above. Figure 6 As shown in , the device 600 includes a question understanding module 610, which is configured to determine the intention and classification of the question, the entities in the question, and at least one business line associated with the question by performing question understanding on the user's question using a language model, and the user is identified based on the organizational structure and the question and answer history. The device 600 also includes a business line determination module 620, which is configured to determine the target business line to be searched in the business knowledge base for the question based on the at least one business line associated with the question, the organizational structure of the user, and the question and answer history. The device 600 also includes an answer acquisition module 630, which is configured to select business knowledge items from the target business knowledge corresponding to the target business line in the business knowledge base based on the intention and classification of the question and the entities in the question by the language model to generate an answer to the question, and the business knowledge base includes multiple sub-knowledge bases based on business knowledge types, and the multiple sub-knowledge bases include a first sub-knowledge base corresponding to structured business knowledge, a second sub-knowledge base corresponding to unstructured business knowledge, and a third sub-knowledge base corresponding to semi-structured business knowledge. The apparatus 600 further includes an answer providing module 640 configured to provide the generated answer to the user in an interactive manner in response to the accuracy of the generated answer satisfying an accuracy threshold.

[0064] In some embodiments, the device 600 may further include a knowledge base updating module configured to: in response to the accuracy of the generated answer satisfying the accuracy threshold, update a corresponding sub-knowledge base in the business knowledge base based on the user's question and at least part of the generated answer.

[0065] In some embodiments, the first business knowledge included in the first sub-knowledge base can be obtained by: performing structured extraction on business knowledge determined as structured data to obtain at least one sub-domain of structured business knowledge; and establishing a mapping between each sub-domain in the at least one sub-domain and the corresponding structured business knowledge.

[0066] In some embodiments, the device 600 may also include: a manual response module, configured to obtain a manual answer to the question by initiating a manual response in response to the accuracy of the generated answer not satisfying the accuracy threshold; an accuracy judgment module, configured to determine whether the obtained manual answer solves the problem based on the accuracy of the obtained manual answer and the score of the manual answer from the user; and a storage module, configured to store the obtained manual answer in response to the obtained manual answer solving the problem.

[0067] In some embodiments, the stored artificial answers can also be used to update the corresponding sub-knowledge base in the business knowledge base. The device 600 also includes: a data type judgment module, configured to determine that the artificial answers belong to unstructured business knowledge; a preprocessing module, configured to obtain the preprocessed artificial answers by performing data preprocessing on the artificial answers; and a knowledge base update module, configured to use the preprocessed artificial answers to update the second business in the second sub-knowledge base in the business knowledge base.

[0068] In some embodiments, the third business knowledge included in the third sub-knowledge base may include business line documents and help documents, and the device 600 also includes: a preprocessing module, configured to obtain the preprocessed business line documents and the help documents by performing data preprocessing on the business line documents and the help documents; and a blocking module, configured to obtain document blocks for the business line documents and the help documents by blocking the preprocessed business line documents and the help documents.

[0069] In some embodiments, the entity in the question may include at least one of the following: a keyword entity, a general knowledge entity, a help entity, a query entity, a large model scene entity, and an enumeration value entity.

[0070] In some embodiments, the device 600 also includes a knowledge selection module, which is configured to multi-way recall a first number of business knowledge items based on the entity in the question; select a second number of business knowledge items from the first number of business knowledge items based on popularity and cumulative quantile regression by a semantic model, and the second number is smaller than the first number; select a third number of business knowledge items from the second number of business knowledge items based on popularity and click-through rate by a cross encoder, and the third number is smaller than the second number; and select a fourth number of business knowledge items from the third number of business knowledge items based on question and answer history by a large model, and the fourth number is smaller than the third number.

[0071] In some embodiments, the device 600 also includes a question clarity judgment module, which is configured to clarify the question by facilitating the addition of context to the question in response to the intent of the question not satisfying the intent clarity threshold; and a question understanding module 610, which is configured to re-execute question understanding based on the clarified question.

[0072] In some embodiments, the device 600 also includes an answer clarity judgment module, which is configured to update the question by clarifying the attributes of the target answer that are different from the attributes of the answer in response to the generated answer not satisfying the answer clarity threshold; and a question understanding module 610, which is configured to re-execute question understanding based on the updated question.

[0073] Figure 7 FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a coprocessor.

[0074] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0075] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0076] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0077] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0078] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0079] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0080] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0082] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] Various embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the various embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

1. A method for knowledge question answering, comprising: The language model determines the intent and classification of the question, entities in the question, and at least one business line associated with the question by performing question understanding on the question of the user, wherein the user is identified based on the organizational structure and the question and answer history; Based on the at least one business line associated with the question, the organizational structure of the user, and the question-and-answer history, determining a target business line to be searched in a business knowledge base for the question, wherein the target business line is associated with both the question and the user and indicates a direction and scope to be searched in the business knowledge base; Among the target business knowledge corresponding to the target business line in the business knowledge base, the language model selects business knowledge items based on the intent and the classification of the question and the entities in the question to generate an answer to the question, the business knowledge base includes a plurality of sub-knowledge bases based on business knowledge types, the plurality of sub-knowledge bases include a first sub-knowledge base corresponding to structured business knowledge, a second sub-knowledge base corresponding to unstructured business knowledge, and a third sub-knowledge base corresponding to semi-structured business knowledge; as well as In response to the accuracy of the generated answer satisfying an accuracy threshold, providing the generated answer to the user in an interactive manner, After the generated answer is determined to be adapted to the question in terms of semantic similarity and contextual adaptation, the question and a portion of the answer are used to update a corresponding sub-knowledge base in the business knowledge base.

2. The method according to claim 1, wherein the first business knowledge included in the first sub-knowledge base is obtained by: Acquire at least one sub-domain of structured business knowledge by performing structured extraction on the business knowledge determined as structured data; and A mapping is established between each sub-domain in the at least one sub-domain and corresponding structured business knowledge.

3. The method according to claim 1, further comprising: In response to the accuracy of the generated answer not satisfying the accuracy threshold, obtaining a manual answer to the question by initiating a manual response; Determining whether the obtained artificial answer solves the problem based on the accuracy of the obtained artificial answer and the score of the artificial answer from the user; as well as In response to the obtained artificial answer solving the problem, the obtained artificial answer is stored.

4. The method according to claim 3, wherein the stored manual answers are also used to update the corresponding sub-knowledge base in the business knowledge base, comprising: Determining that the artificial answer belongs to unstructured business knowledge; Obtaining a preprocessed artificial answer by performing data preprocessing on the artificial answer; as well as The pre-processed artificial answer is used to update the second business in the second sub-knowledge base in the business knowledge base.

5. The method according to claim 1, wherein the third business knowledge included in the third sub-knowledge base includes business line documents and help documents, and the method further comprises: Obtaining the preprocessed business line document and the help document by performing data preprocessing on the business line document and the help document; as well as Document blocks for the business line document and the help document are obtained by dividing the pre-processed business line document and the help document into blocks.

6. The method of claim 1, wherein the entity in the question comprises at least one of the following: Keyword entities, general entities, help entities, query entities, large model scene entities, and enumeration value entities.

7. The method of claim 6, wherein the selecting comprises: Recalling a first number of business knowledge items in multiple ways based on the entity in the question; Selecting, by a semantic model, a second number of business knowledge items from the first number of business knowledge items based on popularity and cumulative quantile regression, wherein the second number is smaller than the first number; Selecting, by a cross encoder, a third number of business knowledge items from the second number of business knowledge items based on popularity and click-through rate, wherein the third number is smaller than the second number; as well as A fourth number of business knowledge items is selected by the large model from the third number of business knowledge items based on the question and answer history, the fourth number being smaller than the third number.

8. The method according to claim 1, further comprising: In response to the intent of the question not satisfying an intent clarity threshold, clarifying the question by facilitating adding context to the question; as well as Problem understanding is re-performed based on the clarified problem.

9. The method according to claim 1, further comprising: In response to the generated answer not satisfying an answer clarity threshold, updating the question by clarifying attributes of a target answer that differ from attributes of the answer; as well as Question understanding is re-performed based on the updated question.

10. A device for controlling a robot, comprising: a question understanding module configured to determine the intent and classification of a question, entities in the question, and at least one business line associated with the question by performing question understanding on a question of a user by a language model, wherein the user is identified based on an organizational structure and a question-answering history; a business line determination module configured to determine a target business line to be searched in a business knowledge base for the question based on the at least one business line associated with the question, the organizational structure of the user, and the question-answering history, wherein the target business line is associated with both the question and the user and indicates a direction and scope to be searched in the business knowledge base; an answer acquisition module, configured to select business knowledge items from the target business knowledge corresponding to the target business line in the business knowledge base, based on the intent and the classification of the question and the entities in the question, by the language model to generate an answer to the question, wherein the business knowledge base includes a plurality of sub-knowledge bases based on business knowledge types, the plurality of sub-knowledge bases including a first sub-knowledge base corresponding to structured business knowledge, a second sub-knowledge base corresponding to unstructured business knowledge, and a third sub-knowledge base corresponding to semi-structured business knowledge; as well as an answer providing module configured to provide the generated answer to the user in an interactive manner in response to the accuracy of the generated answer satisfying an accuracy threshold, After the generated answer is determined to be adapted to the question in terms of semantic similarity and contextual adaptation, the question and a portion of the answer are used to update a corresponding sub-knowledge base in the business knowledge base.

11. An electronic device, comprising: processor; as well as A memory coupled to the processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 9.

12. A machine-readable storage medium having machine-executable instructions stored thereon, wherein the machine-executable instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program which, when executed by a processor of a computer, causes the processor to perform the method according to any one of claims 1 to 9.

Images