An ABI Question Answering Method and System for Unlabeled Data

Abstract

This invention relates to the field of knowledge-based question answering technology, specifically to an ABI question answering method and system for unlabeled data. The method includes: acquiring the query content from a user's question and files in an ABI management system database; acquiring the importance of each keyword in the query keyword set; acquiring the matching degree between each file in the database and the query content; acquiring the core importance of each keyword in each file in the database; constructing a feature matrix of the query content keywords and database files based on the core importance and importance; acquiring the final relevant files for the query content; acquiring the overall attention score of each final relevant file relative to the query content; and selecting answer files based on the overall attention score to generate a response. This invention can handle complex sentence structures or uncommon expressions, providing effective answers to the query content and improving the reference value of the answers.

Description

Technical Field

[0001] This invention relates to the field of knowledge question answering technology, specifically to an ABI question answering method and system for unlabeled data. Background Technology

[0002] For enterprises with diverse knowledge-based question-answering needs, traditional question-answering systems are typically based on preset rules and templates. If the question exceeds the template's scope, the system's answer may lack value. Therefore, a generative ABI platform can be built, providing support for various database types. Administrators or users can configure the enterprise's relevant databases on this platform, enabling information extraction from unlabeled text, answer generation, and understanding of user intent. By combining self-supervised learning, information extraction, and natural language processing algorithms, a more intelligent and flexible question-answering system based on unlabeled data can be constructed.

[0003] In existing technologies, question-answering systems can only recognize and process limited natural language expressions. They cannot understand complex sentence structures or uncommon expressions. If a user's question does not contain the system's preset keywords or sentence patterns, the system may not be able to provide an effective answer. Or, if the question exceeds the scope of the template, the system's answer may not be of reference value. At the same time, the answers are standardized answers based on preset templates and rules, which are applicable to common questions and lack personalization or context sensitivity. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides an ABI question-answering method and system for unlabeled data.

[0005] The ABI question-answering method and system for unlabeled data in this invention adopts the following technical solution:

[0006] One embodiment of the present invention provides an ABI question answering method for unlabeled data, the method comprising the following steps:

[0007] Retrieve user queries for the same question and files from the ABI management system database; extract keywords from the query content;

[0008] Obtain the query keyword sequence and query keyword set; based on the position of the keywords in the query keyword sequence and their frequency in the query content, determine the importance of each keyword in the query keyword set; based on the importance of the keywords and the frequency of their occurrence in the database files, determine the matching degree between each file in the database and the query content;

[0009] Obtain the rarity of each keyword in the query keyword set for each file in the database; based on the matching degree, rarity, and keyword importance, determine the core keyword level of each keyword in each file in the database; construct a feature matrix of query keywords and database files based on core keyword level and importance; build a decision tree based on importance and feature matrix to obtain the final relevant files for the query content;

[0010] Obtain the keyword vectors for the query content and the keyword vectors for each final relevant document; based on the keyword vectors for the query content, the keyword vectors for the final relevant documents, their importance, core importance, and feature matrix, obtain the overall attention score for each final relevant document relative to the query content; filter the answer documents for the query content based on the overall attention score, and generate the answer to the query content.

[0011] Furthermore, the query content includes the most recent query question and historical query questions.

[0012] Furthermore, the specific steps for obtaining the query keyword sequence and query keyword set, and determining the importance of each keyword in the query keyword set based on the position of the keyword in the query keyword sequence and its frequency in the query content, are as follows:

[0013] Arrange all keywords in the query content in chronological order to obtain the query keyword sequence; the set of all different keywords in the query content is used as the query keyword set.

[0014]

[0015] In the formula, X i To find the maximum index of the i-th keyword in the query keyword sequence; N is the total number of keywords in the query keyword sequence; a i To query the temporal parameters of the i-th keyword in the keyword set, if the i-th keyword appeared in the most recent query, the temporal parameter is TH1; if the i-th keyword appeared in a historical query, the temporal parameter is TH2, where TH1 > TH2; F i This query retrieves the frequency of the i-th keyword in the query content; sigmoid() is the sigmoid function used for normalization; Z i To query the importance of the i-th keyword in a keyword set.

[0016] Furthermore, the specific steps for determining the matching degree between each file in the database and the query content based on the importance of the keywords and the frequency of their occurrence in the database files are as follows:

[0017]

[0018] In the formula, M represents the number of keywords in the query keyword set; Z a To query the importance of the a-th keyword in a keyword set; F j,a P represents the number of times the keyword a appears in the j-th file of the database; j This represents the degree of matching between the j-th file in the database and the query content.

[0019] Furthermore, the specific steps for obtaining the rarity of each keyword in the query keyword set within each file of the database, and determining the core keyword level of each keyword in each file of the database based on the matching degree, rarity, and keyword importance, are as follows:

[0020] Using the TF-IDF statistical method, the TF-IDF value corresponding to the b-th keyword in the query keyword set in the j-th file of the database is obtained, and this TF-IDF value is used as the rarity of the b-th keyword in the query keyword set in the j-th file of the database;

[0021]

[0022] In the formula, P j TD represents the degree of matching between the j-th file in the database and the query content; b,j To query the rarity of the b-th keyword in the b-th file of the database; TD j To query the average rarity of all keywords in the keyword set within the j-th file of the database; sigmoid() is the sigmoid function used for normalization; H j,b The core importance of the b-th keyword in the j-th file of the database.

[0023] Furthermore, the specific steps involved in constructing a feature matrix of query content keywords and database files based on their core importance and significance are as follows:

[0024] The product of the core importance of the b-th keyword in the j-th file in the database and the importance of the b-th keyword in the query keyword set is used as the feature value corresponding to the j-th file and the b-th keyword in the feature matrix to be constructed.

[0025] Fill in the feature values ​​corresponding to each file and each keyword in the feature matrix to be constructed according to the corresponding positions to obtain the feature matrix of query content keywords and database files. In the feature matrix to be constructed, the rows represent different files in the database, the columns represent different keywords in the query content, and the intersection of the rows and columns is where the feature values ​​should be filled in.

[0026] Furthermore, the specific steps involved in building a decision tree based on importance and feature matrix to obtain the final relevant documents for the query content are as follows:

[0027] The keyword with the highest importance is used as the root node of the decision tree. Files with feature values ​​greater than a preset first threshold corresponding to the root node keyword are used as related files of the first-level branch node. Other keywords with feature values ​​greater than a preset first threshold corresponding to the related files of the first-level branch node are used as branch nodes of the first level. Branch nodes of other levels are built in sequence until all keywords have been traversed to obtain the decision tree. Files with feature values ​​greater than a preset first threshold corresponding to the keywords of the last leaf node of the decision tree are used as the final related files of the query content.

[0028] Furthermore, the specific steps for obtaining the keyword vector of the query content and the keyword vector of each final related file are as follows:

[0029] The TF-IDF method is used to convert all different keywords in the query content into numerical vector representations, resulting in the keyword vector of the query content. The TF-IDF method is also used to convert different keywords belonging to the query content into numerical vector representations for the f-th final related document, resulting in the keyword vector of the f-th final related document.

[0030] Furthermore, the specific steps for obtaining the overall attention score of each final relevant document relative to the query content based on the keyword vector of the query content, the keyword vector of the final relevant document, its importance, core importance, and feature matrix are as follows:

[0031]

[0032] In the formula, Q is the keyword vector of the query content; K f K is the keyword vector of the f-th final relevant document; f T Z1 is the transpose of the keyword vector of the f-th final relevant document; Z2 is the importance vector of the query content keywords, and the specific method for obtaining the importance vector is as follows: according to the order of the keywords in the query content keyword vector, the importance of all different keywords in the query content is arranged in rows to obtain the importance vector of the query content keywords; H1 fLet f be the core keyword intensity vector of the f-th final relevant document. The specific method for obtaining this core keyword intensity vector is as follows: Arrange the core keyword intensity of all different keywords in the f-th final relevant document in rows according to the order of the keywords in the query keyword vector to obtain the core keyword intensity vector of the f-th final relevant document; ||| is the L2 norm, used to take the absolute value; sigmoid() is the sigmoid function, used for normalization; R f Let S be the feature value vector of the keywords in the f-th final related document. The specific method for obtaining the feature value vector is as follows: In the feature matrix, the vector formed by the feature values ​​of the row containing the f-th final related document is used as the feature value vector of the keywords in the f-th final related document; f The overall attention score for the f-th final relevant document relative to the query content.

[0033] The present invention also proposes an ABI question-answering system for unlabeled data, including a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the steps of the aforementioned method.

[0034] The beneficial effects of the technical solution of this invention are as follows: According to this invention, during the ABI question-answering system process, the importance of keywords in the query keyword set and the matching degree between the database files and the query content can be accurately analyzed, thereby constructing a suitable decision tree and introducing an attention mechanism. This enables the handling of complex sentence structures or uncommon expressions, providing effective answers to the query content, improving the reference value of the answers, and enhancing the personalization or contextual sensitivity of the answers. When determining the importance of keywords in the query keyword set, the importance of each keyword in the query keyword set is determined by analyzing the position of the keywords in the query keyword sequence and their frequency in the query content, improving the scientific rigor, accuracy, and objectivity of keyword importance determination. When determining the matching degree between the database files and the query content, the matching degree is determined by analyzing the importance of keywords and the number of occurrences of keywords in the database files, improving the scientific rigor, accuracy, and objectivity of the matching degree. When building the decision tree, the decision tree is determined by analyzing the rarity of keywords combined with the matching degree and importance, improving the rationality of obtaining the final relevant files for the query content when building the decision tree, and providing a data foundation for the subsequent attention mechanism. When determining the overall attention score of the final relevant documents relative to the query content, the keyword vectors of the query content, the keyword vectors of the final relevant documents, their importance, core importance, and feature matrix are fully considered to determine the overall attention score, thereby improving the comprehensiveness and accuracy of the overall attention score. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart illustrating the steps of an ABI question-answering method for unlabeled data provided in an embodiment of the present invention;

[0037] Figure 2 This is the feature matrix to be constructed for an ABI question-answering method in the case of unlabeled data, as provided in an embodiment of the present invention. Detailed Implementation

[0038] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an ABI question-answering method and system for unlabeled data proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0040] The following description, in conjunction with the accompanying drawings, details the specific solution of the ABI question-answering method and system for unlabeled data provided by this invention.

[0041] Please see Figure 1 The diagram illustrates a flowchart of an ABI question-answering method for unlabeled data according to an embodiment of the present invention. The method includes the following steps:

[0042] Step S001: Obtain the query content submitted by the user for the same question and the files in the ABI management system database; obtain the keywords in the query content.

[0043] It should be noted that the main purpose of this embodiment is to answer user queries regarding the same issue, where the query content consists of multiple questions raised by the user. Before starting the analysis, relevant data will be obtained first.

[0044] Specifically, it retrieves the user's query for the same question and the files in the ABI management system database.

[0045] It should be noted that the query content submitted by a user for the same question includes the most recent query and historical query questions. Historical query questions include multiple queries. In this embodiment, all query questions are collectively referred to as query content. Obtaining the query content and files from the ABI management system database are existing methods and will not be described in detail here.

[0046] Specifically, to retrieve keywords from the query content, the following steps are taken:

[0047] Use the TF-IDF (term frequency-inverse document frequency) extraction strategy to obtain keywords from the query content.

[0048] It should be noted that existing methods for obtaining keywords in the query content using TF-IDF will not be described in this embodiment.

[0049] At this point, the keywords in the query content have been obtained.

[0050] Step S002: Obtain the query keyword sequence and query keyword set; based on the position of the keywords in the query keyword sequence and their frequency in the query content, determine the importance of each keyword in the query keyword set; based on the importance of the keywords and the number of times the keywords appear in the database files, determine the matching degree between each file in the database and the query content.

[0051] It should be noted that when a user submits a query, the ABI system first needs to search for relevant documents in the company's database. During this process, due to the information explosion era, acquiring labeled data is time-consuming and expensive, and the company's relevant data is constantly increasing and expanding. Therefore, unlabeled data is used to build the ABI answer system. Since there is a large amount of information in the database, in order to ensure that the query is answered accurately, the keywords in the query are first searched for relevant documents in the database, and then the search is performed again in combination with the above content.

[0052] It's important to note that in Chinese natural language word order, important content follows a post-position rule, with adjectives appearing first. Keywords appearing later in the query query are considered core, and keywords in preceding query content are more important and valuable the closer they are to the current query. Furthermore, keywords appearing more frequently in preceding query content also have greater importance and reference value. For example, if the query is "Applications of machine learning in medicine," the keyword sequence obtained using the TF-IDF method would be: "1: Machine learning, 2: Medicine, 3: Applications." The answer focuses on how to apply these applications. Therefore, by analyzing the position of keywords in the query keyword sequence and their frequency in the query content, the importance of each keyword in the query keyword set can be determined.

[0053] Specifically, the query keyword sequence and query keyword set are obtained as follows:

[0054] Arrange all keywords in the query content in chronological order to obtain the query keyword sequence; the set of all different keywords in the query content is used as the query keyword set.

[0055] Furthermore, based on the position of each keyword in the query keyword sequence and its frequency in the query content, the importance of each keyword in the query keyword set is determined, as follows:

[0056]

[0057] In the formula, X i To find the maximum index of the i-th keyword in the query keyword sequence; N is the total number of keywords in the query keyword sequence; a i To query the temporal parameters of the i-th keyword in the keyword set, if the i-th keyword appeared in the most recent query, the temporal parameter is TH1; if the i-th keyword appeared in a historical query, the temporal parameter is TH2, where TH1 > TH2. In this embodiment, TH1 = 2 and TH2 = 1 are used for description. i This query retrieves the frequency of the i-th keyword in the query content; sigmoid() is the sigmoid function used for normalization; Z i To query the importance of the i-th keyword in a keyword set.

[0058] It should be noted that in the above formula This indicates the position of the i-th keyword in the query keyword sequence; the later the position, the higher the position. The larger the value, the greater its importance; timing parameter a i Further emphasis is placed on whether the i-th keyword appears in the most recent query. If the i-th keyword appears in the most recent query, it indicates that the later the query appears, the higher its importance. The more frequently the i-th keyword appears in the query content, the more important it is.

[0059] It should be noted that the importance of each keyword in the query keyword set has already been obtained. Combining this importance with the number of times the keyword appears in each file of the database, the matching degree between each file in the ABI management system database and the query content is analyzed. If the keyword in the query keyword set has a high importance and appears frequently in the database files, it indicates a good match between the current file and the query content.

[0060] Specifically, based on the importance of keywords and the frequency of keyword occurrences in the database files, the degree of match between each file in the database and the query content is determined, as follows:

[0061]

[0062] In the formula, M represents the number of keywords in the query keyword set; Z a To query the importance of the a-th keyword in a keyword set; F j,a P represents the number of times the keyword a appears in the j-th file of the database; j This represents the degree of matching between the j-th file in the database and the query content.

[0063] It should be noted that Z a This indicates the importance of the 'a'th keyword in the query keyword set. If the importance of the 'a'th keyword is high, and the number of times the 'a'th keyword appears in the 'j'th file in the database is also large, it means that the 'j'th file in the database matches the 'a'th keyword in the query content well. By analyzing all the keywords in the query keyword set, we can obtain the overall matching degree between the 'j'th file in the database and the query content. The higher the matching degree, the more helpful the current file is to the query content.

[0064] At this point, we have obtained the degree of matching between each file in the database and the query content.

[0065] Step S003: Obtain the rarity of each keyword in the query keyword set in each file of the database; based on the matching degree, rarity, and importance of the keywords, obtain the core keyword degree of each keyword in each file of the database; based on the core keyword degree and importance, construct a feature matrix of query content keywords and database files; based on the importance and feature matrix, establish a decision tree to obtain the final relevant files of the query content.

[0066] It's important to note that the degree of matching between database files and query content can provide insights for the ABI question-answering system from a keyword perspective. To improve answer accuracy, the relevance of file content also needs to be considered. Decision tree models can make decisions based on input features (i.e., keywords in the query content), enabling the model to identify which features best distinguish relevant from irrelevant files. During decision tree construction, the model automatically selects the features that most improve classification accuracy. Keywords extracted from the query content are used to determine file relevance and assess feature importance, prioritizing features highly relevant to the query content to find more accurately relevant database files.

[0067] It should be further explained that when constructing a decision tree model, it is first necessary to determine the feature values ​​in the feature matrix of the query content keywords and the database file, and then construct the feature matrix to build the decision tree.

[0068] Specifically, the rarity of each keyword in the query keyword set is obtained for each file in the database, as follows:

[0069] The specific method for obtaining the rarity level is as follows: Using the TF-IDF (term frequency-inverse document frequency) statistical method, obtain the TF-IDF value corresponding to the b-th keyword in the j-th file of the database. This TF-IDF value is then used as the rarity level of the b-th keyword in the j-th file of the database. It should be noted that obtaining the TF-IDF value corresponding to the b-th keyword in the j-th file of the database using the TF-IDF (term frequency-inverse document frequency) statistical method is an existing method and will not be elaborated upon in this embodiment.

[0070] Furthermore, based on the degree of matching, rarity, and importance of keywords, the core keyword level of each keyword in each file of the database is obtained; based on the core keyword level and importance, the feature values ​​corresponding to each file and each keyword in the feature matrix to be constructed are obtained, as follows:

[0071]

[0072] In the formula, P j TD represents the degree of matching between the j-th file in the database and the query content; b,j To query the rarity of the b-th keyword in the b-th file of the database; TD j To query the average rarity of all keywords in the keyword set within the j-th file of the database; sigmoid() is the sigmoid function used for normalization; H j,b The core importance of the b-th keyword in the j-th file of the database.

[0073] The product of the core importance of the b-th keyword in the j-th file of the database and the importance of the b-th keyword in the query keyword set is used as the feature value corresponding to the j-th file and the b-th keyword in the feature matrix to be constructed.

[0074] It should be noted that, This represents the rarity of the b keyword appearing in the j-th file, based on the matching degree between the j-th file and the query content. It reflects the core importance of the b keyword appearing in the j-th file. By analyzing the importance of the b keyword, the feature values ​​corresponding to the j-th file and the b keyword in the feature matrix to be constructed are obtained. The larger the feature value, the more important the keyword is in the file.

[0075] Furthermore, a feature matrix is ​​constructed based on the feature values ​​for the query keywords and the database file, as follows:

[0076] Fill in the feature values ​​corresponding to each file and each keyword in the feature matrix to be constructed according to the corresponding positions to obtain the feature matrix of query content keywords and database files. In the feature matrix to be constructed, the rows represent different files in the database, the columns represent different keywords in the query content, and the intersection of the rows and columns is where the feature values ​​should be filled in.

[0077] It should be noted that, for a better understanding of the feature matrix, please refer to [link / reference needed]. Figure 2 , Figure 2 The feature matrix to be constructed in this embodiment is shown below. The rows in the matrix represent different files in the database, namely DOC1, DOC2, etc., and the columns represent different keywords in the query content, namely keyword1, keyword2, etc. The intersection of the rows and columns is where the feature values ​​should be filled in.

[0078] Furthermore, a decision tree is built based on the importance and feature matrix to obtain the final relevant documents for the query content, as follows:

[0079] The keyword with the highest importance is used as the root node of the decision tree. Files with feature values ​​greater than a preset first threshold corresponding to the root node keyword are used as related files of the first-level branch node, where the preset first threshold is 0.5. Other keywords with feature values ​​greater than the preset first threshold corresponding to the related files of the first-level branch node are used as branch nodes of the first level. Branch nodes of other levels are built in sequence until all keywords have been traversed to obtain the decision tree. Files with feature values ​​greater than the preset first threshold corresponding to the keywords of the last leaf node of the decision tree are used as the final related files of the query content.

[0080] It should be noted that keywords that have already been used as nodes will not be used to build the decision tree again.

[0081] At this point, the final relevant files for the queried content have been obtained.

[0082] Step S004: Obtain the keyword vector of the query content and the keyword vector of each final related document; based on the keyword vector of the query content, the keyword vector of the final related document, the importance, the core key degree, and the feature matrix, obtain the overall attention score of each final related document relative to the query content; filter the answer documents of the query content based on the overall attention score, and generate the answer to the query content.

[0083] It's important to note that to ensure the ABI question-answering system outputs the most accurate answers, an attention mechanism is needed to further determine the relevance of database files to the query content. Attention mechanisms are widely used in natural language processing, dynamically assigning weights to different elements of the input sequence, allowing the model to selectively input the most important information. Applying an attention mechanism to a question-answering system dynamically assigns weights to keywords, ensuring that highly relevant database files are the most important references for answering the query. Before starting the analysis, it's necessary to obtain the keyword vectors for the query content and the keyword vectors for each file in the database, then analyze their similarity to obtain the attention weights.

[0084] Specifically, the keyword vectors for the query content and the keyword vectors for each final related file are obtained, as follows:

[0085] The TF-IDF method is used to convert all different keywords in the query content into numerical vector representations, resulting in the keyword vector of the query content. The TF-IDF method is also used to convert different keywords belonging to the query content into numerical vector representations for the f-th final related document, resulting in the keyword vector of the f-th final related document.

[0086] It should be noted that obtaining the keyword vector of the query content and the keyword vector of each final related file through the TF-IDF method is an existing method of the TF-IDF method, and will not be described in detail in this embodiment.

[0087] Furthermore, based on the keyword vectors of the query content, the keyword vectors of the final relevant documents, their importance, core importance, and feature matrix, the overall attention score of each final relevant document relative to the query content is obtained, as follows:

[0088]

[0089] In the formula, Q is the keyword vector of the query content; K f K is the keyword vector of the f-th final relevant document; f TZ1 is the transpose of the keyword vector of the f-th final relevant document; Z2 is the importance vector of the query content keywords, and the specific method for obtaining the importance vector is as follows: according to the order of the keywords in the query content keyword vector, the importance of all different keywords in the query content is arranged in rows to obtain the importance vector of the query content keywords; H1 f Let f be the core keyword intensity vector of the f-th final relevant document. The specific method for obtaining this core keyword intensity vector is as follows: Arrange the core keyword intensity of all different keywords in the f-th final relevant document in rows according to the order of the keywords in the query keyword vector to obtain the core keyword intensity vector of the f-th final relevant document; ||| is the L2 norm, used to take the absolute value; sigmoid() is the sigmoid function, used for normalization; R f Let S be the feature value vector of the keywords in the f-th final related document. The specific method for obtaining the feature value vector is as follows: In the feature matrix, the vector formed by the feature values ​​of the row containing the f-th final related document is used as the feature value vector of the keywords in the f-th final related document; f The overall attention score for the f-th final relevant document relative to the query content.

[0090] It should be noted that ||Z1·Q|| represents the magnitude of the query keyword vector obtained by using the importance of the query keywords as weights, and ||H1|| represents the magnitude of the query keyword vector. f ·K f The symbol "‖" indicates that the core importance of the keywords in the final relevant documents is used as the weight to obtain the modulus of the keyword vector of the final relevant documents. The cosine similarity between the query content and the keyword vectors in the final relevant files is represented by the following formula. The importance of the keywords in the query content and the core importance of the keywords in the file are used as weights to calculate the modulus of the file and the query content, respectively. The larger the normalized value of this formula, the higher the similarity and the greater the attention weight. By combining the feature vectors, the overall attention score of each final relevant file relative to the query content is obtained.

[0091] Furthermore, the response files for the query are filtered based on the overall attention score, and a response to the query is generated, as follows:

[0092] A preset attention score threshold is set. Files with an overall attention score greater than the preset threshold are used as answer files for the query. Conversely, files with a lower overall attention score are not used as answer files. All answer files are integrated into a Large Language Model (LLM). The deep learning model GPT is used to generate natural language answers based on the answer files. The generated natural language answers are displayed to users on the ABI management system, along with links to answer files related to the query.

[0093] By following the steps above, an ABI question-answering method for unlabeled data is completed.

[0094] Another embodiment of the present invention provides an ABI question-answering system for unlabeled data, the system including a memory and a processor, wherein the processor, when executing a computer program stored in the memory, performs the following operations:

[0095] The process involves: retrieving user queries for the same question and files from the ABI management system database; retrieving keywords from the query content; retrieving a sequence and set of query keywords; determining the importance of each keyword in the query keyword set based on its position in the query keyword sequence and its frequency in the query content; determining the matching degree between each file in the database and the query content based on the keyword importance and the frequency of keyword occurrences in the database files; determining the rarity of each keyword in the query keyword set within each file in the database; determining the core importance of each keyword in each file in the database based on the matching degree, rarity, and keyword importance; constructing a feature matrix of query content keywords and database files based on core importance and importance; building a decision tree based on importance and the feature matrix to obtain the final relevant files for the query content; retrieving keyword vectors for the query content and keyword vectors for each final relevant file; obtaining an overall attention score for each final relevant file relative to the query content based on the query content keyword vectors, keyword vectors of the final relevant files, importance, core importance, and feature matrix; and filtering the answer files for the query content based on the overall attention score and generating a response to the query content.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An ABI question-answering method for unlabeled data, characterized in that, The method includes the following steps: Retrieve user queries for the same question and files from the ABI management system database; extract keywords from the query content; Obtain the query keyword sequence and query keyword set; based on the position of the keywords in the query keyword sequence and their frequency in the query content, determine the importance of each keyword in the query keyword set; based on the importance of the keywords and the frequency of their occurrence in the database files, determine the matching degree between each file in the database and the query content; Obtain the rarity of each keyword in the query keyword set for each file in the database; based on the matching degree, rarity, and keyword importance, determine the core keyword level of each keyword in each file in the database; construct a feature matrix of query keywords and database files based on core keyword level and importance; build a decision tree based on importance and feature matrix to obtain the final relevant files for the query content; Obtain the keyword vectors for the query content and the keyword vectors for each final relevant document; based on the keyword vectors for the query content, the keyword vectors for the final relevant documents, their importance, core importance, and feature matrix, obtain the overall attention score for each final relevant document relative to the query content; filter the answer documents for the query content based on the overall attention score, and generate the answer to the query content.

2. The ABI question-answering method for unlabeled data as described in claim 1, characterized in that, The query content includes the most recent query question, as well as historical query questions.

3. The ABI question-answering method for unlabeled data as described in claim 2, characterized in that, The specific steps involved in obtaining the query keyword sequence and query keyword set, and determining the importance of each keyword in the query keyword set based on its position in the query keyword sequence and its frequency in the query content, are as follows: Arrange all keywords in the query content in chronological order to obtain the query keyword sequence; the set of all different keywords in the query content is used as the query keyword set. In the formula, X i To find the maximum index of the i-th keyword in the query keyword sequence; N is the total number of keywords in the query keyword sequence; a i To query the time sequence parameter of the i-th keyword in the keyword set, if the i-th keyword appeared in the most recent query, the time sequence parameter is TH1; if the i-th keyword appeared in a historical query, the time sequence parameter is TH2, and TH1>TH2. F i To query the frequency of the i-th keyword in the keyword set in the query content; sigmoid() is the sigmoid function, used for normalization; Z i To query the importance of the i-th keyword in a keyword set.

4. The ABI question-answering method for unlabeled data as described in claim 1, characterized in that, The specific steps involved in determining the matching degree between each file in the database and the query content based on the importance of keywords and the frequency of keyword occurrences in the database files are as follows: In the formula, M represents the number of keywords in the query keyword set; Z a To query the importance of the a-th keyword in a keyword set; F j,a P represents the number of times the keyword a appears in the j-th file of the database; j This represents the degree of matching between the j-th file in the database and the query content.

5. The ABI question-answering method for unlabeled data as described in claim 1, characterized in that, The specific steps for obtaining the rarity of each keyword in the query keyword set within each file of the database, and determining the core keyword level of each keyword in each file of the database based on the matching degree, rarity, and keyword importance, are as follows: Using the TF-IDF statistical method, the TF-IDF value corresponding to the b-th keyword in the query keyword set in the j-th file of the database is obtained, and this TF-IDF value is used as the rarity of the b-th keyword in the query keyword set in the j-th file of the database; In the formula, P j This represents the degree of matching between the j-th file in the database and the query content. TD b,j To query the rarity of the b-th keyword in the b-th file of the database; TD j To query the average rarity of all keywords in the keyword set within the j-th file of the database; sigmoid() is the sigmoid function used for normalization; H j,b The core importance of the b-th keyword in the j-th file of the database.

6. The ABI question-answering method for unlabeled data as described in claim 5, characterized in that, The specific steps involved in constructing a feature matrix of query content keywords and database files based on their core importance and significance are as follows: The product of the core importance of the b-th keyword in the j-th file in the database and the importance of the b-th keyword in the query keyword set is used as the feature value corresponding to the j-th file and the b-th keyword in the feature matrix to be constructed. Fill in the feature values ​​corresponding to each file and each keyword in the feature matrix to be constructed according to the corresponding positions to obtain the feature matrix of query content keywords and database files. In the feature matrix to be constructed, the rows represent different files in the database, the columns represent different keywords in the query content, and the intersection of the rows and columns is where the feature values ​​should be filled in.

7. The ABI question-answering method for unlabeled data as described in claim 1, characterized in that, The specific steps involved in building a decision tree based on importance and feature matrix to obtain the final relevant files for the query content are as follows: The keyword with the highest importance is used as the root node of the decision tree. Files with feature values ​​greater than a preset first threshold corresponding to the root node keyword are used as related files of the first-level branch node. Other keywords with feature values ​​greater than a preset first threshold corresponding to the related files of the first-level branch node are used as branch nodes of the first level. Branch nodes of other levels are built in sequence until all keywords have been traversed to obtain the decision tree. Files with feature values ​​greater than a preset first threshold corresponding to the keywords of the last leaf node of the decision tree are used as the final related files of the query content.

8. The ABI question-answering method for unlabeled data as described in claim 1, characterized in that, The specific steps for obtaining the keyword vector of the query content and the keyword vector of each final related file are as follows: The TF-IDF method is used to convert all different keywords in the query content into numerical vector representations, resulting in the keyword vector of the query content. The TF-IDF method is also used to convert different keywords belonging to the query content into numerical vector representations for the f-th final related document, resulting in the keyword vector of the f-th final related document.

9. The ABI question-answering method for unlabeled data as described in claim 6, characterized in that, The process of obtaining the overall attention score of each relevant document relative to the query content based on the keyword vector of the query content, the keyword vector of the relevant documents, their importance, core importance, and feature matrix includes the following specific steps: In the formula, Q is the keyword vector of the query content; K f K is the keyword vector of the f-th final relevant document; f T Z1 is the transpose of the keyword vector of the f-th final relevant document; Z2 is the importance vector of the query content keywords, and the specific method for obtaining the importance vector is as follows: according to the order of the keywords in the query content keyword vector, the importance of all different keywords in the query content is arranged in rows to obtain the importance vector of the query content keywords; H1 f Let f be the core keyword intensity vector of the f-th final relevant document. The specific method for obtaining this core keyword intensity vector is as follows: Arrange the core keyword intensity of all different keywords in the f-th final relevant document in rows according to the order of the keywords in the query keyword vector to obtain the core keyword intensity vector of the f-th final relevant document; ||| is the L2 norm, used to take the absolute value; sigmoid() is the sigmoid function, used for normalization; R f Let S be the feature value vector of the keywords in the f-th final related document. The specific method for obtaining the feature value vector is as follows: In the feature matrix, the vector formed by the feature values ​​of the row containing the f-th final related document is used as the feature value vector of the keywords in the f-th final related document; f The overall attention score for the f-th final relevant document relative to the query content.

10. An ABI question-answering system for unlabeled data, the system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the ABI question-answering method for unlabeled data as described in any one of claims 1-9.

Images