A method and system for determining standard questions based on similarity

Abstract

The present invention provides a method and system for determining standard questions based on similarity, including step S1: obtaining user question information, calculating the similarity between the user question and the existing training data according to a similarity model, and storing the questions with similarity lower than a specified threshold as data to be processed, and determining the unfamiliar data that does not match the characteristics of the existing standard questions; and step S2: performing cluster analysis on the user questions in the data to be processed, extracting the characteristics of each category as hot words, sorting the hot words from high to low, storing the hot questions with the highest ranking in the database to be annotated, and expanding the standard questions used in the intelligent question-answering system. The system of the present invention includes a determination device and an expansion device, the determination device determines the unfamiliar data that does not match the characteristics of the existing standard questions, and the expansion device expands the standard questions. The expanded standard questions generated by the present invention are clearly distinguished from the existing standard questions, greatly reducing labor costs and memory consumption, and improving accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for determining a standard problem based on similarity. Background Art

[0002] With the in-depth development of computer and artificial intelligence technology and its rapid implementation in various industries, the customer service industry has become a key application scenario in artificial intelligence technology. Many companies use intelligent customer service robots to replace most manual customer service and have established intelligent customer service systems, transforming the original large-scale manual customer service model to an intelligent customer service robot plus a small amount of manual customer service model. Then, in the process of human-computer interaction, in order to understand human language, the artificial intelligence semantic recognition system usually requires a knowledge base. Different companies have their own exclusive knowledge and semantic bases, and the editing and expansion of the knowledge base is particularly important for the artificial intelligence semantic recognition system.

[0003] The most initial form of knowledge in the knowledge base is FAQ (Frequently Asked Questions) that users interact with robots, that is, question-answer pairs. Among them, standard questions are standard questions used to express certain knowledge. For example, "How to apply for a credit card" is a standard question description in the banking industry. When users input into the machine, the ideal situation is to use standard questions, and the robot can understand the user's intention through retrieval methods. However, in reality, due to the richness of language, users often do not use standard questions. Therefore, for semantic recognition, the knowledge base needs to have extended questions of standard questions. The extended questions are slightly different from the standard questions in expression, but express the same meaning.

[0004] In the traditional method, it is necessary to analyze a large number of users' questions and summarize multiple standard questions from them, and then add similar questions to the standard questions as training corpus for the artificial intelligence system. This process consumes a lot of labor costs and leads to low recognition efficiency.

[0005] For example, in the prior art CN 110413749 A "Method and device for determining standard questions", after obtaining multiple representative texts corresponding to multiple user questions, the invention further obtains the feature value of each representative text, so that the difference between the feature values ​​of any two representative texts is negatively correlated with the semantic similarity of the two representative texts; then the representative texts are sorted according to the feature values ​​of each representative text to obtain a text sequence, and the annotator can quickly find a subsequence formed by a continuous arrangement of multiple semantically similar representative texts by judging whether two adjacent representative texts in the text sequence are semantically similar, and annotate each subsequence found. The invention uses a text clustering device to cluster questions, but does not take into account the semantic similarity between user questions and existing corpus. Direct clustering without screening will result in semantic entanglement or conflict between the clustering results and existing standard questions or similar questions, and it is difficult to avoid standard questions with semantic similarity to existing standard questions.

[0006] For example, in the prior art CN 106803092 A "A method and device for determining standard question data", the invention solves the problem of low efficiency in obtaining answers to questions in the prior art. The method includes: according to the obtained personalized question data, from the features associated with the preset standard question data, obtaining the features associated with the personalized question data; inputting the personalized question data and the obtained features into a trained standard question probability model; training according to the preset standard question data and the feature samples associated with the preset standard question data; determining the standard question data from the preset standard question data according to the probability distribution output by the standard question probability model. The invention obtains the features associated with the personalized question data from the features associated with the preset standard question data. This method has no way to fit the new class group well when processing unfamiliar data.

[0007] For example, in the prior art CN 105608199 A "Method and device for expanding standard questions in an intelligent question-answering system", the invention determines a predetermined number of standard questions as hot questions from the standard question database based on the collected log set; and modifies the subject of the similar question set to obtain an expanded question. The invention performs subject modification on the existing standard question sentences to obtain an expanded question for each hot question, and the standard question sentences generated in this way have limitations.

[0008] It can be seen that the existing technology requires a high degree of manual intervention when determining standard questions. The knowledge base manager analyzes the most frequently asked hot questions of the intelligent question-answering system and then subjectively determines the standard questions. Even if there are cluster analysis semantic text questions, it is impossible to avoid generating standard questions similar to existing standard questions, which makes the standard question determination mechanism system inefficient and poor in accuracy.

[0009] Therefore, a semi-supervised method for generating standard questions in intelligent customer service scenarios is needed to significantly reduce the time for manually summarizing and generalizing knowledge points, thereby achieving the goal of reducing costs and increasing efficiency. Summary of the invention

[0010] In order to solve the above problems, the present invention provides a method and system for determining standard questions based on similarity. By combining the similarity algorithm with the text clustering method, knowledge points that can be used as standard questions are automatically generated and recommended to management personnel. The generation of standard questions is assisted by a semi-supervised method, so as to achieve the effect of significantly reducing labor costs, improving accuracy and efficiency, and avoiding duplication.

[0011] To achieve the above object, the present invention provides a method for determining a standard question based on similarity, comprising the following steps:

[0012] S1: Obtain user question information, calculate the similarity between the user question and the existing training data according to the similarity model, store the questions with similarity lower than the specified threshold as the data to be processed, and determine the unfamiliar data that does not match the characteristics of the existing standard questions;

[0013] S2: Perform cluster analysis on user questions in the data to be processed, extract the features of each category as hot words, sort the hot words from high to low, store the hot questions with the highest ranking in the database to be labeled, and expand them for standard questions in the intelligent question-answering system.

[0014] Optionally, step S1 further includes the following steps:

[0015] S1.1: Use the word segmentation model to segment the user's question into Chinese words and use stop words to filter;

[0016] S1.2: Obtain word granularity data of the user's question, search in the bag-of-words search library, and obtain the data that needs to be compared for the user's question;

[0017] S1.3: Calculate the semantic similarity between the user's question and each comparison data obtained, and sort them;

[0018] S1.4: Determine user questions below a specified threshold as unfamiliar data.

[0019] Optionally, the similarity model adopts a Simnet semantic model, including an input layer, a representation layer and a matching layer; the input layer converts a text word sequence into a word embedding vector sequence by retrieving a vocabulary; the representation layer converts the word embedding vector sequence into a semantic vector; the matching layer uses the semantic vector for interactive calculation to obtain a matching score.

[0020] Optionally, a pair-wise Ranking Loss function is used to train the Simnet model, and its training optimization goal is to make S(Q,D+)>S(Q,D-), where Q represents the search query text, D+ represents a related document, D- represents an irrelevant document, S(Q,D+) represents the matching score between document D+ and Q, and S(Q,D-) represents the matching score between document D- and Q.

[0021] Optionally, character granularity and word granularity are used for fusion and complementary training, and mixed granularity word sequences are input to the input layer, or different granularity word segmentation methods are used simultaneously, and different granularity word sequences are input to the input layer respectively.

[0022] Optionally, step S2 further includes the following steps:

[0023] S2.1: Obtain unprocessed unfamiliar data;

[0024] S2.2: Perform cross similarity calculation on the unfamiliar data and perform text clustering processing;

[0025] S2.3: Calculate the average similarity of the user question data whose clustering categories exceed the set number;

[0026] S2.4: Recommend the closest classification question as the expanded standard question.

[0027] In addition, the present invention also provides a system for determining standard questions based on similarity, comprising:

[0028] The determining device determines unfamiliar data that does not match the characteristics of existing standard questions, comprising:

[0029] A training module obtains associated features from the preset standard question data and the acquired annotation data for training the similarity model;

[0030] The inference module obtains user question information, calculates the similarity between the user question and the existing training data based on the similarity model, and stores the questions with similarity below the specified threshold as data to be processed;

[0031] Extension device, which is used to extend the standard questions in the intelligent question answering system, including:

[0032] The clustering module performs cluster analysis on the user questions in the data to be processed, extracts the features of each category as hot words, sorts the hot words from high to low, and stores the hot questions with the highest ranking in the database to be labeled;

[0033] The human-computer collaboration module calculates the data to be labeled using the average similarity algorithm and recommends the closest classification question as the extended standard question.

[0034] Optionally, the reasoning module determines the unfamiliar data by the following steps:

[0035] S1.1: Use the word segmentation model to segment the user's question into Chinese words and use stop words to filter;

[0036] S1.2: Obtain word granularity data of the user's question, search in the bag-of-words search library, and obtain the data that needs to be compared for the user's question;

[0037] S1.3: Calculate the semantic similarity between the user's question and each comparison data obtained, and sort them;

[0038] S1.4: Determine user questions below a specified threshold as unfamiliar data.

[0039] Optionally, the training module trains the similarity model using a Simnet semantic model, including an input layer, a representation layer and a matching layer; the input layer converts a text word sequence into a word embedding vector sequence by retrieving a vocabulary; the representation layer converts the word embedding vector sequence into a semantic vector; the matching layer uses the semantic vector for interactive calculation to obtain a matching score.

[0040] Optionally, the expansion device expands the standard question by the following steps:

[0041] S2.1: Obtain unprocessed unfamiliar data;

[0042] S2.2: Perform cross similarity calculation on the unfamiliar data and perform text clustering processing;

[0043] S2.3: Calculate the average similarity of the user question data whose clustering categories exceed the set number;

[0044] S2.4: Recommend the closest classification question as the expanded standard question.

[0045] The advantages and beneficial effects of the present invention are as follows: compared with the prior art method for determining standard questions that requires a lot of manual intervention and generates similar and repeated standard questions, the present invention provides a method and system for determining standard questions based on similarity, which utilizes the advantages of text similarity algorithm and deep learning neural network model, calculates sparse data through similarity model, and regenerated extended standard questions can be clearly distinguished from existing standard questions, does not rely on existing standard questions, does not need to modify the original standard questions, and the results are more accurate. By combining the similarity algorithm with text clustering technology, the existing annotated data is fully utilized, and the available standard questions are expanded on the basis of the existing standard questions, and the knowledge points that can be used as standard questions are automatically expanded and generated, and recommended to managers. The standard questions generated by the present invention are all different from the original data and generated according to the user's questions. The new standard questions are unique and accurate, and can be directly recommended to knowledge base managers for using the extended knowledge base. This method assists the generation of new standard questions, achieves the effect of greatly reducing labor costs and memory consumption, improving accuracy and efficiency, and effectively avoiding the generation of repeated questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only partial introductions to some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a process flow of determining unfamiliar data in a method for determining a standard problem based on similarity according to an embodiment of the present invention is shown;

[0048] Figure 2 The following schematically shows a flow chart of an extended standard question of a method for determining a standard question based on similarity according to an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the structure of a similarity deep neural network model of a method for determining a standard problem based on similarity according to an embodiment of the present invention is shown;

[0050] Figure 4 A schematic diagram showing the structure of a multi-granularity segmentation model of a word in a method for determining a standard problem based on similarity according to an embodiment of the present invention;

[0051] Figure 5 The following is a schematic diagram showing the structure of a system for determining a standard question based on similarity according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0053] In one embodiment, the present invention provides a method for determining a standard question based on similarity, which is used to expand the standard questions in an intelligent question answering system, comprising the following steps:

[0054] S1: Obtain user question information, calculate the similarity between the user question and the existing training data according to the similarity model, and store the questions with similarity lower than the specified threshold as data to be processed, and determine the unfamiliar data that does not match the characteristics of the existing standard questions. Among them, firstly obtain the user's voice information through the robot's human-computer interaction interface or telephone system, convert the voice into text based on voice recognition technology, and store it as a human-computer interaction log. Since the question asked by the user is a question with multiple expressions, it is difficult to send the question directly to the background or as subsequent process data, so the user's question must be converted into a standard question that can be recognized by the system or subsequent process. However, the standard questions used in the knowledge base in the system or method may not be exhaustive, and various questions may occur in different scenarios, so the standard questions in the knowledge base also need to be continuously updated and expanded by determining new standard questions. When determining new standard questions, in order to avoid the occurrence of problems similar to existing standard questions and reduce repetition as much as possible, the present invention adopts a problem generation strategy that is opposite to the prior art, and determines it as a standard question by generating unfamiliar data that is least similar to the existing standard questions. That is, the similarity between the user's question and the existing training data is calculated based on the similarity model, and the questions with similarity below the specified threshold are stored as data to be processed, and unfamiliar data that does not match the existing standard question features is determined. The similarity model used is trained based on the associated features obtained from the preset standard question data and its acquired annotated data. The acquired annotated data is the data that can be used as a new standard question formed by multiple training of the similarity model and multiple executions of the process, and is used as training data in subsequent processes.

[0055] S2: Perform cluster analysis on the user questions in the data to be processed, extract the features of each category as hot words, sort the hot words from high to low, store the hot questions with the highest ranking in the database to be marked, and expand the standard questions used in the intelligent question-answering system. Since the unfamiliar data determined in step S1 is most likely to form questions different from the existing standard questions, step S2 further performs text cluster analysis on these data to obtain multiple similar question classification results of the user questions, extract the features of each category as hot words, sort the hot words from high to low, and store the hot questions with the highest ranking in the database to be marked. The data stored in the database to be marked can be used as a reserve of standard questions. According to the actual situation, the managers in each application scenario, after further processing, use the questions in the database to be marked as extended questions, that is, the extended new standard questions, to apply to the intelligent question-answering system. The intelligent question-answering system may include a database for storing interactive logs, a standard question database, an extended question database, and an unfamiliar question database.

[0056] In one embodiment, if Figure 1 As shown, step S1 also includes the following steps:

[0057] S1.1: Use the word segmentation model to perform Chinese word segmentation on user questions, and use stop words to filter; this step obtains user questions in the interaction log database, generates a bag-of-words retrieval library based on the data in the standard question database and the extended question database, and uses the Jieba (Jieba, a Chinese word segmentation tool) Chinese word segmentation tool to segment all saved questions, and then uses the stop word list determined according to the specific business to filter stop words; obtain the association between words and questions, and store them in the memory for retrieval. Through screening, the questions that are determined as chat questions in intent recognition and the questions that already exist in the standard question database or the extended question database are filtered. Through the preprocessing of this step, user questions can be preliminarily screened and questions with no relevant word granularity can be filtered, which can greatly reduce the amount of calculation, reduce the calculation cost, and reduce memory consumption.

[0058] S1.2: Obtain word granularity data of the user's question, use the word data to search in the bag-of-words retrieval database, and obtain the data that needs to be compared with the user's question. Bag-of-words (BOW) is a commonly used document representation method in the field of information retrieval. A bag-of-words retrieval database is generated based on the data in the standard question database and the extended question database. In the bag-of-words retrieval database, data that can be compared with the words in the user's question is searched.

[0059] S1.3: Calculate the semantic similarity between the user question and each comparison data obtained, and sort them; after calculating the semantic similarity, sort the obtained similarity scores from high to low, and set a threshold to divide the similarity between the user question and the data in the bag-of-words database. The threshold can be adjusted dynamically as needed.

[0060] S1.4: Determine user questions below a specified threshold as unfamiliar data. Through these steps, data with low similarity to known data is screened out from the massive data of user questions. Because the unfamiliar data is not very similar to the known data, standard questions that are not included in the current knowledge base can be clustered from the unfamiliar data.

[0061] In one embodiment, the similarity model adopts the Simnet (SimilarityNet, text semantic matching / text similarity framework) model, including an input layer, a representation layer and a matching layer; the input layer converts the text word sequence into a word embedding vector sequence by searching the vocabulary; the representation layer converts the word embedding vector sequence into a semantic vector; the matching layer uses the semantic vector for interactive calculation to obtain a matching score. The Simnet model is a supervised neural network semantic matching model, which follows the implicit continuous vector representation method in semantic representation, but performs end-to-end modeling of the semantic matching problem under the deep learning framework, unifying the embedding vector representation of words with the semantic representation of sentences, the vector representation of semantics with the matching degree calculation, the matching degree calculation of text pairs and the supervised learning of pair-wise algorithms in an overall framework. Word embedding vectors represent words or phrases in the vocabulary as vectors composed of real numbers. If a word is regarded as the smallest unit of text, the mapping process of word embedding is: mapping or embedding a word in the text space into another numerical vector space, and each word corresponds to a vector representation.

[0062] The input layer converts the text word sequence into a word embedding vector sequence by searching the vocabulary. The main function of the representation layer is to construct the representation from word to sentence, or to convert the isolated word embedding vector representation of the sequence into one or more low-dimensional dense semantic vectors with global information. The accumulation method of the BOW (Bag of words) model is used to further improve the representation effect. The structure code of the BOW model is as follows:

[0063] emb_layer=layers.EmbeddingLayer(self.dict_size,self.emb_dim,"emb")

[0064] left_emb=emb_layer.ops(left)

[0065] right_emb=emb_layer.ops(right)

[0066] pool_layer=layers.SequencePoolLayer("sum")

[0067] left_pool=pool_layer.ops(left_emb)

[0068] right_pool=pool_layer.ops(right_emb)

[0069] softsign_layer=layers.SoftsignLayer()

[0070] left_soft=softsign_layer.ops(left_pool)

[0071] right_soft=softsign_layer.ops(right_pool)

[0072] bow_layer=layers.FCLayer(self.bow_dim,"relu","fc")

[0073] left_bow=bow_layer.ops(left_soft)

[0074] right_bow=bow_layer.ops(right_soft)

[0075] cos_sim_layer=layers.CosSimLayer()

[0076] pred=cos_sim_layer.ops(left_bow,right_bow)

[0077] The matching layer uses the text representation vector, namely the semantic vector, to perform interactive calculations and finally obtain the matching score.

[0078] In one embodiment, if Figure 3The structural diagram of the similarity deep neural network model shown in the figure uses the Pair-wise Ranking Loss function (pairwise sorting loss function) to train the Simnet model, and its training optimization goal is to make S(Q,D+)>S(Q,D-), where Q represents the search query text, D+ represents a related document, D- represents an irrelevant document, S(Q,D+) represents the matching score between documents D+ and Q, and S(Q,D-) represents the matching score between documents D- and Q. Documents D+, D- and Q are matched and calculated in the matching network. Preferably, a Hinge Loss function (loss function or objective function) based on Max-Margin (maximum interval algorithm) is used:

[0079]

[0080] The loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. This embodiment adopts the HingeLoss function (loss function or objective function) based on Max-Margin (maximum margin algorithm). This loss function is concise and efficient, and can also adjust the discrimination of the model score by setting different margins, and has good adaptability in different application scenarios.

[0081] In one embodiment, if Figure 4The schematic diagram of the structure of the multi-granularity segmentation model of words shown in the figure adopts the fusion and complementary training of word granularity and word granularity, inputs the mixed granularity word sequence to the input layer, or uses different granularity segmentation at the same time, and inputs different granularity word sequences to the input layer respectively. After the fusion and complementary training of word granularity and word granularity, the word granularity input can be close to the matching effect under the word granularity, so that the model can obtain good generalization and can make up for the shortcomings of word granularity input, such as word segmentation errors and unregistered words OOV (Out of Vocabulary, which refers to the abnormality that occurs when a word unknown to a known vocabulary is found). On the other hand, since there is no unique objective standard for word segmentation itself, different segmentation methods can actually complement each other, such as using large-granularity segmentation and fine-granularity segmentation at the same time. In this way, the accuracy requirements for single word segmentation can be less high, and in a sense, the semantic matching task is highly dependent on word segmentation. Therefore, the data processing flow input to the input layer may include the following situations: 1) The word sequences of segmentation granularity one and segmentation granularity two are input to the input layer as mixed granularity, and continue to be represented in the representation layer to construct semantic vectors, and are matched in the matching layer to obtain a matching score; 2) The word sequences of segmentation granularity one and segmentation granularity two are input to separate input layers respectively, and then representation vectors of segmentation granularity one and two are formed in the representation layer, and mixed granularity representation is performed. After the semantics is formed, matching is performed in the matching layer to obtain a matching score; 3) The word sequences of segmentation granularity one and segmentation granularity two are input to separate input layers respectively, and then semantic representation vectors of segmentation granularity one and two are formed in their respective representation layers, and the semantic vectors of segmentation granularity one and two are matched in their respective matching layers, and finally a common matching score is obtained.

[0082] In one embodiment, if Figure 2 Indicates that step S2 performs text clustering analysis based on the unfamiliar data obtained in step S1 to obtain the aggregated results of the user's question. Step S2 also includes the following steps:

[0083] S2.1: Obtain unprocessed unfamiliar data; and obtain features that conflict with known question data based on the obtained unfamiliar questions.

[0084] S2.2: Perform cross-similarity calculation on the unfamiliar data and perform text clustering processing; the text clustering processing is to regard the pairs with similarity scores higher than the specified threshold as the same group, and continue to perform cross-similarity calculation on the questions in the same group until there are no related similar questions. The clustering algorithm code is as follows:

[0085]

[0086] After processing in step S2.2, the data in the unfamiliar question database will form multiple similar question categories after text clustering analysis. The features of each category are extracted as hot words, and the hot words are sorted from high to low. By setting a limit on the number of new standard questions that can be generated, those that do not exceed the limit are temporarily regarded as special case data, stored in the database, and clustered and fitted with future unfamiliar data. Classifications that exceed the limit can be used as data to be labeled, that is, the hot questions with the highest ranking are stored in the database to be labeled. The top hot questions can be one or more top-ranked classification questions that exceed the limit.

[0087] S2.3: Calculate the average similarity of the user question data whose clustering categories exceed the set number, and obtain the questions that can represent the category as the standard questions to be determined for recommendation to the management personnel.

[0088] S2.4: Recommend the closest classification question as the extended standard question, which can be stored in the extended question database, completing the step of determining the new standard question. Based on the recommended extended new standard question, the knowledge base manager can include all the extended standard questions and all user questions under this category into the knowledge base.

[0089] In one embodiment, if Figure 5 As shown, the present invention also provides a system for determining a standard question based on similarity, including a determination device 100 and an expansion device 200, specifically:

[0090] The determining device 100 determines unfamiliar data that does not match the characteristics of existing standard questions. The determining device 100 includes a training module 101 and a reasoning module 102:

[0091] The training module 101 obtains the associated features from the preset standard question data and the annotated data it has obtained, which are used to train the similarity model. The annotated data that has been obtained is the data that can be used as new standard questions formed by multiple training of the similarity model and multiple operations of the system, and is used as training data in subsequent processes. The training module 101 trains the similarity model using the Simnet (SimilarityNet, text semantic matching / text similarity framework) text semantic matching model, including an input layer, a representation layer, and a matching layer; the input layer converts the text word sequence into a word embedding vector sequence by retrieving a vocabulary; the representation layer converts the word embedding vector sequence into a semantic vector; the matching layer uses the semantic vector for interactive calculation to obtain a matching score. The Simnet model is a supervised neural network semantic matching model. It follows the implicit continuous vector representation method in semantic representation, but performs end-to-end modeling of the semantic matching problem under the deep learning framework, unifying the word embedding vector representation and sentence semantic representation, the semantic vector representation and matching calculation, the text pair matching calculation and the supervised learning of the pair-wise algorithm in an overall framework.

[0092] The input layer converts the text word sequence into a word embedding vector sequence by searching the vocabulary. The main function of the representation layer is to construct the representation from word to sentence, or to convert the isolated word embedding vector representation of the sequence into one or more low-dimensional dense semantic vectors with global information. The accumulation method of the BOW (Bag of words) model is used to further improve the representation effect. The structure code of the BOW model is as follows:

[0093] emb_layer=layers.EmbeddingLayer(self.dict_size,self.emb_dim,"emb")

[0094] left_emb=emb_layer.ops(left)

[0095] right_emb=emb_layer.ops(right)

[0096] pool_layer=layers.SequencePoolLayer("sum")

[0097] left_pool=pool_layer.ops(left_emb)

[0098] right_pool=pool_layer.ops(right_emb)

[0099] softsign_layer=layers.SoftsignLayer()

[0100] left_soft=softsign_layer.ops(left_pool)

[0101] right_soft=softsign_layer.ops(right_pool)

[0102] bow_layer=layers.FCLayer(self.bow_dim,"relu","fc")

[0103] left_bow=bow_layer.ops(left_soft)

[0104] right_bow=bow_layer.ops(right_soft)

[0105] cos_sim_layer=layers.CosSimLayer()

[0106] pred=cos_sim_layer.ops(left_bow,right_bow)

[0107] The matching layer uses the text representation vector, namely the semantic vector, to perform interactive calculations and finally obtain the matching score.

[0108] In one embodiment, a pair-wise Ranking Loss function is used to train the Simnet model, and the training optimization goal is to make S(Q, D+)>S(Q, D-), where Q represents the search query text, D+ represents a related document, D- represents an irrelevant document, S(Q, D+) represents the matching score between document D+ and Q, and S(Q, D-) represents the matching score between document D- and Q. Preferably, a HingeLoss function (loss function or objective function) based on Max-Margin (maximum margin algorithm) is used:

[0109]

[0110] The loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. This embodiment adopts the HingeLoss function (loss function or objective function) based on Max-Margin (maximum margin algorithm). This loss function is concise and efficient, and can also adjust the discrimination of the model score by setting different margins, and has good adaptability in different application scenarios.

[0111] The reasoning module 102 obtains user question information, calculates the similarity between the user question and the existing training data according to the similarity model, and stores the questions with similarity below the specified threshold as data to be processed. Among them, the user's voice information is first obtained through the robot's human-computer interaction interface or telephone system, and the voice is converted into text based on the voice recognition technology, and stored in the log library as a human-computer interaction log. Since the question asked by the user is a question with multiple expressions, it is difficult to send the question directly to the background or as subsequent process data, so the user's question must be converted into a standard question that can be recognized by the system or subsequent process. However, the standard questions used in the knowledge base in the system or method may not be exhaustive, and various questions may occur in different scenarios, so the standard questions in the knowledge base also need to be continuously updated and expanded by determining new standard questions. When determining new standard questions, in order to avoid the occurrence of problems similar to existing standard questions and reduce repetition as much as possible, the present invention adopts a problem generation strategy that is opposite to the prior art, and generates unfamiliar data that is least similar to the existing standard questions to determine them as standard questions. That is, the similarity between the user's question and the existing training data is inferred based on the similarity model, and the questions with similarity lower than the specified threshold are stored as data to be processed, and unfamiliar data that does not match the characteristics of existing standard questions is determined.

[0112] In one embodiment, if Figure 1 As shown, the determining device 100 performs the following steps to determine the unfamiliar data:

[0113] S1.1: Use the word segmentation model to perform Chinese word segmentation on user questions, and use stop words to filter; this step obtains user questions in the interaction log database, generates a bag-of-words retrieval library based on the data in the standard question database and the extended question database, and uses the Jieba (Jieba, a Chinese word segmentation tool) Chinese word segmentation tool to segment all saved questions, and then uses the stop word list determined according to the specific business to filter stop words; obtain the association between words and questions, and store them in the memory for retrieval. Through screening, the questions that are determined as chat questions in intent recognition and the questions that already exist in the standard question database or the extended question database are filtered. Through the preprocessing of this step, user questions can be preliminarily screened and questions with no relevant word granularity can be filtered, which can greatly reduce the amount of calculation, reduce the calculation cost, and reduce memory consumption.

[0114] S1.2: Obtain word granularity data of the user's question, use the word data to search in the bag-of-words retrieval database, and obtain the data that needs to be compared with the user's question. Bag-of-words (BOW) is a commonly used document representation method in the field of information retrieval. A bag-of-words retrieval database is generated based on the data in the standard question database and the extended question database. In the bag-of-words retrieval database, data that can be compared with the words in the user's question is searched.

[0115] S1.3: Calculate the semantic similarity between the user question and each comparison data obtained, and sort them; after calculating the semantic similarity, sort the obtained similarity scores from high to low, and set a threshold to divide the similarity between the user question and the data in the bag-of-words database. The threshold can be adjusted dynamically as needed.

[0116] S1.4: Determine user questions below a specified threshold as unfamiliar data. Through these steps, data with low similarity to known data is screened out from the massive data of user questions. Because the unfamiliar data is not very similar to the known data, standard questions that are not included in the current knowledge base can be clustered from the unfamiliar data.

[0117] like Figure 4As shown in the figure, character granularity and word granularity are used for fusion and complementary training, and mixed granularity word sequences are input to the input layer, or different granularity word segmentation is used at the same time, and different granularity word sequences are input to the input layer respectively. After fusion and complementary training of character granularity and word granularity, character granularity input can be close to the matching effect under word granularity, so that the model can obtain good generalization and can make up for the shortcomings of word granularity input, such as word segmentation errors and unregistered words OOV (Out of Vocabulary, which refers to the abnormality that occurs when unknown words are found in a known vocabulary). On the other hand, since there is no unique objective standard for word segmentation itself, different segmentation methods can actually complement each other, such as using large-granularity word segmentation and fine-granularity word segmentation at the same time. In this way, the accuracy requirements for single word segmentation can be less high, and in a sense, the semantic matching task is highly dependent on word segmentation. Therefore, the data processing flow input to the input layer may include the following situations: 1) The word sequences of segmentation granularity one and segmentation granularity two are input to the input layer as mixed granularity, and continue to be represented in the representation layer to construct semantic vectors, and are matched in the matching layer to obtain a matching score; 2) The word sequences of segmentation granularity one and segmentation granularity two are input to separate input layers respectively, and then representation vectors of segmentation granularity one and two are formed in the representation layer, and mixed granularity representation is performed. After the semantics is formed, matching is performed in the matching layer to obtain a matching score; 3) The word sequences of segmentation granularity one and segmentation granularity two are input to separate input layers respectively, and then semantic representation vectors of segmentation granularity one and two are formed in their respective representation layers, and the semantic vectors of segmentation granularity one and two are matched in their respective matching layers, and finally a common matching score is obtained.

[0118] The expansion device 200 is used to expand the standard questions in the intelligent question answering system. The expansion device 200 includes a clustering module 201 and a human-machine collaboration module 202:

[0119] The clustering module 201 performs cluster analysis on the user questions in the data to be processed, extracts the features of each category as hot words, sorts the hot words from high to low, and stores the hot questions with the highest ranking in the database to be labeled. Since the unfamiliar data determined by the determination device 100 is most likely to form questions different from the existing standard questions, the clustering module 201 further performs text cluster analysis on these data, obtains classification results of multiple similar questions of the user questions, extracts the features of each category as hot words, sorts the hot words from high to low, and stores the hot questions with the highest ranking in the database to be labeled.

[0120] The human-computer collaboration module 202 calculates the data to be labeled using an average similarity algorithm and recommends the closest classification question as the extended standard question. The data stored in the database to be labeled can be used as a reserve of standard questions. According to the actual situation, the managers in each application scenario use the questions in the database to be labeled as extended questions, that is, the extended new standard questions, after further processing by the human-computer collaboration module 202, to apply to the intelligent question-answering system. The intelligent question-answering system may include a database for storing interaction logs, a standard question database, an extended question database, and an unfamiliar question database.

[0121] In one embodiment, if Figure 2 As shown, the expansion device 200 performs the following steps to perform text clustering analysis on the unfamiliar data obtained in step S1.4 to obtain the aggregation result of the user's question:

[0122] S2.1: Obtain unprocessed unfamiliar data; and obtain features that conflict with known question data based on the obtained unfamiliar questions.

[0123] S2.2: Perform cross-similarity calculation on the unfamiliar data and perform text clustering processing; the text clustering processing is to regard the pairs with similarity scores higher than the specified threshold as the same group, and continue to perform cross-similarity calculation on the questions in the same group until there are no related similar questions. The clustering algorithm code is as follows:

[0124]

[0125] After processing in step S2.2, the data in the unfamiliar question database will form multiple similar question categories after text clustering analysis. The features of each category are extracted as hot words, and the hot words are sorted from high to low. By setting a limit on the number of new standard questions that can be generated, those that do not exceed the limit are temporarily regarded as special case data, stored in the database, and clustered and fitted with future unfamiliar data. Classifications that exceed the limit can be used as data to be labeled, that is, the hot questions with the highest ranking are stored in the database to be labeled. The top hot questions can be one or more top-ranked classification questions that exceed the limit.

[0126] S2.3: Calculate the average similarity of the user question data whose clustering categories exceed the set number, and obtain the questions that can represent the category as the standard questions to be determined for recommendation to the management personnel.

[0127] S2.4: Recommend the closest classification question as the extended standard question, which can be stored in the extended question database, completing the step of determining the new standard question. Based on the recommended extended new standard question, the knowledge base manager can include all the extended standard questions and all user questions under this category into the knowledge base.

[0128] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0129] Although the spirit and principle of the present invention have been described with reference to the above specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for determining a standard problem based on similarity, characterized in that: The following steps are involved: S1: Obtain user question information, calculate the similarity between the user question and the existing training data according to the similarity model, store the questions with similarity lower than the specified threshold as the data to be processed, and determine the unfamiliar data that does not match the characteristics of the existing standard questions; S2: Perform cluster analysis on user questions in the data to be processed, extract the features of each category as hot words, sort the hot words from high to low, set a limit on the number of new standard questions, and temporarily define those that do not exceed the limit as special case data, store them in the database, and perform cluster fitting with future unfamiliar data. The categories that exceed the limit are regarded as data to be labeled, and store the hot questions with the highest ranking in the database to be labeled, which are expanded to be used as standard questions in the intelligent question-answering system. Wherein step S1 further comprises the following steps: S1.1: Use the word segmentation model to segment the user's question into Chinese words and use stop words to filter; S1.2: Obtain word granularity data of the user's question, search in the bag-of-words search library, and obtain the data that needs to be compared for the user's question; S1.3: Calculate the semantic similarity between the user's question and each comparison data obtained, and sort them; S1.4: Determine user questions below a specified threshold as unfamiliar data; and Wherein step S2 further comprises the following steps: S2.1: Obtain unprocessed unfamiliar data; S2.2: Perform cross similarity calculation on the unfamiliar data and perform text clustering processing; S2.3: Calculate the average similarity of the user question data whose clustering categories exceed the set number; S2.4: Recommend the closest classification question as the expanded standard question.

2. The method for determining standard questions based on similarity as claimed in claim 1, characterized in that: The similarity model adopts the Simnet text semantic matching model, including an input layer, a representation layer and a matching layer; the input layer converts a text word sequence into a word embedding vector sequence by searching a vocabulary; the representation layer converts the word embedding vector sequence into a semantic vector; the matching layer uses the semantic vector for interactive calculation to obtain a matching score.

3. The method for determining standard questions based on similarity as claimed in claim 2, characterized in that: The pair-wiseRanking Loss function is used to train the Simnet model, and its training optimization goal is to make S(Q,D+)>S(Q,D-), where Q represents the search query text, D+ represents a related document, D- represents an irrelevant document, S(Q,D+) represents the matching score between document D+ and Q, and S(Q,D-) represents the matching score between document D- and Q.

4. The method for determining standard questions based on similarity as claimed in claim 2, characterized in that: Use character granularity and word granularity for fusion and complementary training, input mixed granularity word sequences to the input layer, or use different granularity word segmentation at the same time, and input different granularity word sequences to the input layer respectively.

5. A system for determining standard questions based on similarity, characterized in that: include: The determining device determines unfamiliar data that does not match the characteristics of existing standard questions, comprising: A training module obtains associated features from the preset standard question data and the acquired annotation data for training the similarity model; The inference module obtains user question information, calculates the similarity between the user question and the existing training data based on the similarity model, and stores the questions with similarity below the specified threshold as data to be processed; Extension device, which is used to extend the standard questions in the intelligent question answering system, including: The clustering module performs cluster analysis on the user questions in the data to be processed, extracts the features of each category as hot words, sorts the hot words from high to low, and stores the hot questions with the highest ranking in the database to be labeled; The human-computer collaboration module calculates the data to be labeled using the average similarity algorithm and recommends the closest classification question as the extended standard question. The reasoning module determines the unfamiliar data by the following steps: S1.1: Use the word segmentation model to segment the user's question into Chinese words and use stop words to filter; S1.2: Obtain word granularity data of the user's question, search in the bag-of-words search library, and obtain the data that needs to be compared for the user's question; S1.3: Calculate the semantic similarity between the user's question and each comparison data obtained, and sort them; S1.4: Determine user questions below a specified threshold as unfamiliar data; and The expansion device expands the standard question by the following steps: S2.1: Based on unprocessed unfamiliar data; S2.2: Perform cross similarity calculation on the unfamiliar data and perform text clustering processing; S2.3: Calculate the average similarity of the user question data whose clustering categories exceed the set number; S2.4: Recommend the closest classification question as the expanded standard question.

6. The system for determining standard questions based on similarity as claimed in claim 5, characterized in that: The training module trains the similarity model using the Simnet model, which includes an input layer, a representation layer and a matching layer; the input layer converts a text word sequence into a word embedding vector sequence by searching a vocabulary; the representation layer converts the word embedding vector sequence into a semantic vector; the matching layer uses the semantic vector for interactive calculation to obtain a matching score.

Images