Data attribute filling method, device, equipment and computer-readable storage medium
Through preset model and graph structure clustering technology, the data attributes in the dialogue log are automatically filled, which solves the problem of low data attribute filling efficiency in the existing technology, and achieves efficient and accurate attribute supplementation.
Patent Information
- Application Number
- CN202010088080.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-02-12
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-02-12
AI Technical Summary
In the large-scale dialogue log mining scenario, the data attribute filling efficiency is low, and it is impossible to supplement knowledge attributes in batches through one operation, and it is prone to errors and takes a long time.
The initial question data to be answered is predicted through the preset model set, the knowledge owner and the corresponding knowledge base are obtained, and the comprehensive similarity between the initial question data to be answered and the historical question data is calculated. If it is greater than the threshold, it is aggregated in the graph structure. Class processing, automatically populate according to the clustering results and attributes of historical problem data.
It realizes efficient automatic filling of data attributes, improves filling efficiency, reduces manual intervention, and reduces error rate.
Smart Images

Figure CN111339248B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data attribute filling method, device, equipment and computer-readable storage medium. Background Art
[0002] At present, the attributes of data are generally filled in by literal similarity clustering, which is not suitable for large-scale conversation log mining scenarios with multiple knowledge owners. It is impossible to batch supplement the corresponding knowledge attributes with a single operation. Instead, it is necessary to manually set attributes for questions raised by users, which is time-consuming and prone to errors. In addition, the efficiency of attribute filling is low. How to efficiently and automatically fill in data attributes in big data scenarios is a technical problem that needs to be solved urgently in this field. Summary of the invention
[0003] The main purpose of the present invention is to provide a data attribute filling method, device, equipment and computer-readable storage medium, aiming to solve the technical problem of low efficiency of data attribute filling.
[0004] To achieve the above object, the present invention provides a data attribute filling method, the data attribute filling method comprising the following steps:
[0005] Predict the initial question data to be answered using the preset model set to obtain the target prediction result;
[0006] Based on the target prediction result, obtaining the knowledge owner to which the initial question data to be answered belongs, and determining the knowledge base corresponding to the initial question data to be answered according to the knowledge owner;
[0007] Calculating the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base;
[0008] Determining whether the similarity is greater than or equal to a first preset threshold;
[0009] If the similarity is greater than or equal to a first preset threshold, the initial question data to be answered is input into each node of a preset graph G=(V, E), the weight of the initial question data to be answered is determined according to the degree in the graph, and the initial question data to be answered is clustered based on the weight to obtain a clustering result, wherein the question data has the highest weight in the clustering result, and the rest are similar question data, V is a node set, E is an edge set, and the similar question data is data that has a similar relationship with the question data;
[0010] Determining whether the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold;
[0011] If the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold, the attributes of the clustering result are determined, and the attributes are used to fill the clustering result with attributes.
[0012] Optionally, before predicting the initial unanswered question data by using the preset model set to obtain the target prediction result, the following steps are also included:
[0013] Removing punctuation marks from the first initial data set of questions to be answered by using a regular expression to obtain a second initial data set of questions to be answered;
[0014] Performing synonym conversion on the second initial set of questions to be answered by using a preset synonym conversion method to obtain a third initial set of questions to be answered;
[0015] The library function is called to perform literal deduplication processing on the third initial data set of questions to be answered to obtain a target data set of questions to be answered, wherein the target data set of questions to be answered includes at least one initial data set of questions to be answered.
[0016] Optionally, the calling library function performs literal deduplication processing on the third initial question data set to be answered to obtain a target question data set to be answered, wherein the target question data set to be answered includes at least one initial question data set to be answered, comprising the following steps:
[0017] Sort each third initial question data to be answered in the third initial question data set by sentence length by calling a quick sorting algorithm in a library function to obtain a sorted third initial question data set to be answered;
[0018] The sorted third initial data set of questions to be answered is traversed, repeated words are removed, and a target data set of questions to be answered is obtained.
[0019] Optionally, performing synonym conversion on the second initial set of questions to be answered by using a preset synonym conversion method to obtain a third initial set of questions to be answered includes the following steps:
[0020] Performing word segmentation on the second initial set of questions to be answered to obtain word segmentation data;
[0021] Obtaining a feature vector of the word segmentation data, and calculating a cosine angle value between the feature vector and a feature vector of each word in a preset word library;
[0022] Determine whether the cosine angle value is less than a preset angle value;
[0023] If the cosine angle value is less than the preset angle value, then the synonym data of each word in the preset vocabulary is obtained, and the synonym data is used to form a third initial question data set to be answered;
[0024] If the cosine angle value is greater than or equal to the preset angle value, the step of determining whether the cosine angle value is less than the preset angle value is continued until the cosine angle value meets the preset angle value.
[0025] Optionally, predicting the initial to-be-answered question data by using a preset model set to obtain a target prediction result comprises the following steps:
[0026] Predicting the initial question data to be answered by using the language representation BERT model in the preset model set, and determining whether the initial question data to be answered belongs to a valid type;
[0027] If the initial question data to be answered belongs to a valid type, obtaining a valid type prediction result;
[0028] Predicting the initial question data to be answered by using a text classification textcnn model in a preset model set, and determining whether the initial question data to be answered belongs to a chat type;
[0029] If the initial question data to be answered belongs to the chat type, obtaining a chat type prediction result;
[0030] The effective class prediction result and the chat class prediction result are combined to obtain a target prediction result.
[0031] Optionally, if the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold, determining the attribute of the clustering result, and using the attribute to fill the attribute of the clustering result includes:
[0032] If the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold, acquiring an attribute set of the clustering result based on a preset mapping relationship between the attributes of the historical question data and the attributes of the clustering result, the attribute set of the clustering result including at least one attribute of the clustering result;
[0033] Frequent item sets in the attribute set of the clustering result are mined, and attributes of the clustering result are determined based on the frequent item sets.
[0034] Optionally, the calculating of the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base comprises the following steps:
[0035] Calculate the literal similarity between the initial to-be-answered question data and the historical question data in the knowledge base by using term frequency-inverse document frequency TF-IDF;
[0036] Calculating the semantic similarity between the initial to-be-answered question data and the historical question data in the knowledge base through the twin network;
[0037] The literal similarity and the semantic similarity values are prioritized according to the similarity values to obtain a comprehensive similarity.
[0038] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a data attribute filling device, the data attribute filling device comprising the following modules:
[0039] A prediction module, used to predict the initial question data to be answered by using a preset model set to obtain a target prediction result;
[0040] A classification module, configured to obtain, based on the target prediction result, a knowledge owner to which the initial question data to be answered belongs, and determine, according to the knowledge owner, a knowledge base corresponding to the initial question data to be answered;
[0041] An identification module, used to calculate the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base;
[0042] A similarity determination module, used to determine whether the similarity is greater than or equal to a first preset threshold;
[0043] A clustering module, for inputting the initial question data to be answered into each node of a preset graph G=(V, E) if the similarity is greater than or equal to a first preset threshold, determining the weight of the initial question data to be answered according to the degree in the graph, and clustering the initial question data to be answered based on the weight to obtain a clustering result, wherein the question data with the highest weight in the clustering result is the question data, and the rest are similar question data, V is a node set, E is an edge set, and the similar question data is data having a similar relationship with the question data;
[0044] A matching degree judgment module, used to judge whether the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold;
[0045] A filling module is used to determine the attributes of the clustering result and use the attributes to fill the clustering result if the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold.
[0046] Optionally, the data attribute filling device further includes the following modules:
[0047] a punctuation mark removal module, configured to remove punctuation marks in the first initial data set of questions to be answered by using a regular expression, so as to obtain a second initial data set of questions to be answered;
[0048] a synonym conversion module, configured to perform synonym conversion on the second initial set of questions to be answered by using a preset synonym conversion method to obtain a third initial set of questions to be answered;
[0049] The literal deduplication module is used to call a library function to perform literal deduplication processing on the third initial question data set to be answered to obtain a target question data set to be answered, wherein the target question data set to be answered includes at least one initial question data to be answered.
[0050] Optionally, the literal deduplication module includes the following units:
[0051] A sorting unit, configured to sort each third initial question data to be answered in the third initial question data set according to sentence length by calling a quick sorting algorithm in a library function to obtain a sorted third initial question data set to be answered;
[0052] The word deduplication unit is used to traverse the sorted third initial data set of questions to be answered, remove repeated words, and obtain a target data set of questions to be answered.
[0053] Optionally, the synonym conversion module includes the following units:
[0054] A word segmentation unit, used for performing word segmentation on the second initial set of questions to be answered to obtain word segmentation data;
[0055] A cosine angle value calculation unit, used to obtain a feature vector of the word segmentation data, and calculate a cosine angle value between the feature vector and a feature vector of each word in a preset word library;
[0056] A cosine angle value judgment unit, used to judge whether the cosine angle value is less than a preset angle value;
[0057] A synonym data acquisition unit is used to obtain synonym data of each word in the preset vocabulary if the cosine angle value is less than a preset angle value, and to form a third initial set of questions to be answered with the synonym data; if the cosine angle value is greater than or equal to the preset angle value, continue to execute the step of determining whether the cosine angle value is less than the preset angle value until the cosine angle value meets the preset angle value.
[0058] Optionally, the prediction module includes the following units:
[0059] A valid type prediction unit, used to predict the initial question data to be answered by using a language representation BERT model in a preset model set, and determine whether the initial question data to be answered belongs to a valid type;
[0060] A valid class prediction result acquisition unit, configured to acquire a valid class prediction result if the initial question data to be answered belongs to a valid type;
[0061] A chat type prediction unit, used to predict the initial question data to be answered by using a text classification textcnn model in a preset model set, and determine whether the initial question data to be answered belongs to the chat type;
[0062] A chat prediction result acquisition unit, configured to acquire a chat prediction result if the initial question data to be answered belongs to the chat type;
[0063] The prediction result combination unit is used to combine the effective class prediction result and the chat class prediction result to obtain a target prediction result.
[0064] Optionally, the filling module includes:
[0065] an attribute set acquisition unit for clustering results, configured to acquire an attribute set for clustering results based on a preset mapping relationship between the attribute of the historical question data and the attribute of the clustering results if the matching degree between the attribute of the historical question data in the knowledge base and the clustering results is greater than or equal to a second preset threshold, wherein the attribute set for clustering results includes at least one attribute of the clustering results;
[0066] The frequent item set mining unit is used to mine the frequent item sets in the attribute set of the clustering result, and determine the attribute of the clustering result based on the frequent item sets.
[0067] Optionally, the identification module includes the following units:
[0068] A word similarity calculation unit, used for calculating the word similarity between the initial to-be-answered question data and the historical question data in the knowledge base through term frequency-inverse document frequency TF-IDF;
[0069] A semantic similarity calculation unit, used to calculate the semantic similarity between the initial to-be-answered question data and the historical question data in the knowledge base through a twin network;
[0070] The similarity acquisition unit is used to prioritize the literal similarity and the semantic similarity values according to the magnitude of the similarity values to obtain a comprehensive similarity.
[0071] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a data attribute filling device, which includes a memory, a processor, and a data attribute filling program stored in the memory and executable on the processor, and the data attribute filling program, when executed by the processor, implements the steps of the data attribute filling method as described in any one of the above items.
[0072] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a data attribute filling program is stored, and when the data attribute filling program is executed by a processor, the steps of the data attribute filling method as described in any one of the above items are implemented.
[0073] The question data set is clustered through the graph to separate the question data and similar question data. The question data and similar question data have the same attributes, and each attribute corresponds to the corresponding knowledge base. Different data are stored in the corresponding knowledge base according to the attributes. Then, the language representation bert model is used to predict the fluency of the unanswered question and whether it is a valid sentence. The binary classification model trained by the text classification textcnn model is used to identify whether it is a chat, and the valid non-chat part is taken. Secondly, the literal similarity and semantic similarity are calculated respectively through the twin network and word frequency-inverse document frequency TF-IDF, and then the question data that meets the similarity are clustered, and the attributes with a high matching degree with the clustering results are selected from the knowledge base to achieve the purpose of quickly filling the attributes of the data. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 A schematic diagram of the structure of a data attribute filling device operating environment involved in an embodiment of the present invention;
[0075] Figure 2 It is a flowchart of a first embodiment of a method for filling data attributes of the present invention;
[0076] Figure 3 This is a flow chart of a second embodiment of a method for filling in data attributes according to the present invention;
[0077] Figure 4 for Figure 3 A detailed flow chart of an embodiment of step S103;
[0078] Figure 5 for Figure 3 A detailed flow chart of an embodiment of step S102;
[0079] Figure 6 for Figure 2 A detailed flow chart of an embodiment of step S10;
[0080] Figure 7 for Figure 2 A detailed flow chart of an embodiment of step S70;
[0081] Figure 8 for Figure 2 A detailed flow chart of step S30 in an embodiment;
[0082] Fig. 9 This is a schematic diagram of the functional modules of an embodiment of the data attribute filling device of the present invention.
[0083] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0084] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0085] The invention provides a data attribute filling device.
[0086] Reference Figure 1 , Figure 1 A schematic diagram of the structure of the data attribute filling device operating environment involved in the embodiment of the present invention.
[0087] like Figure 1 As shown, the data attribute filling device includes: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or it can be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.
[0088] Those skilled in the art will understand that Figure 1 The hardware structure of the data attribute filling device shown in the figure does not constitute a limitation on the data attribute filling device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0089] like Figure 1As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a data attribute filling program. The operating system is a program that manages and controls the data attribute filling device and software resources, and supports the operation of the data attribute filling program and other software and / or programs.
[0090] exist Figure 1 In the hardware structure of the data attribute filling device shown, the network interface 1004 is mainly used to access the network; the user interface 1003 is mainly used to detect confirmation instructions and editing instructions, etc. The processor 1001 can be used to call the data attribute filling program stored in the memory 1005 and perform the operations of the following data attribute filling method embodiments.
[0091] Based on the above data attribute filling device hardware structure, various embodiments of the data attribute filling method of the present invention are proposed.
[0092] Reference Figure 2 , Figure 2 1 is a flow chart of a first embodiment of a method for filling data attributes of the present invention. In this embodiment, the method for filling data attributes includes the following steps:
[0093] Step S10, predicting the initial question data to be answered by using a preset model set to obtain a target prediction result;
[0094] In this embodiment, a pre-trained prediction model in a preset model set is used to predict the initial data to be answered. The preset model set is used to predict the data to be answered to obtain a valid class prediction result. For example, the preset model set may include: a language representation BERT model and a text classification TextCNN model. Then, different models are used to predict the initial data to be answered to obtain a prediction result. The prediction result may be that the initial data to be answered belongs to the chat class or the valid class.
[0095] Step S20, based on the target prediction result, obtaining the knowledge owner to which the initial question data to be answered belongs, and determining the knowledge base corresponding to the initial question data to be answered according to the knowledge owner;
[0096] In this embodiment, the nature and relationship of things are called the attributes of things. For example, the insurance amount, insurance policy number and insured person can be classified as the attribute of "insurance". In this embodiment, the knowledge owner to which the initial question data to be answered belongs refers to the category to which the predicted initial question data to be answered belongs. Before this, different knowledge bases with different classifications have been pre-set, and there is a mapping relationship between the knowledge base and the initial question data to be answered with different knowledge owners. Therefore, after obtaining the knowledge owner to which the initial question data to be answered belongs, the initial question data to be answered can be distributed to the corresponding knowledge base according to the mapping relationship.
[0097] Step S30, calculating the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base;
[0098] In this embodiment, after the initial question data to be answered with different knowledge owners are distributed to the corresponding knowledge base, it is necessary to calculate the similarity between the distributed initial question data to be answered and the historical question data. The purpose of calculating the similarity is to obtain other data that have an approximate relationship with the current initial question data to be answered. The other data may include literal similarity. For example, the current initial question data to be answered has "insurance" appearing many times, and there is also a historical question data in which "insurance" appears many times in the corresponding knowledge base, which means that there is a certain similarity between the two data. In order to calculate this similarity, a preset similarity algorithm can be used for calculation, for example, word frequency-inverse file frequency.
[0099] Step S40, determining whether the similarity is greater than or equal to a first preset threshold;
[0100] In this embodiment, since there may be multiple historical question data similar to the current initial question data to be answered in the knowledge base, and those multiple historical question data may not all meet the preset similarity, a first preset threshold is set in advance. The value of the first preset threshold is not limited, for example, it can be 90%.
[0101] Step S50: if the similarity is greater than or equal to the first preset threshold, the initial question data to be answered is input into each node of the preset G=(V, E), the weight of the initial question data to be answered is determined according to the degree in the graph, and the initial question data to be answered is clustered based on the weight to obtain a clustering result, wherein the question data with the highest weight in the clustering result is the question data, and the rest are similar question data, V is a node set, E is an edge set, and similar question data is data that has a similar relationship with the question data;
[0102] In this embodiment, a graph is composed of a finite non-empty set of vertices and a set of edges between vertices, which can be expressed as G=(V,E), where V is a set of nodes and E is a set of edges. In this embodiment, a point is each initial question data to be answered, and an edge is the similarity of each initial question data to be answered. The point with the largest degree in the graph, that is, the point with the most central position, is taken as a representative, i.e., the historical question data, and the degree here is the weight of each point.
[0103] Step S60, determining whether the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold;
[0104] In this embodiment, there may be a one-to-one mapping relationship or a one-to-many mapping relationship between the clustering results and the attributes of the historical question data, and these mapping relationships are all preset.
[0105] Step S70: If the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to the second preset threshold, the attributes of the clustering result are determined, and the attributes are used to fill the clustering result with attributes.
[0106] In this embodiment, one knowledge owner corresponds to one knowledge base; there are multiple historical problem data under one knowledge base, and the historical problem data have different attributes. When the matching degree between the attributes of the historical problem data in the knowledge base and the clustering results is greater than or equal to the second preset threshold, the attributes of the historical problems can be filled into the clustering results. The specific filling method is to pre-establish an attribute table to be filled, and when the matching degree is greater than or equal to the second preset threshold, the corresponding attributes are mapped to the attribute table to be filled.
[0107] First, the problem data set is clustered through a graph to separate problem data and similar problem data. Problem data and similar problem data have the same attributes, and each attribute has a corresponding relationship with the corresponding knowledge base. Different data are stored in the corresponding knowledge base according to the attributes, and attributes with a high degree of match with the clustering results are selected from the knowledge base to fill in the problem attributes.
[0108] Reference Figure 3 , Figure 3 1 is a flow chart of a second embodiment of the data attribute filling method of the present invention. In this embodiment, before the initial unanswered question data is predicted by the preset model set in step S10 and the target prediction result is obtained, the data attribute filling method includes the following steps:
[0109] Step S80, removing punctuation marks in the first initial data set of questions to be answered by using a regular expression to obtain a second initial data set of questions to be answered;
[0110] In this embodiment, punctuation marks in the problem data are removed by using regular expressions to obtain punctuation mark-removed problem data.
[0111] Step S90, performing synonym conversion on the second initial data set of questions to be answered by using a preset synonym conversion method to obtain a third initial data set of questions to be answered;
[0112] In this embodiment, a character string is first searched to find different characters or words, and then replaced, similar to a dictionary search, for example, eShengbao->eShengbao; EShenbao->eShengbao, which mainly implements the functions of voice error correction and unified description of insurance types.
[0113] Step S100, calling a library function to perform literal deduplication processing on the third initial data set of questions to be answered, to obtain a target data set of questions to be answered, wherein the target data set of questions to be answered includes at least one initial data set of questions to be answered.
[0114] In this embodiment, the deduplication action is performed through the library function to obtain the literal deduplication problem data. The library function refers to a method of putting functions into a library for use. The method is to compile some commonly used functions and put them into a file for calling.
[0115] The third initial data set of questions to be answered is traversed in sequence to determine whether there are question data with the same words. After the repeated words in a single question are removed, there may still be other question data with the same words, for example, "Can coal miners buy e-life insurance?" appears continuously. If there are question data with the same words, only one question data is saved to obtain the target data set of questions to be answered. In order to avoid duplicate data, only one question data is saved, that is, each initial question data to be answered in the target data set of questions to be answered is unique and different.
[0116] Reference Figure 4 , Figure 4 for Figure 3 In this embodiment, step S100, calling a library function to perform literal deduplication processing on the third initial question data set to be answered to obtain a target question data set to be answered, the target question data set to be answered includes at least one initial question data to be answered, including the following steps:
[0117] Step S1001, sorting each third initial question data to be answered in the third initial question data set according to sentence length by calling a quick sorting algorithm in a library function to obtain a sorted third initial question data set to be answered;
[0118] In this embodiment, the data to be sorted is divided into two independent parts by sorting, wherein the length of all data in one part is smaller than the length of all data in the other part, and then the two parts of data are quickly sorted separately according to this method. The whole sorting process can be performed recursively, so that the whole data can be turned into an ordered sequence.
[0119] Step S1002, traverse the sorted third initial data set of questions to be answered, remove repeated words, and obtain a target data set of questions to be answered.
[0120] In this embodiment, when traversing the sorted data, the two parts of the sorted data can be traversed at the same time, so that it is possible to promptly identify whether there are repeated words. If there are repeated words, they are removed to obtain question data with literal deduplication, that is, the initial question data to be answered.
[0121] Reference Figure 5 , Figure 5 for Figure 3 In this embodiment, step S90, performing synonym conversion on the second initial data set of questions to be answered by using a preset synonym conversion method to obtain a third initial data set of questions to be answered, includes the following steps:
[0122] Step S901, segmenting the second initial data set of questions to be answered to obtain segmented data;
[0123] In this embodiment, the punctuation problem data can be segmented by using the Jieba word segmentation method to obtain the segmented data.
[0124] Step S902, obtaining a feature vector of the word segmentation data, and calculating a cosine angle value between the feature vector and the feature vector of each word in the preset word library;
[0125] In this embodiment, the word segmentation data is first converted into a feature vector form, and then the cosine angle value between the feature vectors is calculated using the cosine formula. The smaller the cosine angle value, the more similar the feature vectors are.
[0126] Step S903, determining whether the cosine angle value is less than a preset angle value;
[0127] In this embodiment, in order to obtain a cosine angle value that meets a preset condition, it is necessary to set a preset angle value, for example, 20°.
[0128] Step S904: if the cosine angle value is less than the preset angle value, then obtain the synonym data of each word in the preset vocabulary, and form the synonym data into a third initial set of question data to be answered; if the cosine angle value is greater than or equal to the preset angle value, then return to step S903.
[0129] In this embodiment, due to the constraint of the preset angle value, data less than the preset angle value can be obtained. If it is greater than or equal to the preset angle value, it is necessary to calculate the cosine angle value between the feature vector of the word segmentation data and the feature vectors of other words in the preset word library.
[0130] Reference Figure 6 , Figure 6 for Figure 2 Detailed flow chart of an embodiment of step S10 in the embodiment. In this embodiment, step S10 predicts the initial unanswered question data by using the preset model set to obtain the target prediction result, including the following steps:
[0131] Step S101, predicting the initial question data to be answered by using the language representation BERT model in the preset model set, and determining whether the initial question data to be answered belongs to a valid type;
[0132] In this embodiment, in order to make the language representation BERT model be able to generate valid question data, it is necessary to train the language representation BERT model before this. That is, the initial BERT model is trained using training sample data of known valid types and invalid types until the initial BERT model can accurately identify the valid type of initial question data to be answered.
[0133] Step S102, if the initial question data to be answered belongs to a valid type, obtaining a valid type prediction result;
[0134] In this embodiment, it can be known from step S101 that the language representation BERT model can identify whether the initial question data to be answered is of a valid type, and then the validity type prediction result can be obtained. The purpose of identifying the initial question data to be answered through the BERT model is to distinguish valid questions from all the initial question data to be answered. For example, valid questions must belong to a certain knowledge base. If the current scenario is a human-computer question-and-answer scenario of purchasing insurance, if the question data of purchasing fruit is received, it is invalid data.
[0135] Step S103, predicting the initial question data to be answered by using the text classification textcnn model in the preset model set, and determining whether the initial question data to be answered belongs to the chat type;
[0136] In this embodiment, the initial text classification model must also be trained for predicting the chat type. Only after the training is completed, that is, after a certain accuracy rate is achieved, can the initial question data to be answered be predicted.
[0137] Step S104, if the initial question data to be answered belongs to the chat type, obtain the chat type prediction result;
[0138] In this embodiment, the text classification textcnn model is trained by pre-set small talk training samples and non-small talk training samples, so that it has the ability to recognize. For example, in a human-computer question-and-answer scenario for purchasing insurance, the received laughter data or sigh data can be used as small talk type data.
[0139] Step S105, combining the effective class prediction result and the chat class prediction result to obtain the target prediction result.
[0140] In this embodiment, the chat class prediction result and the effective class prediction result are taken as components of the effective class prediction result.
[0141] Reference Figure 7 , Figure 7 for Figure 2 In the embodiment of step S70, in step S70, if the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to the second preset threshold, the attribute of the clustering result is determined, and the attribute is used to fill the clustering result with the attribute, including the following steps:
[0142] Step S701, if the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold, then obtaining an attribute set of the clustering result based on a preset mapping relationship between the attributes of the historical question data and the attributes of the clustering result, the attribute set of the clustering result including at least one attribute of the clustering result;
[0143] Step S702: mining frequent item sets in the attribute set of the clustering result, and determining the attribute of the clustering result based on the frequent item sets.
[0144] In this embodiment, the attributes that frequently appear in the attribute set of the clustering result can be mined through the big data mining platform. The standard that can be used as a frequent item set can be pre-set, for example, if the current attribute appears three or more times, the attribute can be set as a frequent item set.
[0145] Reference Figure 8 , Figure 8 for Figure 2 In this embodiment, step S30, calculating the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base, includes the following steps:
[0146] Step S301, calculating the literal similarity between the initial to-be-answered question data and the historical question data in the knowledge base by using term frequency-inverse document frequency TF-IDF;
[0147] In this embodiment, word segmentation is first performed through jieba and organized into a specified format. Then, the gensim library converts the data of each question to be compared into a sparse vector through doc2bow. Secondly, the corpus is processed through term frequency-inverse document frequency TF-IDF, and then the eigenvalue and sparse matrix similarity are used to establish an index. Finally, the literal similarity between the data of each question to be answered is obtained.
[0148] Step S302, calculating the semantic similarity between the initial unanswered question data and the historical question data in the knowledge base through the twin network;
[0149] In this embodiment, the twin network is composed of two networks with the same structure and shared parameters. This model is selected when two sentences come from the same field and have great structural similarity. The spatial similarity between the two sentences is measured by calculating Manhattan distance, Euclidean distance, cosine similarity, etc., and then the semantic similarity is obtained.
[0150] Step S303 , prioritizing the literal similarity and semantic similarity values respectively according to the magnitude of the similarity values to obtain a comprehensive similarity.
[0151] In this embodiment, in order to obtain more accurate attributes, it is necessary to obtain literal similarity and semantic similarity. Only by combining the literal similarity and semantic similarity together can the similarity be obtained. When calculating the similarity between the initial question data to be answered and the historical question data, and judging whether the similarity is greater than or equal to the first preset threshold, both the literal similarity and the semantic similarity are required to be greater than or equal to the first preset threshold before the initial question data to be answered can be input into each node of the graph G = (V, E).
[0152] First, the question data set is clustered through a graph to separate question data and similar question data. Question data and similar question data have the same attributes, and each attribute corresponds to the corresponding knowledge base. Different data are stored in the corresponding knowledge base according to the attributes. Then, the BERT model is used to predict the fluency of unanswered questions and whether they are valid sentences. The binary classification model trained by the TextCNN model is used to identify whether it is idle chat and take the valid non-idle chat part. Secondly, the twin network and word frequency-inverse document frequency TF-IDF respectively calculate the literal similarity and semantic similarity, and then cluster the question data that meet the similarity, and select the attributes with a high match with the clustering results from the knowledge base to fill in the attributes of the question.
[0153] Reference Fig. 9 , Fig. 9 This is a functional module diagram of an embodiment of a data attribute filling device of the present invention. In this embodiment, the data attribute filling device includes:
[0154] A prediction module 10 is used to predict the initial question data to be answered by using a preset model set to obtain a target prediction result;
[0155] A classification module 20, configured to obtain the knowledge owner to which the initial question data to be answered belongs based on the target prediction result, and determine a knowledge base corresponding to the initial question data to be answered according to the knowledge owner;
[0156] The identification module 30 is used to calculate the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base;
[0157] A similarity determination module 40 is used to determine whether the similarity is greater than or equal to a first preset threshold;
[0158] A clustering module 50 is used for inputting the initial question data to be answered into each node of a preset graph G=(V, E) if the similarity is greater than or equal to a first preset threshold value, determining the weight of the initial question data to be answered according to the degree in the graph, and clustering the initial question data to be answered based on the weight to obtain a clustering result, wherein the question data with the highest weight in the clustering result is the question data, and the rest are similar question data, V is a node set, E is an edge set, and the similar question data is data having a similar relationship with the question data;
[0159] A matching degree judgment module 60, used to judge whether the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold;
[0160] The filling module 70 is used to determine the attributes of the clustering result and use the attributes to fill the clustering result if the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold.
[0161] In this embodiment, the purpose of obtaining multiple attributes can be achieved through one operation through the modules in the device, thereby improving the efficiency of classifying different initial unanswered question data into different attributes.
[0162] The present invention also provides a computer-readable storage medium.
[0163] In this embodiment, a data attribute filling program is stored on the computer-readable storage medium, and when the data attribute filling program is executed by the processor, the steps of the data attribute filling method described in any of the above embodiments are implemented.
[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0165] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.
Claims
1. A data attribute filling method, characterized in that: The data attribute filling method comprises the following steps: Predict the initial question data to be answered using the preset model set to obtain the target prediction result; Based on the target prediction result, obtaining the knowledge owner to which the initial question data to be answered belongs, and determining the knowledge base corresponding to the initial question data to be answered according to the knowledge owner; Calculating the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base; Determining whether the similarity is greater than or equal to a first preset threshold; If the similarity is greater than or equal to a first preset threshold, the initial question data to be answered is input into each node of a preset graph G=(V, E), the weight of the initial question data to be answered is determined according to the degree in the graph, and the initial question data to be answered is clustered based on the weight to obtain a clustering result, wherein the question data has the highest weight in the clustering result, and the rest are similar question data, V is a node set, E is an edge set, and the similar question data is data that has a similar relationship with the question data; Determining whether the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold; If the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold, the attributes of the clustering result are determined, and the attributes are used to fill the clustering result with attributes.
2. The data attribute filling method according to claim 1, characterized in that: Before predicting the initial to-be-answered question data by using the preset model set to obtain the target prediction result, the following steps are also included: Removing punctuation marks from the first initial data set of questions to be answered by using a regular expression to obtain a second initial data set of questions to be answered; Performing synonym conversion on the second initial set of questions to be answered by using a preset synonym conversion method to obtain a third initial set of questions to be answered; The library function is called to perform literal deduplication processing on the third initial data set of questions to be answered to obtain a target data set of questions to be answered, wherein the target data set of questions to be answered includes at least one initial data set of questions to be answered.
3. The data attribute filling method according to claim 2, characterized in that: The calling library function performs literal deduplication processing on the third initial data set of questions to be answered to obtain a target data set of questions to be answered, comprising the following steps: Sort each third initial question data to be answered in the third initial question data set by sentence length by calling a quick sorting algorithm in a library function to obtain a sorted third initial question data set to be answered; The sorted third initial data set of questions to be answered is traversed, repeated words are removed, and a target data set of questions to be answered is obtained.
4. The data attribute filling method according to claim 2, characterized in that: The step of performing synonym conversion on the second initial set of questions to be answered by using a preset synonym conversion method to obtain a third initial set of questions to be answered includes the following steps: Performing word segmentation on the second initial set of questions to be answered to obtain word segmentation data; Obtaining a feature vector of the word segmentation data, and calculating a cosine angle value between the feature vector and a feature vector of each word in a preset word library; Determine whether the cosine angle value is less than a preset angle value; If the cosine angle value is less than the preset angle value, then the synonym data of each word in the preset vocabulary is obtained, and the synonym data is used to form a third initial question data set to be answered; If the cosine angle value is greater than or equal to the preset angle value, the step of determining whether the cosine angle value is less than the preset angle value is continued until the cosine angle value meets the preset angle value.
5. The data attribute filling method according to claim 1, characterized in that: The method of predicting the initial question data to be answered by using the preset model set to obtain the target prediction result includes the following steps: Predicting the initial question data to be answered by using the language representation BERT model in the preset model set, and determining whether the initial question data to be answered belongs to a valid type; If the initial question data to be answered belongs to a valid type, obtaining a valid type prediction result; Predicting the initial question data to be answered by using a text classification textcnn model in a preset model set, and determining whether the initial question data to be answered belongs to a chat type; If the initial question data to be answered belongs to the chat type, obtaining a chat type prediction result; The effective class prediction result and the chat class prediction result are combined to obtain a target prediction result.
6. The data attribute filling method according to claim 1, characterized in that: If the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold, determining the attribute of the clustering result, and using the attribute to fill the attribute of the clustering result includes: If the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold, obtaining an attribute set of the clustering result based on a preset mapping relationship between the attributes of the historical question data and the attributes of the clustering result, the attribute set of the clustering result including at least one attribute of the clustering result; Frequent item sets in the attribute set of the clustering result are mined, and attributes of the clustering result are determined based on the frequent item sets.
7. The data attribute filling method according to any one of claims 1 to 6, characterized in that: The calculating of the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base comprises the following steps: Calculate the literal similarity between the initial to-be-answered question data and the historical question data in the knowledge base by using term frequency-inverse document frequency TF-IDF; Calculating the semantic similarity between the initial to-be-answered question data and the historical question data in the knowledge base through the twin network; The literal similarity and the semantic similarity values are prioritized according to the similarity values to obtain a comprehensive similarity.
8. A data attribute filling device, characterized in that: The data attribute filling device comprises the following modules: The prediction module is used to predict the initial question data to be answered through a preset model set to obtain a target prediction result; A classification module, configured to obtain, based on the target prediction result, a knowledge owner to which the initial question data to be answered belongs, and determine, according to the knowledge owner, a knowledge base corresponding to the initial question data to be answered; An identification module, used to calculate the comprehensive similarity between the initial unanswered question data and the historical question data in the knowledge base; A similarity determination module, used to determine whether the similarity is greater than or equal to a first preset threshold; A clustering module, for inputting the initial question data to be answered into each node of a preset graph G=(V, E) if the similarity is greater than or equal to a first preset threshold, determining the weight of the initial question data to be answered according to the degree in the graph, and clustering the initial question data to be answered based on the weight to obtain a clustering result, wherein the question data with the highest weight in the clustering result is the question data, and the rest are similar question data, V is a node set, E is an edge set, and the similar question data is data having a similar relationship with the question data; A matching degree judgment module, used to judge whether the matching degree between the attribute of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold; A filling module is used to determine the attributes of the clustering result and use the attributes to fill the clustering result if the matching degree between the attributes of the historical question data in the knowledge base and the clustering result is greater than or equal to a second preset threshold.
9. A data attribute filling device, characterized in that: The data attribute filling device includes a memory, a processor, and a data attribute filling program stored in the memory and executable on the processor. When the data attribute filling program is executed by the processor, the steps of the data attribute filling method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data attribute filling program, and when the data attribute filling program is executed by a processor, the steps of the data attribute filling method according to any one of claims 1 to 7 are implemented.
