Text data mining method, device and electronic equipment

Abstract

The present disclosure relates to a text data mining method, device and electronic device, and in particular to the field of data mining technology. The method comprises: extracting a first question text and a first reply paragraph corresponding to the first question text from a historical conversation log; inputting the first question text and the first reply paragraph into a target BERT model to obtain a first answer prediction text, wherein the target BERT model is a model obtained after pre-training based on an initial BERT model; storing the first question text and the first answer prediction text in correspondence; wherein the target BERT model annotates the first reply paragraph with at least one of the following first vectors: an attribute judgment vector, used to indicate whether a word in the first reply paragraph is an attribute word; a word-like judgment vector, used to indicate whether a word in the first reply paragraph is a word indicating yes or no; and a repeated word judgment vector, used to indicate whether a word in the reply paragraph appears in the second question text.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data mining, and in particular to a text data mining method, device and electronic device. Background Art

[0002] At present, in order to improve the understanding and response capabilities of customer service robots, it is necessary to mine the data of manual customer service conversations, extract question-answer pairs, and apply them to online customer service robot Q&A to improve the overall performance of customer service robots when answering user questions. However, the current data mining methods may extract incorrect answers to questions (for example, some attribute words, polarity words, etc. cannot be accurately focused and extraction errors occur), so the accuracy of data mining is low. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a text data mining method, device and electronic device, which can improve the accuracy of data mining.

[0004] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0005] In a second aspect, a text data mining method is provided, comprising:

[0006] Extracting a first question text and a first reply paragraph corresponding to the first question text from the historical conversation log;

[0007] Inputting the first question text and the first reply paragraph into a target BERT model, and obtaining a first answer prediction text output by the target BERT model, wherein the target BERT model is a model obtained after pre-training based on an initial BERT model;

[0008] storing the first question text and the first answer prediction text in correspondence;

[0009] The target BERT model annotates the first reply paragraph using at least one of the following first vectors:

[0010] An attribute judgment vector, used to indicate whether a word in the first reply paragraph is an attribute word;

[0011] A yes-or-no word judgment vector, used to indicate whether the word in the first reply paragraph is a word indicating yes or no;

[0012] A repeated word judgment vector is used to indicate whether a word in the first reply paragraph appears in the first question text.

[0013] As an optional implementation of the embodiment of the present disclosure, before inputting the first question text and the first reply paragraph into the target BERT model and obtaining the first answer prediction text output by the target BERT model, the method includes:

[0014] Acquire annotated sample data, wherein the annotated sample data includes a plurality of question texts, reply paragraphs corresponding to the plurality of question texts, and annotated answer text corresponding to each question text;

[0015] Acquire a target question text, a target response paragraph corresponding to the target question text, and a target answer text corresponding to the target question text from the annotated sample data;

[0016] Input the target question text and the target response paragraph into a first BERT model, and obtain a second answer prediction text output by the first BERT model;

[0017] Calculating a loss parameter according to the second answer prediction text and the target answer text;

[0018] Based on the loss parameter, the first BERT model is fine-tuned to obtain the target BERT model.

[0019] As an optional implementation of the embodiment of the present disclosure, before inputting the target question text and the target response paragraph into the first BERT model and obtaining the second answer prediction text output by the first BERT model, the method further includes:

[0020] Obtain target text pair data;

[0021] Pre-training a preset BERT model according to the target text pair data to obtain a second BERT model;

[0022] The at least one first vector is configured for the second BERT model to obtain the first BERT model.

[0023] As an optional implementation of the embodiment of the present disclosure, the pre-training task when pre-training the preset BERT model according to the text pairing data includes at least one of the following:

[0024] Masked language model MLM task, next sentence prediction NSP task.

[0025] As an optional implementation of the embodiment of the present disclosure, the text pair data includes:

[0026] Text pair data in the target domain, and / or text pair data in any domain.

[0027] As an optional implementation of the embodiment of the present disclosure, the step of inputting the first question text and the first reply paragraph into a target BERT model and obtaining a first answer prediction text output by the target BERT model includes:

[0028] The first question text and the first response paragraph determine an initial answer prediction text;

[0029] When it is determined that the first question text is a polarity type to be reversed according to the first question text and the first reply paragraph, extracting a linking verb from the first question text;

[0030] Reversing the polarity of the copula to obtain a polarity-reversed copula;

[0031] The first answer prediction text is determined according to the polarity inversion linking verb and the initial answer prediction text.

[0032] As an optional implementation of the embodiment of the present disclosure, the target BERT model further annotates the first reply paragraph using at least one of the following second vectors:

[0033] A tag vector, used to convert each word in the first reply paragraph into a vector of fixed dimension;

[0034] A segment vector, used to indicate whether the first reply segment is the same sentence;

[0035] A position vector is used to represent the positions of different words in the first reply paragraph in the first reply paragraph.

[0036] In a second aspect, a text data mining device is provided, comprising:

[0037] An extraction module, configured to extract a first question text and a first reply paragraph corresponding to the first question text from the historical conversation log;

[0038] An answer prediction module, used to input the first question text and the first reply paragraph into a target BERT model, and obtain a first answer prediction text output by the target BERT model, wherein the target BERT model is a model obtained after pre-training based on an initial BERT model;

[0039] A storage module, used for storing the first question text and the first answer prediction text in correspondence;

[0040] The target BERT model annotates the first reply paragraph using at least one of the following first vectors:

[0041] An attribute judgment vector, used to indicate whether a word in the first reply paragraph is an attribute word;

[0042] A yes-or-no word judgment vector, used to indicate whether the word in the first reply paragraph is a word indicating yes or no;

[0043] A repeated word judgment vector is used to indicate whether a word in the first reply paragraph appears in the first question text.

[0044] As an optional implementation of the embodiment of the present disclosure, the device further includes:

[0045] An adjustment module is used to obtain annotated sample data before the answer prediction module inputs the first question text and the first reply paragraph into the target BERT model to obtain the first answer prediction text output by the target BERT model, the annotated sample data including multiple question texts, reply paragraphs corresponding to the multiple question texts, and annotated answer text corresponding to each question text; obtain a target question text, a target reply paragraph corresponding to the target question text, and a target answer text corresponding to the target question text from the annotated sample data; input the target question text and the target reply paragraph into the first BERT model, and obtain a second answer prediction text output by the first BERT model; calculate a loss parameter based on the second answer prediction text and the target answer text; and fine-tune the first BERT model based on the loss parameter to obtain the target BERT model.

[0046] As an optional implementation of the embodiment of the present disclosure, the device further includes:

[0047] A pre-training module is used for obtaining target text pair data before the adjustment module inputs the target question text and the target response paragraph into the first BERT model and obtains the second answer prediction text output by the first BERT model; pre-training a preset BERT model according to the target text pair data to obtain a second BERT model; and configuring the at least one first vector for the second BERT model to obtain the first BERT model.

[0048] As an optional implementation of the embodiment of the present disclosure, the pre-training task when pre-training the preset BERT model according to the text pairing data includes at least one of the following:

[0049] Masked language model MLM task, next sentence prediction NSP task.

[0050] As an optional implementation of the embodiment of the present disclosure, the text pair data includes:

[0051] Text pair data in the target domain, and / or text pair data in any domain.

[0052] As an optional implementation of the embodiment of the present disclosure, the answer prediction module is specifically used to:

[0053] The first question text and the first response paragraph determine an initial answer prediction text;

[0054] When it is determined that the first question text is a polarity type to be reversed according to the first question text and the first reply paragraph, extracting a linking verb from the first question text;

[0055] Reversing the polarity of the copula to obtain a polarity-reversed copula;

[0056] The first answer prediction text is determined according to the polarity inversion linking verb and the initial answer prediction text.

[0057] As an optional implementation of the embodiment of the present disclosure, the target BERT model further annotates the first reply paragraph using at least one of the following second vectors:

[0058] A tag vector, used to convert each word in the first reply paragraph into a vector of fixed dimension;

[0059] A segment vector, used to indicate whether the first reply segment is the same sentence;

[0060] A position vector is used to represent the positions of different words in the first reply paragraph in the first reply paragraph.

[0061] In a third aspect, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the text data mining method as described in the second aspect or any one of its optional embodiments is implemented.

[0062] In a fourth aspect, a computer-readable storage medium is provided, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the text data mining method as described in the second aspect or any optional embodiment thereof is implemented.

[0063] In a fifth aspect, a computer program product is provided, characterized in that it includes: when the computer program product is run on a computer, the computer implements the text data mining method as described in the second aspect or any optional embodiment thereof.

[0064] The text data mining method provided by the embodiment of the present disclosure can extract a first question text and a first reply paragraph corresponding to the first question text from a historical conversation log; input the first question text and the first reply paragraph into a target BERT model to obtain a first answer prediction text output by the target BERT model, wherein the target BERT model is a model obtained after pre-training based on an initial BERT model; store the first question text and the first answer prediction text correspondingly; wherein the target BERT model annotates the first reply paragraph with at least one of the following first vectors: an attribute judgment vector, used to indicate whether a word in the first reply paragraph is an attribute word; a whether-or-not word judgment vector, used to indicate whether a word in the first reply paragraph is a word indicating yes or no; a repeated word judgment vector, used to indicate whether a word in the first reply paragraph appears in the first question text. Through this solution, since the target BERT model marks the first reply paragraph through one or more of the attribute judgment vector, the class word judgment vector and the repeated word judgment vector, after the first reply paragraph is marked, the target BERT model can extract the answer corresponding to the first question text from the first reply paragraph. Some attribute words that are very likely to be answers, class words, and repeated class words can be accurately focused on and extracted, thereby improving the accuracy of data mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0067] Figure 1 A schematic diagram of an implementation of the text data mining method provided in the embodiment of the present disclosure;

[0068] Figure 2 A flowchart of a text data mining method provided by an embodiment of the present disclosure;

[0069] Figure 3 A schematic diagram of storing data in a question-answer library provided in an embodiment of the present disclosure;

[0070] Figure 4 A flowchart of another text data mining method provided by an embodiment of the present disclosure;

[0071] Figure 5 A structural block diagram of a text data mining device provided by an embodiment of the present disclosure;

[0072] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0074] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0075] Customer service robots are an important part of the current e-commerce scene, playing a significant role in night duty, quickly answering high-frequency repeated questions, and reducing labor costs. There are currently two product forms: fully automatic robots and intelligent assistance. Fully automatic robots do not require human intervention. The system completely understands the user's query and selects the corresponding reply to return to the user; intelligent assistance will understand the user's query during the manual customer service stage and give some recommended replies. The manual customer service will select the replies that may be sent directly or only require a small amount of modification to send to the user, and then send them directly or after slightly editing.

[0076] Customer service robots face two problems: they cannot fully understand user queries and it is difficult to obtain appropriate answers. That is, both the ability to understand and the ability to respond need to be improved. In actual online business, if the customer service robot cannot answer user questions, the system will transfer these unresolved questions to manual customer service according to some rules. Since manual customer service is a service personnel who has received certain training, the satisfaction of manual customer conversations will be much higher than that of robot customers, and because they are real people, the replies are more empathetic and the answers are more authentic, so it is a very good learning material. This patented method proposes a complete set of processes, which extracts the question-answer (Query-Answer) library by mining the data in the manual customer service conversation offline, and applies it to online robot customer service Q&A (including fully automatic robots and intelligent assistance) to improve the overall performance of the customer service robot.

[0077] Currently, there is a need to mine the data of manual customer service conversations to extract question-answer pairs and apply them to online customer service robot Q&A to improve the overall performance of customer service robots in answering user questions. However, current data mining methods may extract incorrect answers to questions (for example, some attribute words, polarity words, etc. cannot be accurately focused on and incorrect answers are extracted), so the accuracy of data mining is low.

[0078] In order to solve the above problems, the embodiment of the present disclosure provides a text data mining method, device and electronic device. In this scheme, a first question text and a first reply paragraph corresponding to the first question text can be extracted from a historical conversation log; the first question text and the first reply paragraph are input into a target bidirectional encoder representation (Bidirectional Encoder Representations from Transformer, BERT) model based on feature matching, and the first answer prediction text output by the target BERT model is obtained, and the target BERT model is a model obtained after pre-training based on the initial BERT model; the first question text and the first answer prediction text are stored correspondingly. Since the target BERT model annotates the first reply paragraph through one or more of the attribute judgment vector, the class word judgment vector and the repeated word judgment vector, so that after the first reply paragraph is annotated, the target BERT model can extract the answer corresponding to the first question text from the first reply paragraph, and some attribute words that are very likely to be answers, whether there are class words, and repeated class words can be accurately focused and extracted, thereby improving the accuracy of data mining.

[0079] like Figure 1 As shown, it is a schematic diagram of an implementation of the text data mining method provided in an embodiment of the present disclosure. In this method, the historical conversation log between the user and the manual customer service is first obtained, and data mining is performed based on the historical conversation log and the text data mining method provided in the embodiment of the present disclosure. After the questions and answers are obtained, they are saved accordingly in the form of question and answer pairs, so that the customer service robot can use the saved question and answer pairs when conducting online questions and answers with the user in the future.

[0080] The text data mining method provided in the embodiment of the present disclosure can be implemented by a text data mining device or an electronic device, and the text data mining device can be a functional module or functional entity in the electronic device. The electronic device includes but is not limited to: a server, a personal computer, a laptop, a tablet computer, a smart phone, etc. The computer device includes a user device and a network device. Among them, the user device includes but is not limited to a computer, a smart phone, a tablet computer, etc.; the network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of computers or network servers in cloud computing, wherein cloud computing is a kind of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets. Among them, the computer device can be operated alone to implement the present disclosure, and can also be connected to the network and implement the present disclosure through interactive operations with other computer devices in the network. Among them, the network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.

[0081] like Figure 2 FIG. 1 is a flow chart of a text data mining method provided by an embodiment of the present disclosure, and the method comprises the following steps:

[0082] 201. Extract a first question text from a historical conversation log.

[0083] 202. Extract a first reply paragraph corresponding to the first question text from the historical conversation log.

[0084] The above historical conversation log may include conversations between users and human customer service. The conversations between users and human customer service may include but are not limited to: question texts entered by users, text content directly replied by human customer service to users, and text content of replies selected by human customer service through intelligent assistance.

[0085] For example, after a user enters a question text, the manual customer service responds to the user by selecting a pre-selected reply template text, and such reply template text is also included in the conversation between the user and the manual customer service.

[0086] In a conversation, since the answer to a query can only be given after it is asked, based on the time sequence, the query can be located from the historical conversation log first, and then the conversation content after the query can be selected as the reply paragraph of the query to reduce redundant text interference. In other words, in the process of extracting the first question text and the first reply paragraph from the historical conversation log, the first question text entered by the user can be determined, and then one or more texts replied by the manual customer service after the first question text can be selected from the historical conversation log, and these one or more texts can be spliced ​​as the first reply paragraph.

[0087] In some embodiments, the historical conversation log may include a large amount of conversation content. Therefore, the historical conversation log may be divided according to the object type involved in the conversation, and the text content of the same object type may be grouped as a group of conversations.

[0088] For different scenarios, the above object type may be different. For example, assuming that for an e-commerce conversation scenario, the above object type may refer to a commodity type. Conversations for the same commodity may be classified as a group of conversations.

[0089] Furthermore, the historical conversation log may be divided according to the object type involved in the conversation and a preset division time range, and the text contents of the same object type and within the preset division time range are regarded as a group of conversations.

[0090] For example, assume that in an e-commerce conversation scenario, there are 100 conversation messages between the user of product A and the manual customer service in the historical conversation log, of which 40 conversation messages were involved on June 7 and 60 conversation messages were involved on June 8. Then, if product A and each day are used as a time range, then these 100 conversation messages can be divided into two groups of conversations, the 40 conversation messages involved on June 7 as one group of conversation messages, and the 60 conversation messages involved on June 8 as another group of conversation messages; if product A and every 2 days are used as a time range, the above 100 conversation messages can be divided into one group of conversations.

[0091] 203. Input the first question text and the first reply paragraph into the target BERT model to obtain the first answer prediction text output by the target BERT model.

[0092] Among them, the target BERT model is a model obtained after pre-training based on the initial BERT model.

[0093] The target BERT model is a language representation model pre-trained based on the initial BERT model, wherein the initial BERT model refers to the model structure framework of the traditional BERT model. In the process of pre-training based on the initial BERT model, a new Masked Language Model (MLM) task and / or Next Sentence Prediction (NSP) task are used to generate deep bidirectional language representation. The MLM task allows the model to predict the current word through the context, that is, to better learn the vector of the current word through the semantics of the context. NSP randomly shuffles the order of sentences and then judges whether the two sentences are in a contextual relationship to learn the logical and semantic relationship between sentences, thereby improving the representation effect of sentence vectors. The goal of the above pre-training is to use large-scale unlabeled corpus training to obtain the representation of the text containing rich semantic information, that is, the semantic representation of the text.

[0094] In the above pre-training process, texts in any field may be used for pre-training, and / or texts in a specific field may be used for pre-training.

[0095] In some embodiments, an initial BERT model can be pre-trained based on texts in any field to obtain a BERT model after the first stage of pre-training. Then, based on the BERT model after the first stage of pre-training, a second stage of pre-training can be performed based on texts in the target field to obtain a BERT model after pre-training.

[0096] The target field may be any field, for example, e-commerce field, emergency field, etc. The pre-training tasks of the first stage pre-training and the second stage pre-training are the same, and may include: MLM tasks and / or NSP tasks.

[0097] Furthermore, after the pre-training model is obtained based on the initial BERT model, the pre-trained model can be fine-tuned, that is, the semantic representation of the text is adjusted in a specific NLP task (usually referred to as the fine-tuning process), and finally applied to the NLP task. In the fine-tuning process, a small amount of annotated corpus can allow the model to learn specific task data, so that it can achieve better results during testing.

[0098] The traditional BERT model usually represents the input text through a token vector (Token Embedding), a segment vector (Segment Embedding) and a position vector (Position Embedding). In the embodiment of the present disclosure, in addition to using at least one of Token Embedding, Segment Embedding and Position Embedding to represent the input text, at least one first vector is added to represent the input text.

[0099] Among them, the above-mentioned Token Embedding, Segment Embedding and Position Embedding can be referred to as second vectors in the embodiment of the present disclosure. The functions of these three vectors are as follows:

[0100] Token Embedding is used to convert each word into a vector of fixed dimension. In the target BERT, the vector dimension is 768 dimensions, that is, each word has a 1*768-dimensional vector. Special markers of CLS and SEP are added at the beginning, end, and in the middle of the sentence pair (if there are two sentences). For example, if the input is a sentence with 10 words, Token Embedding is a vector of 10+2=12*768; if the input is two sentences, each with 5 words, Token Embedding is a vector of (5+5+1)+2=13*768. In the target BERT in the embodiment of the present disclosure, Token Embedding is specifically used to convert each word in the first reply paragraph into a vector of fixed dimension.

[0101] Segment Embedding is used to tell the model whether the input is the same sentence, that is, to process sentence pairs. Segment Embedding has two values: 0 and 1. If there are two sentences, each word in the first sentence is assigned a value of 0, and each word in the second sentence is assigned a value of 1; if there is one sentence, each word is assigned a value of 0. In the target BERT in the disclosed embodiment, Segment Embedding is specifically used to indicate whether the first reply paragraph is the same sentence.

[0102] Position Embedding is used to indicate that different characters appear at different positions in the text. When different characters appear at different positions in the text, their semantics will be significantly different. For example, the characters in the sentences "Zhang San gave birth to Li Si" and "Li Si gave birth to Zhang San" are the same, but the order has changed. The semantics of the two sentences are very different, so position information is needed to tell the model that different characters appear at different positions in the text. It can be simply considered that Position Embedding uses an incremental method of Roman numerals to indicate the different positions of different characters. In the target BERT in the disclosed embodiment, Position Embedding is specifically used to indicate the positions of different characters in the first reply paragraph.

[0103] The first vector includes but is not limited to at least one of the following:

[0104] Attribute judgment vector (is_Entity_Embedding), in the disclosed embodiment, is_Entity_Embedding is used to indicate whether a word in the first reply paragraph is an attribute word. To determine whether a word is an attribute word, we can compare it with the attribute words in a pre-stored attribute table to determine whether the word in the first reply paragraph is in the word table. If so, the is_Entity_Embedding value is 1, and if not, the is_Entity_Embedding value is 0.

[0105] For example, suppose the user's query asks "What material is this dress made of?", and the manual customer service replies "It's made of silk, very smooth, and this model is selling very well recently. It's better to act than to be tempted~". Since "silk" is pre-stored in the attribute table of the knowledge base, it may be the answer. We mark it with an is_Entity_Embedding value of 1, so that the target BERT can learn it more easily.

[0106] Yes_No_Embedding: In the disclosed embodiment, Yes_No_Embedding is used to indicate whether the word in the first reply paragraph is a word indicating yes or no. If the word in the first reply paragraph appears in the user's Query (i.e., appears in the first question text) in the conversation, it is more likely to be the answer.

[0107] For example, if the user asks "Will it pill?" and the customer service representative answers "No", then "No" will be assigned a value of 1 by Yes_No_Embedding because it is more likely to be the answer.

[0108] Repeated word judgment vector (Exact_Match_Embedding), in the disclosed embodiment, Exact_Match_Embedding is used to indicate whether a word in the first reply paragraph appears in the first question text.

[0109] For example, a user asks "Is this clothing waterproof?" The customer service representative replies "Yes, it is made of waterproof material and will not get wet even when it rains." The word "waterproof" in the customer service representative's answer appears in the query, indicating that it is likely related to the query. We mark it with a Yes_No_Embedding value of 1, making it easier for the target BERT model to learn.

[0110] For some negative questions, in some scenarios, the answer extracted by the target BERT model may have the opposite polarity to the actual answer. For example, the actual answer is positive (yes, will, yes, etc.), but the extracted answer is negative (no, will not, not, etc.).

[0111] To solve the above problem, in some embodiments, the first question text and the first reply paragraph are input into the target BERT model, and obtaining the first answer prediction text output by the target BERT model may include: first, after the first question text and the first reply paragraph are input into the target BERT model, the target BERT model first determines the initial answer prediction text based on the first question text and the first reply paragraph; then, based on the first question text and the first reply paragraph, it is determined whether the first question text is a polarity type to be reversed, and when it is determined that the first question text is a polarity type to be reversed, a copula is extracted from the first question text; and the polarity of the copula is reversed to obtain a polarity-reversed copula; finally, the first answer prediction text is determined based on the polarity-reversed copula and the initial answer prediction text.

[0112] The initial answer prediction text may be extracted from the first question text and / or the first reply paragraph.

[0113] When judging whether the first question text is to be reversed polarity based on the first question text and the first reply paragraph, a classification module can be added to the target BERT model to perform the classification task.

[0114] The classification module can perform multi-task learning on a variety of question texts, response paragraphs, and corresponding annotated answers to determine the following three categories:

[0115] 0: No answer;

[0116] 1: The answer can be extracted;

[0117] 2: The answer can be extracted and the polarity needs to be reversed.

[0118] The first question text and the first reply paragraph can be input into the classification module, and the classification module can be used to obtain the output classification. When the classification is "2: the answer can be extracted and the polarity needs to be reversed", the first question text is determined to be a polarity type to be reversed, and the copula is extracted from the first question text; and the polarity of the copula is reversed to obtain the polarity-reversed copula; finally, the first answer prediction text is determined based on the polarity-reversed copula and the initial answer prediction text. The copula may include but is not limited to: "will", "is", "have", etc.

[0119] The following Table 1 is a schematic diagram for two polarity reversal situations.

[0120] Table 1

[0121]

[0122] As can be seen from Table 1, when the classification label is "2: the answer can be extracted and the polarity needs to be reversed", the copula "yes" will be extracted from the Query "Does the sweatshirt suit have pockets?", and the polarity of the copula will be reversed to obtain "no", and then combined with "no" and the initial answer prediction text "has pockets" to obtain the final first answer prediction text "no pockets"; and when the classification label is "2: the answer can be extracted and the polarity needs to be reversed", the copula "will" will be extracted from the Query "Will it fade?", and the polarity of the copula will be reversed to obtain "no", and then combined with "no" and the initial answer prediction text "will fade" to obtain the final first answer prediction text "no pockets".

[0123] 204. Store the first question text and the first answer prediction text in correspondence with each other.

[0124] After the first answer prediction text is input into the target BERT model, the first answer prediction text can be used as the answer corresponding to the first question text and stored in the question-answer library accordingly.

[0125] It should be noted that, based on other question texts and corresponding reply paragraphs extracted from historical conversation logs, data mining can be performed according to the method shown in 201 to 204 above to determine the answer prediction text corresponding to the question text and save it in the question-answer library.

[0126] The question-answer library may include multiple sets of correspondingly stored question texts and answer prediction texts. Figure 3As shown, it is a schematic diagram of storing data in a question-answer library provided by an embodiment of the present disclosure, wherein the answer content corresponding to question text 1 is answer prediction text 1; the answer content corresponding to question text 2 is answer prediction text 2; the answer content corresponding to question text 3 is answer prediction text 3; and the answer content corresponding to question text 4 is answer prediction text 4.

[0127] In the subsequent online question-and-answer process between the user and the customer service robot, the question text entered by the user can be matched with the question text in the question-answer library. After the matching question text in the question-answer library is determined, the answer prediction text stored corresponding to the matching question text in the question-answer library can be used as the answer to the question text entered by the user and replied to the user by the customer service robot.

[0128] The text data mining method provided by the embodiment of the present disclosure, since the target BERT model marks the first reply paragraph through one or more of the attribute judgment vector, the class word judgment vector and the repeated word judgment vector, after the first reply paragraph is marked, it is convenient for the target BERT model to extract the answer corresponding to the first question text from the first reply paragraph, and some attribute words that are very likely to be the answers, class words, and repeated class words can be accurately focused on and extracted, thereby improving the accuracy of data mining.

[0129] like Figure 4 As shown, another text data mining method provided by an embodiment of the present disclosure may include but is not limited to a pre-training stage, a fine-tuning stage, and an answer prediction stage. The method includes the following steps:

[0130] The pre-training stage includes but is not limited to the following steps 401 and 402:

[0131] 401. Obtain target text pair data.

[0132] The target text pair data includes, but is not limited to: text pair data in a target field, and / or text pair data in any field.

[0133] The above-mentioned text pair data refers to: including multiple groups of text data, each group of text data includes two texts to form a text pair as an input of the preset BERT model.

[0134] 402. Pre-train a preset BERT model based on the target text data to obtain a second BERT model.

[0135] Among them, the pre-training task when pre-training the preset BERT model according to the text pairing data includes at least one of the following: an MLM task and a next sentence prediction NSP task.

[0136] In some embodiments, an initial BERT model can be pre-trained based on texts in any field to obtain a BERT model after the first stage of pre-training. Then, based on the BERT model after the first stage of pre-training, a second stage of pre-training can be performed based on texts in the target field to obtain a BERT model after pre-training.

[0137] Through the above pre-training process, the MLM task can be used to let the model predict the current word through the context, that is, to better learn the vector of the current word through the semantics of the context. NSP learns the logical and semantic relationships between sentences by randomly disrupting the order of sentences and then judging whether the two sentences are in a contextual relationship. Therefore, the pre-selection process can improve the representation effect of the sentence vector. Pre-training the data with text from the target field can improve the representation effect of the sentence vector and, based on the business characteristics of the target field, obtain the representation effect of the sentence vector that conforms to the field.

[0138] The fine-tuning phase includes but is not limited to the following steps 403 to 408:

[0139] 403. Configure at least one first vector for the second BERT model to obtain a first BERT model.

[0140] The above-mentioned Token Embedding, Segment Embedding and Position Embedding already exist in the second BERT model. One of is_Entity_Embedding, Exact_Match_Embedding and Yes_No_Embedding can be added on the basis of the original second BERT model, so that the subsequent target BERT also includes one of is_Entity_Embedding, Exact_Match_Embedding and Yes_No_Embedding.

[0141] 404. Obtain labeled sample data.

[0142] The annotated sample data includes multiple question texts, response paragraphs corresponding to the multiple question texts, and annotated answer text corresponding to each question text.

[0143] The labeled sample data may be data obtained by manual labeling.

[0144] 405. Obtain a target question text, a target response paragraph corresponding to the target question text, and a target answer text corresponding to the target question text from the annotated sample data.

[0145] The target question text is any question text among the multiple question texts in the above-mentioned labeled sample data.

[0146] 406. Input the target question text and the target response paragraph into the first BERT model, and obtain the second answer prediction text output by the first BERT model.

[0147] 407. Calculate a loss parameter based on the second answer prediction text and the target answer text.

[0148] Among them, when calculating the loss parameter based on the second answer prediction text and the target answer text, the loss parameter can be calculated based on the second answer prediction text, the target answer text and a preset loss function.

[0149] The above preset loss function can be selected according to actual needs and is not limited in the embodiments of the present disclosure.

[0150] 408. Based on the loss parameter, fine-tune the first BERT model to obtain a target BERT model.

[0151] After executing one or more rounds of steps 405 to 408 above, when the loss parameter is within the preset loss range, it is considered that the model has converged and the target BERT model is obtained.

[0152] In the above embodiment, after pre-training, the target BERT model is obtained by fine-tuning the model by annotating sample data. The model can be converged through multiple rounds of training with annotated sample data. The accuracy of the target BERT model obtained in this way will be greatly improved, which lays a good foundation for subsequent answer prediction and can further improve the accuracy of data mining.

[0153] The answer prediction stage includes but is not limited to the following steps 409 to 412:

[0154] 409. Extract the first question text from the historical conversation log.

[0155] 410. Extract a first reply paragraph corresponding to the first question text from the historical conversation log.

[0156] 411. Input the first question text and the first reply paragraph into the target BERT model to obtain the first answer prediction text output by the target BERT model.

[0157] 412. Store the first question text and the first answer prediction text in correspondence with each other.

[0158] The description of the above steps 409 to 412 can refer to the above description of steps 201 to 204, which will not be repeated here.

[0159] The text data mining method provided by the embodiment of the present disclosure, since the target BERT model marks the first reply paragraph through one or more of the attribute judgment vector, the class word judgment vector and the repeated word judgment vector, after the first reply paragraph is marked, it is convenient for the target BERT model to extract the answer corresponding to the first question text from the first reply paragraph, and some attribute words that are very likely to be the answers, class words, and repeated class words can be accurately focused on and extracted, thereby improving the accuracy of data mining.

[0160] like Figure 5 FIG. 1 is a block diagram of a text data mining device provided by an embodiment of the present disclosure, and the device includes:

[0161] An extraction module 501 is used to extract a first question text and a first reply paragraph corresponding to the first question text from a historical conversation log;

[0162] An answer prediction module 502 is used to input the first question text and the first reply paragraph into a target BERT model, and obtain a first answer prediction text output by the target BERT model, wherein the target BERT model is a model obtained after pre-training based on an initial BERT model;

[0163] A storage module 503, configured to store the first question text and the first answer prediction text in correspondence;

[0164] The target BERT model annotates the first reply paragraph using at least one of the following first vectors:

[0165] An attribute judgment vector, used to indicate whether a word in the first reply paragraph is an attribute word;

[0166] A yes-or-no word judgment vector, used to indicate whether the word in the first reply paragraph is a word indicating yes or no;

[0167] A repeated word judgment vector is used to indicate whether a word in the first reply paragraph appears in the first question text.

[0168] As an optional implementation of the embodiment of the present disclosure, the device further includes:

[0169] The adjustment module 504 is used to obtain annotated sample data before the answer prediction module 502 inputs the first question text and the first reply paragraph into the target BERT model to obtain the first answer prediction text output by the target BERT model, the annotated sample data including multiple question texts, reply paragraphs corresponding to the multiple question texts, and annotated answer text corresponding to each question text; obtain a target question text, a target reply paragraph corresponding to the target question text, and a target answer text corresponding to the target question text from the annotated sample data; input the target question text and the target reply paragraph into the first BERT model, and obtain a second answer prediction text output by the first BERT model; calculate a loss parameter based on the second answer prediction text and the target answer text; and fine-tune the first BERT model based on the loss parameter to obtain the target BERT model.

[0170] As an optional implementation of the embodiment of the present disclosure, the device further includes:

[0171] The pre-training module 505 is used for obtaining target text pair data before the adjustment module 504 inputs the target question text and the target response paragraph into the first BERT model and obtains the second answer prediction text output by the first BERT model; pre-training a preset BERT model according to the target text pair data to obtain a second BERT model; and configuring the at least one first vector for the second BERT model to obtain the first BERT model.

[0172] As an optional implementation of the embodiment of the present disclosure, the pre-training task when pre-training the preset BERT model according to the text pairing data includes at least one of the following:

[0173] Masked language model MLM task, next sentence prediction NSP task.

[0174] As an optional implementation of the embodiment of the present disclosure, the text pair data includes:

[0175] Text pair data in the target domain, and / or text pair data in any domain.

[0176] As an optional implementation of the embodiment of the present disclosure, the answer prediction module 502 is specifically used to:

[0177] The first question text and the first response paragraph determine an initial answer prediction text;

[0178] When it is determined that the first question text is a polarity type to be reversed according to the first question text and the first reply paragraph, extracting a linking verb from the first question text;

[0179] Reversing the polarity of the copula to obtain a polarity-reversed copula;

[0180] The first answer prediction text is determined according to the polarity inversion linking verb and the initial answer prediction text.

[0181] As an optional implementation of the embodiment of the present disclosure, the target BERT model further annotates the first reply paragraph using at least one of the following second vectors:

[0182] A tag vector, used to convert each word in the first reply paragraph into a vector of fixed dimension;

[0183] A segment vector, used to indicate whether the first reply segment is the same sentence;

[0184] A position vector is used to represent the positions of different words in the first reply paragraph in the first reply paragraph.

[0185] like Figure 6 As shown, it is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure, and the electronic device includes: a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601, and when the computer program is executed by the processor 601, each process of the text data mining method in the above method embodiment is implemented. And the same technical effect can be achieved, and it will not be repeated here to avoid repetition.

[0186] An embodiment of the present disclosure provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, each process of the text data mining method in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0187] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0188] The embodiment of the present disclosure provides a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the text data mining method in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0189] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0190] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0191] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0192] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0193] It should be noted that, in this article, relational terms such as "second" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0194] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text data mining method, characterized in that: include: Extracting a first question text and a first reply paragraph corresponding to the first question text from the historical conversation log; Inputting the first question text and the first reply paragraph into a target BERT model, and obtaining a first answer prediction text output by the target BERT model, wherein the target BERT model is a model obtained after pre-training based on an initial BERT model; storing the first question text and the first answer prediction text in correspondence; The target BERT model annotates the first reply paragraph using at least one of the following first vectors, and the first vector is used to enable the target BERT model to extract the answer corresponding to the first question text from the first reply paragraph: An attribute judgment vector, used to indicate whether a word in the first reply paragraph is an attribute word; A yes-or-no word judgment vector, used to indicate whether the word in the first reply paragraph is a word indicating yes or no; A repeated word judgment vector is used to indicate whether a word in the first reply paragraph appears in the first question text.

2. The method according to claim 1, characterized in that Before inputting the first question text and the first reply paragraph into the target BERT model and obtaining the first answer prediction text output by the target BERT model, the method includes: Acquire annotated sample data, wherein the annotated sample data includes a plurality of question texts, reply paragraphs corresponding to the plurality of question texts, and annotated answer text corresponding to each question text; Acquire a target question text, a target response paragraph corresponding to the target question text, and a target answer text corresponding to the target question text from the annotated sample data; Input the target question text and the target response paragraph into a first BERT model, and obtain a second answer prediction text output by the first BERT model; Calculating a loss parameter according to the second answer prediction text and the target answer text; Based on the loss parameter, the first BERT model is fine-tuned to obtain the target BERT model.

3. The method according to claim 2, characterized in that Before inputting the target question text and the target response paragraph into a first BERT model and obtaining a second answer prediction text output by the first BERT model, the method further includes: Obtain target text pair data; Pre-training a preset BERT model according to the target text pair data to obtain a second BERT model; The at least one first vector is configured for the second BERT model to obtain the first BERT model.

4. The method according to claim 3, characterized in that The pre-training task when pre-training the preset BERT model according to the target text data includes at least one of the following: Masked language model MLM task, next sentence prediction NSP task.

5. The method according to claim 3, characterized in that: The target text pair data includes: Text pair data in the target domain, and / or text pair data in any domain.

6. The method according to claim 3, characterized in that the step of inputting the first question text and the first reply paragraph into a target BERT model and obtaining a first answer prediction text output by the target BERT model comprises: The first question text and the first response paragraph determine an initial answer prediction text; When it is determined that the first question text is a polarity type to be reversed according to the first question text and the first reply paragraph, extracting a linking verb from the first question text; Reversing the polarity of the copula to obtain a polarity-reversed copula; The first answer prediction text is determined according to the polarity inversion linking verb and the initial answer prediction text.

7. The method according to claim 1, characterized in that The target BERT model further annotates the first response paragraph using at least one of the following second vectors: A tag vector, used to convert each word in the first reply paragraph into a vector of fixed dimension; A segment vector, used to indicate whether the first reply segment is the same sentence; A position vector is used to represent the position of different words in the first reply paragraph in the first reply paragraph.

8. A text data mining device, characterized in that: include: An extraction module, configured to extract a first question text and a first reply paragraph corresponding to the first question text from a historical conversation log; An answer prediction module, used to input the first question text and the first reply paragraph into a target BERT model, and obtain a first answer prediction text output by the target BERT model, wherein the target BERT model is a model obtained after pre-training based on an initial BERT model; A storage module, used for storing the first question text and the first answer prediction text in correspondence; The target BERT model annotates the first reply paragraph using at least one of the following first vectors, and the first vector is used to enable the target BERT model to extract the answer corresponding to the first question text from the first reply paragraph: An attribute judgment vector, used to indicate whether a word in the first reply paragraph is an attribute word; A yes-or-no word judgment vector, used to indicate whether the word in the first reply paragraph is a word indicating yes or no; A repeated word judgment vector is used to indicate whether a word in the first reply paragraph appears in the first question text.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the text data mining method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the text data mining method according to any one of claims 1 to 7 is implemented.

Images