A text processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a text processing method and device, electronic equipment and storage medium, belonging to the field of artificial intelligence. The method comprises: obtaining dialogue text; obtaining a first candidate text by text splicing through a first field separator; obtaining a first material text by text encoding the first candidate text according to the context relationship; obtaining a second material text by text encoding the reference text according to the context relationship; obtaining a third material text by text splicing the first material text and the second material text through a second field separator; determining the start position of the target text and the end position of the target text according to the third material text by keyword prediction; and extracting and outputting the target text as the answer to the target question according to the start position and the end position. Embodiments of the present application can start fitting from a more smooth target function, avoid falling into a local minimum, better find a global optimum, and the prediction result is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Dialogue reading comprehension, also known as multi-turn question-answering reading comprehension, requires the model to find the corresponding answer output from the reference text according to the current question. However, unlike general single-turn reading comprehension models, most questions are based on dialogue context, that is, to accurately understand the semantics of the current question, the context of the dialogue needs to be combined.

[0003] The existing solution is usually a combination of a coreference resolution model and a reading comprehension model, that is, the current question is first spliced with the historical question-answering content, and the model is explicitly completed to complete the semantics of the current question, and then the completed question is spliced with the reference text and input into the reading comprehension model for answer generation. However, the existing solution requires explicit coreference resolution data labeling for training, and is prone to error propagation, resulting in inaccurate prediction results. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a text processing method and device, an electronic device, and a storage medium that do not require coreference resolution data labeling and have higher accuracy, aiming to improve the accuracy of the output of the reading comprehension model in the multi-turn question-answering reading comprehension scenario.

[0005] To achieve the above purpose, a first aspect of an embodiment of the present application provides a text processing method, which comprises:

[0006] Obtaining dialogue text, the dialogue text comprising a current question, historical question-answering, and a keyword;

[0007] Text splicing the current question, the historical question-answering, and the keyword by a first field separator to obtain a first candidate text;

[0008] Text encoding the first candidate text according to a context relationship to obtain a first material text;

[0009] Obtaining a reference text, text encoding the reference text according to a context relationship to obtain a second material text, the reference text comprising a target question;

[0010] Text splicing the first material text and the second material text by a second field separator to obtain a third material text, and determining a start position of a target text and an end position of the target text according to the third material text; the target text is the answer content of the target question;

[0011] According to the start position and the end position, the target text is extracted and output.

[0012] In some possible embodiments, the first material text and the second material text are spliced by the second field separator to obtain a third material text, and a start position of the target text and an end position of the target text are determined according to keyword prediction based on the third material text, including:

[0013] The third material text is input into the target question and answer model after training, and the start position of the target text and the end position of the target text are output.

[0014] The training process of the target question and answer model includes the following steps:

[0015] The training text material is divided to obtain a plurality of training data sets;

[0016] The plurality of training data sets are input into a plurality of candidate question and answer models to be trained, and a difficulty score is determined according to a model output result and a preset evaluation index;

[0017] The training data sets are sorted according to difficulty according to the difficulty score to obtain a training set sequence, and the target question and answer model is trained through the training set sequence.

[0018] In some possible embodiments, the dialog text is obtained, including:

[0019] A text dictionary is obtained, and a plurality of text sentences are obtained by regular matching the dialog text according to the text dictionary;

[0020] A directed acyclic graph is constructed according to the plurality of text sentences, a path of each node in the directed acyclic graph is traversed, the keyword is determined according to the path, and the part of speech of the keyword is determined.

[0021] In some possible embodiments, the training process of the target question and answer model further includes the following steps:

[0022] A first starting word and a first ending word of an original answer in the training text material are obtained;

[0023] A second starting word and a second ending word of the target text predicted by the target question and answer model are obtained;

[0024] A first loss value is calculated according to the first starting word and the second starting word through a cross-entropy loss function, and / or a second loss value is calculated according to the first ending word and the second ending word through a cross-entropy loss function;

[0025] The target question and answer model is parameter tuned according to the first loss value and / or the second loss value.

[0026] In some possible embodiments, the target question and answer model comprises an encoder and a classifier; the first material text and the second material text are spliced by a second field separator to obtain a third material text, and a start position of a target text and an end position of the target text are determined according to keyword prediction based on the third material text, comprising:

[0027] The third material text is segmented to obtain a candidate word, and the candidate word is converted into a fixed-dimensional word vector by the encoder;

[0028] The spliced sentence contained in the third material text is represented by a vector by the encoder to obtain a sentence vector;

[0029] The position relationship between the candidate words in the third material text is obtained by the encoder;

[0030] According to the word vector, the sentence vector and the position relationship, the classifier outputs a probability that the candidate word belongs to the start position or a probability that the candidate word belongs to the end position.

[0031] In some possible embodiments, the obtaining of the dialogue text further comprises:

[0032] It is determined that the dialogue text contains an out-of-vocabulary word of the text dictionary;

[0033] The out-of-vocabulary word is input into a hidden Markov model, and a start probability, an emission probability and a transition probability are output by the hidden Markov model;

[0034] The part of speech of the out-of-vocabulary word is determined according to the start probability, the emission probability and the transition probability.

[0035] In some possible embodiments, the segmenting of the third material text to obtain a candidate word and the converting of the candidate word into a fixed-dimensional word vector by the encoder comprise:

[0036] The word vectors are combined to obtain a word vector sequence, and a first identifier is inserted at a start position of the word vector sequence; the first identifier is used to aggregate representation information of the word vectors in the word vector sequence.

[0037] To achieve the above object, a second aspect of the embodiment of the present application provides a text processing device, which comprises:

[0038] A material obtaining unit is configured to obtain dialogue text, wherein the dialogue text comprises a current question, historical question and answer, and a keyword.

[0039] a text processing unit, configured to splice the current question, the history question and answer and the keyword by a first field separator to obtain a first candidate text;

[0040] a first encoding unit, configured to encode the first candidate text according to a context relationship to obtain a first material text;

[0041] a second encoding unit, configured to obtain a reference text, encode the reference text according to the context relationship to obtain a second material text, and the reference text comprises a target question;

[0042] a third encoding unit, configured to splice the first material text and the second material text by a second field separator to obtain a third material text, and determine a start position and an end position of a target text according to the third material text, and the target text is an answer content of the target question;

[0043] an answer output unit, configured to extract and output the target text according to the start position and the end position.

[0044] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0045] To achieve the above object, a fourth aspect of the embodiment of the present application provides a storage medium, which is a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0046] The text processing method and device, electronic device and storage medium provided by the present application, wherein the method first encodes the current question which has interacted with the history question and answer, and then splices the encoded reference text to let the reading comprehension model automatically learn and complete the semantics of the question in the hidden encoding layer; in order to enhance the problem completion ability of the model, the method also introduces the keyword enhancement mode based on the part-of-speech tagging, which explicitly guides the model to focus on the keywords such as nouns and verbs, so as to enhance the problem completion ability of the model, so that the reading comprehension model can start from a more smooth target function, avoid falling into a local minimum point, better find a global optimal point, and the obtained prediction result is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a step flowchart of the text processing method provided by the embodiment of the present application;

[0048] Figure 2 is a model structure schematic diagram in an embodiment of the present application;

[0049] Figure 3 is a step flowchart of a target question and answer model training process in an embodiment of the present application;

[0050] Figure 4 is a step flowchart of model optimization by a loss function in a target question and answer model training process in an embodiment of the present application;

[0051] Figure 5 is a step flowchart of prediction by a trained target question and answer model in an embodiment of the present application;

[0052] Figure 6 is a step flowchart of keyword recognition and part-of-speech tagging in an embodiment of the present application;

[0053] Figure 7 is a step flowchart of processing of an unregistered word in an embodiment of the present application;

[0054] Figure 8 is a structure schematic diagram of a text processing apparatus provided in an embodiment of the present application;

[0055] Figure 9 is a hardware structure schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0057] It should be noted that although the functional modules are divided in the apparatus schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0059] First, several terms involved in the present application are analyzed:

[0060] Artificial intelligence (AI): a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0061] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics, also known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0062] Information extraction: a text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, chapters, and text information is composed of small specific units, such as words, phrases, sentences, paragraphs or combinations of these specific units. Extracting noun phrases, names, places and other information from text data is text information extraction. Of course, the information extracted by the text information extraction technology can be various types of information.

[0063] Based on the foregoing background art or related principles, the embodiments of the present application provide a text processing method and device, electronic equipment and storage medium, aiming to improve the accuracy of the output result of the reading comprehension model.

[0064] The recommendation method and device, electronic equipment and storage medium provided by the embodiments of the present application are specifically explained by the following embodiments. First, the text processing method in the embodiments of the present application is described.

[0065] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0066] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0067] The text processing method provided by the embodiments of the present application relates to the field of artificial intelligence technology. The text processing method provided by the embodiments of the present application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; and the software can be an application for implementing the text processing method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0069] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0070] As described in the background section, reading comprehension models need to find the corresponding answer from the reference text based on the current question. However, unlike typical single-turn reading comprehension models, most questions are based on the dialogue context, meaning that accurately understanding the semantics of the current question requires considering the context of the dialogue. For example, in a multi-turn dialogue, B asks, "What does Tom's cat like to do?"; A replies, "Tom's cat likes to sleep." B asks, "Why is that?", omitting the subject "cat" from the previous context. "Likes to sleep" refers to "likes to sleep." Existing solutions often combine a substitution resolution model with a reading comprehension model. First, the current question is concatenated with historical question-and-answer content, allowing the model to explicitly complete the semantics of the current question. For example, inputting B: "What does Tom's cat like to do?"; A: "Tom's cat likes to sleep," B: "Why is that?", the reading comprehension model is expected to output "Why does Tom's cat like to sleep?". Then, the completed question is concatenated with the reference text and input into the reading comprehension model to generate the answer. This approach requires explicit substitution resolution data annotation for training and is prone to error propagation. The problem setting in this application only considers extractive question-and-answer, meaning the model outputs a fragment from the reference text. For example... Figure 1 The diagram shown is an optional flowchart of a text processing method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106:

[0071] Step S101: Obtain the dialogue text; wherein, the dialogue text includes the current question, historical questions and answers, and keywords;

[0072] Specifically in the embodiments, the dialogue text is text content that can include any interactive dialogue, which can be long-term stored in the collected background data or short-term cached text data. For example, a set of multi-round dialogue content collected in the embodiments is B: "How is the personality of Betty's cat?", A: "Betty's cat is very clingy", B: "Why is this so?" This is a three-round dialogue. In this dialogue text, the current question can be "Why is this so?", the historical question and answer can be "How is the personality of Betty's cat?", and "Betty's cat is very clingy". After the dialogue text, the embodiments can perform simple preprocessing on the text content, for example, identifying and labeling keywords in the text, wherein the keywords include but are not limited to verbs or nouns in the dialogue text. Specifically in the embodiments, the set of words with the part of speech of verbs and nouns in the historical dialogue segment in the current dialogue can be labeled by using three-party part of speech labeling.

[0073] Step S102, the current question, the historical question and answer, and the keywords are spliced by a first field separator to obtain a first candidate text;

[0074] The first candidate text refers to the text obtained by splicing the current question, the historical question and answer, and the keywords. Specifically in the embodiments, the current question, the historical question and answer, and the keywords (set) obtained in step S101 are spliced, and the SEP symbol commonly used in the BERT model can be used for text splicing in the embodiments. SEP is a formal parameter of a function, and in most cases, the SEP parameter is used to specify the separator symbol of characters. For example, according to the regular expression sep='s+', it can be determined that the text content in the embodiments can be spliced by at least one space symbol. For example, the text splicing method in the embodiments is to connect the current question with the keywords by a SEP symbol, then connect the keywords by a SEP symbol, and finally connect the historical question and answer content by a SEP symbol.

[0075] Step S103, text encoding the first candidate text according to the context relationship to obtain a first material text;

[0076] The first material text specifically refers to the text content obtained by vectorization processing and encoding processing. Specifically in the embodiments, the model constructed in the technical scheme of the present application is a unified coreference resolution and reading comprehension model based on a double tower. For example, Figure 2As shown, the double-tower structure refers to at least two encoders in the model. Based on the double-tower structure, the reading comprehension model in the embodiment can divide the prediction output process into two stages. In the first stage, the candidate text obtained by text splicing in step S102 is segmented and encoded, and then input into an encoder to learn the context. Specifically, the encoder in the embodiment can use a pre-trained BERT family encoder, and BERT family models such as Roberta and albert are applicable, but the embodiment only exemplifies the implementation of the encoder and does not limit the implementation thereof. Taking BERT as an example, the input and output of the model are both 512x768 encodings, and the embedding output in the stage is extracted according to the current question part, so that the first material text in the embodiment can be obtained.

[0077] In step S104, the reference text is obtained, and the reference text is text encoded according to the context relationship to obtain the second material text.

[0078] The content of the reference text can include the current question in the dialogue text, i.e., the target question; in addition, the reference text can also include other text materials that have the same or similar semantics as the dialogue text; the second material text refers to the material text obtained by encoding the reference text by another encoder in the double-tower structure in the embodiment. Specifically, the second encoder and the first encoder can adopt the same structure in the embodiment, for example, a pre-trained BERT family encoder, and similarly, the input and output of the model are both 512x768 encodings, and the embedding output in the stage is extracted according to the current question part, so that the second material text in the embodiment can be obtained.

[0079] In step S105, the first material text and the second material text are text spliced by the second field separator to obtain the third material text, and the start position and the end position of the target text are determined according to the third material text; wherein the target text is the answer content of the target question.

[0080] Specifically in the embodiments, in the second stage of answer prediction, firstly, the first material text coded in step S103 and the second material text coded in step S104 need to be spliced; then the spliced material text is input into a pre-trained classification prediction model, for example, an additional BERT model, in which two softmax linear classifiers are used to calculate the probability of each individual word (vector representation) in the reference text belonging to the start and end positions of the answer, and the loss value can be calculated by the cross-entropy function to optimize the parameters of the model. It should be noted that in the splicing process in step S105, the SEP symbol can also be used to splice the first stage of the current question encoding content and the reference text encoding content, and then input into the BERT model for classification prediction.

[0081] In step S106, the target text is extracted and output according to the start position and the end position.

[0082] Specifically in the embodiments, through the BERT model in step S105, according to the context semantics of the text, the classification prediction of each word in the input material text is performed to determine whether it belongs to the start position word or the end position word of the target text. After determining the start position word and the end position word of the target text, the content of the target text is output as the answer to the target question.

[0083] It should be noted that in addition to the BERT model, the language model used in the embodiments can be a transformer model, an LSTM model or a GPT-2 model, etc.

[0084] Taking the transformer model as an example, the transformer model includes an encoder and a decoder. The training process of the language model includes: obtaining a text segment containing a current question and a reference text segment, and inputting the two text segments into the transformer model. The text segment is encoded by the encoder of the transformer model to obtain a hidden feature vector of the text segment, and the hidden feature vector of the sample text segment is decoded by the decoder of the transformer model, so that the hidden feature vector of the sample text segment is multiplied by the reference word vector to obtain a text description word segment. The similarity between the text description word segment and the reference video description word segment is calculated by the loss function of the transformer model, and the loss function of the transformer model is optimized according to the similarity. The model loss of the loss function is back propagated, and the model parameters are continuously adjusted until the similarity is greater than or equal to a similarity threshold, the optimization of the transformer model is stopped, and a required language model is obtained.

[0085] In an embodiment, as shown in Figure 3 The method includes the step S104 of determining the start position and the end position of the target text according to the third material text, by text splicing the first material text and the second material text by the second field separator to obtain the third material text, which includes the step S1041 of:

[0086] inputting the third material text into the target question and answer model trained, and determining the start position and the end position of the target text;

[0087] In an embodiment, the start position and the end position of the target answer are predicted and output by the pre-trained classification prediction model after the spliced material text. In an embodiment, in order to effectively improve the training efficiency and robustness of the model, avoid data noise and difficult samples to cause underfitting of the model, the training process of the target question and answer model can include steps S10411-S10413:

[0088] S10411, cutting the training text material to obtain a plurality of training data sets;

[0089] Specifically, in this embodiment, the final constructed and trained target question-answering model is an extractive question-answering model; for example, the current question and reference text in the historical text data are directly concatenated and input into the question-answering extraction model built based on the BERT model for training. The historical text data serves as training data, and its preprocessing and concatenation methods are the same as those in steps S102-S103, and will not be repeated here. In this embodiment, all the integrated training data is divided into several training datasets. For example, in this embodiment, the data is divided into ten parts, and each training dataset is used to train a separate model.

[0090] S10412. Input several training datasets into several candidate question-answering models to be trained, and determine the difficulty score based on the model output and the preset evaluation index.

[0091] Specifically, in the embodiment, the training data is divided into several training datasets in step S10411. Taking 10 training datasets as an example, in the embodiment, the training dataset is used as a sample. For each sample, inference is performed by the other N-1 non-corresponding models to obtain N-1 results. All results are regarded as a dataset and standardized by the character-level F1 value to obtain the difficulty score of this data sample. The F1 value is a statistical indicator used to measure the accuracy of a binary classification model, which takes into account both the precision and recall of the classification model. Furthermore, the F1 score can be regarded as a harmonic average of the model's precision and recall.

[0092] S10413. Sort the training dataset according to the difficulty score to obtain the training set sequence. Train the candidate question answering model using the training set sequence to obtain the target question answering model.

[0093] Specifically, in the embodiment, in step S10412, for each sample, inference is performed using the other N-1 models, resulting in N-1 results. All results are treated as a dataset, and a character-level F1 score is used to determine the difficulty score of this data sample. The final difficulty is then sorted in ascending order for training from easy to difficult, forming a training set sequence. In this embodiment, the training datasets in the training set sequence are fed to the target question-answering model in an orderly manner from easy to difficult, effectively improving the training efficiency and robustness of the model, and avoiding underfitting caused by data noise and difficult samples.

[0094] like Figure 4 As shown, in some feasible implementations, the training process of the target question-answering model may further include steps S10414-S10416:

[0095] S10414, acquire the first start word and the first end word of the original answer in the training text material;

[0096] In the training process of the target question answering model, in order to make the prediction result of the finally obtained target question answering model more accurate, the loss value between the prediction value of the model in the training process and the original answer in the training text material input into the model can be calculated, and the parameters of the target question answering model in the training stage are optimized by making the loss value converge. In the process of loss value calculation, the word at the start position and the word at the end position of the original answer in the training text material need to be acquired first.

[0097] S10415, acquire the second start word and the second end word of the target text predicted by the target question answering model;

[0098] Secondly, the embodiment collects the word at the start position and the word at the end position of the answer output by the model in the training stage.

[0099] S10416, calculate the first loss value according to the first start word and the second start word through the cross-entropy loss function, and / or calculate the second loss value according to the first end word and the second end word through the cross-entropy loss function;

[0100] S10417, parameter tuning of the target question answering model according to the first loss value and / or the second loss value;

[0101] In the embodiment, the cross-entropy loss function is used to calculate the loss value between the start words and the end words. In the embodiment, the word at the start position and the word at the end position in the training text material are determined by the label annotation method, and the probability distribution P(x) of the word position of the original (real) answer is also determined, so the information entropy is constant. In the calculation process of cross-entropy, the difference between the real probability distribution P(x) and the predicted probability distribution Q(x) is represented by relative entropy, and the smaller the value is, the better the prediction result is, so it is necessary to minimize the relative entropy, and the cross-entropy is equal to the relative entropy plus a constant (information entropy), so the loss value of the model training process is calculated through the cross-entropy, and the model is adjusted according to the calculated loss value, until the loss value converges, and the final target question answering model (parameters) is obtained, which can improve the operation efficiency and accuracy in the prediction process.

[0102] The target question answering model is trained according to steps S10411-S10416, and mainly includes two core functional structures: an encoder and a classifier. The target question answering model constructed based on the BERT model converts each word into a vector form through an input token embedding layer, i.e., an encoding layer, so as to obtain a vectorized representation Token of each word after coding. Two softmax linear classifiers are spliced at the output end of the BERT to calculate the probability of each token in the reference text belonging to the start and end positions of the answer. Further, the target question answering model in the embodiment includes an encoder and a classifier. As shown in FIG. 8, the first material text and the second material text are spliced into a third material text through a second field separator, and the start position and the end position of the target text are determined according to the third material text. The keyword prediction can include steps S1042-S1045: Figure 5

[0103] S1042, the third material text is segmented to obtain candidate words, and the candidate words are converted into fixed-dimensional word vectors by the encoder;

[0104] In the embodiment, the BERT performs tokenization processing on the input text, i.e., the third material text, to obtain a plurality of candidate words. Each candidate word (token) in the third material text is input into the token embedding layer to convert each word into a vector form. More specifically, the token embedding layer in the target question answering model is used to convert each word into a fixed-dimensional vector. In the target question answering model of the embodiment, each word is converted into a 768-dimensional vector representation. The input text needs to be tokenized before being input into the token embedding layer. In addition, another special token is inserted at the beginning of the tokenization result ([CLS]), which can be regarded as a classification task and a division sentence pair service.

[0105] S1043, the spliced sentence in the third material text is represented by the encoder to obtain a sentence vector;

[0106] ​In specific embodiments, the encoder in the target QA model includes two embedding layers in addition to the token embeddings layer, namely, a segment embeddings layer and a position embeddings layer. The target QA model can process classification prediction of the target answer (word position) for the input based on the text material. The task is to determine whether the first text material and the second text material in the concatenation are semantically similar. The segment embeddings layer has only two vector representations. The first vector assigns 0 to each token in the first sentence, and the second vector assigns 1 to each token in the second sentence.

[0107] S1044, obtaining, by the encoder, a position relationship between candidate words in the third material text;

[0108] In specific embodiments, the addition of the position embeddings allows the target QA model to understand that the first "I" and the second "I" in the sentence "I think, therefore I am" should have different vector representations. BERT can process input sequences of up to 512 tokens. The embodiments obtain the sequence order information encoding of the input by allowing the target QA model to learn a vector representation at each position.

[0109] S1045, outputting, by the classifier, a probability that the candidate word belongs to the start position or a probability that the candidate word belongs to the end position according to the word vector, the sentence vector, and the position relationship.

[0110] In specific embodiments, the token embeddings output a word vector representation of (1, n, 768); the segment embeddings layer outputs a (1, n, 768) vector representation to help the model distinguish between the two sentences in the sentence pair; and the position embeddings layer outputs a (1, n, 768) position vector to allow the model to learn the order attribute of the input. The outputs of the three are added element by element to obtain a combined representation with a size of (1, n, 768). Then, the original Masked Language Model task and the Next Sentence Prediction task in the BERT model are executed to obtain the start position candidate word (vector representation) and the end position candidate word (vector representation) from the task output result. The corresponding probabilities are output by the softmax layer.

[0111] In some possible embodiments, the step S1041 of segmenting the third material text to obtain candidate words and converting the candidate words into word vectors of a fixed dimension through an encoder can further include a processing procedure of identifier insertion, i.e., combining the word vectors to obtain a word vector sequence, and inserting a first identifier at a starting position of the word vector sequence; the first identifier is used to aggregate representation information of the word vectors in the word vector sequence.

[0112] In specific embodiments, in order to complete a specific classification task, the author inserts a specific classification token ([CLS]) at the beginning of each sequence of input in addition to the token of the word, and the last output of the classification token corresponding to the last Transformer layer is used to aggregate the representation information of the entire sequence.

[0113] As shown in FIG. 11, in some possible embodiments, since the dialogue text obtained in the step S101 can include the current question, the historical question and answer, and the keywords and the like, after obtaining the dialogue text, it is necessary to perform necessary recognition extraction and part-of-speech tagging on the keywords contained in the text, and the processing procedure can include steps S1011-S1012. Figure 6

[0114] S1011, obtaining a text dictionary, performing regular matching on the dialogue text according to the text dictionary to obtain a plurality of text sentences;

[0115] In specific embodiments, the recognition of complete keywords and part-of-speech tagging can be based on a dictionary lookup algorithm of string matching and a statistical algorithm. Taking the process of part-of-speech tagging implemented by jieba segmentation as an example, in the embodiments, first, a dictionary is initialized, the sentence is converted into UTF-8 or GBK, the input text is separated into sentences according to regular matching.

[0116] S1012, constructing a directed acyclic graph according to the plurality of text sentences, traversing a path of each node in the directed acyclic graph, and determining the keywords and the parts of speech of the keywords according to the path;

[0117] In specific embodiments, based on the sentences separated in the step S1011, first, a directed acyclic graph (DAG) of the sentences is constructed, and then a Route is constructed by dynamic programming, the maximum path probability of each node in the DAG from the end of the sentence to the beginning of the sentence, and the end position of the word corresponding to the node when the probability is maximum are calculated, each node Route is traversed, the word combination is assembled, and the single segmentation and part-of-speech tagging of each sentence are performed.

[0118] In some possible implementation scenarios, there can be an out-of-vocabulary word that is not recorded in the dictionary in the dialogue text obtained in the step S101, for example, Figure 7 ​As shown, for such an out-of-vocabulary word, an HMM hidden Markov model can be used in the embodiment to process, and the processing process includes S1013-S1015:

[0119] S1013, determine that the dialogue text has an out-of-vocabulary word of a text dictionary;

[0120] Specifically, in the embodiment, during the traversal process in S1012, if the word is not in the dictionary, i.e., an out-of-vocabulary word, an HMM hidden Markov model is used for segmentation.

[0121] S1014, input the out-of-vocabulary word into the hidden Markov model, and output the start probability, emission probability and transition probability through the hidden Markov model;

[0122] Specifically, in the embodiment, the start probability start_p, the emission probability emit_p and the transition probability trans_p can be obtained through the viterbi algorithm and large-scale corpus statistics. Specifically, in the algorithm implementation of the embodiment, three files prob_start.py, prob_emit.py and prob_trans.py are respectively corresponding to the probabilities of the word in the four cases of BEMS and each part of speech.

[0123] S1015, determine the part of speech of the out-of-vocabulary word according to the start probability, the emission probability and the transition probability;

[0124] Specifically, in the embodiment, the viterbi algorithm is then used to obtain the three probabilities, and the observation sequence (the segmented sentence) is converted to obtain the hidden sequence (the part of speech annotation sequence). In this way, the embodiment completes the part of speech annotation of the out-of-vocabulary word.

[0125] In combination with the drawings in the specification Figure 2 The specific implementation process of the technical solution of the present application is described in detail:

[0126] For a certain dialogue-based human-computer interaction scene, the embodiment first uses three-party part of speech annotation to annotate the word set with the part of speech of verb and noun in the historical dialogue segment in the current dialogue, uses the SEP symbol to splice the current question, the keyword and the historical question and answer segment, and then performs segmentation and encoding. Then input the ENCODER to learn the context. The ENCODER can use a pre-trained BERT family encoder, which can be but is not limited to BERT. The model pre-trained in a large-scale corpus can fully interact with the text. Taking BERT as an example, the input and output of the model are both 512*768 encodings. The embodiment extracts the embedding output by the stage corresponding to the current question part. For the reference text, the embodiment also encodes the text using BERT to generate a corresponding length of text encoding.

[0127] In the second stage embodiment, the SEP symbol is used to splice the current question learned in the first stage and the reference text encoding into an additional BERT for interaction. At the output end of the BERT, two softmax linear classifiers are spliced to calculate the probability of each token in the reference text belonging to the start and end positions of the answer, respectively, and the cross-entropy function is used for calculation.

[0128] The BERT model used in the second stage is trained as follows: the current question and the reference text in the training data are spliced and input into the BERT question and answer extraction model for training. The training data is divided into ten parts, and a model is trained for each part. For each sample, N-1 other non-corresponding models are used for reasoning, i.e., N-1 results are obtained. All results are regarded as a data set, and the character-level F1 value is used for scoring to obtain the difficulty score of the data sample. The final difficulty is ranked in ascending order for training from easy to difficult.

[0129] Please refer to Figure 8 The embodiments of the present application also provide a text processing device, which can implement the above-mentioned text processing method. The device comprises:

[0130] A material acquisition unit is configured to acquire dialogue text, wherein the dialogue text comprises a current question, historical question and answer, and a keyword;

[0131] A text processing unit is configured to splice the current question, the historical question and answer, and the keyword by using a first field separator to obtain a first candidate text;

[0132] A first encoding unit is configured to encode the first candidate text according to a context relationship to obtain a first material text;

[0133] A second encoding unit is configured to acquire a reference text, and encode the reference text according to the context relationship to obtain a second material text, wherein the reference text comprises a target question;

[0134] A third encoding unit is configured to splice the first material text and the second material text by using a second field separator to obtain a third material text, and determine a start position of a target text and an end position of the target text according to the third material text, wherein the target text is an answer content of the target question;

[0135] An answer output unit is configured to extract and output the target text according to the start position and the end position.

[0136] The specific implementation of the text processing device is basically the same as that of the above-mentioned text processing method, and will not be repeated here.

[0137] The electronic device of another embodiment is illustrated in the hardware structure, and the electronic device includes:

[0138] Please refer to Figure 9 , Figure 9 The electronic device of another embodiment is illustrated in the hardware structure, and the electronic device includes:

[0139] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0140] The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the text processing method of the embodiments of the present application.

[0141] The input / output interface 903 is configured to realize information input and output.

[0142] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0143] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0144] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between them in the device.

[0145] The embodiments of the present application further provide a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program. The computer program is executed by a processor to implement the above-mentioned text processing method.

[0146] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include memory that is remotely located with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The text processing method, text processing device, electronic equipment and storage medium provided by the embodiments of the present application encode the current question interacted with the question and answer history, then splice the encoded reference text, and let the reading comprehension model automatically learn to complete the semantics of the question in the hidden encoding layer; in order to enhance the problem completion ability of the model, the method also introduces a key word enhancement method based on part-of-speech tagging, which explicitly guides the model to focus on key words such as nouns and verbs, to enhance the problem completion ability of the model, so that the reading comprehension model can start fitting from a more smooth objective function, avoid falling into a local minimum, and better find a global optimum.

[0148] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0149] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0150] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0151] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0152] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but is used to connect between similar objects.

[0153] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0154] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0155] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the application.

[0156] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0157] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0158] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A text processing method characterized by, The method comprises: acquiring dialogue text, the dialogue text comprising a current question, historical questions and answers, and a keyword; concatenating the current question, the historical questions and answers, and the keyword by a first field separator to obtain a first candidate text; text encoding the first candidate text according to a context relationship to obtain a first material text; acquiring a reference text, text encoding the reference text according to a context relationship to obtain a second material text, the reference text comprising a target question; concatenating the first material text and the second material text by a second field separator to obtain a third material text, and determining a start position and an end position of a target text according to the third material text, the target text being an answer to the target question; extracting and outputting the target text according to the start position and the end position; determining the start position and the end position of the target text by inputting the third material text into a target question and answer model trained, wherein the target question and answer model comprises an encoder and a classifier, and the method comprises the following steps: segmenting the third material text to obtain candidate words, and converting the candidate words into word vectors of a fixed dimension by the encoder; vectorizing the third material text by the encoder to obtain a sentence vector; acquiring a position relationship between the candidate words in the third material text by the encoder; outputting, by the classifier, a probability that the candidate words belong to a start position or an end position according to the word vectors, the sentence vector, and the position relationship. The method comprises: acquiring a text dictionary, and performing regular matching on the dialogue text according to the text dictionary to obtain a plurality of text sentences; constructing a directed acyclic graph according to the plurality of text sentences, traversing a path of each node in the directed acyclic graph, and determining the keyword and the part of speech of the keyword according to the path. The training process of the target question and answer model further comprises the following steps: ​ ​ 2. The text processing method of claim 1, wherein, ​ ​ ​ 3. The text processing method of claim 1, wherein, ​ obtaining a first start word and a first end word of an original answer in the training text material; obtaining a second start word and a second end word of the target text predicted by the target question and answer model; calculating a first loss value according to the first start word and the second start word by using a cross-entropy loss function, and / or calculating a second loss value according to the first end word and the second end word by using the cross-entropy loss function; parameter tuning the target question and answer model according to the first loss value and / or the second loss value.

4. The text processing method of claim 2, wherein, The obtaining of the dialogue text further comprises: determining that the dialogue text contains an out-of-vocabulary word of the text dictionary; inputting the out-of-vocabulary word into a hidden Markov model to output a start probability, an emission probability and a transition probability by using the hidden Markov model; determining the part of speech of the out-of-vocabulary word according to the start probability, the emission probability and the transition probability.

5. The text processing method of claim 1, wherein, The tokenization of the third material text to obtain a candidate word and the conversion of the candidate word into a fixed-dimensional word vector by using the encoder comprise: combining the word vectors to obtain a word vector sequence, and inserting a first identifier at a start position of the word vector sequence; the first identifier is used to aggregate the representation information of the word vectors in the word vector sequence.

6. A text processing apparatus characterized by comprising: The device comprises: a material acquisition unit configured to acquire dialogue text, the dialogue text comprising a current question, a history question and answer, and a keyword; a text processing unit configured to concatenate the current question, the history question and answer, and the keyword by using a first field separator to obtain a first candidate text; a first encoding unit configured to encode the first candidate text according to a context relationship to obtain a first material text; a second encoding unit configured to acquire a reference text, encode the reference text according to a context relationship to obtain a second material text, and the reference text comprising a target question; a third encoding unit configured to concatenate the first material text and the second material text by using a second field separator to obtain a third material text, and determine a start position of a target text and an end position of the target text according to the third material text; the target text being an answer content of the target question; an answer output unit configured to extract and output the target text according to the start position and the end position; The third encoding unit further comprises: inputting the third material text into a target question and answer model trained to determine the start position of the target text and the end position of the target text; The training process of the target question and answer model comprises the following steps: segmenting training text material to obtain a plurality of training data sets; inputting a plurality of training data sets into a plurality of candidate question and answer models to be trained, and determining a difficulty score according to a model output result and a preset evaluation index; sorting the training data sets according to the difficulty score to obtain a training set sequence, and training the target question and answer model by using the training set sequence; The target question and answer model comprises an encoder and a classifier; and the third encoding unit further comprises: The third material text is segmented to obtain a candidate word, and the candidate word is converted into a fixed-dimension word vector by the encoder; The third material text is vectorized to obtain a sentence vector by the encoder; The position relationship between the candidate words in the third material text is obtained by the encoder; The classifier outputs a probability that the candidate word belongs to a start position or a probability that the candidate word belongs to an end position according to the word vector, the sentence vector, and the position relationship.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the text processing method in any one of claims 1 to 5 when executing the computer program.

8. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the text processing method in any one of claims 1 to 5.

Images