Text processing method and device based on multiple rounds of question and answer, equipment and storage medium

Abstract

Embodiments of the present application provide a text processing method and device based on multi-round question and answer, equipment and storage medium, belonging to the technical field of financial technology. The method comprises: obtaining a first question text and a first answer text in a historical round to splice and construct a first text set; inputting the first text set into a to-be-trained model for dialogue modeling training to obtain a target model; inputting the first text set corresponding to a plurality of historical rounds into the target model to output semantic information; generating a first vector of the historical round according to the semantic information, combining the first vector with a second vector of a second dialogue text to obtain a first fusion vector; and performing target answer prediction to determine a start position, an end position and a target answer in the second text. The present application scheme can have stronger historical analysis capability; and by simulating the memory mode, the historical dialogue is modeled, which can effectively reduce the noise and improve the accuracy and interpretability of the prediction output result.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a text processing method, apparatus, device and storage medium based on multi-turn question answering. Background Technology

[0002] With the development of computer technology, more and more technologies (such as big data, cloud computing, or blockchain) are being applied in the financial field, and the traditional financial industry is gradually transforming into fintech. For example, in financial transactions, models based on conversational reading comprehension are frequently used to provide interactive services. Conversational reading comprehension, also known as multi-turn question-and-answer reading comprehension, requires the model to find the corresponding answer from the reference text based on the current question. However, unlike typical single-turn reading comprehension models, most questions are based on the dialogue context; that is, accurately understanding the semantics of the current question requires considering the context of the dialogue.

[0003] Historical modeling and unreferencing are the biggest challenges facing existing multi-turn dialogue systems. Historical modeling refers to the interconnectedness of questions and answers across different turns in a dialogue; for a long dialogue, a question or answer from the beginning might be mentioned at the end. Unreferencing is similar to historical modeling but fundamentally different. Unreferencing occurs when a pronoun in a question actually refers to a subject from a previous turn, while historical modeling focuses more on the changes and retention of explicit dialogue information itself.

[0004] In the relevant technical solutions, the most common approach in the historical modeling process of semantic models is to concatenate the question and answer from all previous rounds with the current question and then let the model perform reasoning. This approach is prone to introducing a lot of noise, resulting in low accuracy of prediction and reasoning results, poor interpretability, and unsatisfactory interactive service with customers. Summary of the Invention

[0005] The main objective of this application is to propose a text processing method, apparatus, device, and storage medium based on multi-turn question answering, thereby improving the accuracy of answer reasoning during multi-turn question answering.

[0006] To achieve the above objectives, a first aspect of this application proposes a text processing method based on multi-turn question answering, the method comprising the following steps:

[0007] Obtain the first dialogue text of the historical rounds, concatenate the first question text and the first answer text in the first dialogue text, and construct the first text set based on the concatenated text;

[0008] Construct a model to be trained by inputting the first text set into the model to be trained for dialogue modeling training, and obtain the trained target model.

[0009] Input the first set of texts corresponding to several historical rounds into the target model, and output semantic information;

[0010] A first vector corresponding to the historical round is generated based on the semantic information, and the first vector is combined with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector; the first vector represents the memory information of the historical round determined based on the semantic information, and the second vector is an intermediate vector obtained by feature encoding of the first text set through the target model;

[0011] The target answer is predicted based on the first fusion vector, the first start position and the first end position in the second text are determined, and the target answer is determined based on the first start position and the first end position.

[0012] In some feasible embodiments, obtaining the first dialogue text of historical rounds, concatenating the first question text and the first answer text in the first dialogue text, and constructing a first text set based on the concatenated text includes:

[0013] Extract the first question text and the first answer text from the first dialogue text;

[0014] Based on the position information of the first answer text in the first dialogue text, a position tag is generated, the position tag including the second start position and the second end position of the first answer text in the first dialogue text;

[0015] Sample data was constructed based on the first question text, the first answer text, and the location tags;

[0016] The first text set is constructed by sorting the sample data.

[0017] In some feasible embodiments, the step of constructing sample data based on the first question text, the first answer text, and location tags includes:

[0018] The first question text and the first answer text are concatenated using the first connector symbol to obtain the first concatenated text;

[0019] The position label corresponding to the first answer text in the first concatenated text is determined to obtain the sample data.

[0020] In some feasible embodiments, the step of constructing the model to be trained, which involves inputting the first text set into the model to be trained for dialogue modeling training to obtain the trained target model, includes:

[0021] The model to be trained is obtained for dialogue modeling training, and the predicted answer is output. The third starting position and the third ending position of the predicted answer are determined.

[0022] A first loss value is calculated based on the third starting position and the second starting position using the cross-entropy loss function, and / or a second loss value is calculated based on the third ending position and the second ending position using the cross-entropy loss function;

[0023] The target model is then optimized based on the first loss value and / or the second loss value.

[0024] In some feasible embodiments, the step of inputting the first text set corresponding to several historical rounds into the target model and outputting semantic information includes:

[0025] The first embedding matrix is ​​constructed based on the number of dialogue turns in the first text set, the length of the text material in the first text set, and the dimension of the hidden layer in the model to be trained.

[0026] The first embedding matrix is ​​transformed according to the number of dialogue turns to obtain the second embedding matrix;

[0027] The semantic information is obtained by performing unidirectional self-attention calculation based on the second embedding matrix.

[0028] In some feasible embodiments, generating a first vector corresponding to the historical round based on the semantic information, and combining the first vector with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector includes:

[0029] The third embedding matrix of the semantic information is copied, and the first vector is determined based on the copied third embedding matrix and the weight matrix.

[0030] The second dialogue text of the current round is input into the target model, and the second vector is obtained by encoding it through the target model.

[0031] The first vector and the second vector are combined to obtain the first fused vector.

[0032] In some feasible embodiments, after copying the third embedding matrix of the semantic information and determining the first vector based on the copied third embedding matrices and the weight matrix, the method further includes:

[0033] The first fusion vector is combined with the third vector of the third dialogue text to obtain the second fusion vector; the third dialogue text includes the dialogue text of the next round of the second dialogue text; the third vector is an intermediate vector obtained by feature encoding of the third dialogue text through the target model.

[0034] The target answer is predicted based on the second fusion vector.

[0035] To achieve the above objectives, a second aspect of this application provides a text processing apparatus based on multi-turn question-and-answer, the apparatus comprising:

[0036] The sample construction unit is used to obtain the first dialogue text of the historical rounds, concatenate the first question text and the first answer text in the first dialogue text, and construct the first text set based on the concatenated text;

[0037] The model training unit is used to construct the model to be trained. The first text set is input into the model to be trained to perform dialogue modeling training, and the trained target model is obtained.

[0038] The semantic acquisition unit is used to input the first set of texts corresponding to several historical rounds into the target model and output semantic information.

[0039] A semantic fusion unit is used to generate a first vector corresponding to the historical round based on the semantic information, and combine the first vector with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector; the first vector represents the memory information of the historical round determined based on the semantic information, and the second vector is an intermediate vector obtained by feature encoding of the first text set through the target model;

[0040] The answer prediction unit is used to predict the target answer based on the first fusion vector, determine the first start position and the first end position in the second text, and determine the target answer based on the first start position and the first end position.

[0041] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0042] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0043] The text processing method, apparatus, device, and storage medium based on multi-turn question answering proposed in this application, by simulating human memory mechanisms, interacts with historical memory information for each round of input dialogue content, combines the memory information to output the specific location of the target answer in the text, and determines the final answer text based on this location. The scheme ensures that each dialogue can be explicitly and fully modeled in a temporal sequence through dynamic training, giving the model a stronger historical analysis capability. Furthermore, by simulating human memory and explicitly modeling historical dialogues, noise can be effectively reduced, and the accuracy and interpretability of the predicted output results can be improved, thereby enhancing the interactive service effect for customers in the financial field. Attached Figure Description

[0044] Figure 1 This is a flowchart of a text processing method based on multi-turn question answering provided in an embodiment of this application;

[0045] Figure 2 yes Figure 1 The flowchart of step S101 in the text;

[0046] Figure 3 yes Figure 1 The flowchart of step S102 in the document;

[0047] Figure 4 yes Figure 3 The flowchart of step S1023 in the process;

[0048] Figure 5 yes Figure 1 The flowchart of step S103 in the process;

[0049] Figure 6 yes Figure 1 The flowchart of step S104 in the process;

[0050] Figure 7 yes Figure 6 The flowchart following step S1041;

[0051] Figure 8 This is a schematic diagram of the structure of a text processing device based on multi-turn question answering provided in an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0056] First, let's analyze some of the terms used in this application:

[0057] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0058] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0059] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0060] Based on this, embodiments of this application provide a text processing method, apparatus, device, and storage medium based on multi-turn question answering, aiming to improve the accuracy of answer reasoning in the multi-turn question answering process.

[0061] The text processing methods, apparatuses, electronic devices, and storage media provided in this application are specifically described through the following embodiments. First, the text processing methods in the embodiments of this application are described.

[0062] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0063] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0064] The text processing method based on multi-turn question answering provided in this application relates to the field of financial technology. For example, in this application embodiment, it can be used in general question answering systems aimed at finding accurate information from structured (such as knowledge bases, tables) and unstructured (such as documents) sources to answer customers' financial questions; task- or goal-oriented dialogue systems that need to achieve a specific task or goal through interaction, such as various intelligent assistants, ticketing systems, etc.; this application does not limit the specific financial business involved, such as loan processing, fund consultation, etc.

[0065] The text processing method based on multi-turn question-and-answer provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the text processing method based on multi-turn question-and-answer, etc., but is not limited to the above forms.

[0066] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0067] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0068] As indicated in the background technical solution, current solutions in the financial field typically concatenate all historical question-and-answer rounds with the current question and feed it to the model for reasoning. This approach not only easily introduces a lot of noise but also lacks interpretability. Furthermore, this method directly delegates the task of dereference resolution to the model itself. Due to the inherent diversity of question-and-answer tasks, the model will largely not learn historical modeling and referencing as we expect. Therefore, explicit modeling becomes essential, using functions or simulating human thought processes to explicitly and directly tell the model how to learn.

[0069] To address the aforementioned problems of historical modeling and referential resolution, the core of this application's technical solution lies in simulating the human memory mechanism. Each round of information input interacts with historical memory, resulting in an output, which is simultaneously stored in the mind. Furthermore, it cleverly combines the knowledge inherent in existing large-scale pre-trained models, using BERT to simulate human neurons, proposing a novel historical dialogue memory modeling scheme. Within this scheme, a dedicated historical modeling training method for dialogue tasks is proposed, using each dialogue round as a batch. This dynamic batching approach ensures that all dialogue rounds within a dialogue can fully interact and be memorized sequentially.

[0070] like Figure 1 The diagram shown is an optional flowchart of a text processing method based on multi-turn question answering provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105:

[0071] Step S101: Obtain the first dialogue text of the historical rounds, concatenate the first question text and the first answer text in the first dialogue text, and construct the first text set based on the concatenated text.

[0072] The first dialogue text in the historical rounds can refer to the text content that is stored long-term or cached in the background data during the human-computer interaction process, especially during dialogue or question-and-answer sessions; that is, the first dialogue text in the embodiment can include the text content of any interactive dialogue. Further, in the embodiment, the first text set is a collection of text materials formed by concatenating the question text and the answer text.

[0073] For example, in this embodiment, the collected multi-turn dialogue content is as follows: B: "How is the return on Fund S?", A: "The return on Fund S is very good", B: "Why is that?". This content constitutes a multi-turn dialogue. The first question text may include "How is the return on Fund S?" and "Why is that?". The first answer text in the first dialogue text may include "The return on Fund S is very good". Further, in this embodiment, for the acquired dialogue text, and the question and answer texts within the dialogue text, the correspondence between the question and answer is determined based on keyword matching. Based on this correspondence, the question and answer texts are concatenated to obtain text materials, and all text materials are integrated to obtain a text set.

[0074] Step S102: Construct a model to be trained by inputting the first text set into the model to be trained for dialogue modeling training, and obtain the trained target model.

[0075] Specifically, in the embodiments, the models to be trained, i.e. the target models obtained after training, are all text processing models, including the BERT model in the content. The text models used in the embodiments can be transformer models, LSTM models, or GPT-2 models, etc.

[0076] For example, in this embodiment, the BERT model is used as the model to be trained. Its training process mainly involves classifying and predicting each word in the text set based on the contextual semantics of the dialogue text, determining whether it belongs to the beginning or end of the target text. After determining the beginning and end words of the target text, the target text content is output as the answer to the target question. The BERT model may include an encoder and a decoder. The specific training process for the encoder and decoder may include: inputting each text segment (the concatenated text of the question and answer) from the text set into the model; encoding the text segment using the BERT model's encoder to obtain the hidden feature vector of the text segment; and decoding the hidden feature vector of the sample text segment using the BERT model's decoder, such that the hidden feature vector of the sample text segment is multiplied by a reference word vector to obtain the text description segment. Furthermore, the implementation example can also calculate the similarity between the text description segment and the original answer in the text material through the model's loss function, optimize the model's loss function based on the similarity, backpropagate the model loss of the loss function, and continuously adjust the model parameters until the similarity is greater than or equal to the similarity threshold, at which point the optimization of the BERT model stops, and a text processing model that meets the requirements, i.e., the target model, is obtained.

[0077] Step S103: Input the first text set corresponding to several historical rounds into the target model and output semantic information.

[0078] Specifically, in this embodiment, step S102 first enables the model to learn simple dialogue modeling, that is, to determine the correspondence between questions and answers in the dialogue, as well as the referential relationship between questions and answers. After obtaining a stable learning function, the model then learns more difficult long-turn dialogues. In this embodiment, the set of text materials corresponding to the dialogue content of all turns is input into the target model, such as the BERT model, for interactive learning of semantic information.

[0079] Step S104: Generate a first vector corresponding to the historical round based on the semantic information, and combine the first vector with the second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector.

[0080] Wherein, the first vector represents the memory information of the historical rounds determined based on the semantic information, and the second vector is an intermediate vector obtained by feature encoding of the first text set through the target model.

[0081] Specifically, in the embodiment, after learning and outputting semantic information through step S103, the embedding vector representing the (historical) semantic information is broadcast to each subsequent round. In the process of encoding the input to obtain the output vector through the text processing model, the embedding vector of the semantic information is accumulated to the embedding of each output vector of the corresponding round of question and answer through a certain layer of neurons in the text processing model or a multilayer perceptron (MLP). That is, the fused embedding vector is output, which is the first fused vector.

[0082] Step S105: Predict the target answer based on the first fusion vector, determine the first start position and the first end position in the second text, and determine the target answer based on the first start position and the first end position.

[0083] Specifically, in this embodiment, for the output of the last layer of the text processing model (target model), two classification heads can be connected after the embedding layer. These are used to predict the start and end positions of the corresponding answer in the document given the input question, respectively, and are specifically for the output vector of the document. In this embodiment, softmax can be used to obtain the position with the highest probability value. The different highest probability positions under the start and end position classification heads represent the start and end positions of the answer, respectively.

[0084] In some other feasible implementation methods, such as Figure 2As shown, method step S101, which involves obtaining the first dialogue text from the historical rounds, concatenating the first question text and the first answer text in the first dialogue text, and constructing a first text set based on the concatenated text, may include steps S1011-S1014:

[0085] S1011. Extract the first question text and the first answer text from the first dialogue text;

[0086] Specifically, in the embodiments, during the processing of question and answer texts, punctuation can be identified within massive amounts of text data. By analyzing the punctuation marks between sentences, the text data can be broken down into question and answer texts at the sentence level. In some other embodiments, during the collection of dialogue data in human-computer interaction, question and answer texts can be collected according to a pre-defined question-and-answer pattern.

[0087] It should be further explained that the granularity of the answer text in this embodiment can be at the word level. Furthermore, after obtaining the answer text statements in this embodiment, simple preprocessing can be performed on the statements, such as word segmentation and identification and annotation of keywords in the segmentation results. These keywords include, but are not limited to, verbs or nouns present in the dialogue text. Based on the correspondence between keywords and words in the question text—for example, if a word in the question text contains the same word as a keyword—the keyword can be identified as the target word in the answer text. Additionally, manual annotation can also be used to annotate the questions and answers in the dialogue text.

[0088] S1012. Generate location tags based on the location information of the first answer text in the first dialogue text;

[0089] The location tags include the second start position and the second end position of the first answer text in the first dialogue text;

[0090] Specifically, in this embodiment, after determining the target word of the answer text through step S1011, it is necessary to perform position annotation or determine the position information of the answer text within the complete dialogue text to obtain the position label of the target word. During the position annotation or determination of its position information, this embodiment needs to determine the word preceding the target word to obtain the starting position of the target word; and it also needs to determine the word following the target word to obtain the ending position of the target word.

[0091] S1013. Sample data is constructed based on the first question text, the first answer text, and the location label. The first text set is constructed by sorting the sample data.

[0092] Specifically, in this embodiment, after obtaining position labels by position annotation of the answer text in step S1012, the embodiment uses the question text and the answer text with position labels as training samples for the model, and combines several training samples into a batch for the training process. In this embodiment, when loading a batch, it is ensured that sequentially input questions are grouped into a batch according to their temporal order from front to back. Each batch requires that all dialogue content sentences of a dialogue be combined into a batch and appear in sequence. In this embodiment, all batches are trained on the model in ascending order of batch size, allowing the model to first learn simple dialogue modeling and obtain a stable learning function before learning more difficult long-turn dialogues.

[0093] It should be noted that, in the example implementation, the data is fitted using the batch size during the back propagation process for model training.

[0094] In some other feasible implementation methods, such as Figure 3 As shown, in method step S1013, the process of constructing sample data based on the first question text, the first answer text, and the location tag may include steps S10131-S10132:

[0095] S10131. The first question text and the first answer text are concatenated using the first connecting symbol to obtain the first concatenated text;

[0096] S10132. Determine the position label corresponding to the first answer text in the first concatenated text to obtain the sample data.

[0097] Specifically, in this embodiment, for all input questions in a multi-turn question-and-answer dialogue, the SEP symbol, commonly used in the BERT model, can be used to concatenate the text content. SEP is a formal parameter of a function, and in most cases, the SEP parameter is used to specify the character delimiter. For example, based on the regular expression sep = '\s+', it can be determined that the text content in this embodiment can be concatenated using at least one space character. For example, in this embodiment, the text concatenation method is as follows: the current question is connected to a keyword using a SEP symbol, then another keyword is connected using a SEP symbol, and finally, the historical question-and-answer content is connected using a SEP symbol.

[0098] For example, when loading a batch, it is necessary to ensure that the sequential input questions are grouped into a batch in chronological order from front to back. In this embodiment, each batch is required to be composed of all the dialogue content sentences of a dialogue and appear in order. The sample format of each sentence is "Input: current round question + [sep] + document tag: corresponding answer position". The dialogue text in this embodiment can contain multiple questions and corresponding answers. Then, each dialogue has a complete reference text containing all dialogue (information) content.

[0099] In some other feasible implementation methods, such as Figure 4 As shown, in method step S102, the process of constructing the model to be trained by inputting the first text set into the model to be trained for dialogue modeling training, and obtaining the trained target model, may include steps S1021-S1023:

[0100] S1021. Obtain the model to be trained and perform dialogue modeling training, output the predicted answer, and determine the third starting position and the third ending position of the predicted answer.

[0101] Specifically, in this embodiment, we examine a text processing model. For example, during the training of the BERT model, to ensure more accurate predictions from the final target model, we can calculate the loss between the model's predicted values ​​and the original answers in the training text. By continuously converging the loss value, we can optimize the parameters of the target question-answering model during the training phase. In calculating the loss value, we first need to obtain the words at the beginning and end of the original answers in the training text. Secondly, this embodiment will collect the words at the beginning and end of the answers predicted by the model during the training phase.

[0102] It should be noted that the model to be trained constructed in the embodiments can be an improved model based on the BERT model, such as a BERT model + multi-head classifier answer extraction model. Furthermore, the learning function in the simple dialogue modeling process in the embodiments can be the loss value calculated by the cross-entropy using the loss function. In the embodiments, the stability of the learning function can mean that the change of the loss value during the training process is stable and there will be no large fluctuations.

[0103] S1022. Calculate a first loss value based on the third starting position and the second starting position using the cross-entropy loss function, and / or calculate a second loss value based on the third ending position and the second ending position using the cross-entropy loss function;

[0104] S1023. Optimize the parameters of the target model based on the first loss value and / or the second loss value.

[0105] Specifically, in this embodiment, the cross-entropy loss function is used to calculate the loss values ​​between the starting words and the ending words. In this embodiment, the words at the starting and ending positions in the training text material have been determined by labeling. Therefore, the probability distribution P(x) of the word positions of the original (true) answer has also been determined, so the information entropy is constant. In the calculation of cross-entropy, the difference between the true probability distribution P(x) and the predicted probability distribution Q(x) is represented by relative entropy. The smaller the value, the better the prediction result. Therefore, it is necessary to minimize the relative entropy. Cross-entropy is equal to the relative entropy plus a constant (information entropy). Therefore, the loss value of the model training process is calculated by cross-entropy, and the model parameters are adjusted according to the calculated loss value until the loss value converges, thus obtaining the final target question-answering model (parameters). This can improve the computational efficiency and accuracy in the prediction process.

[0106] In some other feasible implementation methods, such as Figure 5 As shown, method step S103, the process of inputting the first text set corresponding to several historical rounds into the target model and outputting semantic information, may include steps S1031-S1033:

[0107] S1031. Construct a first embedding matrix based on the number of dialogue turns in the first text set, the length of the text material in the first text set, and the dimension of the hidden layer in the model to be trained.

[0108] Specifically, in the embodiment, after the target model obtains a stable learning function, the embodiment can input the dialogue data into the BERT model in the target model in order of batch size to learn semantic information interactively. In the last layer output of the BERT model, the embedding N*S*M matrix is ​​used, where N represents the number of rounds of the current dialogue, S is the input length of each round of question and answer in the current dialogue, and M is the dimension of the BERT hidden embedding.

[0109] S1032. Perform a matrix transformation on the first embedding matrix according to the number of dialogue rounds to obtain a second embedding matrix;

[0110] S1033. Perform unidirectional self-attention calculation based on the second embedding matrix, and output the semantic information.

[0111] Specifically, in this embodiment, the CLS vector of each round of dialogue is extracted to obtain an N*1*M matrix. After transforming it into an N*M matrix, i.e., the second embedding matrix, a unidirectional self-attention mechanism is calculated in the N dimension. That is, the CLS of each round of question-and-answer in this dialogue can only interact with the CLS of the previous question-and-answer. It should be noted that the self-attention calculation in this embodiment is based on the attention mechanism in the BERT model, which can be implemented using attention calculation processes that are already disclosed and widely used in this technical field; it will not be elaborated here. It should also be noted that the CLS vector in this embodiment refers to the insertion of a specific classification token ([CLS]) at the beginning of each word text sequence in the target model during classification prediction, in addition to the word tokens. This classification token is used to aggregate the representation information of the entire sequence, and the vector representation of this sequence is the CLS vector.

[0112] In some feasible implementations, such as Figure 6 As shown, step S104 in the method, which involves generating a first vector corresponding to the historical round based on the semantic information and combining the first vector with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector, may include steps S1041-S1043:

[0113] S1041. Copy the third embedding matrix of the semantic information, and determine the first vector based on the copied third embedding matrix and the weight matrix.

[0114] Specifically, in this embodiment, after performing unidirectional self-attention calculation on the CLS vector of each round of question-and-answer in step S103, the memory information of historical questions and answers has been obtained. The embodiment further broadcasts this memory information to all output vectors. The embodiment copies the second embedding matrix N*M to N*S*M, obtaining the third embedding matrix, and then multiplies it by a weight matrix for the next interaction processing step. Here, the weight matrix is ​​essentially a parameter matrix, which can be learned by the model itself and can be simply understood as a layer of neurons or an MLP layer. For example, in this embodiment, the CLS vector of each round of question-and-answer after the interaction has already obtained the memory of historical questions and answers. The memory of historical questions and answers has been condensed into the CLS vector. For a particular question in a dialogue stream, it may be necessary to refer to historical questions and answers for better understanding; for example, a pronoun in the question may refer to an answer in a historical question. This weight matrix is ​​obtained through model training.

[0115] S1042. Input the second dialogue text of the current round into the target model, and encode it through the target model to obtain the second vector;

[0116] S1043. Combine the first vector and the second vector to obtain the first fused vector;

[0117] Specifically, in the embodiment, the amount of information of the memory information that needs to be acquired in the model can be determined by the weight matrix in step S1041. Then, the filtered CLS is accumulated on the embedding of each output vector of the corresponding round of question and answer. Then, the last layer of the target model in the embodiment outputs to the two classification head row answer inference.

[0118] In some feasible implementations, such as Figure 7 As shown, after copying the third embedding matrix of the semantic information and determining the first vector based on the copied third embedding matrices and the weight matrix, the method may further include steps S1044-S1045:

[0119] S1044. Combine the first fusion vector with the third vector of the third dialogue text to obtain the second fusion vector;

[0120] Wherein, the third dialogue text includes the dialogue text of the next round of the second dialogue text; the third vector is an intermediate vector obtained by feature encoding the third dialogue text through the target model;

[0121] Specifically, in the implementation example, during the answer reasoning process through the target model, the CLS vector of each question and answer can be pre-stored to realize the memory mechanism and single question and answer. The application scenario in the implementation example is a real dialogue, because each question is asked sequentially. Real dialogue processes questions according to the actual order of the dialogue. Therefore, after fusing the historical memory with the embedding vector of the current round each time, the CLS vector corresponding to the current input question can be stored. When the next round of questions is input, it can be directly retrieved instead of re-inputting the text of the previous round into the target model.

[0122] S1045. Predict the target answer based on the second fusion vector;

[0123] It should be noted that the target model in this embodiment mainly includes two core functional structures: an encoder and a classifier. The target question-answering model, built based on the BERT model, inputs to the model's tokenembedding layer (the encoding layer) to convert each word into a vector form, obtaining a vectorized representation of each word as a token. At the output of this BERT model, two softmax linear classifiers are concatenated to calculate the probability that each token in the reference text belongs to the start and end positions of the answer.

[0124] Specifically, in this embodiment, BERT segments the input text (the question for the next round) into several candidate words (tokens) and feeds them into the token embedding layer, converting each word into a vector form. In the target model of this embodiment, each word is converted into a 768-dimensional vector representation. The input text undergoes tokenization before being fed into the token embedding layer. In addition to the token embedding layer, the encoder in the target question answering model includes two embedding layers: a segment embedding layer and a position embedding layer. The target question answering model can classify and predict the target answer (word position) based on the input text. The target model outputs (1, n, 768) word vector representations through TokenEmbeddings; the Segment Embeddings layer outputs (1, n, 768) vector representations to help the model distinguish between the two sentences in a sentence pair; and the Position Embeddings layer outputs (1, n, 768) position vectors, allowing the model to learn the order attributes of the input. The outputs of the three are element-wise summed to obtain a composite representation of size (1, n, 768). Then, the original Masked Language Model task and Next SentencePrediction task in the BERT model are executed, and the start position candidate word (vector representation) and end position candidate word (vector representation) are obtained from the task output. The corresponding probabilities are then obtained by passing the output through a softmax layer.

[0125] The complete implementation process of the technical solution of this application is described in full and detail below with reference to the accompanying drawings:

[0126] First, in this implementation, for all input questions in a multi-turn question-and-answer dialogue, the question and document are concatenated using the [SEP] symbol. When loading a batch, it is ensured that sequential input questions are grouped into a batch in chronological order. Specifically, this implementation trains all batches in ascending order of batch size, allowing the model to first learn simple dialogue modeling and obtain a stable learning function before learning more complex long-turn dialogues.

[0127] Then, in the implementation example, the dialogue data is input into the BERT model in batch order to learn semantic information through interaction. In the N*S*M embedding matrix output by the last layer of the BERT model, N represents the current dialogue round number, S is the input length of each question-and-answer round in the current dialogue, and M is the dimension of the BERT hidden embedding. The implementation example extracts the CLS vector for each round of dialogue, resulting in an N*1*M matrix. After transforming it into an N*M matrix, a unidirectional self-attention mechanism is calculated along the N dimension, ensuring that the CLS of each question-and-answer round in this dialogue can interact with the CLS of previous question-and-answer rounds only.

[0128] In the implementation example, the CLS vector of each round of question-and-answer interaction has acquired the memory of historical questions and answers. We want to broadcast this information to all output vectors. We duplicate it from N*M to N*S*M, then multiply it by a weight matrix for interaction; this allows the model to control the amount of information it needs to acquire. Then, the filtered CLS is accumulated and added to the embedding of each output vector of the corresponding round of question-and-answer. The output of the final layer is handed over to the classification head for answer inference. In inference, the implementation example can pre-store the CLS vector of each question and answer, thus realizing the memory mechanism and inference of a single question and answer under the interaction of historical memory.

[0129] For the output of the final layer, the model in this example uses two classification heads, one for predicting the start position and the other for predicting the end position of the corresponding answer in the document given the input question. Specifically, for the output vector of the document, softmax is used to obtain the position with the highest probability value. The different highest probability positions under the start and end position classification heads represent the start and end positions of the answer, respectively. The corresponding answer can be decoded by mapping it to the original text. For training, cross-entropy can be used to train the model and the parameters can be tuned.

[0130] Please see Figure 8 This application also provides a text processing apparatus based on multi-turn question-and-answer, which can implement the above-mentioned text processing method based on multi-turn question-and-answer. The apparatus includes:

[0131] The sample construction unit is used to obtain the first dialogue text of the historical rounds, concatenate the first question text and the first answer text in the first dialogue text, and construct the first text set based on the concatenated text;

[0132] The model training unit is used to construct the model to be trained. The first text set is input into the model to be trained to perform dialogue modeling training, and the trained target model is obtained.

[0133] The semantic acquisition unit is used to input the first set of texts corresponding to several historical rounds into the target model and output semantic information.

[0134] A semantic fusion unit is used to generate a first vector corresponding to the historical round based on the semantic information, and combine the first vector with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector; the first vector represents the memory information of the historical round determined based on the semantic information, and the second vector is an intermediate vector obtained by feature encoding of the first text set through the target model;

[0135] The answer prediction unit is used to predict the target answer based on the first fusion vector, determine the first start position and the first end position in the second text, and determine the target answer based on the first start position and the first end position.

[0136] The specific implementation of this text processing device is basically the same as the specific embodiment of the text processing method based on multi-turn question answering described above, and will not be repeated here.

[0137] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned text processing method based on multi-turn question-and-answer. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0138] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0139] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0140] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the multi-turn question-and-answer-based text processing method of the embodiments of this application.

[0141] The input / output interface 903 is used to implement information input and output;

[0142] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0143] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0144] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0145] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described text processing method based on multi-turn question-and-answer.

[0146] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0147] The text processing method, text processing device, electronic device, and storage medium based on multi-turn question answering provided in this application simulate the human memory mechanism. For each round of information input, it interacts with historical memory to obtain an output, and the information of this current output is also stored in the mind. Simultaneously, it cleverly combines the knowledge contained in existing large-scale pre-trained models, using BERT to simulate human neurons, proposing a new historical dialogue memory modeling scheme. Furthermore, under this scheme, a historical modeling training method specifically for dialogue tasks is proposed, using the number of turns in each dialogue as a batch, i.e., using dynamic batching to ensure that all dialogue turns in the dialogue can fully interact and be remembered in chronological order.

[0148] The solution has at least the following advantages or can bring corresponding beneficial effects:

[0149] 1. By simulating human memory, the historical dialogue is explicitly modeled, allowing the model to acquire the information needed to understand the current output on its own, rather than relying on manually pieced-together information input.

[0150] 2. By storing CLS vectors, the human memory mechanism is simulated, which accelerates the understanding and reasoning of single data points instead of having to integrate all historical rounds of questions and answers into a batch for each round of reasoning.

[0151] 3. By learning through courses with a range of epochs from small to large, the model can better fit the data and avoid the loss explosion caused by the early appearance of long and difficult data.

[0152] 4. By using dynamic batch training, it is ensured that each dialogue can be explicitly and fully modeled in time sequence, giving the model a stronger historical analysis capability.

[0153] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0154] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0157] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0158] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0160] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0162] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A text processing method based on multi-turn question answering, characterized in that, The method includes: Obtain the first dialogue text of the historical rounds, concatenate the first question text and the first answer text in the first dialogue text, and construct the first text set based on the concatenated text; Construct a model to be trained by inputting the first text set into the model to be trained for dialogue modeling training, and obtain the trained target model. Input the first set of texts corresponding to several historical rounds into the target model, and output semantic information; A first vector corresponding to the historical round is generated based on the semantic information, and the first vector is combined with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector; the first vector represents the memory information of the historical round determined based on the semantic information, and the second vector is an intermediate vector obtained by feature encoding of the second dialogue text through the target model; Based on the first fusion vector, the target answer is predicted, the first start position and the first end position in the second text are determined, and the target answer is determined based on the first start position and the first end position. The step of inputting the first text set corresponding to several historical rounds into the target model and outputting semantic information includes: The first embedding matrix is ​​constructed based on the number of dialogue turns in the first text set, the length of the text material in the first text set, and the dimension of the hidden layer in the model to be trained. The first embedding matrix is ​​transformed according to the number of dialogue turns to obtain the second embedding matrix; The semantic information is obtained by performing unidirectional self-attention calculation based on the second embedding matrix.

2. The method according to claim 1, characterized in that, The step of obtaining the first dialogue text from the historical rounds involves concatenating the first question text and the first answer text from the first dialogue text, and constructing a first text set based on the concatenated text, including: Extract the first question text and the first answer text from the first dialogue text; Based on the position information of the first answer text in the first dialogue text, a position tag is generated, the position tag including the second start position and the second end position of the first answer text in the first dialogue text; Sample data is constructed based on the first question text, the first answer text, and the location label. The first text set is then constructed by sorting the sample data.

3. The method according to claim 2, characterized in that, The sample data constructed based on the first question text, the first answer text, and the location tags includes: The first question text and the first answer text are concatenated using the first connector symbol to obtain the first concatenated text; The position label corresponding to the first answer text in the first concatenated text is determined to obtain the sample data.

4. The method according to claim 2, characterized in that, The step of constructing the model to be trained involves inputting the first text set into the model to be trained for dialogue modeling training, thereby obtaining the trained target model, including: The model to be trained is obtained for dialogue modeling training, and the predicted answer is output. The third starting position and the third ending position of the predicted answer are determined. A first loss value is calculated based on the third starting position and the second starting position using the cross-entropy loss function, and / or a second loss value is calculated based on the third ending position and the second ending position using the cross-entropy loss function; The target model is then optimized based on the first loss value and / or the second loss value.

5. The method according to claim 1, characterized in that, The step of generating a first vector corresponding to the historical round based on the semantic information, and combining the first vector with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector includes: The third embedding matrix of the semantic information is copied, and the first vector is determined based on the copied third embedding matrix and the weight matrix. The second dialogue text of the current round is input into the target model, and the second vector is obtained by encoding it through the target model. The first vector and the second vector are combined to obtain the first fused vector.

6. The method according to claim 5, characterized in that, After copying the third embedding matrix of the semantic information and determining the first vector based on the copied third embedding matrices and the weight matrix, the method further includes: The first fusion vector is combined with the third vector of the third dialogue text to obtain the second fusion vector; the third dialogue text includes the dialogue text of the next round of the second dialogue text; the third vector is an intermediate vector obtained by feature encoding of the third dialogue text through the target model. The target answer is predicted based on the second fusion vector.

7. A text processing device based on multi-turn question-and-answer, characterized in that, The device includes: The sample construction unit is used to obtain the first dialogue text of the historical rounds, concatenate the first question text and the first answer text in the first dialogue text, and construct the first text set based on the concatenated text; The model training unit is used to construct the model to be trained. The first text set is input into the model to be trained to perform dialogue modeling training, and the trained target model is obtained. The semantic acquisition unit is used to input the first set of texts corresponding to several historical rounds into the target model and output semantic information. A semantic fusion unit is used to generate a first vector corresponding to the historical round based on the semantic information, and combine the first vector with a second vector corresponding to the second dialogue text of the current round to obtain a first fusion vector; the first vector represents the memory information of the historical round determined based on the semantic information, and the second vector is an intermediate vector obtained by feature encoding of the first text set through the target model; The answer prediction unit is used to predict the target answer based on the first fusion vector, determine the first start position and the first end position in the second text, and determine the target answer based on the first start position and the first end position. The semantic acquisition unit is specifically used for: The first embedding matrix is ​​constructed based on the number of dialogue turns in the first text set, the length of the text material in the first text set, and the dimension of the hidden layer in the model to be trained. The first embedding matrix is ​​transformed according to the number of dialogue turns to obtain the second embedding matrix; The semantic information is obtained by performing unidirectional self-attention calculation based on the second embedding matrix.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the text processing method based on multi-turn question answering as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the text processing method based on multi-turn question answering as described in any one of claims 1 to 6.

Images