Language model training method, language processing method and electronic device
Through quantum theory and measurement operators to measure the interactive information between sentence pairs, the language model is trained, and the problem of insufficient accuracy of language model processing in the prior art is solved, and higher sentence pair interaction information capture and dialogue system performance improvement is achieved.
Patent Information
- Application Number
- CN202011463278.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-11
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-12-11
AI Technical Summary
The accuracy of the processing results of the language model in the prior art cannot meet the needs, especially in a task-driven multi-round dialogue system, it is impossible to effectively extract the interactive information between sentence pairs.
By introducing quantum theory, using measurement operators to measure the interactive information between sentence pairs, a joint representation of sentence pairs is obtained, and the language model is trained based on this, and the backpropagation algorithm and batch stochastic gradient descent method are used to adjust the model parameters to improve the language model Accuracy of processing results.
It effectively improves the accuracy of the processing results of the language model, captures the interactive information between sentence pairs more accurately, and improves the performance of multiple rounds of dialogue systems.
Smart Images

Figure CN113761145B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and more specifically, to a language model training method, a language processing method, and an electronic device. Background Art
[0002] Deep learning technology has made great progress in fields such as object classification, text processing, recommendation engines, image search, facial recognition, age recognition and speech recognition, human-computer dialogue and emotional computing. Human-computer dialogue has become an important research and application hotspot in the field of speech recognition and natural language processing. With the development of human-computer dialogue systems and the gradual deepening of industrial applications, task-driven multi-round dialogue systems have received more attention from the industry.
[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the prior art: the accuracy of the processing results of the language model in the related technology cannot meet the requirements. Summary of the invention
[0004] In view of this, the present disclosure provides a language model training method, a language processing method and an electronic device that can improve the accuracy of the processing results of the language model.
[0005] One aspect of the present disclosure provides a language model training method, including: processing sentence pair samples to obtain a joint representation of the sentence pairs, the sentence pair samples include paired questions and answer sentences, and the joint representation of the sentence pairs includes interaction information between the paired questions and answer sentences; and training a language model based on the joint representation of the sentence pairs, wherein the interaction information includes mutual detection results between the paired questions and answer sentences, and the mutual detection results include detection results of questions in the sentence pair samples for answer sentences, and detection results of answers in the sentence pair samples for questions.
[0006] According to an embodiment of the present disclosure, the mutual detection result between a pair of question sentences and answer sentences is determined in the following manner: the answer sentence is measured based on a measurement operator or a random operator for the question sentence to obtain a first measurement result, and the question sentence is measured based on a measurement operator or a random operator for the answer sentence to obtain a second measurement result; and the mutual detection result between the pair of question sentences and answer sentences is determined based on the product of the first measurement result and the second measurement result.
[0007] According to an embodiment of the present disclosure, measuring an answer sentence based on a measurement operator or a random operator for a question sentence to obtain a first measurement result includes: determining the first measurement result based on a trace algorithm, a sentence representation of the answer sentence and a measurement operator for the question sentence; and measuring the question sentence based on a measurement operator or a random operator for the answer sentence to obtain a second measurement result includes: determining the second measurement result based on a trace algorithm, a sentence representation of the question sentence and a measurement operator for the answer sentence.
[0008] According to an embodiment of the present disclosure, a measurement operator for a question is determined in the following manner: a word representation tool based on global word frequency statistics is used to determine a first matrix for the question, the first matrix including word vectors for each word in the question; the binary norm of the word vectors for each word in the question included in the first matrix is obtained; and a measurement operator for the question is determined based on the ratio of the word vectors for each word in the question to their respective binary norms.
[0009] According to an embodiment of the present disclosure, the sentence representation of a question is determined in the following manner: a second matrix is determined based on the first matrix and the binary norm of the word vectors of each word in the question; and at least two second matrices are mixed based on the weights of each of the at least two second matrices to obtain a third matrix for characterizing the sentence representation of the question.
[0010] According to an embodiment of the present disclosure, training a language model based on the joint representation of sentence pairs includes: based on a back-propagation algorithm and / or a batch stochastic gradient descent method, training a language model using the joint representation of sentence pairs to obtain values of model parameters of the language model, wherein the model parameters include respective weights of at least two second matrices.
[0011] According to an embodiment of the present disclosure, a language model includes a convolutional layer, a fully connected layer and a classification layer; based on a back-propagation algorithm and / or a batch stochastic gradient descent method, the language model is trained using the joint representation of sentence pairs to obtain values of model parameters of the language model, including: adjusting the values of the model parameters of the language model, and for each of a plurality of sentence pair samples, using a convolutional layer to process the joint representation of the sentence pairs to obtain a state representation feature for the joint representation of the sentence pairs, using a fully connected layer to process the state representation feature to obtain a representation vector, and using a classification layer to process the representation vector to obtain category information; repeating the above operations until the category information for the plurality of sentence pair samples meets a stop condition.
[0012] According to an embodiment of the present disclosure, the stopping condition includes but is not limited to the convergence of the language model.
[0013] According to an embodiment of the present disclosure, the training method also includes: obtaining a sample data set, the sample data set including multiple sentence pair samples; determining a sample data training set from the sample data set, or determining a sample data training set and at least one of a sample data verification set and a sample data test set from the sample data set; using sentence pair samples in the sample data training set as training samples to train a language model; if there is a sample data verification set, using sentence pair samples in the sample data verification set to verify the trained language model to determine the optimal values of the model parameters; and if there is a sample data test set, using sentence pair samples in the sample data test set to test the language model using the optimal values of the model parameters to obtain a test result.
[0014] According to an embodiment of the present disclosure, the training method also includes: after determining a sample data training set from a sample data set, or determining a sample data training set and at least one of a sample data verification set and a sample data test set from a sample data set, preprocessing at least part of the sentence pair samples in the sample data training set, the sample data verification set, and the sample data test set to remove stop words and / or specific symbols in at least part of the sentence pair samples.
[0015] According to an embodiment of the present disclosure, obtaining a sample data set includes at least one of the following: obtaining sentence pair samples to construct a sample data set based on multiple sentence pair samples, the sentence pair samples including paired question sentences and answer sentences; and loading a corpus, the corpus including multiple sentence pair samples.
[0016] One aspect of the present disclosure provides a language processing method comprising: obtaining an input question sentence; and processing the input question sentence using a language model to determine an output answer sentence; wherein the language model is trained based on the above training method.
[0017] According to an embodiment of the present disclosure, the input question sentence comes from a question answering system.
[0018] One aspect of the present disclosure provides a language model training device, including: a sentence pair sample processing module, used to process sentence pair samples to obtain a joint representation of sentence pairs, the sentence pair samples include paired questions and answer sentences, and the joint representation of sentence pairs includes interactive information between the paired questions and answer sentences; and a language model training module, used to train a language model based on the joint representation of sentence pairs.
[0019] Another aspect of the present disclosure provides a language processing device, including: an input question acquisition module, used to acquire an input question; and processing the input question using a language model to determine an output answer sentence, wherein the language model is trained based on the above training method.
[0020] Another aspect of the present disclosure provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, and when the executable instructions are executed by the processor, the method described above is implemented.
[0021] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.
[0022] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement the method described above when executed.
[0023] The embodiments of the present disclosure obtain interactive information between paired question sentences and answer sentences, and perform model training based at least on the interactive information, which helps to improve the accuracy of the processing results of the language model.
[0024] The embodiments of the present disclosure introduce a joint representation of sentence pairs, which effectively improves the shortcoming that the joint representation of density matrices in related technologies cannot extract the interactive information between sentence pairs. Quantum theory is introduced, and a measurement operator is used to measure each sentence pair, so that the content of one sentence guides the representation of another sentence, and finally the two sentences are modeled together, which helps to improve the accuracy of the processing results of the language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0026] Figure 1 A schematic diagram of applicable scenarios of a language model training method, a language processing method, and an electronic device according to an embodiment of the present disclosure;
[0027] Figure 2 An exemplary system architecture to which a language model training method, a language processing method, and an electronic device can be applied according to an embodiment of the present disclosure.
[0028] Figure 3 is a flow chart of a language model training method according to an embodiment of the present disclosure;
[0029] Figure 4 is a schematic diagram of the structure of a language model according to an embodiment of the present disclosure;
[0030] Figure 5 is a schematic diagram of a first matrix according to an embodiment of the present disclosure;
[0031] Figure 6 is a schematic diagram of a second matrix according to an embodiment of the present disclosure;
[0032] Figure 7 is a schematic diagram of a third matrix according to an embodiment of the present disclosure;
[0033] Figure 8 is a schematic diagram of a sample data set according to an embodiment of the present disclosure;
[0034] Fig. 9 is a flow chart of a language processing method according to an embodiment of the present disclosure;
[0035] Fig.10 is a block diagram of a language model training device according to an embodiment of the present disclosure;
[0036] Fig.11is a block diagram of a language processing device according to an embodiment of the present disclosure; and
[0037] Fig.12 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0038] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0039] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0040] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0041] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0042] The related technology may use a network including a convolutional layer to perform feature extraction and classification, etc. In order to facilitate understanding of the embodiments of the present disclosure, a convolutional neural network and its working process are first exemplified.
[0043] In order to facilitate understanding of the embodiments of the present disclosure, natural language processing is first exemplified.
[0044] With the rapid development of science and technology, especially the rapid development of the Internet and social networks, a large amount of information has flooded the network platform. People may obtain a lot of information through the Internet, but how to improve the speed and accuracy of obtaining answers to questions has become a problem of concern to people. A common way to improve efficiency management is matching, which shows that text matching contains huge social value. The embodiments of the present disclosure mainly involve matching between sentence pairs.
[0045] Language modeling (LM) is a basic research topic in the field of natural language processing. The Quantum Language Model (QLM) borrows the concept of density matrix in quantum mechanics to encode the probabilistic uncertainty of single terms and term dependencies into the density matrix. Compared with the traditional Chinese language model (such as the n-gram model), the quantum language model does not increase the parameter scale when representing n-order dependencies. However, QLM also has some limitations. For example, the representation of each word in QLM is a one-hot vector, which can only encode local events but cannot consider global semantic information; in addition, QLM represents text (such as query, document) through a density matrix, which is solved iteratively rather than analytically, so it is difficult to update and optimize the density matrix in an end-to-end design; in addition, the representation, training and matching of the density matrix in QLM cannot be optimized together, which limits the applicability of QLM.
[0046] The neural network classification method based on semantic vector representation aims to distinguish the semantic information of a given sentence. Of course, both the industry and academia are aware of the importance of interactive information between sentence pairs, and try to better obtain them by designing a series of interactive models. For example, the neural network-based quantum language model (NNQLM) proposes to use density matrices to represent questions and answers, integrate the joint representation of density matrices into an end-to-end mechanism, and use word vectors as state vectors of individual words. A single word represents a pure state system, and the entire sentence can correspond to a mixed state system represented by the density matrix. In this way, the density matrix can be derived without iterative estimation, and the density matrix representation can also be integrated into the neural network structure and automatically updated through the back-propagation algorithm. However, this method only introduces the concept of quantum language model, cannot encode the interactive information between sentence pairs, and the measurement, which is an indispensable part of quantum mechanics, is also ignored.
[0047] Matching tasks play an important role in natural language processing tasks. The development of matching tasks has attracted widespread attention from the industry and academia. The disclosed embodiments not only discuss how to obtain similarity information between sentence pairs, but also determine the matching relationship between sentence pairs, which is a fine-grained task in the field of text matching. For example, "Q: What do you think of the male protagonist in the movie?", "A: The serial killer Jeffrey Dahmer in the movie is boring." The matching relationship of this sentence pair is mainly determined by the word "movie".
[0048] The technical solution provided by the embodiments of the present disclosure is intended to improve at least some of the above-mentioned problems.
[0049] Figure 1 The present invention is a schematic diagram of applicable scenarios of a language model training method, a language processing method, and an electronic device according to an embodiment of the present disclosure.
[0050] like Figure 1 As shown, the user inputs information through the interactive interface, or asks the intelligent customer service through voice interaction and other methods: What should I do if I want to return the mobile phone? After the question is processed by the server side of the intelligent customer service, the reply information can be automatically determined and output, such as Hello, JD.com provides you with online return and exchange services, please click on the after-sales application... Intelligent customer service effectively reduces the customer service post's dependence on manual customer service, or can assist manual customer service in their work, and reduce the requirements for the business proficiency level of manual customer service. In addition, the replies given by intelligent customer service can be more standardized, which helps to improve the customer experience. Specifically, the embodiments of the present disclosure can measure the interaction relationship between sentence pairs, such as using measurement operators to measure the interaction relationship between sentence pairs so that the language model can be trained based on the interaction relationship between the sentence pairs. In addition, matching information can also be extracted through a convolutional neural network to provide accurate answer sentences.
[0051] The language model training method, language processing method and electronic device provided by the embodiments of the present disclosure include a sentence pair sample processing process and a model training process. In the sentence pair sample processing process, the sentence pair samples are processed to obtain a joint representation of the sentence pairs, the sentence pair samples include paired question sentences and answer sentences, and the joint representation of the sentence pairs includes the interaction information between the paired question sentences and answer sentences. After completing the sentence pair sample processing process, the model training process is entered to train the language model based on the joint representation of the sentence pairs.
[0052] Figure 2 The following is an exemplary system architecture to which the language model training method, language processing method and electronic device according to the embodiments of the present disclosure can be applied. It should be noted that: Figure 2 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0053] like Figure 2 As shown, the system architecture 200 according to this embodiment may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 is used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0054] The user can use the terminal devices 201, 202, 203 to interact with the server 205 through the network 204 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 201, 202, 203, such as navigation applications, language processing applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).
[0055] The terminal devices 201 , 202 , and 203 may be various electronic devices with language processing capabilities, including but not limited to vehicles, smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0056] The server 205 may be a server that provides various services, such as a background management server (only as an example) for answering statements requested by the terminal devices 201, 202, and 203. The background management server may analyze and process the received request data, and feed back the processing results (such as answering statements, model parameter information, model verification results, etc.) to the terminal device.
[0057] It should be noted that the language model training method provided in the embodiment of the present disclosure can be applied to the server, and the language processing method provided in the embodiment of the present disclosure can be executed by the server 205. The language model training method and language processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 205 and can communicate with the terminal devices 201, 202, 203 and / or the server 205.
[0058] It should be understood that the number of terminal devices, networks and servers is only illustrative. Depending on the implementation requirements, there can be any number of terminal devices, networks and servers.
[0059] Figure 3 The present invention is a flowchart of a language model training method according to an embodiment of the present invention.
[0060] like Figure 3 As shown, the language model training method may include operations S302 to S304.
[0061] In operation S302, sentence pair samples are processed to obtain joint representations of sentence pairs, where the sentence pair samples include paired question sentences and answer sentences, and the joint representations of sentence pairs include interactive information between the paired question sentences and answer sentences.
[0062] In this embodiment, the sentence pair samples may be paired question sentences and answer sentences with annotation information, and the annotation information may represent, for example, whether the question sentence and the answer sentence match or do not match, etc. The sentence pair samples may be obtained from historical question and answer data or may be manually constructed, which is not limited here.
[0063] The joint representation of sentence pairs is used to characterize the interaction information between paired question and answer sentences, which can be used to determine whether the question and answer sentences match.
[0064] For example, the interactive information between the paired question and answer may include: mutual detection results between the paired question and answer, wherein the mutual detection results may include: detection results of the question against the answer in the sentence pair sample, and detection results of the answer against the question in the sentence pair sample.
[0065] The detection result may be determined based on, for example, a density matrix, an operator, etc. The density matrix may be used as a representation of a sentence, for example, a representation of a question sentence or a representation of an answer sentence. The operator may be a random operator or a measurement operator, wherein the measurement operator may represent the superposition state of each word in the sentence.
[0066] In operation S304, a language model is trained based on the joint representation of the sentence pairs.
[0067] In this embodiment, the language model may include multiple networks, such as a convolutional neural network (CNN), a fully connected neural network, a classification neural network, etc. A training model such as a back propagation algorithm and / or a batch stochastic gradient descent method may be used to make the output category of the language model approach the category represented by the label of the sentence pair sample. The activation function, the loss function, etc. may adopt a variety of related technologies, which are not limited here.
[0068] Figure 4 Schematic diagram of the structure of a language model according to an embodiment of the present disclosure.
[0069] like Figure 4 As shown, the language model may include a convolutional layer, a fully connected layer, and a classification layer, wherein the convolutional layer may be used to extract the features of the joint representation of the sentence pair, such as the state representation feature. The fully connected layer may output a representation vector, so that the classification layer may give a classification result based on the representation vector.
[0070] In the embodiments of the present disclosure, based on the shortcoming that the joint representation of density matrices in the current question-answering system QA cannot extract the interactive information between sentence pairs, quantum theory is introduced to process sentence pair samples to obtain the joint representation of sentence pairs, so that the content of one sentence guides the representation of another sentence, and finally the two sentences are modeled together.
[0071] The following is an exemplary description of determining the joint representation of a sentence pair.
[0072] In one embodiment, the mutual detection result between the paired question sentence and answer sentence can be determined in the following manner.
[0073] First, the answer is measured based on the measurement operator or random operator for the question to obtain a first measurement result, and the question is measured based on the measurement operator or random operator for the answer to obtain a second measurement result. For example, the measurement operator can be determined based on the two-norm of the word vector after determining the word vector of each word in the sentence.
[0074] Then, based on the product of the first measurement result and the second measurement result, the mutual detection result between the paired question and answer is determined. By multiplying the first measurement result and the second measurement result, the mutual detection result between the question and answer can be obtained, and the interactive information between the question and answer can be characterized.
[0075] For example, measuring the answer sentence based on a measurement operator or a random operator for the question sentence to obtain a first measurement result includes: determining the first measurement result based on a trace algorithm, a sentence representation of the answer sentence, and a measurement operator for the question sentence. The trace algorithm can be expressed as Tr(), where Tr(A) represents the trace of matrix A, and the trace is the sum of the main diagonal elements of the matrix.
[0076] For another example, measuring the question sentence based on a measurement operator or a random operator for the answer sentence to obtain the second measurement result includes: determining the second measurement result based on a trace algorithm, a sentence representation of the question sentence and a measurement operator for the answer sentence.
[0077] The following uses a file as an example to illustrate the process of determining a measurement operator.
[0078] In one embodiment, the measurement operator for the question sentence may be determined in the following manner.
[0079] First, a word representation tool based on global word frequency statistics determines a first matrix for the question, where the first matrix includes word vectors of each word in the question.
[0080] For example, use the Glove tool to get the word vector of each word in the text , which can be regarded as a linear combination (superposition state) of independent latent semantics and weights, and the distributed representation of each word can be regarded as a quantum state after being measured. Among them, Glove is a mainstream vector representation tool for training word vectors, generating and extracting information.
[0081] Figure 5 is a schematic diagram of a first matrix according to an embodiment of the present disclosure.
[0082] like Figure 5 As shown, the first matrix may include multiple array elements, wherein the value of each array element may be a, b, c, d, etc. Among them, a, b, c, d, etc. may be complex-valued word vectors. The word vectors in the first matrix can be regarded as a linear combination (superposition state) of independent latent semantics and weights, and the distributed representation of each word can be regarded as a quantum state after measurement. In order to obtain a unit state vector from the complex-valued word vector, we take the 2-norm of each word vector. Among them, what is obtained after Glove processing is a matrix, which is the distributed representation. The 2-norm of the matrix S is the square root value of the maximum characteristic root of the product of the transposed conjugate matrix of S and the matrix S, which refers to the straight-line distance between two vector matrices in space.
[0083] Then, obtain the bi-norm of the word vector of each word in the question included in the first matrix. For example, it can be expressed as .
[0084] Next, based on the ratio of the word vector of each word in the question to its respective two-norm, the measurement operator for the question is determined. For example, if the measurement operator is represented by Π, it can be expressed as shown in formula (1).
[0085] Formula (1)
[0086] The measurement operator for the question sentence can be determined in the above manner. It should be noted that the method for determining the measurement operator for the answer sentence can be the same, and is not limited here.
[0087] In one embodiment, the sentence representation of a question sentence is determined as follows.
[0088] First, the second matrix is determined based on the first matrix and the binary norm of the word vector of each word in the question.
[0089] Figure 6 FIG. 1 is a schematic diagram of a second matrix according to an embodiment of the present disclosure. Figure 6 As shown, the value of each element in the first matrix can be replaced by its own second norm, such as e, f, g, h, etc.
[0090] Then, at least two second matrices are mixed based on respective weights of the at least two second matrices to obtain a third matrix for representing a sentence representation of the question sentence.
[0091] The weights of at least two second matrices can be automatically determined during the model training process. The mixing process can be implemented in a weighted superposition manner.
[0092] Figure 7 FIG. 1 is a schematic diagram of a third matrix according to an embodiment of the present disclosure. Figure 7 As shown, weighted superposition is performed on the values of corresponding array elements in multiple second matrices.
[0093] In one embodiment, training a language model based on the joint representation of sentence pairs may include: based on a back-propagation algorithm and / or a batch stochastic gradient descent method, training the language model using the joint representation of sentence pairs to obtain values of model parameters of the language model, wherein the model parameters include respective weights of at least two second matrices.
[0094] For example, refer to Figure 4 As shown, the language model includes a convolutional layer, a fully connected layer and a classification layer. Accordingly, based on the back propagation algorithm and / or the batch stochastic gradient descent method, the language model is trained using the joint representation of the sentence pairs to obtain the values of the model parameters of the language model, which may include the following operations.
[0095] Adjust the values of the model parameters of the language model, and for each of the multiple sentence pair samples, first, use the convolution layer to process the joint representation of the sentence pair to obtain the state representation feature of the joint representation of the sentence pair, then use the fully connected layer to process the state representation feature to obtain the representation vector, and then use the classification layer to process the representation vector to obtain the category information. In this way, by repeatedly performing the above adjustment of the values of the model parameters and performing model training operations until the category information for the multiple sentence pair samples meets the stopping condition. For example, the stopping condition includes but is not limited to: the model output result converges, the similarity between the model output result and the label information is greater than the preset similarity threshold, etc.
[0096] The above method can realize the joint representation training language model based on sentence pairs.
[0097] In one embodiment, the training method may further include the following operations.
[0098] First, a sample dataset is obtained, where the sample dataset includes multiple sentence pair samples.
[0099] Then, a sample data training set is determined from the sample data set, or at least one of a sample data training set and a sample data verification set and a sample data test set is determined from the sample data set.
[0100] Next, sentence pair samples in the sample data training set are used as training samples to train the language model.
[0101] If there is a sample data validation set, the trained language model is validated using sentence pairs in the sample data validation set to determine the optimal values of the model parameters.
[0102] If there is a sample data test set, the language model with the optimal model parameter values is used to test the sample using the sentences in the sample data test set to obtain a test result.
[0103] In order to improve the fairness and rationality of the verification results and the test results, the training method may also include: after determining a sample data training set from the sample data set, or determining a sample data training set and at least one of a sample data verification set and a sample data test set from the sample data set, preprocessing at least part of the sentence pair samples in the sample data training set, the sample data verification set, and the sample data test set to remove stop words and / or specific symbols in at least part of the sentence pair samples.
[0104] Figure 8 A schematic diagram of a sample data set according to an embodiment of the present disclosure.
[0105] like Figure 8 As shown in the figure, 80% of the samples are randomly selected from the text classification corpus as the sample data training set, 10% of the samples are divided into the sample data validation set and the remaining 10% of the samples are divided into the sample data test set, and the training set, validation set and test set are preprocessed respectively to remove stop words and special symbols from each sentence pair.
[0106] In one embodiment, the sample data set may be an existing sample data set or a self-constructed data set, which is not limited here.
[0107] For example, obtaining a sample data set includes at least one of the following: for example, obtaining sentence pair samples to construct a sample data set based on a plurality of sentence pair samples, wherein the sentence pair samples include paired question sentences and answer sentences. For another example, loading a corpus, wherein the corpus includes a plurality of sentence pair samples.
[0108] The following is an illustrative explanation using the question-answering system scenario as an example.
[0109] First, the “crawler” technology is used to collect and construct a question-answering system corpus. The total number of samples in this corpus is N, and each sample contains a question and an answer.
[0110] Then, from the text classification corpus, 80%*N samples are randomly selected as the training set, 10%*N samples are divided into the validation set and the remaining 10%*N samples are divided into the test set, and the training set, validation set and test set are preprocessed respectively to remove stop words and special symbols from each sentence pair.
[0111] Next, determine the joint representation of the sentence pair, which can specifically include the following four steps.
[0112] Step 1: Use the glove tool to get the word vector of each word in the text , which can be regarded as a linear combination (superposition state) of independent latent semantics and weights. The distributed representation of each word can be regarded as a quantum state after measurement. To obtain the unit state vector from the complex-valued word vector, take the bi-norm of each word vector and then calculate the measurement operator Π, as shown in formula (1).
[0113] Step 2: Each sentence is represented by a mixed state density matrix. For a pure density matrix, the diagonal elements represent the distribution of amplitude (semantics), while the non-diagonal elements represent phase interference (semantic combination). The mixed state density matrix is a mixture of pure state density matrices with a certain probability. The representation of the sentence can be obtained from the formula of the density matrix as shown in Equation (2).
[0114] Formula (2)
[0115] Where i is a positive integer greater than or equal to 1, is the weight, is the conjugate transpose.
[0116] Step 3: You can use the superposition state representation of each word in the question as a measurement operator to measure the answer sentence, and you can get the measurement result of the question for the answer sentence. , as shown in formula (3).
[0117] Formula (3)
[0118] in, is the measurement operator, is the weight, which is determined by model training. tr() is the trace function, as shown in formula (4).
[0119] Formula (4)
[0120] Step 4: Similarly, we can get the representation of the answer sentence after measuring the question, and the representation of the sentence pair can be obtained by the joint representation of the two sentences, that is, .in, It is the measurement result of the answer sentence against the question sentence.
[0121] Then, the joint representation of the sentence pair is input into the convolutional neural network (CNN) network. The specific calculation formula is shown in formula (5).
[0122] Formula (5)
[0123] in represents the sigmoid activation function, represents different convolution kernels, and Then it represents the bias, and by using the above formula, the joint representation of each sentence pair is input , respectively get the network hidden layer state representation .
[0124] Next, use the network hidden layer output Input into a nonlinear fully connected neural network layer to get the neural network representation vector , as shown in formula (6). Then, Input to The classification layer outputs the final category , as shown in formula (7).
[0125] Formula (6)
[0126] in, represents the hyperbolic tangent function. represents the weight, Indicates bias.
[0127] Formula (7)
[0128] In order to train the language model, the neural network model loss function is defined as shown in formula (8).
[0129] Formula (8)
[0130] in, represents the true category label, Represents the model prediction result. The model is trained by back propagation algorithm and batch stochastic gradient descent method.
[0131] Verify the model effect on the validation set, and record and save the model parameters when the effect on the validation set is optimal.
[0132] Next, the best model can be used to test the samples in the test set, and finally the prediction results of each sentence pair are obtained, compared with the test labels, and the classification accuracy is calculated. The classification accuracy can be shown in Table 1.
[0133] Table 1 Classification accuracy
[0134]
[0135] In Table 1, TREC-QA and WIKIQA are the names of the test sets. MAP and MRR represent the test methods, respectively. For example, MAP is the average of the test results of multiple samples.
[0136] Method is the type of language model used, such as CNN is a convolutional neural network, LSTM is a long short-term memory language model, QLM is a quantum language model, and NNQLM-1 is a type 1 quantum language model. MBQLM is a measurement-based quasi-quantum language model used in the embodiments of the present disclosure. It can be seen that the test results of MBQLM used in the embodiments of the present disclosure for different test sample sets are the best, which effectively improves the classification accuracy.
[0137] The disclosed embodiment also provides a language processing method.
[0138] Fig. 9 The figure is a flow chart of a language processing method according to an embodiment of the present disclosure.
[0139] like Fig. 9 As shown, the language processing method may include operations S902 to S904.
[0140] In operation S902, an input question is obtained.
[0141] In operation S904, the input question sentence is processed using the language model to determine an output answer sentence.
[0142] The language model may be trained based on the above training method. For example, the input question comes from a question-answering system. The disclosed embodiment uses measurement quanta to capture the interactive information between sentence pairs, thereby improving the accuracy of sentence representation and the effect of the model.
[0143] It should be noted that the measurement quantum, the joint representation of sentence pairs, etc. can be as shown above and will not be repeated here.
[0144] The disclosed embodiment also provides a language model training device.
[0145] Fig.10 4 is a block diagram of a language model training device according to an embodiment of the present disclosure.
[0146] like Fig.10 As shown, the language model training device 1000 may include a sentence pair sample processing module 1010 and a language model training module 1020 .
[0147] The sentence pair sample processing module 1010 is used to process sentence pair samples to obtain a joint representation of sentence pairs. The sentence pair samples include paired question sentences and answer sentences. The joint representation of sentence pairs includes interactive information between paired question sentences and answer sentences.
[0148] The language model training module 1020 is used to train a language model based on the joint representation of sentence pairs.
[0149] Based on the shortcoming that the joint representation of density matrices in the current question-answering system QA cannot extract the interactive information between sentence pairs, the disclosed embodiment introduces quantum theory and uses a measurement operator to measure each sentence pair, so that the content of one sentence guides the representation of another sentence, and finally the two sentences are modeled together, which effectively improves.
[0150] The functions that can be implemented by each module in the embodiment of the present disclosure are shown in the above-mentioned related method part and will not be repeated here.
[0151] Another aspect of the present disclosure provides a language processing device.
[0152] Fig.11 4 is a block diagram of a language processing device according to an embodiment of the present disclosure.
[0153] like Fig.11 As shown, the language processing device 1100 may include: an input question acquisition module 1110 and an answer output module 1120.
[0154] The input question acquisition module 1110 is used to acquire an input question.
[0155] The answer output module 1120 is used to process the input question sentence using a language model to determine the output answer sentence, wherein the language model is trained based on the above training method.
[0156] The functions that can be implemented by each module in the embodiment of the present disclosure are shown in the above-mentioned related method part and will not be repeated here.
[0157] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.
[0158] For example, any multiple of the sentence pair sample processing module 1010 and the language model training module 1020 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the sentence pair sample processing module 1010 and the language model training module 1020 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the sentence pair sample processing module 1010 and the language model training module 1020 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.
[0159] Fig.12 is a block diagram of an electronic device according to an embodiment of the present disclosure. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0160] like Fig.12 As shown, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 to a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include an onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0161] In RAM 1203, various programs and data required for the operation of electronic device 1200 are stored. Processor 1201, ROM 1202 and RAM 1203 are connected to each other via bus 1204. Processor 1201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1202 and / or RAM 1203. It should be noted that the program can also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.
[0162] According to an embodiment of the present disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 may further include one or more of the following components connected to the I / O interface 1205: an input portion 1206 including a keyboard, a mouse, etc.; an output portion 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1208 including a hard disk, etc.; and a communication portion 1209 including a network interface card such as a LAN card, a modem, etc. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage portion 1208 as needed.
[0163] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.
[0164] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0165] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.
[0166] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0167] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0168] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A language model training method, include: Introducing quantum theory to perform the following operations: Processing sentence pair samples to obtain joint representations of the sentence pairs, the sentence pair samples comprising paired question sentences and answer sentences, the joint representations of the sentence pairs comprising interactive information between the paired question sentences and answer sentences; and training the language model based on the joint representation of the sentence pairs; The interactive information includes mutual detection results between the paired question sentences and answer sentences, and the mutual detection results include detection results of question sentences against answer sentences in the sentence pair samples, and detection results of answer sentences against question sentences in the sentence pair samples; The mutual detection result between the paired question and answer sentences is determined in the following manner: Measuring the answer sentence based on the measurement operator for the question sentence to obtain a first measurement result, and measuring the question sentence based on the measurement operator for the answer sentence to obtain a second measurement result; and Determining a mutual detection result between a pair of question sentences and answer sentences based on a product of the first measurement result and the second measurement result; Wherein: measuring the answer sentence based on the measurement operator for the question sentence to obtain the first measurement result comprises: determining the first measurement result based on a trace algorithm, a sentence representation of the answer sentence and a measurement operator for the question sentence; and Measuring the question sentence based on the measurement operator for the answer sentence to obtain the second measurement result includes: determining the second measurement result based on a trace algorithm, a sentence representation of the question sentence and a measurement operator for the answer sentence; The sentence representation of the question is determined in the following manner: Determining a second matrix based on the first matrix and the binary norm of the word vector of each word in the question sentence, wherein the first matrix includes the word vector of each word in the question sentence; and The at least two second matrices are mixed based on their respective weights to obtain a third matrix for representing the sentence representation of the question sentence.
2. The training method according to claim 1, in, The measurement operator for the question is determined in the following manner: Determine the first matrix for the question sentence using a word representation tool based on global word frequency statistics; Obtaining the binary norm of the word vector of each word in the question sentence included in the first matrix; as well as Based on the word vector of each word in the question and the two-norm ratio of each word, a measurement operator for the question is determined.
3. The training method according to claim 1, in, The training of the language model based on the joint representation of the sentence pairs comprises: Based on a back propagation algorithm and / or a batch stochastic gradient descent method, the language model is trained using the joint representation of the sentence pairs to obtain values of model parameters of the language model, wherein the model parameters include respective weights of the at least two second matrices.
4. The training method according to claim 3, in, The language model includes a convolutional layer, a fully connected layer and a classification layer; The method of training the language model based on the back propagation algorithm and / or the batch stochastic gradient descent method using the joint representation of the sentence pairs to obtain the values of the model parameters of the language model includes: Adjust the value of the model parameter of the language model, and for each of the plurality of sentence pair samples, Processing the joint representation of the sentence pair using the convolutional layer to obtain a state representation feature for the joint representation of the sentence pair, Processing the state representation features using the fully connected layer to obtain a representation vector, and Processing the representation vector using the classification layer to obtain category information; The above operations are repeatedly performed until the category information of the plurality of sentence pair samples meets the stopping condition.
5. The training method according to claim 1, further comprising: include: Acquire a sample data set, wherein the sample data set includes a plurality of sentence pair samples; Determine a sample data training set from the sample data set, or determine a sample data training set and at least one of a sample data verification set and a sample data test set from the sample data set; Using sentence pair samples in the sample data training set as training samples to train the language model; If there is a sample data validation set, the trained language model is validated using the sentence pairs in the sample data validation set to determine the optimal model parameter values; as well as If there is a sample data test set, the language model using the optimal model parameter values is tested on the sample using the sentences in the sample data test set to obtain a test result.
6. The training method according to claim 5, further comprising: include: After determining a sample data training set from the sample data set, or determining a sample data training set and at least one of a sample data validation set and a sample data test set from the sample data set, Preprocess at least part of the sentence pair samples in the sample data training set, the sample data validation set, and the sample data test set to remove stop words and / or specific symbols in the at least part of the sentence pair samples.
7. A language processing method, include: Get the input question; as well as Processing the input question sentence using a language model to determine an output answer sentence; Wherein, the language model is trained based on the training method according to any one of claims 1 to 6.
8. The language processing method according to claim 7, in, The input question comes from a question answering system.
9. An electronic device, include: one or more processors; A storage device for storing executable instructions, which, when executed by the processor, implements the language model training method according to any one of claims 1 to 6, or implements the language processing method according to any one of claims 7 to 8.
