Natural language pre-training model training method, device, equipment and storage medium

Abstract

The present application provides a method, device, apparatus, and storage medium for training a natural language pre-training model, which includes: tokenizing text using a dictionary and converting words into one-hot encoding; inputting the one-hot encoding into a word embedding layer and mapping it through the word embedding layer to obtain a corresponding static word vector for each word; adding the corresponding static word vector, paragraph embedding vector, and position embedding vector for each word to obtain an input vector for each word, and using the input vector as the input of the natural language pre-training model to obtain a corresponding dynamic word vector for each word; calculating the similarity between the corresponding static word vector and dynamic word vector for each word and using the similarity calculation result as a constraint term; adjusting the original loss function of the natural language pre-training model according to the constraint term, and training the natural language pre-training model with the adjusted original loss function.

Description

[Technical Field]

[0001] The present application relates to the field of natural language processing technology, and in particular to a natural language pre-training model training method, device, apparatus and storage medium. [Background technology]

[0002] Currently, the mainstream self-attention pre-training model with the BERT (Bidirectional Encoder Representation from Transformers) structure performs random masking on the words input in the text, and then the model predicts the cover words, thereby taking into account the contextual relationships in the obtained word vectors. Currently, many pre-training models improved by BERT improve model performance by adding corpora and expanding the model scale.

[0003] In the training process of natural language pre-training models, words have different meanings in different contexts, but the meanings of words in different contexts are derived from the meaning of the word itself. Therefore, the meaning of a word in a specific context is usually inferred from the meaning of the word itself. However, currently, when designing pre-trained models using BERT, the influence of the meaning of the word itself on the word vectors obtained after training is not fully considered. If the original meaning (static meaning) of the word is not fully considered, not only will the model training time increase, but the model's accuracy performance may also decrease.

[0004] In view of the problems existing in the prior art, there is an urgent need to provide a training method for a natural language pre-training model that takes into account the contextual meaning of words while fully considering the meaning of the words themselves, thereby improving the training effect of the natural language pre-training model and enabling the model to achieve higher accuracy and generalization performance. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a natural language pre-training model training method, device, equipment and storage medium to solve the problem in the prior art that the meaning of the words themselves is not fully taken into consideration, which reduces the training effect of the natural language pre-training model and prevents the model from achieving higher accuracy and generalization performance.

[0006] In a first aspect of an embodiment of the present application, a natural language pre-training model training method is provided, including the steps of: tokenizing text using a dictionary of a natural language pre-training model, and converting words in the text into corresponding one-hot encodings; inputting the corresponding one-hot encodings of the text into a word embedding layer and mapping them through the word embedding layer to obtain corresponding static word vectors for each word; adding the corresponding static word vectors, paragraph embedding vectors, and position embedding vectors for each word to obtain corresponding input vectors for each word, and using the input vectors as inputs to the natural language pre-training model to obtain corresponding dynamic word vectors for each word; calculating the similarity between the corresponding static word vectors and dynamic word vectors for each word, and using the similarity calculation results as constraint terms; adjusting an original loss function of the natural language pre-training model according to the constraint terms, and training the natural language pre-training model with the adjusted original loss function.

[0007] In a second aspect of an embodiment of the present application, there is provided a natural language pre-training model training device, including: a conversion module, configured to tokenize a text using a dictionary of the natural language pre-training model, and convert words in the text into corresponding one-hot encodings; a mapping module, configured to input the corresponding one-hot encodings of the text into a word embedding layer and map them through the word embedding layer to obtain corresponding static word vectors for each word; an input module, configured to add the corresponding static word vectors, paragraph embedding vectors, and position embedding vectors for each word to obtain corresponding input vectors for each word, and use the input vectors as inputs of the natural language pre-training model to obtain corresponding dynamic word vectors for each word; a calculation module, configured to calculate similarities between the corresponding static word vectors and dynamic word vectors for each word, and use the similarity calculation results as constraint terms; and an adjustment module, configured to adjust an original loss function of the natural language pre-training model according to the constraint terms, and train the natural language pre-training model with the adjusted original loss function.

[0008] In a third aspect of an embodiment of the present application, there is provided an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the electronic device implementing the steps of the above method when the processor executes the computer program.

[0009] In a fourth aspect of an embodiment of the present application, there is provided a computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the above method when executed by a processor.

[0010] The at least one technical solution used in the embodiments of the present application has the following beneficial effects:

[0011] The present application uses the dictionary of the natural language pre-training model to tokenize the text and convert the words in the text into corresponding one-hot encodings; input the corresponding one-hot encodings of the text into a word embedding layer and map them using the word embedding layer to obtain corresponding static word vectors for each word; add the corresponding static word vectors, paragraph embedding vectors, and position embedding vectors for each word to obtain corresponding input vectors for each word, and use the input vectors as inputs to the natural language pre-training model to obtain corresponding dynamic word vectors for each word; calculate the similarity between the corresponding static word vectors and dynamic word vectors for each word and use the similarity calculation results as constraint terms; adjust the original loss function of the natural language pre-training model according to the constraint terms, and train the natural language pre-training model with the adjusted original loss function. Through these steps, the present application takes into account the contextual meaning of the words and at the same time fully considers the meaning of the words themselves, thereby improving the training effect of the natural language pre-training model and enabling the model to achieve higher accuracy and generalization performance. [Brief explanation of the drawings]

[0012] In order to more clearly explain the technical solutions in the embodiments of the present application, the following briefly introduces drawings necessary for describing the embodiments or prior art. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative work.

[0013] [Figure 1] 1 is a flowchart of a natural language pre-training model training method provided in an embodiment of the present application. [Figure 2] FIG. 1 is a schematic diagram of the calculation process of constraint terms in a practical application scenario provided in an embodiment of the present application. [Figure 3] FIG. 1 is a schematic configuration diagram of a natural language pre-training model training device provided in an embodiment of the present application. [Figure 4]1 is a schematic configuration diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0014] In the following description, for purposes of explanation, not limitation, specific details, such as particular system structures and techniques, are provided to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary details.

[0015] In recent years, with the continuous development of artificial intelligence and natural language technology, natural language pre-training models have been widely used in various fields, such as text classification and speech recognition, to solve natural language processing tasks in real-world scenarios. Currently, the mainstream self-attention pre-training model with the Bidirectional Encoder Representation from Transformers (BERT) structure performs random masking on the words input in the text, and then the model predicts the cover words, taking into account the contextual relationships in the obtained word vectors. Currently, many pre-training models improved by BERT use methods such as adding corpora and expanding the model scale to improve model performance.

[0016] In the field of natural language processing, the mainstream BERT pre-training model currently uses the context of a word to obtain a dynamic word vector for that word. This method takes into account the different meanings of a word in different contexts, but does not take into account the inherent meaning of the word itself. In natural language, words have different meanings in different contexts, but the contextual meanings of a word are derived from the meaning of the word itself. Therefore, the meaning of a word in a specific context is usually inferred from the meaning of the word itself. However, currently, BERT pre-training models are designed without sufficient consideration of the influence of the meaning of the word itself on the word vector obtained after training. Failure to fully consider the original meaning (static meaning) of a word not only increases the model training time, but can also reduce the model's accuracy performance. Therefore, traditional training methods for natural language pre-training models suffer from long training times, low training effectiveness, and poor model accuracy and generalization performance.

[0017] In view of the problems in the prior art, the present application provides an improved natural language pre-training model training method, in which, before training the natural language pre-training model, the corresponding static word vector and dynamic word vector for each word are first obtained, and the similarity between the dynamic word vector obtained taking into account the context and the static word vector of the word itself is calculated, thereby extracting the representation of the two word vectors in semantic space. The original loss function of the natural language pre-training model is adjusted using the similarity calculation result as a constraint term, and the natural language pre-training model with the adjusted original loss function is trained. The trained model takes into account the context and fully considers the meaning of the word itself, thereby improving the effectiveness of the natural language pre-training model and achieving higher accuracy and general performance.

[0018] 1 is a flowchart of a natural language pre-training model training method provided in an embodiment of the present application. The natural language pre-training model training method of FIG. 1 may be executed by a server. As shown in FIG. 1, the natural language pre-training model training method specifically includes: S101, which tokenizes the text using a dictionary of a natural language pre-trained model and converts words in the text into their corresponding one-hot encodings; S102: inputting the corresponding one-hot encoding of the text into a word embedding layer, and mapping it through the word embedding layer to obtain the corresponding static word vector of each word; S103: add the corresponding static word vector, paragraph embedding vector and position embedding vector of each word to obtain the corresponding input vector of each word, and use the input vector as the input of the natural language pre-training model to obtain the corresponding dynamic word vector of each word; S104: calculating a similarity between the static word vector and the dynamic word vector corresponding to each word, and using the similarity calculation result as a constraint term; S105: adjusting the original loss function of the natural language pre-trained model according to the constraint term, and training the natural language pre-trained model with the adjusted original loss function.

[0019] Specifically, the one-hot encoding in the embodiments of the present application, also known as one-bit valid coding, uses an N-bit status register to code N states, each with its own register bit, only one of which is valid at any given time. The embodiments of the present application convert each word in the text into a corresponding one-hot encoding, so that the entire text corresponds to a series of one-hot encodings (permutations of one-hot encodings according to the order of the words).

[0020] Furthermore, in the embodiments of the present application, different word vectors obtained by a word in different contexts are referred to as dynamic word vectors, and word vectors obtained without considering the context of a word are referred to as static word vectors of the word, where dynamic word vectors can represent the meaning of a word in different contexts, while static word vectors can represent the meaning of the word itself.

[0021] It should be noted that the following embodiments of the present application will be described in detail using an example in which a BERT-based self-attention pre-training model (BERT pre-training model or BERT model) is used as the natural language pre-training model. However, it should be understood that the natural language pre-training model in the embodiments of the present application is limited to the BERT pre-training model, and any model that can be used for natural language processing tasks is applicable to the present application, and the type of natural language pre-training model does not limit the technical solution of the present application.

[0022] In some embodiments, the step of inputting the corresponding one-hot encodings of the text into a word embedding layer and mapping them through the word embedding layer to obtain corresponding static word vectors for each word includes the steps of: generating a series of corresponding one-hot encodings of the text based on the corresponding one-hot encodings of each word in the text; inputting the series of one-hot encodings into the word embedding layer; mapping the series of one-hot encodings through the word embedding layer to obtain corresponding raw vector representations of each word; and using the raw vector representations of each word as the static word vectors.

[0023] Specifically, before mapping through the word embedding layer to obtain the corresponding static word vectors for each word, the input text is first tokenized based on the dictionary of a natural language pre-trained model (BERT pre-trained model), and then converted into the corresponding one-hot encoding of the words according to the vocabulary of the BERT pre-trained model.

[0024] Furthermore, after obtaining the corresponding one-hot encoding of each word, a series of corresponding one-hot encodings of the text is generated based on the corresponding one-hot encoding of each word and the order of each word in the text, and this series of one-hot encodings is input into the word embedding layer of the BERT pre-training model and mapped to obtain the corresponding original vector representation of each word, that is, the corresponding static word vector of each word. The static word vector can represent the meaning of the word itself.

[0025] In some embodiments, the step of adding the corresponding static word vector, paragraph embedding vector, and position embedding vector of each word to obtain a corresponding input vector of each word, using the input vector as an input of a natural language pre-training model to obtain a corresponding dynamic word vector of each word includes the steps of: obtaining a corresponding paragraph embedding vector and position embedding vector in the text for each word, respectively mapping the static word vector, paragraph embedding vector, and position embedding vector in the same dimensional space, adding the static word vector, paragraph embedding vector, and position embedding vector in the same dimensional space to obtain a corresponding input vector of each word; and inputting the input vector into the natural language pre-training model, training the natural language pre-training model on the word masking task and the context task, and outputting a corresponding dynamic word vector of each word in the text.

[0026] Specifically, after mapping using the word embedding layer to obtain the corresponding static word vector of each word, the static word vector, segment embedding vector, and position embedding vector of each word are respectively mapped into the same dimensional space. For example, each vector is mapped into a 768-dimensional space, that is, each vector is mapped into a 768-dimensional vector. Then, the static word vector, segment embedding vector, and position embedding vector in the same dimension are added (i.e., vector addition) to obtain the corresponding input vector of each word.

[0027] Furthermore, the input vector is input into a BERT pre-trained model, and the BERT pre-trained model is used to train the word masking task and the context task, and finally the BERT pre-trained model is used to output the corresponding dynamic word vector of each word in the text.

[0028] The BERT model (Bidirectional Encoder Representations from Transformer) is a language representation model pre-trained with a bidirectional encoder representation by Transformer. Rather than using a traditional unidirectional language model or pre-training by shallowly combining two unidirectional language models, it employs a new masked language model (MLM) to generate deep bidirectional language representations. The BERT model targets training and obtaining representations of text (i.e., semantic representations of text) containing rich semantic information using a large-scale unlabeled corpus, and then fine-tuning the semantic representations for specific NLP tasks and finally using them for the NLP tasks.

[0029] Furthermore, the BERT official model uses two tasks as pre-training to learn semantic information, that is, it introduces two core tasks, namely, random static mask language model training task (Masked LM) and next sentence prediction task (Next Sentence Prediction), into the pre-training of the BERT model. Since this application does not improve or adjust the structure and training task of the BERT model itself, the BERT model will not be described in detail here.

[0030] In some embodiments, the step of calculating the similarity between the corresponding static word vector and dynamic word vector of each word and using the similarity calculation result as a constraint term includes the step of calculating a vector dot product between the static word vector and dynamic word vector of each word, using the vector dot product as a similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term constructed from the static word vector, wherein the static word vector and the dynamic word vector have the same dimension.

[0031] Specifically, after obtaining the corresponding static word vector and dynamic word vector of each word, the constraints (i.e., constraint terms) in the BERT model training process are increased by calculating the vector similarity between the static word vector and the dynamic word vector. In practical application, the embodiments of the present application preferably use the dot product between vectors to represent the similarity between vectors, and the larger the dot product of the vectors, the greater the similarity.

[0032] Furthermore, when using the vector dot product to measure the similarity between the static word vector and the dynamic word vector, the vector dot product can be calculated using the following formula: JPEG2025526865000002.jpg31170

[0033] where R represents the vector dot product, N represents the number of words (or letters) in the sentence, i represents the position of the word (or letter) in the sentence, and Ve i represents the static word vector, and Vt i represents the dynamic word vector.

[0034] In addition, the examples of the present application are as follows: JPEG2025526865000003.jpg36170, where i is the position of the word or character in the sentence, generally starting from 0, and there are a total of N words or characters in the sentence. The calculated and obtained R is the subsequent constraint term, and the constraint term is also called a constraint condition.

[0035] In some embodiments, the step of calculating the similarity between the corresponding static word vector and dynamic word vector of each word and using the similarity calculation result as a constraint term includes the step of calculating the cosine similarity or Manhattan distance between the static word vector and dynamic word vector of each word, using the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term.

[0036] Specifically, in the embodiment of the present application, in addition to using the vector dot product to represent the similarity between vectors, the cosine similarity or Manhattan distance between vectors can also be used to represent the similarity, i.e., the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector can be used as a constraint. Although the calculation method of the cosine similarity or Manhattan distance is not described here, of course, other calculation methods of the similarity between vectors besides the cosine similarity or Manhattan distance can also be applied to the present application.

[0037] According to the technical solution provided in the embodiments of the present application, the embodiments of the present application use vector dot product, cosine similarity, Manhattan distance, etc. to measure the similarity between vectors, thereby increasing the similarity in semantic space between dynamic word vectors and static word vectors, and finally, not only integrating contextual information into the obtained word vector, but also fully referring to the static meaning of the word itself.

[0038] In some embodiments, adjusting the original loss function of the natural language pre-training model according to the constraint term comprises adjusting the original loss function using the following equation: loss=(1-α)·suploss-α·regulation

[0039] Here, loss represents the adjusted loss function, suploss represents the original loss function, α represents the distribution coefficient for adjusting the model training accuracy, and regulation represents the constraint term constructed from static word vectors.

[0040] Specifically, after calculating and obtaining the constraint terms based on the static word vectors, the original loss function of the natural language pre-training model (BERT pre-training model) in the downstream natural language processing task is adjusted based on the constraint terms, i.e., the original loss function suploss is adjusted according to the above formula, and the adjusted loss function loss is obtained.

[0041] In practical applications, loss is the refined (i.e., adjusted) loss function, suploss is the original supervised learning loss function (e.g., cross-entropy loss function), regulation is the constraint term constructed from static word (character) vectors mentioned in the previous sentence, and α is a distribution coefficient for adjusting the model training accuracy, which is in the open interval of 0 to 1 and empirically ranges from 0.1 to 0.2 and needs to be adjusted according to different tasks.

[0042] The above content has described in detail the complete embodiment of the technical solution of the present application, and the training process of the natural language pre-training model of the present application will now be described with reference to the drawings and specific embodiments. Figure 2 is a schematic diagram of the calculation process of constraint terms in an actual application scenario provided in the embodiment of the present application. As shown in Figure 2, the calculation process of constraint terms in the actual application scenario may specifically include the following content:

[0043] In a specific example, for a sentence consisting of six original characters, "CLS Longhu Group SEP", first, each word (or character) is converted into a corresponding one-hot encoding, and then the one-hot encoding is mapped as a static word vector by an embedding mapping layer (i.e., a word embedding layer), that is, mapped as static word vectors Ve0 to Ve5, respectively. Then, the corresponding input vector of each word is input to a multi-layer self-attention neural network (i.e., a BERT model network), and the BERT model network outputs the corresponding dynamic word vector of each word, and the corresponding dynamic word vector of each word (or character) is denoted as Vt0 to Vt5, respectively.

[0044] The input vector of a word is obtained by adding the static word vector, paragraph embedding vector, and position embedding vector, which are mapped to the same dimension. For example, all vectors are mapped as 768-dimensional vectors, so the static word vectors Ve0 to Ve5 and the dynamic word vectors Vt0 to Vt5 have the same dimension. The static word vectors represent the static meaning of each word. The dynamic word vectors are generated using an attention mechanism, which integrates contextual information. Therefore, the dynamic word vectors contain the dynamic meaning of each word.

[0045] Then, from the static word vector and dynamic word vector of each word, the vector dot product between the static word vector and the dynamic word vector is calculated using the vector dot product calculation formula provided in the above-mentioned embodiment, and the vector dot product is used as a constraint term. The original loss function of the BERT model in the supervised learning natural language processing task is adjusted using the constraint term, and the BERT model with the adjusted loss function is trained, thereby providing the trained BERT model with higher accuracy and generalization performance.

[0046] According to the technical solutions provided in the embodiments of the present application, the embodiments of the present application have at least the following advantages:

[0047] (1) In this application, during training of the pre-training model using BERT, constraint terms are calculated from the static word vectors and dynamic word vectors of the token (word), the original loss function of the BERT pre-training model is adjusted according to the constraint terms, and the BERT pre-training model with the adjusted original loss function is trained, thereby shortening the training time of the BERT pre-training model.

[0048] (2) In this application, a constraint term is introduced into the training process of the BERT pre-training model to increase the similarity between the dynamic word vector and the static word vector of each word in a sentence, thereby achieving the purpose of shortening the distance between the dynamic word vector and the static word vector in the semantic space.

[0049] (3) This application applies to various pre-trained models similar to BERT (including various improved models) based on multi-layer self-attention mechanisms, making it applicable in a wide range of applications.

[0050] (4) Using the model training method provided in this application, after fine-tuning training of the model in downstream tasks, the model can have higher accuracy and general performance than models that do not use this solution.

[0051] Below are apparatus embodiments of the present application for carrying out the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0052] 3 is a schematic diagram of a natural language pre-training model training device provided in an embodiment of the present application. As shown in FIG. 3, the natural language pre-training model training device includes: a transformation module 301 arranged to tokenize the text utilizing a dictionary of a natural language pre-trained model and transform words in the text into their corresponding one-hot encodings; a mapping module 302 configured to input the corresponding one-hot encoding of the text into the word embedding layer and map it through the word embedding layer to obtain a corresponding static word vector for each word; an input module 303 configured to sum the corresponding static word vector, paragraph embedding vector and position embedding vector of each word to obtain a corresponding input vector of each word, and take the input vector as an input of a natural language pre-training model to obtain a corresponding dynamic word vector of each word; a calculation module 304 configured to calculate a similarity between the static word vector and the dynamic word vector corresponding to each word, and use the similarity calculation result as a constraint term; An adjustment module 305 configured to adjust the original loss function of the natural language pre-trained model according to the constraint term and train the natural language pre-trained model with the adjusted original loss function.

[0053] In some embodiments, the mapping module 302 in FIG. 3 generates a corresponding series of one-hot encodings of the text based on the corresponding one-hot encoding of each word in the text, inputs the series of one-hot encodings into a word embedding layer, maps the series of one-hot encodings through the word embedding layer to obtain a corresponding raw vector representation of each word, and sets the raw vector representation of each word as a static word vector.

[0054] In some embodiments, the input module 303 in FIG. 3 obtains the corresponding paragraph embedding vector and position embedding vector in the text for each word, respectively maps the static word vector, paragraph embedding vector, and position embedding vector in the same dimensional space, adds the static word vector, paragraph embedding vector, and position embedding vector in the same dimensional space to obtain the corresponding input vector for each word, inputs the input vector into a natural language pre-training model, trains the natural language pre-training model on the word masking task and the context task, and outputs the corresponding dynamic word vector for each word in the text.

[0055] In some embodiments, the calculation module 304 of FIG. 3 includes calculating a vector dot product between the static word vector and the dynamic word vector for each word, taking the vector dot product as a similarity calculation result between the static word vector and the dynamic word vector, and taking the similarity calculation result as a constraint term constructed from the static word vector, where the static word vector and the dynamic word vector have the same dimension.

[0056] In some embodiments, the calculation module 304 of FIG. 3 calculates the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector of each word, and uses the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and the dynamic word vector, and uses the similarity calculation result as a constraint term.

[0057] In some implementations, the adjustment module 305 of FIG. 3 adjusts the original loss function using the following equation: loss=(1-α)·suploss-α·regulation

[0058] Here, loss represents the adjusted loss function, suploss represents the original loss function, α represents the distribution coefficient for adjusting the model training accuracy, and regulation represents the constraint term constructed from static word vectors.

[0059] It should be understood that the magnitude of the numbers of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and does not arbitrarily limit the implementation process of the embodiments of the present application.

[0060] 4 is a schematic diagram of an electronic device 4 provided in an embodiment of the present application. As shown in FIG. 4, the electronic device 4 of the embodiment includes a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable by the processor 401. When the processor 401 executes the computer program 403, it performs the steps in each of the above method embodiments. Alternatively, when the processor 401 executes the computer program 403, it performs the functions of each module / unit in each of the above device embodiments.

[0061] For example, the computer program 403 may be divided into one or more modules / units, which may be stored in the memory 402 and executed by the processor 401 to accomplish the present application. The one or more modules / units may be a series of computer program command sections capable of performing specific functions, which are intended to explain the process of the computer program 403 being executed in the electronic device 4.

[0062] The electronic device 4 may be an electronic device such as a desktop computer, a laptop computer, a palmtop computer, or a cloud server. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. As will be understood by those skilled in the art, FIG. 4 is merely an example of the electronic device 4 and is not intended to limit the electronic device 4, which may include more or fewer components than those shown, or may combine certain components or different components; for example, the electronic device may include input / output devices, network access devices, buses, etc.

[0063] Processor 601 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, any common processor, etc.

[0064] The memory 402 may be an internal storage unit of the electronic device 4, such as a hard disk or RAM of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash card, etc., that is provided in the electronic device 4. Furthermore, the memory 402 may include not only the internal storage unit of the electronic device 4 but also an external storage device. The memory 402 is intended to store computer programs and other programs and data required by the electronic device. The memory 402 may also be used to temporarily store data that has been output or that is to be output.

[0065] Those skilled in the art will understand that, for convenience and brevity, only the division of the above functional units and modules has been used as an example. However, in actual applications, the above functions can be assigned to different functional units or modules as needed, i.e., all or part of the above-described functions can be achieved by dividing the internal structure of the device into different functional units or modules. The functional units and modules in the embodiments may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit. The integrated unit may be implemented in the form of hardware or software functional units. The specific names of the functional units and modules are provided solely for the purpose of distinguishing them from one another and do not limit the scope of protection of the present application. For the specific operating processes of the units and modules in the above system, reference may be made to the corresponding processes in the above-described method embodiments, and further description will be omitted here.

[0066] In the above embodiments, the description of each embodiment has its own emphasis, and for the details or parts not described in an embodiment, reference can be made to the relevant descriptions of other embodiments.

[0067] Those skilled in the art can recognize that the units and algorithm steps of each example described in the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software is determined by the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to realize the described functions for each specific application, but such realization should not be considered to go beyond the scope of this application.

[0068] It should be understood that the disclosed devices / computer devices and methods in the embodiments provided in this application can be implemented in other ways. For example, the device / computer device embodiments described above are merely illustrative, and the division into modules or units is merely a logical division of functions. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into other systems, or some features may be omitted or not implemented. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through several interfaces, devices, or units, and may be electrical, mechanical, or other types.

[0069] Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, i.e., located in one location or distributed across multiple network units, some or all of which may be selected according to actual needs to achieve the objectives of the solutions of this embodiment.

[0070] In addition, each functional unit in the present application may be integrated into one processing unit, each unit may exist physically independently, or two or more units may be integrated into one unit. The integrated unit may be realized in the form of hardware or in the form of a software functional unit.

[0071] The integrated module / unit may be realized in the form of a software functional unit and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the present application recognizes that the realization of all or part of the processes in the above-described method embodiments can be accomplished by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-described method embodiments can be realized. The computer program may include computer program code, which may be in source code format, object code format, an executable file, or some intermediate format. The computer-readable storage medium may include any entity or device capable of carrying computer program code, such as a recording medium, a U-disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier wave signal, an electrical communication signal, and a software distribution medium. Furthermore, the content contained on a computer-readable storage medium may be increased or decreased as required by the legislation and patent practice of a jurisdiction. For example, in some jurisdictions, the legislation and patent practice may require that a computer-readable storage medium not include electrical carrier signals and telecommunications signals.

[0072] The above-mentioned embodiments are only for illustrating the technical solutions of the present invention, but are not intended to limit the same. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art may still amend the technical solutions described in the above-mentioned embodiments or equivalently replace some technical features therein, but it should be understood that such amendments or replacements shall not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and all of them shall fall within the protection scope of the present invention.

Claims

1. Tokenizing text using a dictionary of a natural language pre-trained model and converting words in the text into their corresponding one-hot encodings; inputting the corresponding one-hot encoding of the text into a word embedding layer, and mapping it through the word embedding layer to obtain a corresponding static word vector for each word; summing the corresponding static word vector, paragraph embedding vector and position embedding vector of each word to obtain a corresponding input vector of each word, and using the input vector as an input of the natural language pre-training model to obtain a corresponding dynamic word vector of each word; a step of calculating a similarity between the static word vector and the dynamic word vector corresponding to each word, and using the similarity calculation result as a constraint term; adjusting an original loss function of the natural language pre-trained model according to the constraint term, and training the natural language pre-trained model with the adjusted original loss function. A method for training a natural language pre-training model, comprising:

2. The step of inputting the corresponding one-hot encoding of the text into a word embedding layer and mapping it through the word embedding layer to obtain a corresponding static word vector for each word includes: According to the corresponding one-hot encoding of each word in the text, generate a corresponding series of one-hot encodings of the text, input the series of one-hot encodings into the word embedding layer, map the series of one-hot encodings through the word embedding layer to obtain a corresponding raw vector representation of each word, and use the raw vector representation of each word as a static word vector.

2. The method of claim 1 .

3. the step of adding the corresponding static word vector, paragraph embedding vector, and position embedding vector of each word to obtain a corresponding input vector of each word, and using the input vector as an input of the natural language pre-training model to obtain a corresponding dynamic word vector of each word; Obtaining a corresponding paragraph embedding vector and a position embedding vector in the text for each word, respectively mapping the static word vector, the paragraph embedding vector and the position embedding vector into the same dimensional space, and adding the static word vector, the paragraph embedding vector and the position embedding vector in the same dimensional space to obtain a corresponding input vector for each word; inputting the input vector into the natural language pre-training model, training the natural language pre-training model on a word masking task and a contextual task, and outputting a corresponding dynamic word vector for each word in the text.

2. The method of claim 1 .

4. the step of calculating a similarity between the static word vector and the dynamic word vector corresponding to each word and using the similarity calculation result as a constraint term, a step of calculating a vector dot product between the static word vector and the dynamic word vector for each word, using the vector dot product as a similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term composed of the static word vector, wherein the static word vector and the dynamic word vector have the same dimension.

2. The method of claim 1 .

5. the step of calculating a similarity between the static word vector and the dynamic word vector corresponding to each word and using the similarity calculation result as a constraint term, Calculating the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector for each word, and using the cosine similarity or Manhattan distance as a similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term.

2. The method of claim 1 .

6. The step of adjusting the original loss function of the natural language pre-training model according to the constraint term comprises adjusting the original loss function using the following equation: loss=(1-α)・suploss-α・regulation Here, loss represents the adjusted loss function, suploss represents the original loss function, α represents a distribution coefficient for adjusting the model training accuracy, and regulation represents a constraint term constructed from the static word vector.

5. The method of claim 4.

7. As the natural language pre-training model, a self-attention pre-training model based on BERT is used.

2. The method of claim 1 .

8. a conversion module configured to tokenize text utilizing a dictionary of a natural language pre-trained model and convert words in the text into their corresponding one-hot encodings; a mapping module configured to input the corresponding one-hot encoding of the text into a word embedding layer and map it through the word embedding layer to obtain a corresponding static word vector for each word; an input module configured to sum a corresponding static word vector, a paragraph embedding vector, and a position embedding vector for each word to obtain a corresponding input vector for each word, and use the input vector as an input of the natural language pre-training model to obtain a corresponding dynamic word vector for each word; a calculation module configured to calculate a similarity between the static word vector and the dynamic word vector corresponding to each word, and to use a result of the similarity calculation as a constraint term; an adjustment module configured to adjust an original loss function of the natural language pre-trained model according to the constraint term and train the natural language pre-trained model with the adjusted original loss function. A natural language pre-training model training device comprising:

9. 10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the electronic device implementing the method of claim 1 when the processor executes the computer program.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being capable of implementing the method of claim 1 when executed by a processor.

Images