Natural language pre-training model training method, apparatus, device, and storage medium

Abstract

The present application provides a method, device, apparatus, and storage medium for training a natural language pre-training model, which includes: tokenizing text using a dictionary and converting words into one-hot encoding; inputting the one-hot encoding into a word embedding layer and mapping it through the word embedding layer to obtain a corresponding static word vector for each word; adding the corresponding static word vector, paragraph embedding vector, and position embedding vector for each word to obtain an input vector for each word, and using the input vector as the input of the natural language pre-training model to obtain a corresponding dynamic word vector for each word; calculating the similarity between the corresponding static word vector and dynamic word vector for each word and using the similarity calculation result as a constraint term; adjusting the original loss function of the natural language pre-training model according to the constraint term, and training the natural language pre-training model with the adjusted original loss function.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and particularly to a method, apparatus, device, and storage medium for training a natural language pre-training model.

Background Art

[0002] Currently, the self-attention pre-training model based on the mainstream BERT (Bidirectional Encoder Representation from Transformers) structure makes the model consider the contextual relationship in the obtained word vectors by randomly masking the words input to the text and then predicting the covered words by the model. Currently, many pre-training models improved by BERT improve the performance of the model by methods such as adding corpora and expanding the model scale.

[0003] In the training process of the natural language pre-training model, although a word has different meanings in different contexts, the different contextual meanings of the word are derived from the meaning of the word itself. Therefore, usually, the meaning of the word in a specific context is estimated from the meaning of the word itself. However, currently, the pre-training model based on BERT does not fully consider the influence on the word vector obtained after training from the meaning of the word itself during design. If the original meaning (static meaning) of the word is not fully considered, it will not only increase the training time of the model, but also may reduce the accuracy performance of the model.

[0004] In view of the problems existing in the prior art, it is necessary to urgently provide a training solution for a natural language pre-training model that can consider the contextual meaning of words while fully considering the meaning of the words themselves, improve the training effect of the natural language pre-training model, and enable the model to achieve higher accuracy and generalization performance.

Summary of the Invention

[0005] In view of this, the embodiments of this application provide a method, apparatus, device, and storage medium for training a natural language pre-trained model in order to solve the problem in the prior art of not adequately considering the meaning of the words themselves, which reduces the training effect of the natural language pre-trained model and prevents the model from achieving higher accuracy and generalization performance.

[0006] A first embodiment of the present invention provides a method for training a natural language pre-trained model, comprising the steps of: tokenizing text using a dictionary of a natural language pre-trained model and converting the words in the text into corresponding one-hot encodings; inputting the corresponding one-hot encodings of the text into a word embedding layer and mapping them using the word embedding layer to obtain corresponding static word vectors for each word; adding the corresponding static word vectors, paragraph embedding vectors, and position embedding vectors for each word to obtain corresponding input vectors for each word, using the input vectors as input to a natural language pre-trained model to obtain corresponding dynamic word vectors for each word; calculating the similarity between the corresponding static word vectors and dynamic word vectors for each word and using the similarity calculation results as constraint terms; and adjusting the original loss function of the natural language pre-trained model using the constraint terms and training the natural language pre-trained model with the adjusted original loss function.

[0007] A second embodiment of the present invention provides a natural language pretraining model training device comprising: a conversion module configured to tokenize text using a dictionary of a natural language pretraining model and convert the words in the text into corresponding one-hot encodings; a mapping module configured to input the corresponding one-hot encodings of the text into a word embedding layer and map them using the word embedding layer to obtain corresponding static word vectors for each word; an input module configured to add the corresponding static word vectors, paragraph embedding vectors, and position embedding vectors for each word to obtain corresponding input vectors for each word, use the input vectors as input to a natural language pretraining model, and obtain corresponding dynamic word vectors for each word; a calculation module configured to calculate the similarity between the corresponding static word vectors and dynamic word vectors for each word and use the similarity calculation results as constraint terms; and an adjustment module configured to adjust the original loss function of the natural language pretraining model using the constraint terms and train the natural language pretraining model with the adjusted original loss function.

[0008] A third embodiment of the present invention provides an electronic device comprising memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the steps of the above method are realized when the processor executes the computer program.

[0009] A fourth embodiment of the present application provides a computer-readable storage medium in which a computer program is stored, and which enables the steps of the above method when the computer program is executed by a processor.

[0010] The at least one of the above technical solutions used in the embodiments of this application provides the following beneficial effects.

[0011] The present invention improves the training effectiveness of a natural language pre-training model by considering the contextual meaning of words while also fully considering the meaning of words themselves, thereby enabling the model to achieve higher accuracy and generalization performance. This is achieved through the following steps: tokenizing text using a dictionary of a natural language pre-training model and converting the words in the text into corresponding one-hot encodings; inputting the corresponding one-hot encodings of the text into a word embedding layer and mapping them using the word embedding layer to obtain corresponding static word vectors for each word; adding the corresponding static word vectors, paragraph embedding vectors, and position embedding vectors for each word to obtain corresponding input vectors for each word, using the input vectors as input to a natural language pre-training model, and obtaining corresponding dynamic word vectors for each word; calculating the similarity between the corresponding static word vectors and dynamic word vectors for each word and using the similarity calculation results as constraint terms; and adjusting the original loss function of the natural language pre-training model using the constraint terms and training the natural language pre-training model with the adjusted original loss function. [Brief explanation of the drawing]

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the following is a brief introduction of the drawings necessary for describing the embodiments or the prior art. Clearly, the drawings described below represent only a few embodiments of the present invention, and those skilled in the art can obtain other drawings based on these, without any creative work.

[0013] [Figure 1] This is a flowchart of the natural language pre-training model training method provided in the embodiment of this application. [Figure 2] This is a schematic diagram of the process for calculating constraint terms in an actual application scenario provided in the embodiments of this application. [Figure 3] This is a schematic diagram of the natural language pre-training model training device provided in the embodiment of this application. [Figure 4]This is a schematic diagram of the electronic device provided in the embodiment of this application. [Modes for carrying out the invention]

[0014] In the following description, the embodiments of this application will be thoroughly understood by providing specific details, such as particular system structures and technologies, for illustrative purposes rather than being limiting. However, those skilled in the art should understand that the application can be realized in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, apparatus, circuits and methods will be omitted to avoid unnecessary details interfering with the description of this application.

[0015] In recent years, with the continuous development of artificial intelligence and natural language technology, natural language pre-trained models are widely used in various fields such as text classification and speech recognition to solve natural language processing tasks in real-world scenarios. Currently, the mainstream BERT (Bidirectional Encoder Representation from Transformers) structure self-attention pre-trained models consider contextual relationships in the obtained word vectors by performing random masking on words input into text and then having the model predict cover words. Many improved BERT pre-trained models now enhance model performance through methods such as adding corpus and increasing model size.

[0016] In the current field of natural language processing, the mainstream BERT pre-training model obtains dynamic word vectors of words based on their context. This method considers different meanings of words in different contexts, but does not give much consideration to the intrinsic meaning of the words themselves. In natural language, words have different meanings in different contexts, but the meaning of a word in a different context is derived from the meaning of the word itself. Therefore, the meaning of a word in a specific context is usually estimated from the meaning of the word itself. However, currently, BERT pre-training models do not adequately consider the influence of the meaning of the word itself on the word vectors obtained after training during the design phase. If the intrinsic meaning of a word (static meaning) is not adequately considered, it not only increases the training time of the model but can also decrease the accuracy performance of the model. Therefore, conventional training methods for natural language pre-training models have the problems of long training times, low training effectiveness, and low model accuracy and generalization performance.

[0017] In view of the problems in the prior art, this application provides an improved natural language pre-training model training method. In this application, before training the natural language pre-training model, the corresponding static and dynamic word vectors for each word are first obtained, and the representation of the two word vectors in the semantic space is drawn in by calculating the similarity between the dynamic word vector, which is obtained taking context into consideration, and the static word vector of the word itself. The original loss function of the natural language pre-training model is adjusted using the similarity calculation result as a constraint term, and the natural language pre-training model with the adjusted original loss function is trained. The trained model takes context into full consideration as well as the meaning of the word itself, improving the effectiveness of the natural language pre-training model and resulting in higher accuracy and general performance.

[0018] Figure 1 is a flowchart of a natural language pre-training model training method provided in an embodiment of this application. The natural language pre-training model training method of Figure 1 may be executed by a server. As shown in Figure 1, the natural language pre-training model training method specifically includes: S101 tokenizes text using a dictionary from a pre-trained natural language model and converts the words in the text into their corresponding one-hot encodings. S102 inputs the corresponding one-hot encoding of the text into a word embedding layer, and the word embedding layer maps it to obtain the corresponding static word vector for each word. S103 involves adding the corresponding static word vector, paragraph embedding vector, and position embedding vector for each word to obtain the corresponding input vector for each word, using the input vector as input to a natural language pre-trained model, and obtaining the corresponding dynamic word vector for each word. S104 calculates the similarity between the corresponding static word vector and dynamic word vector for each word, and uses the similarity calculation result as a constraint term. This includes step S105, which adjusts the original loss function of the pre-trained natural language model using constraints, and then trains the pre-trained natural language model with the adjusted original loss function.

[0019] Specifically, the one-hot encoding of the embodiment of this application, also known as one-bit valid coding, works by using an N-bit status register to encode N states, with each state having its own register bit, and only one of these bits being valid at any given time. In the embodiment of this application, each word in the text is converted to its corresponding one-hot encoding, so the entire text corresponds to a series of one-hot encodings (a permutation of one-hot encodings according to the order of the words).

[0020] Furthermore, in the embodiments of the present application, different word vectors obtained for a word in different contexts are referred to as dynamic word vectors, and word vectors obtained without considering the context of the word are referred to as static word vectors of the word. Here, dynamic word vectors can represent the meaning of a word in different contexts, while static word vectors can represent the meaning of the word itself.

[0021] Note that the following embodiments of the present application will be described in detail by taking the self-attention pre-training model by BERT (BERT pre-training model or BERT model) as the natural language pre-training model. However, it should be understood that the natural language pre-training model in the embodiments of the present application is not limited to the BERT pre-training model, and any model that can be used for natural language processing tasks can be applied to the present application, and the type of the natural language pre-training model does not limit the technical solution of the present application.

[0022] In some embodiments, the step of inputting the corresponding one-hot encoding of the text into the word embedding layer and obtaining the corresponding static word vector of each word by mapping through the word embedding layer includes generating a series of corresponding one-hot encodings of the text based on the corresponding one-hot encoding of each word in the text, inputting the series of one-hot encodings into the word embedding layer, mapping the series of one-hot encodings through the word embedding layer to obtain the corresponding original vector representation of each word, and using the original vector representation of each word as the static word vector.

[0023] Specifically, before obtaining the corresponding static word vector of each word by mapping through the word embedding layer, first, based on the dictionary of the natural language pre-training model (BERT pre-training model), the input text is tokenized and then converted into the corresponding one-hot encoding (One-Hot Encoding) of the word by the vocabulary of the BERT pre-training model.

[0024] Furthermore, after obtaining the corresponding one-hot encoding of each word, based on the corresponding one-hot encoding of each word and the order of each word in the text, a corresponding series of one-hot encodings of the text are generated. This series of one-hot encodings are input into the word embedding layer of the BERT pre-training model and mapped to obtain the corresponding original vector representation of each word, that is, the corresponding static word vector of each word. The static word vector can represent the meaning of the word itself.

[0025] In some embodiments, the step of adding the corresponding static word vector, paragraph embedding vector and position embedding vector of each word to obtain the corresponding input vector of each word, and using the input vector as the input of the natural language pre-training model to obtain the corresponding dynamic word vector of each word includes: obtaining the corresponding paragraph embedding vector and position embedding vector of each word in the text, mapping the static word vector, paragraph embedding vector and position embedding vector into the same dimensional space respectively, adding the static word vector, paragraph embedding vector and position embedding vector in the same dimensional space, obtaining the corresponding input vector of each word, inputting the input vector into the natural language pre-training model, training the word masking task and context task by the natural language pre-training model, and outputting the corresponding dynamic word vector of each word in the text.

[0026] Specifically, after obtaining the corresponding static word vector of each word by mapping through the word embedding layer, the static word vector, paragraph embedding vector and position embedding vector of each word are mapped into the same dimensional space respectively. For example, each vector is mapped into a 768-dimensional space, that is, each vector is mapped into a 768-dimensional vector. Furthermore, the static word vector, paragraph embedding vector (segment embedding) and position embedding vector (position embedding) in the same dimension are added (i.e., the vectors are added) to obtain the corresponding input vector of each word.

[0027] Furthermore, the input vectors are fed into a BERT pre-trained model, the BERT pre-trained model is used to train on word masking and context tasks, and finally, the BERT pre-trained model is used to output the corresponding dynamic word vectors for each word in the text.

[0028] The BERT model (Bidirectional Encoder Representations from Transformer) is a pre-trained language representation model using bidirectional encoder representations based on Transformers. Unlike conventional methods that use traditional one-way language models or shallowly combine two one-way language models for pre-training, it employs a new masked language model (MLM) to generate deep bidirectional language representations. The BERT model targets a large unlabeled corpus to train and acquire a rich semantic representation (i.e., a semantic representation of the text) of the text. Subsequently, the semantic representation of the text is fine-tuned for a specific NLP task and ultimately used for that NLP task.

[0029] Furthermore, the official BERT model uses two tasks as pre-training to learn semantic information; specifically, it introduces two core tasks into the pre-training of the BERT model: a random static mask language model training task (Masked LM) and a next sentence prediction task (Next Sentence Prediction). This application does not improve or adjust the structure of the BERT model or the training tasks themselves, so the BERT model will not be described in detail here.

[0030] In some embodiments, the step of calculating the similarity between the corresponding static word vector and dynamic word vector for each word and using the similarity calculation result as a constraint term includes the step of calculating the dot product of the static word vector and dynamic word vector for each word, using the dot product of the vector as the similarity calculation result between the static word vector and dynamic word vector, and using the similarity calculation result as a constraint term composed of static word vectors, where the static word vector and dynamic word vector have the same dimension.

[0031] Specifically, after obtaining the corresponding static and dynamic word vectors for each word, the BERT model training process is characterized by increasing the constraints (i.e., constraint terms) by calculating the vector similarity between the static and dynamic word vectors. In practical applications, preferably, the embodiment of this application can use the dot product between vectors to represent the similarity between them, with a larger dot product indicating greater similarity.

[0032] Furthermore, when using the dot product of vectors to measure similarity between static and dynamic word vectors, the dot product can be calculated using the following formula. JPEG0007891586000001.jpg31170

[0033] Here, R represents the dot product of vectors, N represents the number of words (or letters) in the sentence, i represents the position of the word (or letter) in the sentence, and Ve i Vt represents a static word vector, i This represents a dynamic word vector.

[0034] Furthermore, the embodiments described in this application are JPEG0007891586000002.jpg36170 Here, i is the position of the word or character in the sentence, generally starting from 0, and there are a total of N words or characters in the sentence. The calculated and obtained R will be used as the constraint term, and the constraint term is also called the constraint condition.

[0035] In some embodiments, the step of calculating the similarity between the corresponding static word vector and dynamic word vector for each word and using the similarity calculation result as a constraint term includes the step of calculating the cosine similarity or Manhattan distance between the static word vector and dynamic word vector for each word, using the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and dynamic word vector, and using the similarity calculation result as a constraint term.

[0036] Specifically, in the embodiments of this application, in addition to representing the similarity between vectors using the dot product of vectors, the similarity between vectors may also be represented using the cosine similarity or Manhattan distance; that is, the cosine similarity or Manhattan distance between a static word vector and a dynamic word vector may be used as a constraint term. The method for calculating the cosine similarity or Manhattan distance will not be described here, but of course, other methods for calculating the similarity between vectors, in addition to the cosine similarity or Manhattan distance, are also applicable to this application.

[0037] According to the technical solutions provided in the embodiments of this application, the semantic similarity between dynamic and static word vectors is increased by measuring the similarity between vectors using the dot product of vectors, cosine similarity, or Manhattan distance, and the resulting word vector not only fuses contextual information but also makes full use of the static meaning of the words themselves.

[0038] In some embodiments, the step of adjusting the original loss function of a natural language pre-trained model by constraint terms includes the step of adjusting the original loss function using the following equation. loss=(1-α)·suploss-α·regulation

[0039] Here, `loss` represents the adjusted loss function, `suploss` represents the original loss function, `α` represents the distribution coefficient for adjusting the model training accuracy, and `regulation` represents the constraint terms composed of static word vectors.

[0040] Specifically, after calculating and obtaining the constraint terms using static word vectors, the original loss function of the natural language pre-training model (BERT pre-training model) in the downstream natural language processing task is adjusted by the constraint terms, that is, the original loss function suploss is adjusted according to the above formula, and the adjusted loss function loss is obtained.

[0041] In actual applications, loss is the improved (i.e., adjusted) loss function, suploss is the original loss function of supervised learning (e.g., cross-entropy loss function), regulation is the constraint term composed of the static word (character) vectors mentioned above, α is the distribution coefficient for adjusting the model training accuracy, which is within the open interval of 0 to 1, and empirically, it is in the range of 0.1 to 0.2 and needs to be adjusted according to different tasks.

[0042] The above content has described the complete implementation example of the technical solution of this application in detail. Next, the training process of the natural language pre-training model of this application will be described while referring to the drawings and specific examples. Figure 2 is a schematic diagram of the calculation process of the constraint terms in the actual application scenario provided by the embodiment of this application. As shown in Figure 2, the calculation process of the constraint terms in the actual application scenario may specifically include the following content.

[0043] In a specific embodiment, for a sentence consisting of 6 original characters "CLS Longhu Group SEP", first, each word (or character) is respectively converted into the corresponding one-hot encoding, and further, the one-hot encoding is mapped into static word vectors by an embedding mapping layer (i.e., word embedding layer), that is, mapped into static word vectors Ve0 to Ve5 respectively. Then, the corresponding input vectors of each word are used as the input of a multi-layer self-attention neural network (i.e., BERT model network), and the BERT model network outputs the corresponding dynamic word vectors of each word. The corresponding dynamic word vectors of each word (or character) are denoted as Vt0 to Vt5 respectively.

[0044] The input vector for a word is obtained by adding static word vectors, paragraph embedding vectors, and position embedding vectors, all mapped to the same dimension. For example, all vectors are mapped as 768-dimensional vectors. Therefore, static word vectors Ve0~Ve5 and dynamic word vectors Vt0~Vt5 have the same dimension. Static word vectors represent the static meaning of each word, while dynamic word vectors are generated using an attention mechanism, so contextual information is integrated. Consequently, dynamic word vectors contain the dynamic meaning of each word.

[0045] Subsequently, the dot product between the static and dynamic word vectors is calculated from the static and dynamic word vectors of each word using the dot product calculation formula provided in the aforementioned embodiment. This dot product is then used as a constraint term, and the original loss function of the BERT model in supervised learning natural language processing tasks is adjusted using this constraint term. The BERT model with the adjusted loss function is then trained, thereby providing the trained BERT model with higher accuracy and generalization performance.

[0046] The technical solutions provided in the embodiments of this application have at least the following advantages:

[0047] (1) In this application, during the training of a pre-trained model using BERT, constraint terms are calculated from the static and dynamic word vectors of tokens (words), the original loss function of the BERT pre-trained model is adjusted using the constraint terms, the BERT pre-trained model with the adjusted original loss function is trained, and the training time of the BERT pre-trained model is shortened.

[0048] (2) In this application, constraints are introduced into the training process of the BERT pre-trained model to increase the similarity between the dynamic word vector and the static word vector of each word in a sentence, thereby achieving the objective of shortening the distance between the dynamic word vector and the static word vector in the semantic space.

[0049] (3) This application has a broad scope of application, as it can be applied to various pre-training models (including various improved models) similar to BERT based on a multilayer self-attention mechanism.

[0050] (4) By using the model training method provided in this application, after fine-tuning training the model in downstream tasks, the model can be made to have higher accuracy and general performance than a model in which this solution is not used.

[0051] The following is an apparatus example of this application for carrying out the method example of this application. For details not disclosed in the apparatus example of this application, please refer to the method example of this application.

[0052] Figure 3 is a schematic diagram of the natural language pre-training model training device provided in the embodiment of this application. As shown in Figure 3, the natural language pre-training model training device is A transformation module 301 is positioned to tokenize text using a dictionary from a pre-trained natural language model and convert the words in the text into their corresponding one-hot encodings. A mapping module 302 is positioned to input the corresponding one-hot encoding of the text into a word embedding layer, and to map it using the word embedding layer to obtain the corresponding static word vector for each word. An input module 303 is configured to obtain the corresponding input vector for each word by adding the corresponding static word vector, paragraph embedding vector, and position embedding vector for each word, and to use the input vector as input to a natural language pre-training model to obtain the corresponding dynamic word vector for each word. A calculation module 304 is configured to calculate the similarity between the corresponding static word vector and dynamic word vector for each word, and to use the similarity calculation result as a constraint term. The system includes an adjustment module 305 that adjusts the original loss function of a pre-trained natural language model using constraint terms, and is configured to train the pre-trained natural language model with the adjusted original loss function.

[0053] In some embodiments, the mapping module 302 in Figure 3 generates a series of corresponding one-hot encodings for the text based on the corresponding one-hot encoding of each word in the text, inputs the series of one-hot encodings into a word embedding layer, the word embedding layer maps the series of one-hot encodings, obtains the corresponding original vector representation for each word, and uses the original vector representation for each word as a static word vector.

[0054] In some embodiments, the input module 303 in Figure 3 obtains the corresponding paragraph embedding vector and position embedding vector for each word in the text, maps the static word vector, paragraph embedding vector and position embedding vector into the same dimensional space, adds the static word vector, paragraph embedding vector and position embedding vector in the same dimensional space to obtain the corresponding input vector for each word, inputs the input vector into a natural language pre-trained model, trains the natural language pre-trained model on word masking tasks and context tasks, and outputs the corresponding dynamic word vector for each word in the text.

[0055] In some embodiments, the calculation module 304 in Figure 3 includes the steps of calculating the dot product of the static word vector and the dynamic word vector for each word, using the dot product as the similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term composed of the static word vector, where the static word vector and the dynamic word vector have the same dimension.

[0056] In some embodiments, the calculation module 304 in Figure 3 calculates the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector of each word, uses the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and the dynamic word vector, and uses the similarity calculation result as a constraint term.

[0057] In some embodiments, the adjustment module 305 in Figure 3 adjusts the original loss function using the following formula. loss=(1-α)·suploss-α·regulation

[0058] Here, `loss` represents the adjusted loss function, `suploss` represents the original loss function, `α` represents the distribution coefficient for adjusting the model training accuracy, and `regulation` represents the constraint terms composed of static word vectors.

[0059] It should be understood that the numbering of each step in the above embodiment does not indicate the order of execution, and the execution order of each process should be determined by its function and inherent logic, and does not arbitrarily limit the implementation process of the embodiment of this application.

[0060] Figure 4 is a schematic diagram of the electronic device 4 provided in an embodiment of this application. As shown in Figure 4, the electronic device 4 of the embodiment comprises a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable by the processor 401. When the processor 401 executes the computer program 403, it realizes the steps in each of the above method embodiments. Alternatively, when the processor 401 executes the computer program 403, it realizes the functions of each module / unit in each of the above device embodiments.

[0061] As an example, the computer program 403 may be divided into one or more modules / units, one or more modules / units stored in memory 402 and executed by processor 401 to complete the present application. One or more modules / units may be a series of computer program command sections capable of performing specific functions, the command sections being for describing the processes by which the computer program 403 is executed on the electronic device 4.

[0062] The electronic device 4 may be a desktop computer, a notebook computer, a palmtop computer, or a cloud server. The electronic device 4 may include a processor 401 and memory 402, but is not limited to them. As those skilled in the art will understand, Figure 4 is merely an example of the electronic device 4 and is not limiting to it. It may include more or fewer components than shown, or a combination of certain components or different components. For example, the electronic device may include input / output devices, network access devices, buses, etc.

[0063] The processor 601 may be a Central Processing Unit (CPU), or it may be another general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor may be a microprocessor or any general-purpose processor, etc.

[0064] Memory 402 may be the internal storage unit of the electronic device 4, for example, the hard disk or RAM of the electronic device 4. Memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., installed in the electronic device 4. Furthermore, memory 402 may include not only the internal storage unit of the electronic device 4 but also external storage devices. Memory 402 is for storing computer programs and other programs and data required by the electronic device. Memory 402 may also be used to temporarily store output or data to be output.

[0065] Those skilled in the art will understand that, for the convenience and brevity of explanation, only the division of each functional unit and module described above has been given as an example. However, in actual application, the above functions can be completed by assigning them to different functional units and modules as needed, that is, by dividing the internal structure of the device into different functional units or modules, all or some of the functions described above can be completed. Each functional unit and module in the embodiment may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit. The integrated unit may be implemented in hardware form or in the form of a software functional unit. The specific names of each functional unit and module are for ease of distinction from one another and do not limit the scope of protection of this application. For the specific operating processes of the units and modules in the above system, refer to the corresponding processes in the embodiments of the method described above, and the explanation is omitted here.

[0066] In the above embodiments, each embodiment has its own emphasis, and for parts that are not explained or described in detail in one embodiment, you can refer to the relevant explanations in other embodiments.

[0067] Those skilled in the art will be aware that each example unit and algorithmic step described in the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software will depend on the specific application of the technical solution and the design constraints. Those skilled in the art may implement the described functions using different methods for each specific application, but such implementations should not be considered to be beyond the scope of this application.

[0068] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the embodiments of the apparatus / computer equipment described above are merely illustrative, and the division into modules or units is merely a logical functional division, and other division methods may be applied in actual implementation, for example, multiple units or components may be combined or integrated into other systems, or some features may be ignored or not performed. Also, the coupling or direct coupling or communication connection between the entities shown or discussed may be an indirect coupling or communication connection by several interfaces, devices or units, and may be in electrical, mechanical or other forms.

[0069] The units described as separating members may or may not be physically separated, and the members shown as units may or may not be physical units, that is, they may be located in one place or distributed among multiple network units. Depending on the actual needs, some or all of these units can be selected to achieve the objectives of the scheme of this embodiment.

[0070] In this application, each functional unit may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit. The integrated unit may be implemented in hardware form or in the form of a software functional unit.

[0071] The integrated module / unit may be implemented in the form of a software function unit and may be stored in a computer-readable storage medium when sold or used as an independent product. With this understanding, the present application can implement all or part of the processes in the above embodiments by directing the relevant hardware with a computer program, the computer program may be stored in a computer-readable storage medium, and the steps of each of the above embodiments can be implemented when the computer program is executed by a processor. The computer program may include computer program code, which may be in source code format, object code format, executable file format, or some intermediate format. The computer-readable storage medium may include any entity or device capable of carrying the computer program code, recording media, U disks, removable hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunications signals, and software distribution media. Furthermore, the contents contained in a computer-readable storage medium may be increased or decreased as appropriate in accordance with the requirements of legislation and patent practice within the jurisdiction. For example, in a certain jurisdiction, based on legislation and patent practice, the computer-readable storage medium may not contain electrical carrier signals or telecommunications signals.

[0072] The embodiments described above are merely for illustrating the technical solutions of the present invention and are not intended to limit them. While this application has been described in detail with reference to the embodiments described above, those skilled in the art should understand that the technical solutions described in each of the embodiments described above may still be amended, or some of their technical features may be replaced with equivalent ones. However, such amendments or replacements should not cause the substance of the corresponding technical solutions to deviate from the idea and scope of the technical solutions of each embodiment of the present invention, and should all be included within the scope of protection of the present invention.

Claims

1. A processor tokenizes text using a dictionary of a pre-trained natural language model and converts the words in the text into corresponding one-hot encodings. The process involves a processor inputting the corresponding one-hot encoding of the text into a word embedding layer, and the word embedding layer mapping it to obtain the corresponding static word vector for each word. The processor adds the corresponding static word vector, paragraph embedding vector, and position embedding vector for each word to obtain the corresponding input vector for each word, uses the input vector as input to the natural language pre-training model, and obtains the corresponding dynamic word vector for each word. The processor calculates the similarity between the corresponding static word vector and the dynamic word vector for each word, and uses the similarity calculation result as a constraint term. The processor includes the steps of adjusting the original loss function of the natural language pre-trained model according to the constraints, and training the natural language pre-trained model with the adjusted original loss function. A method for training a natural language pre-trained model, characterized by the following features.

2. The step of a processor inputting the corresponding one-hot encoding of the text into a word embedding layer and mapping it by the word embedding layer to obtain the corresponding static word vector for each word is: The process includes the steps of: the processor generating a corresponding set of one-hot encodings for the text based on the corresponding one-hot encoding of each word in the text; inputting the set of one-hot encodings into the word embedding layer; the word embedding layer mapping the set of one-hot encodings; obtaining the corresponding original vector representation for each word; and converting the original vector representation for each word into a static word vector. The method according to feature 1.

3. The step of the processor adding the corresponding static word vector, paragraph embedding vector and position embedding vector for each word to obtain the corresponding input vector for each word, using the input vector as input to the natural language pre-training model to obtain the corresponding dynamic word vector for each word is: The processor obtains the corresponding paragraph embedding vector and position embedding vector for each word in the text, maps the static word vector, paragraph embedding vector and position embedding vector into the same dimensional space, adds the static word vector, paragraph embedding vector and position embedding vector in the same dimensional space, and obtains the corresponding input vector for each word. The processor includes the steps of inputting the input vector into the natural language pre-training model, training the natural language pre-training model on a context task which is a word masking task and a next sentence prediction task, and outputting a corresponding dynamic word vector for each word in the text. The method according to feature 1.

4. The step of the processor calculating the similarity between the corresponding static word vector and the dynamic word vector for each word, and using the similarity calculation result as a constraint term, The process includes the steps of the processor calculating the dot product of the static word vector and the dynamic word vector for each word, using the dot product as the similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term composed of the static word vector, wherein the static word vector and the dynamic word vector have the same dimension. The method according to feature 1.

5. The step of the processor calculating the similarity between the corresponding static word vector and the dynamic word vector for each word, and using the similarity calculation result as a constraint term, The processor includes the steps of calculating the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector for each word, using the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term. The method according to feature 1.

6. The step of the processor adjusting the original loss function of the natural language pre-trained model by the constraint term and training the natural language pre-trained model with the adjusted original loss function includes the step of the processor adjusting the original loss function using the following formula: loss=(1-α)・suploss-α・regulation Here, `loss` represents the adjusted loss function, `suploss` represents the original loss function, `α` represents the distribution coefficient for adjusting the model training accuracy, and `regulation` represents the constraint term composed of the static word vectors. The method according to feature 4.

7. As the aforementioned natural language pre-training model, the BERT-based self-attention pre-training model is used. The method according to feature 1.

8. A transformation module is configured to tokenize text using a dictionary of a pre-trained natural language model and to convert the words in the text into their corresponding one-hot encodings. A mapping module is configured to input the corresponding one-hot encoding of the aforementioned text into a word embedding layer, and to map it using the word embedding layer to obtain the corresponding static word vector for each word. An input module is configured to add the corresponding static word vector, paragraph embedding vector, and position embedding vector for each word to obtain the corresponding input vector for each word, use the input vector as input to the natural language pre-training model, and obtain the corresponding dynamic word vector for each word. A calculation module is configured to calculate the similarity between the corresponding static word vector and the dynamic word vector for each word, and to use the similarity calculation result as a constraint term. The system includes an adjustment module that adjusts the original loss function of the natural language pre-trained model according to the constraints and is configured to train the natural language pre-trained model with the adjusted original loss function. A natural language pre-training model training device characterized by the following features.

9. An electronic device comprising memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the electronic device realizes the method according to claim 1 when the processor executes the computer program.

10. A computer-readable storage medium in which a computer program is stored, characterized in that the method according to claim 1 is implemented when the computer program is executed by a processor.

Images