Natural language pre-training model training method and device, equipment and storage medium

Abstract

The application provides a natural language pre-training model training method and device, equipment and a storage medium. The method comprises the following steps: performing word segmentation on a text by using a dictionary and converting words into one-hot encoding; inputting the one-hot encoding into a word embedding layer to obtain a static word vector corresponding to each word by using the word embedding layer; adding the static word vector corresponding to each word, a paragraph embedding vector and a position embedding vector to obtain an input vector of each word; taking the input vector as an input of a natural language pre-training model to obtain a dynamic word vector corresponding to each word; calculating a similarity between the static word vector corresponding to each word and the dynamic word vector; taking the similarity calculation result as a constraint term; adjusting an original loss function of the natural language pre-training model by using the constraint term, and training the natural language pre-training model after adjusting the original loss function. The application improves the training effect of the natural language pre-training model, and makes the model obtain better precision and generalization performance.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method, apparatus, device and storage medium for training a natural language pre-training model. Background Technology

[0002] Current mainstream self-attention pre-trained models based on the BERT (Bidirectional Encoder Representation from Transformers) architecture obtain word vectors that take contextual relationships into account by randomly masking words in the input text and then having the model predict the masked words. Most current BERT-based improved pre-trained models enhance performance by increasing the corpus size and scaling up the model.

[0003] In the training process of natural language pre-training models, the meaning of a word varies in different contexts. However, the meaning of a word in different contexts is derived from its original meaning. Therefore, the meaning of a word in a certain context is usually inferred from its original meaning. However, current BERT-based pre-training models do not fully consider the impact of the original meaning of the word on the word vectors obtained after training. Failure to fully consider the original meaning of the word (static meaning) may not only increase the training time of the model but also reduce the accuracy of the model.

[0004] Given the problems existing in the current technology, there is an urgent need to provide a natural language pre-training model training scheme that can take into account both the contextual meaning of words and the meaning of the words themselves, thereby improving the training effect of natural language pre-training models and enabling the models to obtain better accuracy and generalization performance. Summary of the Invention

[0005] In view of this, embodiments of this application provide a natural language pre-training model training method, apparatus, device, and storage medium to solve the problem that the prior art fails to fully consider the meaning of the words themselves, thus reducing the training effect of the natural language pre-training model and preventing the model from obtaining better accuracy and generalization performance.

[0006] A first aspect of this application provides a method for training a natural language pre-training model, comprising: segmenting text using a dictionary of the natural language pre-training model and converting the words in the text into corresponding one-hot codes; inputting the one-hot codes corresponding to the text into a word embedding layer and using the word embedding layer to map and obtain static word vectors corresponding to each word; adding the static word vectors, paragraph embedding vectors, and position embedding vectors corresponding to each word to obtain an input vector corresponding to each word, using the input vectors as input to the natural language pre-training model to obtain dynamic word vectors corresponding to each word; calculating the similarity between the static word vectors and dynamic word vectors corresponding to each word, and using the similarity calculation result as a constraint term; adjusting the original loss function of the natural language pre-training model using the constraint term, and training the natural language pre-training model after adjusting the original loss function.

[0007] A second aspect of this application provides a natural language pre-training model training apparatus, comprising: a conversion module configured to segment text using a dictionary of the natural language pre-training model and convert words in the text into corresponding one-hot codes; a mapping module configured to input the one-hot codes corresponding to the text into a word embedding layer and map the text using the word embedding layer to obtain static word vectors corresponding to each word; an input module configured to add the static word vectors, paragraph embedding vectors, and position embedding vectors corresponding to each word to obtain an input vector corresponding to each word, and use the input vectors as input to the natural language pre-training model to obtain dynamic word vectors corresponding to each word; a calculation module configured to calculate the similarity between the static word vectors and dynamic word vectors corresponding to each word and use the similarity calculation result as a constraint term; and an adjustment module configured to adjust the original loss function of the natural language pre-training model using the constraint term and train the natural language pre-training model after adjusting the original loss function.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0011] This paper utilizes the dictionary of a pre-trained natural language processing (NLP) model to segment text and convert each word into a corresponding one-hot encoding. The one-hot encoding is then input into a word embedding layer, which maps the results to obtain static word vectors for each word. The static word vectors, paragraph embedding vectors, and positional embedding vectors for each word are summed to obtain the input vector for each word. This input vector is then used as the input to the NLP pre-training model to obtain dynamic word vectors for each word. The similarity between the static and dynamic word vectors for each word is calculated, and the similarity result is used as a constraint. This constraint is then used to adjust the original loss function of the NLP pre-training model, which is then trained. This application considers both the contextual meaning and the intrinsic meaning of words, thereby improving the training effect of the NLP pre-training model and enabling it to achieve better accuracy and generalization performance. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the natural language pre-training model training method provided in this application embodiment;

[0014] Figure 2 This is a schematic diagram illustrating the calculation process of constraint terms in a real-world application scenario provided in the embodiments of this application;

[0015] Figure 3 This is a schematic diagram of the structure of the natural language pre-training model training device provided in the embodiments of this application;

[0016] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0018] In recent years, with the continuous development of artificial intelligence and natural language processing technologies, pre-trained natural language models have been widely applied to various fields to solve real-world natural language processing tasks, such as text classification and speech recognition. Current mainstream self-attention pre-trained models based on the BERT (Bidirectional Encoder Representation from Transformers) structure incorporate contextual relationships into the word vectors by randomly masking words in the input text and then having the model predict the masked words. Most current BERT-based improved pre-trained models enhance performance by increasing the corpus and expanding the model size.

[0019] In the current field of natural language processing, mainstream BERT-based pre-trained models obtain dynamic word vectors by training a word within its context. While this approach considers the different meanings of a word in different contexts, it pays less attention to the inherent meaning of the word itself. In natural language, the meaning of a word varies across different contexts, but these meanings are derived from the word's original meaning. Therefore, the meaning of a word in a given context is usually inferred from its original meaning. However, current BERT-based pre-trained models do not fully consider the impact of the word's original meaning on the resulting word vectors. This lack of consideration for the word's original meaning (static meaning) may not only increase training time but also reduce model accuracy. Therefore, existing training methods for natural language pre-training models suffer from long training times, poor training results, and low model accuracy and generalization performance.

[0020] In view of the problems existing in the prior art, this application provides an improved method for training natural language pre-training models. Before training the natural language pre-training model, this application first obtains the static word vector and dynamic word vector corresponding to each word. By calculating the similarity between the dynamic word vector obtained considering the context and the static word vector of the word itself, the two types of word vectors are brought closer together in semantic space. The similarity calculation result is used as a constraint to adjust the original loss function of the natural language pre-training model, and the natural language pre-training model with the adjusted original loss function is then trained. This allows the trained model to fully consider the meaning of the word itself while considering the context, thereby improving the performance of the natural language pre-training model and giving it better accuracy and generalization performance.

[0021] Figure 1 This is a flowchart illustrating the natural language pre-training model training method provided in this application embodiment. Figure 1 The training method for natural language pre-trained models can be performed by the server. For example... Figure 1As shown, the training method for this natural language pre-training model can specifically include:

[0022] S101 uses the dictionary of a pre-trained natural language model to segment the text and converts the words in the text into corresponding one-hot codes;

[0023] S102, input the one-hot encoding of the text into the word embedding layer, and use the word embedding layer to map and obtain the static word vector corresponding to each word;

[0024] S103, add the static word vector, paragraph embedding vector and position embedding vector corresponding to each word to obtain the input vector corresponding to each word, and use the input vector as the input to the natural language pre-training model to obtain the dynamic word vector corresponding to each word;

[0025] S104, calculate the similarity between the static word vector and the dynamic word vector corresponding to each word, and use the similarity calculation result as a constraint term;

[0026] S105, adjust the original loss function of the natural language pre-training model using constraint terms, and train the natural language pre-training model with the adjusted original loss function.

[0027] Specifically, the one-hot encoding in this application embodiment, also known as one-bit valid encoding, works by using an N-bit state register to encode N states. Each state has its own independent register bit, and at any given time, only one bit is valid. This application embodiment converts each word in the text into a corresponding one-hot code, thus the entire text corresponds to a series of one-hot codes (arranged according to the word order).

[0028] Furthermore, in this embodiment of the application, the different word vectors obtained by a word in different contexts are called dynamic word vectors, and the word vectors obtained without considering the context of the word are called static word vectors. Among them, dynamic word vectors can represent the meaning of a word in different contexts, while static word vectors can represent the meaning of the word itself.

[0029] It should be noted that the following embodiments of this application use a BERT-based self-attention pre-trained model (hereinafter referred to as the BERT pre-trained model or BERT model) as an example for detailed description of the natural language pre-training model. However, it should be understood that the natural language pre-training model in the embodiments of this application is not limited to the BERT pre-trained model. Any model that can be applied in natural language processing tasks is applicable to this application. The type of natural language pre-training model does not constitute a limitation on the technical solution of this application.

[0030] In some embodiments, the one-hot encoding corresponding to the text is input into the word embedding layer, and the static word vector corresponding to each word is obtained by mapping using the word embedding layer. This includes: generating a series of one-hot encodings corresponding to the text based on the one-hot encodings corresponding to each word in the text, inputting the series of one-hot encodings into the word embedding layer, mapping the series of one-hot encodings using the word embedding layer to obtain the original vector representation corresponding to each word, and using the original vector representation of each word as the static word vector.

[0031] Specifically, before using the word embedding layer to map and obtain the static word vector corresponding to each word, the input text is first segmented according to the dictionary of the natural language pre-trained model (BERT pre-trained model), and then converted into one-hot encoding corresponding to each word by the vocabulary of the BERT pre-trained model.

[0032] Furthermore, after obtaining the one-hot encoding for each word, a series of one-hot encodings for the text are generated based on the one-hot encoding of each word and the order of each word in the text. This series of one-hot encodings is then input into the word embedding layer of the BERT pre-trained model. After mapping, the original vector representation corresponding to each word is obtained, i.e., the static word vector corresponding to each word. The static word vector can express the meaning of the word itself.

[0033] In some embodiments, the static word vector, paragraph embedding vector, and position embedding vector corresponding to each word are added together to obtain the input vector corresponding to each word. The input vector is then used as the input to a natural language pre-training model to obtain the dynamic word vector corresponding to each word. This includes: obtaining the paragraph embedding vector and position embedding vector corresponding to each word in the text; mapping the static word vector, paragraph embedding vector, and position embedding vector to the same dimensional space; adding the static word vector, paragraph embedding vector, and position embedding vector in the same dimensional space to obtain the input vector corresponding to each word; inputting the input vector into the natural language pre-training model; using the natural language pre-training model to train word masking and context tasks; and outputting the dynamic word vector corresponding to each word in the text.

[0034] Specifically, after obtaining the static word vector corresponding to each word by mapping using the word embedding layer, the static word vector, segment embedding vector, and position embedding vector of each word are mapped to the same dimensional space. For example, each vector is mapped to a 768-dimensional space, that is, each vector is mapped to a 768-dimensional vector. Then, the static word vector, segment embedding vector, and position embedding vector in the same dimension are added together (i.e., vector addition) to obtain the input vector corresponding to each word.

[0035] Furthermore, the input vector is fed into the BERT pre-trained model, which is then used to train word masking and sentence context tasks. Finally, the BERT pre-trained model is used to output the dynamic word vector corresponding to each word in the text.

[0036] BERT (Bidirectional Encoder Representations from Transformer) is a pre-trained language representation model based on Transformer bidirectional encoder representations. It emphasizes a departure from traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training. Instead, it employs a novel masked language model (MLM) to generate deep bidirectional language representations. The goal of BERT is to train on large-scale unlabeled corpora to obtain representations of text containing rich semantic information (i.e., semantic representations of the text), then fine-tune these semantic representations for specific NLP tasks before finally applying them to those tasks.

[0037] Furthermore, to learn semantic information, the official BERT model uses two tasks for pre-training: the two core tasks introduced in the pre-training of the BERT model are: the language model training task with random static masks (Masked LM) and the next sentence prediction task (Next Sentence Prediction). Since this application does not improve or adjust the structure of the BERT model or the training tasks themselves, the BERT model will not be discussed in detail here.

[0038] In some embodiments, the similarity between the static word vector and the dynamic word vector corresponding to each word is calculated, and the similarity calculation result is used as a constraint term. This includes: calculating the vector dot product between the static word vector and the dynamic word vector of each word, using the vector dot product as the similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term constructed based on the static word vector; wherein the static word vector and the dynamic word vector have the same dimension.

[0039] Specifically, after obtaining the static and dynamic word vectors for each word, the constraints (i.e., constraint terms) used to enhance the BERT model training process are determined by calculating the vector similarity between the static and dynamic word vectors. In practical applications, preferably, the similarity between vectors can be represented by the inner product of vectors, with a larger inner product indicating greater similarity.

[0040] Furthermore, when using the vector dot product to measure the similarity between static and dynamic word vectors, the vector dot product can be calculated using the following formula:

[0041]

[0042] Where R represents the vector inner product, N represents the number of words (or characters) in the sentence, i represents the position of the word (or character) in the sentence, and Ve i Vt represents static word vectors. i This represents dynamic word vectors.

[0043] It should be noted that in this embodiment of the application, static word vectors (or static character vectors) are denoted as Ve. i Where i represents the position of the word or character in the sentence, generally starting from 0; the dynamic vector corresponding to the word or character obtained after mapping through a multi-layer self-attention neural network (BERT model network) is denoted as Vt. i , where i is the position of the word or character in the sentence, usually starting from 0, and there are N words or characters in the sentence. The calculated R is used as the subsequent constraint term, also known as the constraint condition.

[0044] In some embodiments, the similarity between the static word vector and the dynamic word vector corresponding to each word is calculated, and the similarity calculation result is used as a constraint term. This includes: calculating the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector of each word, using the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and the dynamic word vector, and using the similarity calculation result as a constraint term.

[0045] Specifically, in addition to using the vector dot product to represent the similarity between vectors, this application embodiment can also use cosine similarity or Manhattan distance to represent the similarity between vectors, that is, the cosine similarity or Manhattan distance between static word vectors and dynamic word vectors as a constraint term. The calculation method of cosine similarity or Manhattan distance will not be described here. Of course, in addition to cosine similarity or Manhattan distance, other methods for calculating the similarity between vectors are also applicable to this application.

[0046] According to the technical solution provided in the embodiments of this application, the embodiments of this application use vector inner product, cosine similarity or Manhattan distance to measure the similarity between vectors, thereby narrowing the similarity between dynamic word vectors and static word vectors in semantic space, so that the final word vectors not only integrate the contextual information of the context, but also fully refer to the static meaning of the word itself.

[0047] In some embodiments, the original loss function of the natural language pre-trained model is adjusted using constraint terms, including adjusting the original loss function using the following formula:

[0048] loss=(1-α)·suploss-α·regulation

[0049] Where loss represents the adjusted loss function, suploss represents the original loss function, α represents the partition coefficient used to adjust the model training accuracy, and regulation represents the constraint term constructed based on static word vectors.

[0050] Specifically, after calculating the constraint terms based on static word vectors, the original loss function of the natural language pre-trained model (BERT pre-trained model) in the downstream natural language processing task is adjusted using the constraint terms. That is, the original loss function suploss is adjusted using the above formula to obtain the adjusted loss function loss.

[0051] In practical applications, loss is the modified (i.e., adjusted) loss function, suploss is the original supervised learning loss function (such as the cross-entropy loss function), regulation is the constraint term constructed based on static word (character) vectors mentioned earlier, and α is the partition coefficient, used to adjust the model training accuracy. It is in the open interval of 0 to 1, and empirically can be taken between 0.1 and 0.2. This value needs to be adjusted according to different tasks.

[0052] The above content provides a detailed description of the complete embodiments of the technical solution of this application. The training process of the natural language pre-training model of this application is described below with reference to the accompanying drawings and specific embodiments. Figure 2 This is a schematic diagram illustrating the calculation process of constraint terms in a real-world application scenario provided in the embodiments of this application, such as... Figure 2 As shown, the calculation process of the constraint terms in this practical application scenario may specifically include:

[0053] In one specific embodiment, assuming that for a sentence composed of six original characters “CLS Longhu Group SEP”, each word (or character) is first converted into its corresponding one-hot encoding, and then the one-hot encoding is mapped to static word vectors using an embedding mapping layer (i.e., word embedding layer), that is, mapped to static word vectors from Ve0 to Ve5 respectively; then, the input vector corresponding to each word is used as the input of a multi-layer self-attention neural network (i.e., BERT model network), and the dynamic word vector corresponding to each word is output using the BERT model network, and the dynamic word vector corresponding to each word (or character) is denoted as Vt0 to Vt5 respectively.

[0054] Since the input vector of a word is obtained by adding static word vectors, paragraph embedding vectors, and position embedding vectors that are mapped to the same dimension, for example, mapping all vectors to 768-dimensional vectors, the static word vectors Ve0 to Ve5 and the dynamic word vectors Vt0 to Vt5 have the same dimension. The static word vectors represent the static meaning of each word, while the dynamic word vectors are generated using an attention mechanism, so they integrate contextual information. Therefore, the dynamic word vectors contain the dynamic meaning of each word.

[0055] Subsequently, based on the static and dynamic word vectors of each word, the vector inner product between the static and dynamic word vectors is calculated using the vector inner product calculation formula provided in the aforementioned embodiment. The vector inner product is used as a constraint term. The original loss function of the BERT model in the supervised learning natural language processing task is adjusted using the constraint term, and the BERT model with the adjusted loss function is trained, thereby enabling the trained BERT model to obtain better accuracy and generalization performance.

[0056] According to the technical solutions provided in the embodiments of this application, the embodiments of this application have at least the following advantages:

[0057] (1) This application proposes to calculate constraint terms based on static word vectors and dynamic word vectors of tokens during the training of BERT-based pre-trained models, adjust the original loss function of the BERT pre-trained model using the constraint terms, and train the BERT pre-trained model with the adjusted original loss function, thereby shortening the training time of the BERT pre-trained model.

[0058] (2) This application improves the similarity between the dynamic word vectors and static word vectors of each word in a sentence by introducing constraint terms during the training process of the BERT pre-trained model, thereby narrowing the distance between the dynamic word vectors and static word vectors in the semantic space.

[0059] (3) This application can be applied to various pre-trained models based on multi-layer self-attention mechanism similar to BERT (including various improved models), and has a wide range of applications;

[0060] (4) By using the model training method provided in this application, after fine-tuning the model in the downstream task, the model can achieve better accuracy and generalization performance than the model that does not use this scheme.

[0061] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0062] Figure 3 This is a schematic diagram of the structure of the natural language pre-training model training device provided in the embodiments of this application.

[0063] like Figure 3 As shown, the training device for this natural language pre-training model includes:

[0064] The conversion module 301 is configured to segment the text using the dictionary of the natural language pre-trained model and convert the words in the text into corresponding one-hot codes;

[0065] The mapping module 302 is configured to input the one-hot encoding of the text into the word embedding layer and use the word embedding layer to map and obtain the static word vector corresponding to each word.

[0066] The input module 303 is configured to add the static word vector, paragraph embedding vector and position embedding vector corresponding to each word to obtain the input vector corresponding to each word, and use the input vector as the input to the natural language pre-training model to obtain the dynamic word vector corresponding to each word.

[0067] The calculation module 304 is configured to calculate the similarity between the static word vector and the dynamic word vector corresponding to each word, and use the similarity calculation result as a constraint term.

[0068] The adjustment module 305 is configured to adjust the original loss function of the natural language pre-trained model using constraint terms, and then train the natural language pre-trained model with the adjusted original loss function.

[0069] In some embodiments, Figure 3 The mapping module 302 generates a series of one-hot codes for each word in the text based on the one-hot code corresponding to each word. The series of one-hot codes are input into the word embedding layer. The word embedding layer is used to map the series of one-hot codes to obtain the original vector representation of each word. The original vector representation of each word is used as a static word vector.

[0070] In some embodiments, Figure 3 The input module 303 obtains the paragraph embedding vector and position embedding vector corresponding to each word in the text, maps the static word vector, paragraph embedding vector and position embedding vector to the same dimension space, and adds the static word vector, paragraph embedding vector and position embedding vector in the same dimension space to obtain the input vector corresponding to each word; the input vector is input into the natural language pre-training model, and the natural language pre-training model is used to train the word masking task and the sentence context task, and outputs the dynamic word vector corresponding to each word in the text.

[0071] In some embodiments, Figure 3The calculation module 304 calculates the dot product between the static word vector and the dynamic word vector for each word, and uses the dot product as the similarity calculation result between the static word vector and the dynamic word vector, and uses the similarity calculation result as the constraint term constructed based on the static word vector; wherein, the static word vector and the dynamic word vector have the same dimension.

[0072] In some embodiments, Figure 3 The calculation module 304 calculates the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector of each word, uses the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and the dynamic word vector, and uses the similarity calculation result as a constraint term.

[0073] In some embodiments, Figure 3 The adjustment module 305 adjusts the original loss function using the following formula:

[0074] loss=(1-α)·suploss-α·regulation

[0075] Where loss represents the adjusted loss function, suploss represents the original loss function, α represents the partition coefficient used to adjust the model training accuracy, and regulation represents the constraint term constructed based on static word vectors.

[0076] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0077] Figure 4 This is a schematic diagram of the structure of the electronic device 4 provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 of this embodiment includes a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module / unit in the various device embodiments described above.

[0078] For example, computer program 403 may be divided into one or more modules / units, which are stored in memory 402 and executed by processor 401 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 403 in electronic device 4.

[0079] Electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 4 may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0080] Processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0081] The memory 402 can be an internal storage unit of the electronic device 4, such as a hard disk or RAM. The memory 402 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 402 can include both internal and external storage units of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device. The memory 402 can also be used to temporarily store data that has been output or will be output.

[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0083] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0084] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0085] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0089] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for training a natural language pre-training model, characterized in that, include: The text is segmented using a dictionary from a pre-trained natural language model, and the words in the text are converted into corresponding one-hot codes. The one-hot encoding corresponding to the text is input into the word embedding layer, and the static word vector corresponding to each word is obtained by mapping using the word embedding layer. The static word vector, paragraph embedding vector, and position embedding vector corresponding to each word are added together to obtain the input vector corresponding to each word. The input vector is used as the input to the natural language pre-training model to obtain the dynamic word vector corresponding to each word. Calculate the similarity between the static word vector and the dynamic word vector corresponding to each word, and use the similarity calculation result as a constraint term; The original loss function of the natural language pre-training model is adjusted using the aforementioned constraint terms, and the natural language pre-training model with the adjusted original loss function is then trained.

2. The method according to claim 1, characterized in that, The step of inputting the one-hot encoding corresponding to the text into the word embedding layer and using the word embedding layer to map and obtain the static word vector corresponding to each word includes: Based on the one-hot encoding corresponding to each word in the text, a series of one-hot encodings corresponding to the text are generated. The series of one-hot encodings are input into the word embedding layer. The word embedding layer is used to map the series of one-hot encodings to obtain the original vector representation corresponding to each word. The original vector representation of each word is used as a static word vector.

3. The method according to claim 1, characterized in that, The step of adding the static word vector, paragraph embedding vector, and position embedding vector corresponding to each word to obtain the input vector corresponding to each word, and using the input vector as the input to the natural language pre-training model to obtain the dynamic word vector corresponding to each word, includes: Obtain the paragraph embedding vector and position embedding vector corresponding to each word in the text, map the static word vector, the paragraph embedding vector and the position embedding vector to the same dimension space, and add the static word vector, the paragraph embedding vector and the position embedding vector in the same dimension space to obtain the input vector corresponding to each word; The input vector is fed into the natural language pre-training model, which is then used to train word masking and sentence context tasks, and outputs dynamic word vectors corresponding to each word in the text.

4. The method according to claim 1, characterized in that, The step of calculating the similarity between the static word vector and the dynamic word vector corresponding to each word, and using the similarity calculation result as a constraint term, includes: Calculate the dot product between the static word vector and the dynamic word vector for each word, and use the dot product as the similarity calculation result between the static word vector and the dynamic word vector. Use the similarity calculation result as a constraint term constructed based on the static word vector; wherein the static word vector and the dynamic word vector have the same dimension.

5. The method according to claim 1, characterized in that, The step of calculating the similarity between the static word vector and the dynamic word vector corresponding to each word, and using the similarity calculation result as a constraint term, includes: Calculate the cosine similarity or Manhattan distance between the static word vector and the dynamic word vector for each word, and use the cosine similarity or Manhattan distance as the similarity calculation result between the static word vector and the dynamic word vector, and use the similarity calculation result as a constraint term.

6. The method according to claim 4, characterized in that, The adjustment of the original loss function of the natural language pre-trained model using the constraint term includes adjusting the original loss function using the following formula: loss=(1-α)·suploss-α·regulation Where loss represents the adjusted loss function, suploss represents the original loss function, α represents the partition coefficient used to adjust the model training accuracy, and regulation represents the constraint term constructed based on the static word vectors.

7. The method according to any one of claims 1 to 6, characterized in that, The natural language pre-training model adopts a BERT-based self-attention pre-training model.

8. A natural language pre-training model training device, characterized in that, include: The conversion module is configured to segment the text using a dictionary from a pre-trained natural language model and convert the words in the text into corresponding one-hot codes. The mapping module is configured to input the one-hot encoding corresponding to the text into the word embedding layer, and use the word embedding layer to map and obtain the static word vector corresponding to each word. The input module is configured to add the static word vector, paragraph embedding vector and position embedding vector corresponding to each word to obtain the input vector corresponding to each word, and use the input vector as the input of the natural language pre-training model to obtain the dynamic word vector corresponding to each word; The calculation module is configured to calculate the similarity between the static word vector and the dynamic word vector corresponding to each word, and use the similarity calculation result as a constraint term. The adjustment module is configured to adjust the original loss function of the natural language pre-trained model using the constraint terms, and to train the natural language pre-trained model with the adjusted original loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images