A Bert model pre-training method, device, system and storage medium

Abstract

The application discloses a Bert model pre-training method and device, a system and a storage medium, and the method comprises the following steps: generating a word / character correspondence table by randomly combining words / characters in a word / character library of an initial Bert model; mapping three original data sets to generate a general text data set, a domain text data set and a domain sensitive text data set according to the word / character correspondence table; and training the initial Bert model by using the general text data, the domain text data and the sensitive text data in sequence to obtain a sensitive data Bert model. According to the application, any element in the word / character library table of the Bert model is mapped to another element in the word / character library table, and then the three types of data subjected to disorder processing are used to train the Bert model, so that the problems of the Bert model training, such as the fact that the classified document cannot directly participate in the model development and the fact that the sample data is too small, are solved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, system, and storage medium for pre-training a BERT model based on domain-sensitive text data. Background Technology

[0002] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. Its research focuses on various theories and methods for enabling effective communication between humans and computers using natural language. It aims to extract information from textual data, allowing computers to process or "understand" natural language to perform tasks such as automatic translation, text classification, and sentiment analysis.

[0003] The BERT model (Bidirectional Encoder Representations from Transformers) is a language representation model. Its main structure consists of stacked encoder layers from transformer models, employing a two-stage framework: pre-training and fine-tuning for specific tasks. It emphasizes a departure from traditional unidirectional language models or shallow concatenation of two unidirectional models for pre-training. Instead, it utilizes a novel MLM (masked language model) to generate deep bidirectional language representations. The main advantages of this model are: 1) Pre-training bidirectional Transformers using MLM to generate deep bidirectional language representations. 2) After pre-training, only an additional output layer needs to be added for fine-tuning, achieving state-of-the-art performance across a wide range of downstream tasks. This process does not require task-specific structural modifications to the BERT model.

[0004] Pre-training using BERT models is a popular solution for training NLP language models. It improves upon general language models trained on unlabeled text to perform specific tasks. The idea is that the model's parameters are no longer randomly initialized, but rather trained on a task to obtain a set of model parameters. These parameters are then used to initialize the model before training to obtain better predictive insights.

[0005] In some fields, such as banking and government agencies, there is a lot of classified data. To process this data, it is necessary to first de-identify it. However, the text information is distorted after de-identification. Fine-tuning directly on the existing data is not effective. Moreover, data within the field is often insufficient to meet the needs of BERT model pre-training. Summary of the Invention

[0006] This invention provides a method, apparatus, system, and storage medium for pre-training a BERT model based on domain-sensitive text data, in order to at least address the aforementioned shortcomings of existing technologies in processing sensitive text data.

[0007] According to a first aspect of the present invention, a BERT model pre-training method is provided, comprising: randomly combining words / characters in a word / character database of an initial BERT model to generate a word / character lookup table; mapping a general text dataset, a domain text dataset, and a domain-sensitive text dataset to the word / character lookup table to generate a general text dataset, a domain text dataset, and a domain-sensitive text dataset, respectively; training the initial BERT model using general text data from the general text dataset to obtain a general data BERT model; training the general data BERT model using domain text data from the domain text dataset to obtain a domain data BERT model; and training the domain data BERT model using sensitive text data from the domain-sensitive text dataset to obtain a sensitive data BERT model.

[0008] According to a second aspect of the present invention, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs the method described in any of the above embodiments.

[0009] According to a third aspect of the present invention, a BERT model pre-training apparatus is also provided, comprising: a lookup table generation module, configured to randomly combine words / characters in the word / character database of an initial BERT model to generate a word / character lookup table; a dataset mapping module, configured to map a general text dataset, a domain text dataset, and a domain sensitive text dataset to the general text dataset, a domain text dataset, and a domain sensitive text dataset respectively, according to the word / character lookup table; a general text data training module, configured to train the initial BERT model using general text data in the general text dataset to obtain a general data BERT model; a domain text data training module, configured to train the general data BERT model using domain text data in the domain text dataset to obtain a domain data BERT model; and a sensitive text data training module, configured to train the domain data BERT model using sensitive text data in the domain sensitive text dataset to obtain a sensitive data BERT model.

[0010] According to a fourth aspect of the present invention, a BERT model pre-training system is also provided, comprising: a processor; and a memory connected to the processor, the memory storing a program in which the processor executes any of the methods provided in the first aspect of the present invention when the program is executed.

[0011] The above-mentioned technical solution of the present invention maps any element in the BERT model word / character database to another element in the word / character database that is not the element itself, for the purpose of disordering text data. Then, the BERT model is pre-trained using three types of disordered data, which solves the problems of BERT model training based on sensitive information, such as classified documents not being able to directly participate in model development and the small amount of sample data. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0013] Figure 1 This is a hardware structure block diagram of a computing device used to implement the Bert model pre-training method according to Embodiment 1 of the present invention;

[0014] Figure 2 This is a flowchart illustrating the BERT model pre-training method according to Embodiment 1 of the present invention;

[0015] Figure 3 This is a schematic diagram of the Bert model pre-training device according to Embodiment 2 of the present invention;

[0016] Figure 4 This is a schematic diagram of the Bert model pre-training system according to Embodiment 3 of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Example 1

[0020] According to this embodiment, an embodiment of a BERT model pre-training method is provided.

[0021] The method embodiments provided in this example can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Figure 1 A hardware block diagram of a computing device for implementing the BERT model pre-training method provided in this embodiment is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), memory for storing data, and transmission devices for communication functions. In addition, it may also include: a display, input / output interfaces (I / O interfaces), a universal serial bus (USB) port (which may be included as one of the ports in the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0022] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within the computing device.

[0023] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the BERT model pre-training method in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the domain-sensitive data processing method of the aforementioned application. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0024] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computing device's communications provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0025] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of the computing device.

[0026] It should be noted here that, in some optional embodiments, the above... Figure 1 The computing device shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computing devices.

[0027] Under the above operating environment, this embodiment provides a method for pre-training a BERT model. Figure 2 A flowchart of the method is shown. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0028] The method provided in this embodiment takes text classification as an example, and its logic is as follows:

[0029] Using the BERT model's word / character database, words and characters are randomly combined to generate a word / character lookup table. Referring to this table, three new datasets are generated: a general text dataset, a domain text dataset, and a domain-sensitive text dataset. The general text data from the general text dataset is used to train the BERT model corresponding to the aforementioned word / character database, resulting in a general data BERT model. Next, the domain text data from the domain text dataset is used to train the general data BERT model, resulting in a domain data BERT model. Finally, the sensitive text data from the domain-sensitive text dataset is used to train the domain data BERT model, resulting in a sensitive data BERT model. The obtained sensitive data BERT model can then be integrated into subsequent tasks. For example, in this embodiment, the sensitive data BERT model is trained using a fully connected layer and SoftMax to obtain a sensitive text classification model.

[0030] refer to Figure 2 As shown, the specific steps of the method provided in this embodiment are as follows:

[0031] S201: Randomly combine the words / characters in the initial BERT model's word / character database to generate a word / character lookup table.

[0032] BERT models, such as Bert-base-cased and Bert-base-chinese, each have a vocabulary table or character set. In this invention, the vocabulary / character set is the vocabulary table or character set, and the generated word / character lookup table is the word lookup table or character lookup table. The word / character lookup table is generated, for example, by mapping any element in the vocabulary / character set to another element in the vocabulary / character set that is not the element itself. Assuming the size of the vocabulary / character set is N, there can be N factorial mapping schemes:

[0033]

[0034] For example, the number of elements in the bert-base-cased word / character library is 28,996, and the number of elements in the bert-base-chinese word / character library is 21,128. It can be seen that the number of mapping schemes of random combinations is close to infinity, and sensitive information can be well shielded through this scrambling method.

[0035] Taking the Bert-base-chinese model as an example, it corresponds to a character library with a length of 21,128, and a look-up table is mapped using a random strategy. 21,128 characters will generate 21,128 pairs of character mapping relationships. According to the mapping relationships in the look-up table, each character has a corresponding character. Suppose a sample data is "The sky is cyan waiting for the misty rain", and here we can make the following assumptions:

[0036] sky -> earth, cyan -> white, color -> gold, mist -> fog, rain -> wind

[0037] Then the original sentence after scrambling is: earth white gold fog wind.

[0038] In the present invention: The Bert model not only refers to the basic Bert model, but also includes adjusted models such as the ALBert model, the RoBerta model, and the TinyBert model. These models can be collectively referred to as the initial Bert model, and the word / character library table can be a subset of the word / character library corresponding to the Bert model.

[0039] S202: According to the above-mentioned word / character look-up table, the general text original dataset, the domain text original dataset, and the domain sensitive text original dataset are respectively mapped to generate the general text dataset, the domain text dataset, and the domain sensitive text dataset.

[0040] In this execution, according to the previously generated word / character look-up table, three pre-obtained datasets, namely the general text original dataset, the domain text original dataset, and the domain sensitive text original dataset, are mapped to generate three scrambled new datasets, namely the general text dataset, the domain text dataset, and the domain sensitive text dataset. As the name implies, among them:

[0041] The data in the general text original dataset is not limited to the domain and task. In this embodiment, it is not limited to the classification task, and thus a large amount of text can be obtained.

[0042] The data in the domain text original dataset is the data within the domain related to the data in the domain sensitive text original dataset, not limited to the task. In this embodiment, it is not limited to the classification task, and may include or not include the data in the domain sensitive text original dataset.

[0043] In this embodiment, the data in the domain sensitive text original dataset is the domain sensitive text original data corresponding to the classification task of this embodiment.

[0044] This execution completed the data transformation of three datasets. The sensitive text data was generated as plaintext data after the original sensitive text data was scrambled, thus successfully masking sensitive information. Although the new text is distorted in readability, it still implicitly contains the original text structure and order. Therefore, with subsequent pre-training of a large amount of data using the BERT model, feature extraction can still be completed for downstream classification tasks.

[0045] S203: Train the initial BERT model using general text data from the general text dataset to obtain the general data BERT model.

[0046] The initial BERT model was trained using general text data, which met the requirement of BERT model to have a large amount of text data and overcame the limitation of small amount of sensitive text data in a single task.

[0047] S204: Use the domain text data in the domain text dataset to train the general data BERT model to obtain the domain data BERT model.

[0048] Using domain-specific text data to train a general-purpose BERT model can accelerate convergence and optimize the model.

[0049] S205: Use sensitive text data from the domain sensitive text dataset to train the domain data BERT model to obtain the sensitive data BERT model.

[0050] During this process, unsupervised training can be performed on the domain data BERT model based on sensitive text data to obtain the sensitive data BERT model.

[0051] Subsequently, this sensitive data can be used in the BERT model for downstream tasks. For example, in this embodiment, it can be connected to a fully connected layer + SoftMax to obtain a text classification model. In addition, downstream tasks such as NER (Named Entity Recognition) and NSP (NextSentence Prediction) can also be performed.

[0052] In addition, refer to Figure 1 As shown, according to a second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0053] In this embodiment of the invention, any element in the BERT model word / character database is mapped to another element in the word / character database that is not the element itself, for the purpose of disordering the text data. Then, the BERT model is trained using three types of disordered data, which solves the problems of training BERT models based on sensitive information, such as classified documents not being able to directly participate in model development and the small amount of sample data.

[0054] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0056] Example 2

[0057] Figure 3 A BERT model pre-training apparatus according to this embodiment is shown, which corresponds to the method described according to the first aspect of Embodiment 1. Reference Figure 3 As shown, the device includes:

[0058] The lookup table generation module 301 is used to randomly combine words / characters in the initial BERT model's word / character database to generate a word / character lookup table. For example, the word / character lookup table can be generated by mapping any element in the word / character database to another element in the word / character database that is not the element itself.

[0059] The dataset mapping module 302 is used to map the general text dataset, the domain text dataset, and the domain-sensitive text dataset to the above-mentioned word / character lookup table to generate the general text dataset, the domain text dataset, and the domain-sensitive text dataset, respectively.

[0060] The general text data training module 303 is used to train the initial BERT model using general text data from the general text dataset to obtain the general data BERT model.

[0061] The domain text data training module 304 is used to train the general data BERT model using domain text data from the domain text dataset to obtain the domain data BERT model.

[0062] The sensitive text data training module 305 is used to train the domain data BERT model using sensitive text data from the domain sensitive text dataset to obtain the sensitive data BERT model.

[0063] Although the description of Embodiment 2 is not fully elaborated herein, those skilled in the art will understand that its specific implementation corresponds to the method in Embodiment 1, and can obtain the same technical advantages and effects.

[0064] Example 3

[0065] Figure 4 A BERT model pre-training system according to this embodiment is illustrated, which corresponds to the method described according to Embodiment 1. Reference Figure 4 As shown, the system includes:

[0066] Processor 401; and

[0067] A memory 402 connected to the processor stores a program that is executed by the processor 402 during program execution, providing any of the methods described in Embodiment 1.

[0068] Although the description of Embodiment 3 is not fully elaborated herein, those skilled in the art will understand that its specific implementation corresponds to the method in Embodiment 1, and can obtain the same technical advantages and effects.

[0069] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0070] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0074] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0075] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A BERT model pre-training method, characterized in that, include: The words / characters in the initial BERT model's word / character database are randomly combined to generate a word / character mapping table, and any element in the BERT model's word / character database table is mapped to another element in the word / character database table that is not the element itself; Based on the aforementioned word / character mapping table, the general text dataset, the domain text dataset, and the domain-sensitive text dataset are respectively mapped to generate the general text dataset, the domain text dataset, and the domain-sensitive text dataset. The initial BERT model is trained using general text data from the general text dataset to obtain a general data BERT model; The domain-specific text data in the domain-specific text dataset is used to train the general data BERT model to obtain the domain-specific data BERT model. The sensitive text data in the domain sensitive text dataset is used to train the domain data BERT model to obtain the sensitive data BERT model.

2. The BERT model pre-training method according to claim 1, characterized in that: The generated word / character lookup table maps any element in the word / character database to another element in the word / character database that is not the element itself.

3. The BERT model pre-training method according to claim 1, characterized in that: Training the domain data BERT model is unsupervised training of the domain data BERT model.

4. The BERT model pre-training method according to claim 1, characterized in that: Downstream tasks of the method include text classification, named entity recognition, or next sentence prediction.

5. A storage medium, characterized in that, The storage medium includes a stored program, wherein the method according to any one of claims 1-4 is executed by a processor when the program is run.

6. A BERT model pre-training device, characterized in that, include: The lookup table generation module is used to randomly combine words / characters in the initial BERT model's word / character database to generate a word / character lookup table, and to map any element in the BERT model's word / character database table to another element in the word / character database table that is not the element itself; The dataset mapping module is used to map the general text dataset, the domain text dataset, and the domain-sensitive text dataset to the general text dataset, the domain text dataset, and the domain-sensitive text dataset respectively according to the word / character lookup table; A general text data training module is used to train the initial BERT model using general text data in the general text dataset to obtain a general data BERT model. The domain text data training module is used to train the general data BERT model using domain text data in the domain text dataset to obtain the domain data BERT model. The sensitive text data training module is used to train the domain data BERT model using sensitive text data from the domain sensitive text dataset to obtain the sensitive data BERT model.

7. The BERT model pre-training device according to claim 6, characterized in that: The generated word / character lookup table maps any element in the word / character database to another element in the word / character database that is not the element itself.

8. The BERT model pre-training device according to claim 6, characterized in that: Training the domain data BERT model is unsupervised training of the domain data BERT model.

9. A BERT model pre-training system, characterized in that, include: processor; as well as A memory connected to the processor, the memory storing a program, which, when executed by the processor, describes the method according to any one of claims 1-4.

Images