BERT-based sequence generation method and device

Abstract

The present invention discloses a sequence generation method and device based on BERT, which relates to the technical field of natural language processing. The main purpose of the present invention is to use the BERT model to realize the processing of sequence generation tasks. The main technical scheme of the present invention is: obtaining a sequence generation model constructed based on the BERT model; setting the iteration parameters of the sequence generation model; inputting first sequence data into the sequence generation model; and the sequence generation model generates second sequence data according to the first sequence data and the iteration parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a BERT-based sequence generation method and device. Background Art

[0002] The BERT model is a language model proposed by Google that is built on bidirectional Transformers. The BERT model combines a pre-trained model with a downstream task model, which means that the BERT model is still used when doing downstream tasks. BERT stands for Bidirectional Encoder Representations from Transformers, which is a bidirectional encoder representation based on Transformers. Bidirectional means that when processing a word, it can take into account the information of the words before and after the word to obtain the semantics of the context. BERT pre-trains deep bidirectional representations by jointly adjusting the context in all layers. Therefore, the pre-trained BERT representation can be fine-tuned through an additional output layer to quickly adapt to downstream tasks.

[0003] However, according to the above structure of the BERT model, the natural language processing tasks that the model can currently handle are mainly concentrated in text classification tasks, such as sentiment classification; sequence labeling tasks, such as real-time word segmentation recognition, part-of-speech tagging, etc., and cannot handle sequence generation tasks, such as sentence simplification, machine translation, etc. The existing sequence generation model uses a unidirectional left-to-right decoding method when processing sequence generation tasks, which makes the training objectives and generation methods of the BERT model significantly different from those of the existing sequence generation models, and the BERT model cannot be well applied to sequence generation tasks. Summary of the invention

[0004] In view of the above problems, the present invention proposes a BERT-based sequence generation method and device, the main purpose of which is to use the BERT model to realize the processing of sequence generation tasks.

[0005] In order to achieve the above object, the present invention mainly provides the following technical solutions:

[0006] On the one hand, the present invention provides a BERT-based sequence generation method, specifically comprising:

[0007] Get the sequence generation model built based on the BERT model;

[0008] Setting iteration parameters of the sequence generation model;

[0009] Inputting first sequence data into the sequence generation model;

[0010] The sequence generation model generates second sequence data according to the first sequence data and iteration parameters.

[0011] On the other hand, the present invention provides a BERT-based sequence generation device, specifically comprising:

[0012] An acquisition unit, used to acquire a sequence generation model built based on a BERT model;

[0013] A setting unit, used to set the iteration parameters of the sequence generation model obtained by the acquisition unit;

[0014] An input unit, used for inputting first sequence data into the sequence generation model obtained by the acquisition unit;

[0015] The generating unit is used to generate second sequence data by the sequence generating model according to the first sequence data input by the input unit and the iteration parameter obtained by the setting unit.

[0016] On the other hand, the present invention provides a BERT-based sequence generation model, which is composed of a cascade of a first BERT model and a second BERT model, wherein the input of the first BERT model is the input of the sequence generation model, and the output of the first BERT model is the length of the output sequence of the sequence generation model and the output sequence vector; the input of the second BERT model is the output sequence vector output by the first BERT model and preset mask parameters, wherein the preset mask parameters are used to perform a mask operation on a specified vector in the output sequence vector, and the output of the second BERT model is the prediction result of the specified vector after the mask operation.

[0017] On the other hand, the present invention provides a processor, which is used to run a program, wherein the program executes the above-mentioned BERT-based sequence generation method when running.

[0018] By means of the above technical solution, the present invention provides a BERT-based sequence generation method and device, which realizes the application of the BERT model to the sequence generation task in the natural language processing task by acquiring the sequence generation model constructed based on the BERT model, thereby improving the processing effect of the sequence generation task. When processing the sequence generation task, in addition to the need to construct a sequence generation model based on the BERT model, the present invention also needs to specify the specific application method of the model. Compared with the existing BERT model that is only applied to processing classification tasks, the present invention controls the number of iterations of the model by setting the model iteration parameters, thereby using the output of the original BERT model to gradually determine the sequence data to be generated, thereby achieving the purpose of applying the BERT model to the sequence generation task.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0021] Figure 1 A flowchart of a BERT-based sequence generation method proposed in an embodiment of the present invention is shown;

[0022] Figure 2 A structural block diagram of a sequence generation model proposed in an embodiment of the present invention is shown;

[0023] Figure 3 A schematic diagram showing an iterative process in which a sequence generation model proposed in an embodiment of the present invention processes a sequence generation task;

[0024] Figure 4 A block diagram of a BERT-based sequence generation device according to an embodiment of the present invention is shown;

[0025] Figure 5 A block diagram showing the composition of another BERT-based sequence generation device proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0027] Natural language processing is a model for studying language ability and language application. It is achieved by establishing a computer algorithm framework, and is improved through training and evaluation, and finally used in various practical systems. The scenarios in which natural language processing models are applied include information retrieval, machine translation, document classification, information extraction, text mining, etc. Therefore, the training of language models plays an important role in natural language processing tasks. At present, pre-trained representation methods have made significant progress in handling natural language processing tasks, and the existing pre-trained representation methods are all one-way language models. That is to say, each word can only be trained using the word information before the word. Such constraints severely limit the representation ability of pre-trained representations. Therefore, the BERT model can solve the above constraints. BERT is a MASK model. BERT replaces the words in the input sequence with random masks. The goal is to use context information (compared with the traditional left-to-right language model, the BERT model can use the context of the left and right of the masked word for prediction) to predict the masked word. However, the current use of the BERT model is limited to the processing of classification tasks and cannot handle sequence generation tasks.

[0028] Based on the above understanding of the BRET model, an embodiment of the present invention provides a BERT-based sequence generation method, which is improved on the basis of the BERT model so that the improved BERT model can effectively handle sequence generation tasks. The specific steps of this method are as follows: Figure 1 As shown, the method includes:

[0029] Step 101: Obtain a sequence generation model based on the BERT model.

[0030] Since the sequence generation model in the embodiment of the present invention is constructed based on the BERT model, the sequence generation model does not predict in one direction during the sequence generation process, but predicts the output sequence through bidirectional contextual relationship prediction. In other words, the sequence generation method of the sequence generation model is not the traditional left-to-right prediction, but predicts the output sequence through the left and right contexts, so that the prediction effect of the model is better.

[0031] In addition, since the BERT model can quickly adapt to natural language processing tasks in different application scenarios by fine-tuning model parameters, the sequence generation model in this step can also be used to fine-tune model parameters to adapt to sequence generation tasks in different application scenarios. That is, when there are insufficient training samples in an application scenario, the sequence generation model in this step can also be adapted to the application scenario by fine-tuning model parameters, making it suitable for new application scenarios that lack training samples. Therefore, after training, the sequence generation model in this step can be used as a pre-trained model and quickly applied to other sequence generation task scenarios by fine-tuning model parameters.

[0032] Step 102: Set the iteration parameters of the sequence generation model.

[0033] Since the existing BERT model is mainly used for the classification task of the input sequence, and its output is mainly the corresponding category representation, therefore, when the sequence generation model in the embodiment of the present invention processes the input sequence, its output is mainly reflected in the prediction of the length of the output sequence and the vector value prediction of each vector in the output sequence, that is, the category representation of the output in BERT is applied to the prediction of individual vectors in the sequence to be generated. Since the length of the output sequence is generally greater than the classification type of the category representation, in this embodiment, the output sequence is gradually determined through multiple iterative calculations, and the number of iterations determines the number of sequence vectors that need to be determined in each iteration.

[0034] Step 103: Input the first sequence data into the sequence generation model.

[0035] Step 104: The sequence generation model generates second sequence data according to the first sequence data and iteration parameters.

[0036] Among them, in the specific processing process of the sequence generation model, the processing method for word vectors in the BERT model is mainly applied. Specifically, the length of the output sequence is first determined according to the first sequence data, and then the vector in the output sequence determined after each processing is determined according to the set iteration parameters. The complete output sequence is determined through multiple iterative processing, and it is determined as the second sequence data and output.

[0037] Through the description of the above embodiments, the BERT-based sequence generation method provided by the present invention is to obtain a bidirectional language model that can be used for sequence generation tasks by improving the BERT model, and the word vectors in the sequence outputted are obtained by the left and right context information in the sequence, so that the outputted second sequence data has better natural language representation and is more conducive to understanding. The embodiments of the present invention can be applied to various scenarios of sequence generation tasks, such as sentence translation, sentence interpretation or simplification, question-answering dialogue, etc.

[0038] Furthermore, for the above Figure 1 The sequence generation model described in the embodiment of the present invention is created based on the BERT model. Specifically, the sequence generation model can be constructed by combining multiple BERT models. One implementation method of its structure is as follows: Figure 2 As shown, it includes a first BERT model and a second BERT model, and a sequence generation model is formed by cascading the first BERT model and the second BERT model, wherein the input of the first BERT model is the input of the sequence generation model, and the output of the first BERT model is the length of the output sequence of the sequence generation model and the output sequence vector; the input of the second BERT model is the output sequence vector and preset mask parameters output by the first BERT model, and the preset mask parameters are used to perform a mask operation on a specified vector in the output sequence vector, and the output of the second BERT model is the prediction result of the specified vector after the mask operation. Corresponding to Figure 2 , assuming that sequence X is the input sequence of the sequence generation model (first sequence data); sequence Y is the output sequence generated by the sequence generation model (second sequence data), then the input of the first BERT model is sequence X, and the output is Y l And the sequence vector T corresponding to the sequence Y, and the input of the second BERT model is the sequence vector T and Y o The output is Y m , where Y l is the length of the sequence vector T output by the first BERT model, Y o is the preset mask parameter of the second BERT model, through which the number of vectors to mask the vector sequence T can be determined, Y m is the vector value obtained by predicting the masked vector.

[0039] From the above structural description of the sequence generation model, it can be seen that the first BERT model is used to predict the length of the output sequence according to the output sequence, and to initialize the prediction of the vector value of each position vector in the output sequence, and use the prediction result as the input of the second BERT model; and the second BERT model is used to iteratively predict each position vector in the output sequence and determine the vector value of each position vector. o It is only set during the training phase of the sequence generation model, that is, the Y o The value of is provided by the training sample. Therefore, the training sample of the sequence generation model proposed in the embodiment of the present invention needs to include: an input sequence, a preset mask parameter and a vector value, wherein the preset mask parameter is the number of vectors for masking the specified vector in the first BERT model output sequence, and the vector value is the vector value corresponding to the specified vector after the mask operation. Figure 2As shown, a training sample should consist of (X,Y o ,Y m ).

[0040] When the sequence generation model is trained and applied to the sequence generation task, the sequence vector T output by the first BERT model will be completely masked, that is, Y o The length Y of T will be l Determine, and gradually determine the vector value of each position vector in T through multiple iterations.

[0041] Furthermore, in the training process of the sequence generation model in the embodiment of the present invention, since it is necessary to set a preset mask parameter, it is necessary to determine the output sequence length of the model, but the existing BERT model does not need to predict this. For this reason, the embodiment of the present invention adds a preset identifier at the beginning of the input sequence of the first BERT model, so that the first BERT model can determine the length of the output sequence according to the preset identifier, wherein the input sequence includes adding a preset identifier before the input X of the sample, and adding a preset identifier before the first sequence data. Specifically, the preset identifier is a fixed identifier, which does not change with the input sequence of the model. By identifying the preset identifier, the sequence generation model can determine the length of the output sequence vector according to the input sequence vector and the environmental parameters applied by the model. For example, for the same input sequence: hello, in the translation environment, its output sequence is: hello, with a length of 1, and in the dialogue environment, its output sequence can be: good morning, with a length of 3. It can be seen that determining the length of the output sequence by the preset identifier is an environmental parameter that requires training the model.

[0042] Furthermore, in the process of the sequence generation model processing the sequence generation task, the prediction of the second sequence data mainly applies the mask prediction algorithm in the BERT model, that is, masking individual vectors in the input sequence and predicting the vector value of the masked vector. Specifically, the processing process of the sequence generation model includes:

[0043] First, the length of the second sequence data is determined according to a preset identifier at the beginning of the first sequence data.

[0044] To this end, it is necessary to identify and judge the vector at the beginning position of the first sequence data. When it is determined that the vector is a preset identifier, the first sequence data is processed and the length of the second sequence data is determined.

[0045] Afterwards, the masked vector sequence is used as the second sequence data according to the length of the second sequence data.

[0046] In this embodiment, the [MASK] identifier is used to replace all vectors in the second sequence data, that is, a sequence consisting of the same number of [MASK] identifiers is generated according to the above-determined length.

[0047] Third, the number n of vectors that need to be confirmed after each iteration is determined according to the iteration parameter and the above length.

[0048] The iteration parameter is specifically the number of iterations. The specific determination method is not limited and can be an average or a percentage. For example, if the length is 10 and the number of iterations is 3, then the number of vectors to be confirmed in each iteration is determined to be at least 3 based on the average, that is, 3 vectors are determined in the first iteration, 3 vectors are determined in the second iteration, and 4 vectors are determined in the third iteration. If determined based on the percentage (50%), 5 vectors are determined in the first iteration, 3 vectors are determined in the second iteration, and 2 vectors are determined in the third iteration.

[0049] Finally, according to the number of vectors n, in each iteration process, the n vectors with the highest prediction probability are determined as the corresponding vectors in the second sequence data.

[0050] Based on the above description of the sequence generation model processing sequence generation task, the following will be Figure 3 The illustrated contents illustrate a process of iteratively determining the second sequence data when processing a sequence generation task.

[0051] After receiving the first sequence data, the sequence generation model identifies the preset identifier at the beginning of the sequence data, i.e. Figure 2 The identifier [CLS] at the beginning of X in the sequence determines the length of the second sequence data. Assuming that the length is 5, the input of the second BERT model in the sequence generation model is a vector sequence of length 5, and each vector in the vector sequence is replaced by the [MASK] identifier, that is, Figure 3After the first iteration, the input sequence in the middle and lower part obtains the vector value at each position and the corresponding probability, that is, the probability that the vector at the position is the vector value. Assuming that the iteration parameter of the sequence generation model is 3 times, then according to the calculation, after the first iteration, the number of vectors n that need to be determined is 2. If the predicted sequence vector is: "what causes causes headache!", the corresponding probability is "0.7, 0.5, 0.1, 0.8, 0.5", then select the two vectors with the highest probability, that is, the first and the fourth, and obtain the vector sequence "what[MASK][MASK]headache[MASK]", which is used as the input sequence of the second iteration. Repeat the calculation. After 3 iterations, all 5 vectors are determined, thus obtaining the second sequence data: "what causes a headache?". The description of the last two iterations is omitted here, and the rest of the first iteration is processed in the same way.

[0052] Furthermore, for Figure 1 In step 102 of the illustrated embodiment, the specific manner of setting the iteration parameter is not limited in this embodiment. The iteration parameter may be set manually or determined by calculation, that is, the iteration parameter is set according to the predicted length of the second sequence data. In this way, the number of iterations may change with the length. In this way, for a longer sequence, the vector value of each position vector may be determined through more iterations, so that the semantics of the natural language represented by the second sequence data is more accurate and easier to understand.

[0053] Furthermore, as a response to the above Figure 1 In order to realize the method shown in the figure, the embodiment of the present invention provides a sequence generation device based on BERT, and the main purpose of the device is to use the BERT model to realize the processing of sequence generation tasks. For the sake of ease of reading, the present device embodiment will not repeat the details of the above method embodiment one by one, but it should be clear that the device in this embodiment can realize all the contents of the above method embodiment. The device is as follows Figure 4 As shown, specifically including:

[0054] An acquisition unit 21 is used to acquire a sequence generation model built based on a BERT model;

[0055] A setting unit 22, used to set the iteration parameters of the sequence generation model obtained by the acquisition unit 21;

[0056] An input unit 23, used for inputting first sequence data into the sequence generation model obtained by the acquisition unit 21;

[0057] The generating unit 24 is configured to generate second sequence data by the sequence generating model according to the first sequence data input by the input unit 23 and the iteration parameters obtained by the setting unit 22 .

[0058] Further, such as Figure 5 As shown, the device also includes:

[0059] A creating unit 25, configured to create a sequence generation model, wherein the sequence generation model includes a first BERT model and a second BERT model;

[0060] The training unit 26 is used to train the sequence generation model constructed by the creation unit 25 using preset training samples.

[0061] Furthermore, the creation unit 25 is specifically used to cascade the first BERT model and the second BERT model to form a sequence generation model, wherein the input of the first BERT model is the input of the sequence generation model, and the output of the first BERT model is the length of the output sequence of the sequence generation model and the output sequence vector; the input of the second BERT model is the output sequence vector and preset mask parameters output by the first BERT model, the preset mask parameters are used to perform a mask operation on a specified vector in the output sequence vector, and the output of the second BERT model is the prediction result of the specified vector after the mask operation.

[0062] Furthermore, the preset training sample includes: an input sequence, a preset mask parameter and a vector value, wherein the preset mask parameter is the number of vectors on which a specified vector in the output sequence of the first BERT model is masked, and the vector value is the vector value corresponding to the specified vector after the mask operation.

[0063] Further, such as Figure 5 As shown, the training unit 26 is further used to add a preset identifier at the beginning of the input sequence, wherein the first BERT model determines the length of the output sequence according to the preset identifier.

[0064] Furthermore, the input unit 23 is further configured to add a preset identifier at the beginning of the first sequence data, and input the first sequence data containing the preset identifier into the sequence generation model.

[0065] Furthermore, the generating unit 24 is specifically used for:

[0066] The sequence generation model determines the length of the second sequence data according to the preset identifier at the beginning of the first sequence data;

[0067] According to the length, the masked vector sequence is used as the second sequence data;

[0068] Determine the number n of vectors that need to be confirmed after each iteration according to the iteration parameter and the length;

[0069] According to the number of vectors n, in each iteration process, the n vectors with the highest prediction probabilities are determined as the corresponding vectors in the second sequence data.

[0070] In addition, an embodiment of the present invention further provides a processor, which is used to run a program, wherein the program executes the BERT-based sequence generation method provided by any one of the above embodiments when running.

[0071] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0072] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0074] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the preferred embodiment of the present invention.

[0075] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0076] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0077] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0081] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0082] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0083] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0085] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A BERT-based sequence generation method, the method comprising: Get the sequence generation model built based on the BERT model; Setting iteration parameters of the sequence generation model; Inputting first sequence data into the sequence generation model; The sequence generation model generates second sequence data according to the first sequence data and iteration parameters, wherein the sequence generation model is composed of a cascade of a first BERT model and a second BERT model, the input of the first BERT model is the first sequence data, and the output of the first BERT model is the length of the output sequence of the sequence generation model and the sequence vector corresponding to the output sequence; the input of the second BERT model is the sequence vector corresponding to the output sequence and a preset mask parameter, and the output of the second BERT model is the second sequence data.

2. The method according to claim 1, characterized in that: The method further comprises: Creating a sequence generation model, wherein the sequence generation model includes a first BERT model and a second BERT model; The sequence generation model is trained using preset training samples.

3. The method according to claim 2, characterized in that The creating a sequence generation model comprises: The first BERT model and the second BERT model are cascaded to form a sequence generation model, wherein the input of the first BERT model is the input of the sequence generation model, and the output of the first BERT model is the length of the output sequence of the sequence generation model and the output sequence vector; the input of the second BERT model is the output sequence vector and preset mask parameters output by the first BERT model, the preset mask parameters are used to perform a mask operation on a specified vector in the output sequence vector, and the output of the second BERT model is the prediction result of the specified vector after the mask operation.

4. The method according to claim 2, characterized in that: The preset training sample includes: an input sequence, a preset mask parameter and a vector value, wherein the preset mask parameter is the number of vectors for performing a mask operation on a specified vector in the first BERT model output sequence, and the vector value is the vector value corresponding to the specified vector after the mask operation.

5. The method according to claim 4, characterized in that The method further comprises: A preset identifier is added at the beginning of the input sequence, wherein the first BERT model determines the length of the output sequence according to the preset identifier.

6. The method according to claim 1, characterized in that The inputting the first sequence data into the sequence generation model comprises: A preset identifier is added to the beginning of the first sequence data, and the first sequence data containing the preset identifier is input into the sequence generation model.

7. The method according to claim 6, characterized in that The sequence generation model generates second sequence data according to the first sequence data and an iteration parameter, including: The sequence generation model determines the length of the second sequence data according to the preset identifier at the beginning of the first sequence data; According to the length, the masked vector sequence is used as the second sequence data; Determine the number n of vectors that need to be confirmed after each iteration according to the iteration parameter and the length; According to the number of vectors n, in each iteration process, the n vectors with the highest prediction probabilities are determined as the corresponding vectors in the second sequence data.

8. A BERT-based sequence generation device, the device comprising: An acquisition unit, used to acquire a sequence generation model built based on a BERT model; A setting unit, used to set the iteration parameters of the sequence generation model obtained by the acquisition unit; An input unit, used for inputting first sequence data into the sequence generation model obtained by the acquisition unit; A generating unit, configured to generate second sequence data by the sequence generating model according to the first sequence data input by the input unit and the iteration parameter obtained by the setting unit, wherein the sequence generating model is composed of a first BERT model and a second BERT model in cascade, the input of the first BERT model is the first sequence data, the output of the first BERT model is the length of the output sequence of the sequence generating model and the sequence vector corresponding to the output sequence; the input of the second BERT model is the sequence vector corresponding to the output sequence and a preset mask parameter, and the output of the second BERT model is the second sequence data.

9. The device according to claim 8, characterized in that The device also includes: A creating unit, configured to create a sequence generation model, wherein the sequence generation model includes a first BERT model and a second BERT model; The training unit is used to train the sequence generation model constructed by the creation unit using preset training samples.

10. The device according to claim 9, characterized in that The creation unit is specifically used to cascade the first BERT model and the second BERT model to form a sequence generation model, wherein the input of the first BERT model is the input of the sequence generation model, and the output of the first BERT model is the length of the output sequence of the sequence generation model and the output sequence vector; the input of the second BERT model is the output sequence vector output by the first BERT model and preset mask parameters, the preset mask parameters are used to perform a mask operation on a specified vector in the output sequence vector, and the output of the second BERT model is a prediction result of the specified vector after the mask operation.

11. The device according to claim 9, characterized in that The preset training sample includes: an input sequence, a preset mask parameter and a vector value, wherein the preset mask parameter is the number of vectors for performing a mask operation on a specified vector in the first BERT model output sequence, and the vector value is the vector value corresponding to the specified vector after the mask operation.

12. The device according to claim 11, characterized in that The training unit is also used to add a preset identifier at the beginning of the input sequence, wherein the first BERT model determines the length of the output sequence according to the preset identifier.

13. The device according to claim 8, characterized in that The input unit is further used to add a preset identifier at the beginning of the first sequence data, and input the first sequence data containing the preset identifier into the sequence generation model.

14. The device according to claim 13, characterized in that The generating unit is specifically used for: The sequence generation model determines the length of the second sequence data according to the preset identifier at the beginning of the first sequence data; According to the length, the masked vector sequence is used as the second sequence data; Determine the number n of vectors that need to be confirmed after each iteration according to the iteration parameter and the length; According to the number of vectors n, in each iteration process, the n vectors with the highest prediction probabilities are determined as the corresponding vectors in the second sequence data.

15. A BERT-based sequence generation model, wherein the sequence generation model is composed of a first BERT model and a second BERT model cascaded, wherein: The input of the first BERT model is the input of the sequence generation model, and the output of the first BERT model is the length of the output sequence and the output sequence vector of the sequence generation model; The input of the second BERT model is the output sequence vector output by the first BERT model and preset mask parameters, the preset mask parameters are used to perform a mask operation on a specified vector in the output sequence vector, and the output of the second BERT model is the prediction result of the specified vector after the mask operation.

16. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the BERT-based sequence generation method described in any one of claims 1 to 7.

Images