A method for fine-tuning a Bert model for a sequence labeling task

Abstract

The application provides a Bert model fine-tuning method for a sequence labeling task. The method comprises the following steps: defining hierarchical labels of tokens, which are denoted as {tag1, tag2, tag3,..., tagn, tag}; wherein, tag represents original category labels of the tokens, tag1, tag2, tag3,..., tagn represent newly-added other semantic information labels of the tokens, and n represents a layer where the labels are located; the token represents a word or a word in a text sequence to be labeled; n new decoders are added, which are connected with the first n layers of encoders in the last layer of the Bert model one by one, so that the output of the corresponding layer encoder is used as the input of the decoder connected therewith; wherein, the decoding result of the nth decoder corresponds to the nth label in the hierarchical labels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia information content processing, and in particular to a Bert model fine-tuning method for sequence labeling tasks. BACKGROUND

[0002] Sequence labeling is one of the most basic and important tasks in the field of natural language processing, which aims to input a text as an observation sequence and output a state sequence for the observation sequence through model learning. Typical sequence labeling tasks include named entity recognition, Chinese word segmentation, and text summarization.

[0003] In recent years, with the popularity of pre-training language models such as Bert, various tasks in the field of natural language processing have gradually begun to adopt the pre-training + fine-tuning mode. Pre-training refers to pre-training a model with a large amount of data; fine-tuning refers to the process of adapting the parameters of the pre-trained model to a specific task. Currently, sequence labeling tasks also use this mode, that is, a pre-trained language model such as Bert is used as an encoder, and then one or more decoders are added after the pre-trained language model, and the model training task is completed using sequence labeling task data. However, this approach has the following two potential problems, which limit the performance of sequence labeling tasks:

[0004] First, pre-trained language models such as Bert usually use massive amounts of data to complete pre-training, and their neural network weight parameters already contain a large amount of semantic information. The decoder is usually composed of a neural network with randomly initialized weight parameters, and the decoder often cannot fully match the encoding results of the pre-trained language model during fine-tuning.

[0005] Second, pre-trained language models such as Bert are usually formed by stacking multiple neural networks. Studies have shown that different levels of neural networks often focus on different semantic information. However, in existing applications, regardless of any downstream task, the decoder directly uses the topmost representation as input, without fully utilizing the semantic differences of each layer. SUMMARY

[0006] To address the above two potential problems of existing pre-trained language model fine-tuning methods such as Bert, the present application provides a Bert model fine-tuning method for sequence labeling tasks.

[0007] The present application provides a Bert model fine-tuning method for sequence labeling tasks, which includes:

[0008] Define the hierarchical label of the token, denoted as {tag1, tag2, tag3, …, tagn, tag}; wherein, tag represents the original class label of the token, tag1, tag2, tag3, …, tagn represent the other semantic information labels of the newly added token, and n represents the layer where the label is located; token represents a word or a word in the text sequence to be labeled;

[0009] n new decoders are added, which are respectively connected in one-to-one correspondence with the first n layers of the last layer of the encoder in the Bert model, so that the output of the corresponding layer encoder is used as the input of the decoder connected therewith; wherein, the decoding result of the nth decoder corresponds to the nth layer label in the hierarchical label.

[0010] Further, tag1, tag2, tag3 respectively represent the segmentation information, part-of-speech information and syntax information of the token.

[0011] Further, when outputting the result of the sequence labeling task, each layer label of the hierarchical label is further clipped as needed.

[0012] Further, the Bert model can be replaced by any Encoder-only pre-training language model based on the Transformer structure.

[0013] The beneficial effects of the present application are:

[0014] (1) The original label of the sequence labeling task is hierarchically expanded to obtain a hierarchical label, which provides a way to introduce multi-level semantic information, so that the semantic differences of each layer of the encoder can be fully utilized;

[0015] (2) Multiple decoders are added, and each decoder is connected to a different encoder layer of the Bert model, realizing the connection of multi-level labels and different neural network layers of the Bert model, realizing the complete matching of the encoder and the decoder in the fine-tuning process, and the multi-level semantic information guiding the model to fine-tune based on the task data, fully utilizing the semantic information of different levels of Bert, and finally improving the effect of the sequence labeling task. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of a Bert model fine-tuning method for a sequence labeling task provided by an embodiment of the present application is shown in the figure;

[0017] Figure 2 The original fine-tuning method of the Bert model provided by the embodiment of the present application is shown in the figure;

[0018] Figure 3 The improved fine-tuning method of the Bert model provided by the embodiment of the present application is shown in the figure. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Example 1

[0021] This invention proposes a method for fine-tuning the BERT model for sequence labeling tasks, comprising the following steps:

[0022] S101: Define the hierarchical tags of the token, denoted as { tag1, tag2, tag3, ..., tagn, tag}; where tag represents the original category tag of the token, tag1, tag2, tag3, ..., tagn represent other semantic information tags of the newly added token, and n represents the level of the tag; token represents a character or word in the text sequence to be labeled;

[0023] Specifically, the traditional labeling format for sequence labeling tasks is generally (token, tag), where the token is a character or word in the text sequence, and the tag refers to the category corresponding to the token.

[0024] For example, for the text sequence "Xiaoming is from the Public Security Bureau.", the result of tagging according to the traditional tagging format would be: (Xiaoming, PER), (Public Security Bureau, ORG), where PER and ORG represent the person and the organization, respectively. Therefore, the tag here is single-layered.

[0025] In order to distinguish the semantic characteristics of different encoding layers in the pre-trained model, this invention proposes to adopt hierarchical tags. That is, the original tags are transformed into a multi-level tag form {tag1,tag2,tag3,...,tagn,tag} based on the role of their corresponding tokens in the text, which enriches the tag information of the original tags.

[0026] S102: Add n decoders, each of which is connected to the first n encoders of the last encoder in the BERT model, such that the output of the corresponding encoder is used as the input of the decoder connected to it; wherein, the decoding result of the nth decoder corresponds to the nth layer label in the hierarchical label.

[0027] Specifically, after the labels of the tags are expanded in a hierarchical manner, the application correspondingly sets multiple decoders, respectively accesses the outputs of different encoding layers of Bert, so as to decode the encoding results of different layers, wherein the decoding result of each layer decoder corresponds to one layer of the multi-level tags. That is, when fine-tuning, each encoder in the Bert model not only passes the encoding result to the next layer encoder, but also passes the encoding result of the current layer to the corresponding decoder to complete the decoding task, so that the encoding result of the last layer encoder in the Bert model can introduce the semantic information corresponding to different levels of tags, and the prediction accuracy of the category information is improved.

[0028] The Bert model fine-tuning method for sequence labeling tasks provided by the embodiment of the application can simultaneously utilize the information of different encoding layers in the Bert model by expanding the hierarchy of the original labels and correspondingly setting multiple decoders to introduce the semantic information corresponding to different levels of tags, thereby achieving better fine-tuning effect when fine-tuning Bert for sequence labeling tasks.

[0029] Embodiment 2

[0030] On the basis of the above-mentioned embodiments, the hierarchical tags in the embodiment of the application are 4 layers, denoted as { tag1,tag2,tag3, tag}, which respectively represent word segmentation information, part-of-speech information, syntax information and category information.

[0031] Correspondingly, as shown in Figure 2 , the Bert model has 12 layers, and the traditional fine-tuning method is to directly take the output of the 12th layer encoder to participate in fine-tuning.

[0032] According to the fine-tuning method provided by the application, as shown in Figure 2 , the outputs of the 9th, 10th and 11th layers of encoders are directly involved in fine-tuning.

[0033] Based on the improved fine-tuning method of the embodiment of the application, when performing sequence labeling tasks, the multi-level tags { tag1,tag2,tag3,tag} defined by the application will be obtained.

[0034] Considering that the expanded tags of the application are redundant with respect to the output of the sequence labeling task, when outputting the result of the sequence labeling task, the multi-level tags may need to be trimmed according to the task requirements to obtain the same result as the original tags, and only tag is output as the final result.

[0035] Embodiment 3

[0036] In order to facilitate understanding, the fine-tuning method provided by the application is further described by taking a named entity recognition task as an example.

[0037] The named entity recognition in the embodiment adopts an open-source Weibo dataset, and the labels thereof include (B-ORG.NOM, M-ORG.NOM, E-ORG.NOM, B-PER.NAM, M-PER.NAM, E-PER.NAM, B-PER.NOM, B-PER.NOM, M-PER.NOM, E-PER.NOM, B-ORG.NAM, M-ORG.NAM, E-ORG.NAM, B-LOC.NAM, M-LOC.NAM, E-LOC.NAM, B-LOC.NOM, M-LOC.NOM, E-LOC.NOM, O). Wherein the first letters "B", "M" and "E" respectively represent the beginning, middle and end of a complete sequence; ORG, PER and LOC respectively represent an organization, a person and an address; NAM and NOM respectively represent a real reference and a virtual reference. O represents a non-entity word.

[0038] For example, for the text sequence "Liu XX is XX person", the original label is (B-PER.NAM, M-PER.NAM, E-PER.NAM, O, B-PER.NOM, M-PER.NOM, E-PER.NOM).

[0039] After introducing the word segmentation information, the part-of-speech information and the syntax information, the original label is layered and modified into (b-n-sub-B-PER-NAM, m-n-sub-M-PER-NAM, e-n-sub-E-PER-NAM, s-v-pre-O-O-O, b-n-obj-B-PER-NOM, m-n-obj-M-PER-NOM, e-n-obj-E-PER-NOM).

[0040] Wherein, (b, m, e) in the first layer represents the word segmentation information; b is the beginning of a word, m is the middle part of a word, and e is the end of a word. (n, v) in the second layer represents the part-of-speech information; n represents a noun, and v represents a verb. (sub, pre, obj) in the third layer represents the syntax information; sub represents the subject, pre represents the predicate, and obj represents the object.

[0041] In this way, the original label is expanded from 3 layers to 6 layers. In addition to the original 3-layer label, a layer representing word segmentation information, a layer representing part-of-speech information, and a layer representing syntax information are added. The above-mentioned 6-layer labels are respectively connected to 6-layer decoders for model training. After the model training is completed, when the application model is used for inference, the prediction results of the last three decoders are taken and spliced into the original label form.

[0042] Embodiment 4

[0043] For ease of understanding, the fine-tuning method provided by the present application is further described in an entity relation extraction task.

[0044] Unlike the named entity recognition task in Embodiment 1, for an entity relation extraction task, the original label not only includes entity category information, but also includes entity relation information.

[0045] For example, for the text sequence "Liu XX is XX", the original label is (B-PER-NAM-NAN-NAN, M-PER-NAM-NAN-NAN, E-PER-NAM-NAN-NAN, O-O-O-NAN-NAN, B-PER-NOM-identity-0, M-PER-NOM-identity-0, E-PER-NOM-identity-0).

[0046] After introducing the word segmentation information, part-of-speech information and syntax information, the original label is layered into (b-n-sub-B-PER-NAM-NAN-NAN, m-n-sub-M-PER-NAM-NAN-NAN, e-n-sub-E-PER-NAM-NAN-NAN, s-v-pre-O-O-O-NAN-NAN, b-n-obj-B-PER-NOM-identity-0, m-n-obj-M-PER-NOM-identity-0, e-n-obj-E-PER-NOM-identity-0).

[0047] In this way, the original label is expanded to 8 layers, which is the same as Embodiment 3. In addition to the original category label, a layer representing word segmentation information, a layer representing part-of-speech information, and a layer representing syntax information are added. The 8-layer labels are connected to the 8-layer decoder for model training. After the model training is completed, when the application model is used for inference, the prediction results of the last five decoders are taken and spliced into the original label form.

[0048] It should be noted that in the process of multi-level label expansion, not only can word segmentation information, part-of-speech information and syntax information be introduced, but also can be adjusted according to the actual needs of the task, so as to better introduce related additional features and make full use of the hierarchical semantics of each layer of the Bert pre-training language model.

[0049] It can be understood that the fine-tuning method of the present application to the Bert model is also applicable to other Encoder-only pre-training language models based on the Transformers structure.

[0050] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for fine-tuning a Bert model for a sequence labeling task, characterized in that, Comprise: Hierarchical tags defining tokens, denoted as {tag1, tag2, tag3, …, tagn, tag}; wherein, tag represents the original class label of the token, tag1, tag2, tag3, …, tagn represent other semantic information labels of the newly added token, and n represents the layer where the label is located; token represents a word or a word in the text sequence to be labeled; n new decoders are connected one by one with the first n layers of the last layer encoder in the Bert model, so that the output of the corresponding layer encoder is used as the input of the decoder connected therewith, so as to decode the encoding results of different layers; wherein, the decoding result of the nth decoder corresponds to the nth layer of the hierarchical label; when fine-tuning, each layer encoder in the Bert model not only passes the encoding result to the next layer encoder, but also passes the encoding result of the current layer to the corresponding decoder to complete the decoding task, so that the encoding result of the last layer encoder in the Bert model can introduce semantic information corresponding to different levels of labels, thereby improving the prediction accuracy of the class information.

2. The method of claim 1, wherein the method is characterized in that, tag1, tag2, tag3 represent the segmentation information, part-of-speech information and syntax information of the token respectively.

3. The method of claim 1, wherein the method is characterized in that, Also comprise: When outputting the result of the sequence labeling task, each layer of the hierarchical label is trimmed as needed.

4. The method of claim 1 to 3, wherein, Replace the Bert model with any Encoder-only pre-training language model based on the Transformer structure.

Images