Sequence labeling method and training method, device, equipment and medium

Abstract

The application provides a sequence labeling method and a training method, a device, equipment and a medium, and the sequence labeling optimization method comprises the following steps: processing natural language based on a deep learning model, and assigning a sequence label to each word unit; wherein the word unit comprises at least one character or at least one word; determining whether the sequence label conforms to a sequence label rule based on a preset rule; if the sequence label does not conform to the sequence label rule, the sequence label is sent to a linear chain conditional random field (CRF) layer for correction; and the sequence label to be inferred by the model is pre-judged whether to be input into the linear chain conditional random field (CRF) layer, so that the calculation number of the CRF layer can be greatly reduced, and the calculation accuracy is ensured.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to methods, training methods, apparatus, devices and media for sequence labeling. Background Technology

[0002] Sequence labeling is a model processing task in natural language processing. The algorithms used in sequence labeling employ deep learning models combined with Conditional Random Fields (CRFs) to process sequence labels. However, CRFs are based on the Viterbi algorithm and require computation on each sequence individually, which is much more computationally expensive than deep learning models.

[0003] Therefore, there is a need for devices that can improve the computational performance of CRF. Summary of the Invention

[0004] This disclosure provides a sequence labeling method, a sequence labeling training method, an apparatus, an electronic device, and a storage medium.

[0005] According to a first aspect of the present disclosure, a method for sequence labeling is provided, the method comprising:

[0006] Natural language is processed using a deep learning model, and a sequence label is assigned to each word unit; wherein, the word unit includes: at least one character or at least one word;

[0007] Based on preset rules, determine whether the sequence label conforms to the sequence label rules;

[0008] If the sequence label does not conform to the sequence label rule, the sequence label is sent to a linear conditional random field (CRF) for correction.

[0009] Optionally, the sequence tag includes:

[0010] Location tags, indicating their position relative to components of the word;

[0011] Type label, indicating the target type label for the sequence labeling task.

[0012] Optionally, determining whether the sequence label conforms to the sequence label rule based on a preset rule includes:

[0013] According to the preset rules, the position labels are detected to determine whether there are any erroneous position labels in the sequence labels;

[0014] According to the preset rules, the type label is detected to determine whether there is an erroneous type label in the sequence label;

[0015] If the sequence label contains at least one erroneous position label and / or at least one erroneous type label, the sequence label is determined to be inconsistent with the sequence label rules.

[0016] Optionally, the step of detecting whether there is an erroneous position label in the sequence label obtained by the position label according to the preset rule includes:

[0017] According to the first rule of integrity verification in the preset rules, the B tag indicating the entity head is assigned the value N, the E tag indicating the entity tail is assigned the value -N, the I tag indicating the entity middle position is assigned the value 0, and the O tag that is irrelevant to the task is assigned the value 0; wherein, the entity refers to the task extraction target in the sequence labeling task.

[0018] Perform the addition operation after assignment to obtain the result;

[0019] If the result of the calculation is not 0, then the B label or the E label in the location label is incorrect.

[0020] Optionally, the step of detecting whether there is an erroneous position label in the sequence label obtained by the position label according to the preset rule includes:

[0021] According to the second rule of integrity correction in the preset rules, the B tag indicating the entity head is assigned a value of 0, the E tag indicating the entity tail is assigned a value of 0, the I tag indicating the middle position of the entity is assigned a value of M, and the O tag, which is not relevant to the task, is assigned a value of -M.

[0022] After assigning values, perform a polynomial summation operation on any two adjacent terms to obtain multiple operation results;

[0023] If any of the multiple operation results is 0, then the sequence label includes the error location label.

[0024] Optionally, the step of detecting whether there is an erroneous type label in the sequence label obtained from the type label according to the preset rule includes:

[0025] According to the third rule of category consistency correction in the preset rules, the S tags corresponding to single words in the sequence tags are filtered to obtain the filtered sequence tags;

[0026] In the sequence labels after filtering S-labels, if the category label of the Kth sequence label is different from the category labels of the (K-1)th and (K+1)th sequence labels, then it is determined that there is an erroneous type label in the sequence labels.

[0027] Optionally, the step of detecting whether there are erroneous type tags in the sequence tags obtained by the type tags according to the preset rules includes: dividing the tags in the sequence tags into BIE structures indicating entities, S tag structures indicating single words, and O tag structures indicating tasks that do not care about, according to the fourth rule of category consistency correction in the preset rules.

[0028] Determine the number of type tags in each segmented structure of the sequence tags. If the number of type tags in a structure is greater than 1, then it is determined that there are erroneous type tags in the sequence tags.

[0029] According to a second aspect of this disclosure, a method for training sequence labeling is provided, the method comprising:

[0030] The dataset is trained by combining the first deep learning model with the CRF layer model to obtain the parameters of the first deep learning model and the CRF model. The obtained parameters of the first deep learning model are discarded, and the obtained parameters of the CRF model are retained.

[0031] The parameters of the second deep learning model are obtained by training it using a separate second deep learning model.

[0032] Using the obtained CRF layer model parameters and the second deep learning model parameters, and adding the preset rules from the aforementioned sequence labeling method, a trained CRF model with error correction is obtained; wherein, the error correction CRF model includes: determining whether the sequence label conforms to the sequence labeling rules based on the preset rules from the aforementioned sequence labeling method; if the sequence label does not conform to the sequence labeling rules, then correcting it according to the CRF layer and outputting the corrected sequence label;

[0033] The text to be labeled is input into a sequence labeling model with error correction CRF trained to obtain the sequence result.

[0034] According to a third aspect of this disclosure, an apparatus for sequence labeling is provided, the apparatus comprising:

[0035] The natural language processing module is used to process natural language based on a deep learning model and assign sequence labels to each word unit; wherein, the word unit includes: at least one character or at least one word;

[0036] The determination module is used to determine whether the sequence label conforms to the sequence label rules based on preset rules.

[0037] The correction module is used to send the sequence label to a Conditional Random Field (CRF) for correction if the sequence label does not meet the sequence label rules.

[0038] According to a fourth aspect of this disclosure, a sequence labeling training apparatus is provided, the apparatus comprising:

[0039] The first training module is used to train the dataset by combining the first deep learning model with the CRF layer model, to obtain the parameters of the first deep learning model and the CRF model, discard the obtained first deep learning model parameters, and retain the obtained CRF model parameters;

[0040] The second training module is used to train a separate second deep learning model to obtain the parameters of the second deep learning model.

[0041] The error-correcting CRF model determination module is used to utilize the obtained CRF layer model parameters and the second deep learning model parameters, and add the preset rules in claims 1 to 7 to obtain a trained CRF model with error correction; wherein, the error-correcting CRF model includes: determining whether the sequence label conforms to the sequence label annotation rules based on the preset rules in claims 1 to 7; if the sequence label does not conform to the sequence label annotation rules, then correcting it according to the CRF layer and outputting the corrected sequence label;

[0042] The module is used to input the text to be labeled into a sequence labeling model with error correction CRF that has been trained, and to obtain the sequence results.

[0043] According to a fifth aspect of this disclosure, an apparatus is provided, the apparatus comprising:

[0044] Memory;

[0045] The processor, connected to the memory, is configured to execute computer instructions stored in the memory, enabling it to implement the steps in the sequence labeling method provided in the first aspect or the training method for sequence labeling provided in the second aspect.

[0046] According to a sixth aspect of this disclosure, a medium is provided that stores computer-executable instructions; when executed by a processor, the computer-executable instructions are able to implement the steps in the sequence labeling method provided in the first aspect or the steps in the sequence labeling training method provided in the second aspect.

[0047] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0048] In this embodiment of the disclosure, natural language is processed based on a deep learning model, and a sequence label is assigned to each word unit; wherein, the word unit includes: at least one character or at least one word;

[0049] Based on preset rules, it is determined whether the sequence label conforms to the sequence label rules. If the sequence label does not conform to the sequence label rules, the sequence label is sent to a Linear Conditional Random Field (CRF) for correction. This allows for pre-judgment of whether the data to be input for CRF correction should be fed into the CRF layer. Compared to existing methods that require all data to be input into the CRF, this significantly reduces the number of CRF layer calculations while ensuring accuracy, thus improving CRF computational performance. This is especially beneficial for datasets with large amounts of natural language text, improving computational efficiency while maintaining accuracy and saving computational time and space resources.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0051] Figure 1 A schematic diagram of the first process of a sequence labeling method as illustrated in an exemplary embodiment;

[0052] Figure 2 A schematic diagram of the second process of a sequence labeling method as an exemplary embodiment;

[0053] Figure 3 A schematic diagram of the third process of a sequence labeling method as an exemplary embodiment;

[0054] Figure 4 A schematic diagram of the fourth process of a sequence labeling method as illustrated in an exemplary embodiment;

[0055] Figure 5 A schematic diagram of the fifth process of a sequence labeling method as an exemplary embodiment;

[0056] Figure 6 A schematic diagram of the sixth process of a sequence labeling method as an exemplary embodiment;

[0057] Figure 7 A flowchart illustrating a sequence labeling training method as an exemplary embodiment;

[0058] Figure 8 A probability graph of a sequence labeling method with respect to a linear chain CRF, as illustrated in an exemplary embodiment;

[0059] Figure 9 A schematic diagram of the seventh process of a sequence labeling method, as shown in an exemplary embodiment;

[0060] Figure 10 The eighth flowchart of a sequence labeling method is illustrated as an exemplary embodiment.

[0061] Figure 11A schematic diagram of the structure of a sequence labeling device shown in an exemplary embodiment;

[0062] Figure 12 This is a schematic diagram of the structure of a sequence labeling training device, as shown in an exemplary embodiment. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this disclosure as detailed in the appended claims.

[0064] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0065] 1) Word segmentation: This process divides a sentence (a sequence of characters) into a sequence of words. For example, the sentence "Secretary Zhang is currently employed by the Water Resources Bureau." is broken down into "Secretary Zhang", "currently", "employed", "in", "Water Resources Bureau", and ".".

[0066] 2) Part-of-speech tagging: Assign a part of speech to each word. Such as noun, verb, adjective, etc. In the previous example, "Secretary Zhang" and "Water Resources Bureau" are nouns, "currently" is an adverb, "in office" is a verb, "at" is a preposition, and "." is a punctuation mark.

[0067] 3) Named entity recognition: Extracting specific types of lexical entities from the text. For example, "Secretary Zhang" (type: person's name) and "Water Resources Bureau" (type: organization) in the text, while the rest of the text is not an entity.

[0068] Word segmentation, part-of-speech tagging, and named entity recognition can all be transformed into sequence labeling problems. Sequence labeling achieves segmentation and classification by assigning a label to each unit (character or word, referred to as a token). The most commonly used labeling methods include (BIEOS) tags, such as:

[0069] 1) Word segmentation: “Zhang (B) Shu (I) Ji (E) Mu (B) Qian (E) Ren (B) Zhi (E) Yu (S) Shui (B) Li (I) Ju (E). (B)”, where B represents Begin, I represents Inside, E represents End, and S represents Single. The places where B or S appears are the boundaries. By identifying the boundaries marked by B, S, and E, the words can be extracted: “Zhang Xiaochuan”, “currently”, “in office”, “Yu”, “Shui Li Ju”, “.”.

[0070] 2) Part-of-speech tagging: “Zhang (B-NR) Xiao (I-NR) Chuan (E-NR) currently (B-ADV) serves (BV) at (S-PREP) Water Conservancy (I-NG) Bureau (E-NG). (B-WJ)”, where B in B-NR represents the boundary and NR represents the category, here indicating a noun or personal name. This method distinguishes both the boundary and the category, i.e., the part of speech.

[0071] 3) Entity Recognition: The tag “Zhang Xiaochuan (E-Person) currently (O) works (O) at (O) the Water Resources Bureau (E-Organization)” is similar to part-of-speech tagging, except it includes an additional tag “O” to indicate “Outside,” meaning it's not a category relevant to the task. By processing the tag, the entities “Zhang Xiaochuan” (type: Person) and “Water Resources Bureau” (type: Time) can be extracted.

[0072] In addition, sequence labeling tasks include various other task types such as event subject identification, feature identification, key phrase extraction, and text error detection. The modeling methods for labeling these tasks are highly similar to those for entity recognition. For example, in event recognition, the sentence "Zhang Xiaochuan is currently employed at the Water Resources Bureau." is similar to entity recognition. In this example, the subject, action, and object can be extracted.

[0073] This disclosure provides a sequence labeling method, combined with... Figure 1 As shown, the method includes:

[0074] Step S101: Process natural language based on a deep learning model and assign sequence labels to each word unit; wherein, the word unit includes: at least one character or at least one word;

[0075] Step S103: Based on preset rules, determine whether the sequence tag conforms to the sequence tag rules;

[0076] Step S105: If the sequence label does not conform to the sequence label rule, the sequence label is sent to a linear conditional random field (CRF) for correction.

[0077] In this embodiment of the disclosure, the deep learning model may be, but is not limited to, Bi-Short Memory Network (Bi-LSTM), Bi-Gated Recurrent Unit (Bi-GRU), Multi-Layer Convolutional Neural Network (CNN), Multi-Layer Bi-LSTM, Transformer, and other common feature extractors.

[0078] In this embodiment of the disclosure, the deep learning model can also be: BERT model, Albert model, GPT model, etc. Such models use Transformer as feature extractor, have a deep number of model layers, strong feature representation ability, and have a much higher processing power than RNN-based models.

[0079] In this embodiment, the CRF is a linear chain conditional random field (CRF). The linear chain CRF layer mainly replaces the feature functions with feature vectors. In sequence labeling tasks, the main function of the linear chain CRF is to optimize the relationship between labels and find the label sequence with the highest probability (generally using the Viterbi algorithm for decoding). For example, a B-Person label is unlikely to be followed by an I-Orgnization label or an O label, but is more likely to be followed by an I-Person label or an E-Person label. After optimization by the CRF layer, the sequence labeling accuracy is improved.

[0080] In this embodiment, the essential function of the CRF layer applied to the deep learning model is to perform logical correction on two adjacent predicted label values. This logic is obtained through statistical training. Logical correction is indispensable in the Bi-LSTM-CRF model because logical errors in adjacent labels account for a very large proportion of the total incorrect labeling.

[0081] In the embodiments of the present disclosure, deep learning models such as Bi-GRU, multi-layer convolutional neural network CNN, multi-layer Bi-LSTM, Transformer, and BERT model, Albert model, GPT model, when connected with a CRF layer in a specific sequence labeling task, have high application value for overcoming connection errors between sequence tags. However, for the CRF layer, there is great room for optimization in the time and space resources required for its calculation.

[0082] In the method for sequence labeling provided by the embodiments of the present disclosure, in step S101, based on a deep learning model, each word or character unit of the input natural language sequence is labeled with a tag to obtain a labeled sequence of sequence tags. For example, in one embodiment, the input sentence is labeled according to the BIEOS tag method, where B represents Begin, I represents Inside, E represents End, and S represents Single word or entity. The boundaries are where B or S appear. By the boundaries marked by the B, S, and E tags, the words "Zhang Xiaochuan", "currently", "works", "at", "Water Conservancy Bureau", "." can be extracted.

[0083] In the embodiments of the present disclosure, the CRF layer plays a role in correcting tags, correcting incorrect tags to correct ones. For example, in one embodiment, the input sequence "Zhang Xiaochuan currently works at the Water Conservancy Bureau" passes through the Bi-LSTM layer to obtain a corresponding vector matrix of the sequence, and probability selection is performed according to softmax to obtain the sequence:

[0084] (B-Person)(I-Person)(O)(O)(O)(O)(O)(O)(B-Organiztion)(I-Organiztion)(E-Organiztion)(O).

[0085] After passing through the CRF layer, the final sequence can be obtained:

[0086] (B-Person)(I-Person)(E-Person)(O)(O)(O)(O)(O)(B-Organiztion)(I-Organiztion)(E-Organiztion)(O).

[0087] That is, the CRF layer plays a role in correcting tags, and it corrects tags according to the logical rules between tags. For example, an I or E must follow a B tag, an E must follow an I tag, and an I must not follow an O tag, etc. Finally, two entity results, "Zhang Xiaochuan" and "Water Conservancy Bureau", can be determined.

[0088] In this embodiment of the disclosure, the sequence label is generally executed according to the BIOES label system. For example, a certain label is called B-Person, where B is the position label, which is the first unit (token) of the person entity, and Person is the type label, indicating that the entity corresponding to the label is a person's name.

[0089] In this embodiment of the disclosure, the CRF layer corrects erroneous labels according to the logic of the sequence labeling. The logic of the sequence labeling is as follows:

[0090] B+nI+E is the most common positional label structure, where n = 1, 2, 3, ..., k, and k is a non-negative integer. Type labels within the same positional label structure must be consistent; that is, within the same complete positional label structure, the type labels under the B positional label structure must be consistent with the type labels under the I and E positional labels.

[0091] In the output sequence (where one unit represents one entity), without the S tag, the type tag cannot appear only in isolated sequence tags; there must be the same type tag before and after it.

[0092] The S tag and the O tag each constitute a positional structure on their own.

[0093] The S-structure, O-structure, and BIE-structure appear in sequence.

[0094] In this embodiment of the disclosure, the following problem occurs when there is no error correction at the CRF layer:

[0095] An incomplete BIE structure is caused by, for example, the absence of any one of the B, I, or E tags in the BIE structure. Another example is the lack of B and / or E tags. Yet another example is that in the B+n*I+E structure, one or more of the n I tags are missing, or the tags that should be labeled I are labeled with other tags, such as O or S tags.

[0096] In the BIE structure, the type tags of the B, I, and E tags are inconsistent. For example, the type tag under the B and I tags is Organization, while the type tag under the E tag is Company.

[0097] In this embodiment of the disclosure, when processing natural language, the correction of sequence labels by the Conditional Random Field (CRF) can improve the accuracy of the label sequence output. However, directly inputting sequence labels without label errors into the CRF for correction wastes computational resources. Therefore, in steps S103 and S105, based on preset rules, sequence labels that do not require CRF correction can be identified, and the corresponding sequence labels and their corresponding sequence labels can be directly output. This can significantly improve the computational efficiency of the CRF and save time and space resources.

[0098] In this embodiment of the disclosure, the sequence tag includes:

[0099] Location tags, indicating their position relative to components of the word;

[0100] Type label, indicating the type label corresponding to the location label.

[0101] In this embodiment of the disclosure, a sequence tag in the natural language sequence tag includes position tags such as a B tag located at the beginning of the entity structure, an E tag located at the end of the entity structure, and an I tag located in the middle of the entity structure. For a word containing only two characters, the B and E tags form a complete position structure consisting of a word beginning and a word ending. For a word containing three or more characters, the B, I, and E tags form a complete position structure consisting of a word beginning, a word ending, and a word middle. In a complete B+n*I+E position tag structure corresponding to a word, the correct value of n is: the number of characters contained in the entity structure n+2 minus the number of B and E tags corresponding to 2, which gives n.

[0102] In this embodiment of the disclosure, type labels and location labels are marked in correspondence. For example, one type label corresponds to one location label, and the types of type labels should be consistent in an entity structure.

[0103] In this embodiment of the disclosure, the type label can be a label representing the meaning of an entity, such as a person's name, place name, organization, institution, or company, or it can be a label representing the part of speech of a word, such as a noun, verb, or adjective.

[0104] In this embodiment of the disclosure, location tags can be used to separate words with multiple characters in a sentence, and type tags can be used to classify and label the category of a word. This also facilitates the labeling of the meaning or part of speech of a word entity, improving the efficiency of classification tasks.

[0105] In this embodiment of the disclosure, combined with Figure 2 As shown, step S103, determining whether the sequence tag conforms to the sequence tag rule based on the preset rule, includes:

[0106] Step S1031: According to the preset rules, detect whether there are erroneous position tags in the sequence tags obtained by the position tags;

[0107] Step S1032: According to the preset rules, detect whether there is an erroneous type label in the sequence label obtained by the type label;

[0108] Step S1033: When the sequence label contains at least one erroneous position label and / or at least one erroneous type label, it is determined that the sequence label does not conform to the sequence label rule.

[0109] In this embodiment of the disclosure, the preset rule is a rule used to detect whether the location label and / or type label is incorrect, and can be any pre-configured rule.

[0110] In this embodiment of the disclosure, in step S1031, according to preset rules, it is detected whether there are erroneous position tags in the sequence tags. This mainly involves detecting errors in the B, I, and E tags in the BIE tag structure, that is, detecting the integrity of the position tag structure for the same word. For example, in the BIE structure, there may be errors in the B and / or E tags, or in the I tag.

[0111] In this embodiment of the disclosure, in step S1032, according to preset rules, it is detected whether there are erroneous type tags in the sequence tags. The main purpose is to detect whether there are inconsistent type tags in the same BIE position tags, that is, to detect whether there are inconsistent type tags in the same word.

[0112] In this embodiment of the disclosure, in step S1032, after the input sequence passes through a deep learning model, such as a Bi-LSTM, Bert, Albert, GPT, etc., the Softmax layer outputs sequence labels. If the sequence labels contain at least one erroneous position label and / or at least one erroneous type label according to preset rules, it indicates that the sequence labels do not conform to the sequence label rules of the CRF layer and need to be output to the CRF layer for correction.

[0113] In this embodiment, erroneous position tags are detected according to preset rules, which can detect incomplete tag structures. Erroneous type tags are also detected according to preset rules, which can detect type tags with inconsistent categories within the same word. Erroneous sequence tags are identified as sequence tags to be corrected by the Conditional Random Field (CRF) layer, while error-free sequence tags can be directly output. Adding a logic judgment module before inputting the sequence tags to the CRF layer to determine whether to input them for correction reduces the computational resource consumption of the CRF layer, ensuring accuracy while also improving computational efficiency and performance.

[0114] In this embodiment of the disclosure, combined with Figure 2 and Figure 3 As shown, step S1031, which involves detecting whether there are erroneous position tags in the sequence tags obtained from the position tags according to the preset rules, includes:

[0115] Step S10311: According to the first rule of integrity verification in the preset rules, assign the value N to the B tag indicating the entity head, assign the value -N to the E tag indicating the entity tail, assign the value 0 to the I tag indicating the entity middle position, and assign the value 0 to the O tag that is irrelevant to the task; wherein, the entity refers to the task extraction target in the sequence labeling task.

[0116] Step S10312: Perform the addition operation after assignment to obtain the calculation result;

[0117] Step S10313: If the calculation result is not 0, then the B label or the E label in the location label is incorrect.

[0118] In this embodiment of the disclosure, in step S10311, in a complete BIE structure, the B tag and the E tag always appear in pairs. Therefore, when detecting errors in the B tag or the E tag, the value of the B tag and the value of the E tag are set to opposite numbers, and the other tags such as the O tag and the I tag are set to 0. This makes it easier to detect errors in the B tag or the E tag.

[0119] In this embodiment of the disclosure, the value of N is a non-zero number, which can be an integer, a fraction or a decimal, and can be a positive number or a negative number. As long as the B tag is assigned the value of N and the E tag is assigned the value of -N, that is, as long as the assigned value of the B tag is the opposite of the assigned value of the E tag, and the other tags such as the O tag and the I tag are 0, the values ​​of the assigned sequence tags can be summed in step S10312, and the result will be 0 or not 0.

[0120] In this embodiment of the disclosure, if the calculation result is 0, it indicates that the sequence is real and valid and no error has occurred.

[0121] In this embodiment of the disclosure, if the calculation result is not 0, for example, N or -N, or kN or -kN, where k is any value of a positive integer 1, 2, 3, ..., n, it indicates that there is an incomplete BIE structure, such as the lack of B or E tags, and the B or E tags in the position tags are incorrect.

[0122] In this embodiment of the disclosure, the erroneous sequence labels of B or E tags are obtained by numerical summation operations in the above steps. The erroneous sequence labels are then input into the CRF for correction, which can improve computational efficiency and save computational resources.

[0123] In this embodiment of the disclosure, combined with Figure 2 and Figure 4 As shown, step S1031, which involves detecting whether there are erroneous position tags in the sequence tags obtained from the position tags according to the preset rules, includes:

[0124] Step S10314: According to the second rule of integrity correction in the preset rules, the B tag indicating the entity head is assigned a value of 0, the E tag indicating the entity tail is assigned a value of 0, the I tag indicating the middle position of the entity is assigned a value of M, and the O tag that is not of interest to the task is assigned a value of -M.

[0125] In step S10315, after assigning values, a polynomial summation operation is performed on any two adjacent terms to obtain multiple operation results;

[0126] Step S10316: If any of the multiple operation results is 0, then the sequence label includes the error location label.

[0127] In this embodiment, M is a non-zero number, which can be an integer, a fraction, or a decimal, and can be positive or negative. As long as the O label is assigned a value of -M and the I label is assigned a value of M, that is, as long as the O label and the I label are opposites, and the other labels such as the B label and the E label are 0, the error situation where the I label in the B+nI+E structure is mistakenly labeled as the O label, or the I label is directly adjacent to the O label, can be detected. For example, if the label is BIIOIIE, the I label is incorrectly labeled. In this case, step S10315 is executed to calculate the sum of two adjacent values ​​in the sequence, for example, the sum of x(n) + x(n-1), to obtain a new numerical sequence. If all elements in the numerical sequence are 0, it means that the sequence is definitely wrong and the I label is incorrect.

[0128] In this embodiment of the disclosure, by performing the above steps, the erroneous sequence label of the I label is obtained. The erroneous sequence label is then input into the CRF for correction, which can improve computational efficiency and save computational resources.

[0129] In this embodiment of the disclosure, combined with Figure 2 and Figure 5 As shown, step S1032, which involves detecting whether there is an erroneous type label in the sequence label obtained from the type label according to the preset rule, includes:

[0130] Step S10321: According to the third rule of category consistency correction in the preset rules, the S tags corresponding to the single word units in the sequence tags are filtered to obtain the filtered sequence tags;

[0131] Step S10322. Among the sequence tags after filtering the S tags, if the category tag of the Kth sequence tag is different from the category tags of the (K - 1)th and (K + 1)th sequence tags, it is determined that there is an incorrect type tag in the sequence tags.

[0132] In the embodiments of the present disclosure, it should be noted that the number of type tags corresponding to an entity structure must be unique. For example, for the type tags appearing in an entity structure, if the same type tag appears, the number of type tags under this entity structure is counted as 1. If multiple different type tags appear, such as n type tags, the number of type tags under this entity structure is counted as n, where n is a natural number. If there are more than 2 type tags in a word, the type tag annotation of this sequence is incorrect.

[0133] In the embodiments of the present disclosure, the S (Single) tag represents that a single-word unit independently forms an entity structure.

[0134] In the embodiments of the present disclosure, the consistency detection of type tags is performed for an entity structure containing more than two word units. Therefore, it is necessary to filter the S tags to save calculation time and resources when subsequent calculations need to be performed.

[0135] In the embodiments of the present disclosure, in step S10321, after filtering the S tags in the sequence tags, in the filtered sequence tags, the sequence tags contain entity structures composed of more than two word units. If there are more than 2 type tags in an entity structure, the type tags are inconsistent, indicating that the type tags of the sequence tags are incorrect. For example, for a word such as "government office", the corresponding tags are: "政(B - Organization)府(I - Organization)办(I - Organization)公(I - Organization)室(E - Company)", which indicates that the type tag annotation of "室(E - Company)" is incorrect.

[0136] In the embodiments of the present disclosure, in step S10322, among the sequence tags after filtering the S tags, if the category tag of the Kth sequence tag is different from the category tags of the (K - 1)th and (K + 1)th sequence tags, it indicates that there is an incorrect type tag in the sequence tags.

[0137] In the embodiments of the present disclosure, for calculating that the category tag of the Kth sequence tag is different from the category tags of the (K - 1)th and (K + 1)th sequence tags, the following calculation methods may be used:

[0138] If there are M types of type tags in the sequence tags, then let the values ​​of the M type tags be 1, 2, 3, ..., m...M. If, in the sequence tags obtained after filtering the S tags, neither the type tag value before nor after the value m is m, it indicates that there are erroneous type tags in the sequence tags. For example, in one embodiment, a sequence tag after filtering the S tags is structured as follows: "(B-Person)(I-Person)(E-Person)(B-Organization)(I-Company)(E-Company)". The type tag value of (B-Organization) is m. If the type tag value of (E-Person) is not m, and the type tag value of (I-Company) is also not m, then it indicates that there are more than or equal to three types of type tags in the two entity structures, thus indicating that the sequence tags are erroneous.

[0139] In this embodiment of the disclosure, preferably, steps S10321 and S10322 are applicable to tasks such as named entity recognition, event trigger word extraction, and event element extraction. Steps S10321 and S10322 are not limited to named entity recognition tasks; any task involving the labeling and correction of type tags can use steps S10321 and S10322 to detect erroneous type tags.

[0140] In this embodiment of the disclosure, identifying erroneous type labels within the sequence labels can reduce the error rate caused by type label errors and improve the accuracy of category recognition in sequence labeling tasks. It also facilitates inputting the sequence labels containing erroneous type labels into the CRF layer for correction. The CRF corrects the erroneous type labels to the correct type labels, resulting in correct sequence labels and outputting the sequence results.

[0141] In this embodiment of the disclosure, step S1032, which involves detecting whether there is an erroneous type label in the sequence label obtained from the type label according to the preset rule, includes:

[0142] According to the fourth rule of category consistency correction in the preset rules, the tag step S10323 in the sequence tag is divided into BIE structure indicating multi-word words, S tag structure indicating single-word words, and O tag structure indicating task-independent words.

[0143] Step S10324: Determine the number of type tags in each segmented structure of the sequence tag. If the number of type tags in a structure is greater than 1, then it is determined that there are erroneous type tags in the sequence tag.

[0144] In this embodiment, the fourth rule, compared to the third rule, requires calculation on a case-by-case basis, resulting in a slower calculation speed. However, the fourth rule is designed to address the issue of type labeling errors, which are more prevalent in part-of-speech tagging tasks. For example, in part-of-speech tagging, the type label corresponding to each entity structure displays the part of speech of the entity structure. For instance, prepositions are labeled "prep," adverbs are labeled "adv," and nouns can be labeled "NL," "NR," "NG," "N," etc., representing nouns, personal names, or place names. Therefore, the labeling of nouns may all be noun type labels, but it is possible that more than one noun-related type label appears under a single noun. In this case, it is necessary to determine the error rate of the type label for each word unit individually.

[0145] In this embodiment of the disclosure, for example, the part-of-speech tag for the statement "Wantong Interconnect Technology Co., Ltd." is displayed as: "Wantong Interconnect Technology Co., Ltd.". Therefore, in the above tag, the position tag of Interconnect (B-NR) is correct, and the type tag shows that they are both nouns. However, by combining the position tag and the type tag, two different type tags related to nouns appear under the same entity structure. Therefore, this tagging method is incorrect.

[0146] In this embodiment of the disclosure, the type labels of the aforementioned erroneous annotation labels need to be detected through steps S10323 and S10324. That is, each entity structure needs to be divided into separate structures, and for each divided structure, the number of type labels needs to be determined. It should be noted that the number of type labels refers to the number of types of type labels. If the types of type labels in an entity structure are consistent, for example, all are NR, then the number of type labels for this entity structure is recorded as 1. If the types of type labels in an entity structure are inconsistent, and there are multiple different types of type labels, such as n types of type labels, then the number of type labels under this entity structure is counted as n, where n is a natural number.

[0147] In this embodiment of the disclosure, in step S10324, it is detected whether the number of type tags in each segmented entity structure is greater than 1. If it is greater than 1, it indicates that an entity structure has more than two type tags, and it is determined that there are erroneous type tags in the sequence tags.

[0148] In this embodiment, although the fourth rule requires comparing the number of type tags for each entity structure one by one, and the processing speed is slower than the third rule, it can more accurately detect the inconsistency of type tags in part-of-speech tagging tasks, which are more prone to such errors.

[0149] In this embodiment, the third and fourth rules offer different advantages for tasks labeled with different types of tags. The third or fourth rule can be selected based on the specific task type, the required computational speed, or the required accuracy to detect errors in the type tags, allowing for targeted detection and correction. Therefore, setting more detection rules for type tags provides more options for different task needs.

[0150] In this embodiment of the disclosure, step S103, which determines whether the sequence tag conforms to the sequence tag rules based on preset rules, includes:

[0151] Based on rule 5, determining whether the sequence label conforms to the sequence label rules includes:

[0152] Based on the first rule and the second rule, the sequence label of the position label without error is obtained;

[0153] Based on the sequence label containing the error-free position label, the sequence label containing the error-free position label is determined to be a complete sequence label;

[0154] In the sequence labels, based on the third rule or the fourth rule, when the sequence labels with inconsistent type labels in the updated sequence labels are determined, and a sequence label containing at least one erroneous type label is obtained, it is determined that the sequence labels contain sequence labels that need to be corrected by the Conditional Random Field (CRF).

[0155] In this embodiment of the disclosure, the fifth rule instructs that errors in B or E tags be detected first according to the first rule, and then errors in I tags be detected according to the second rule. In this way, in the BIE structure, according to the rule that B and E tags must appear in pairs, after detecting whether B or E tags are incorrect, it is possible to then detect whether I tags are mistakenly labeled as O tags.

[0156] In this embodiment of the disclosure, the fifth rule instructs that after detecting whether the location label is erroneous according to the first and second rules, the type label should be detected in the sequence label where the location label without error is located. When the location label is detected to be complete according to the first and second rules, it is more effective and meaningful to perform the consistency detection of the type label. In this way, the integrity of the location label can be initially screened by the first and second rules. Entity structures with incomplete location label structures are directly input into the CRF layer for correction. For sequence labels with complete location label structures, performing the type label detection of the third or fourth rule can save more computation time and computational resources.

[0157] In this embodiment of the disclosure, combined with Figure 7As shown, a training method for sequence labeling is provided. The method includes:

[0158] Step S201: Combine a first deep learning model with a CRF layer model to train a data set, obtain first deep learning model parameters and CRF model parameters, discard the obtained first deep learning model parameters, and retain the obtained CRF model parameters.

[0159] Step S203: Use a separate second deep learning model for training to obtain the parameters of the second deep learning model.

[0160] Step S205: Use the obtained CRF layer model parameters and the parameters of the second deep learning model, and add the preset rules in the foregoing sequence labeling method to obtain a trained CRF model with error correction. Among them, the error correction CRF model includes: based on the preset rules in the foregoing sequence labeling method, determine whether the sequence label conforms to the sequence label annotation rules; if the sequence label does not conform to the sequence label annotation rules, then perform correction according to the CRF layer and output the corrected sequence label.

[0161] In the embodiments of the present disclosure, in step S201, the deep learning model can be a bidirectional long short-term memory network Bi-LSTM, a bidirectional gated recurrent unit structure Bi-GRU, a multi-layer convolutional neural network CNN, a multi-layer Bi-LSTM, Transformer, BERT model, Albert model, GPT model, etc., which are used to perform a classification task on the input text. For example, assign a score to various categories that a word may belong to. For example, the word "Zhang" corresponds to a 7-dimensional vector, where the score of the "B-Person" dimension is the highest, indicating that the probability of this word corresponding to this category is higher, and the word "Zhang" will be classified as the "B-Person" category.

[0162] In the embodiments of the present disclosure, the error correction CRF model refers to a model that includes a logical judgment step for determining whether to input a sequence label sequence into the CRF. Based on the foregoing first rule, second rule, third rule, fourth rule, and fifth rule included in the preset rules, determine whether the sequence label contains a sequence label to be corrected by the CRF; if the sequence label does not have a sequence label to be corrected by the CRF, output the corresponding sequence label sequence label sequence label. In this way, the correction efficiency of the CRF layer can be improved, saving the time resources and space resources required for calculation, thereby greatly reducing the number of calculations of the CRF layer, and ensuring the accuracy of the calculation. Finally, the calculation performance of the CRF layer can be greatly improved.

[0163] In this embodiment of the disclosure, the structure of the deep learning model and the linear chain CRF can be a combination of any of the following deep learning models and linear chain CRFs: Bi-LSTM, Bi-GRU (Bi-Gated Recurrent Unit), CNN (Convolutional Neural Network), Bi-LSTM, Transformer, BERT, Albert, and GPT.

[0164] In this embodiment, the CRF is followed by a deep learning model. Compared to the deep learning model, the CRF layer mainly replaces the feature functions with feature vectors. The CRF model followed by the deep learning model is preferably a Linear Chain Conditional Random Field (LCRF). The probability graphical structure of the LCR is as follows: Figure 8 As shown, in Figure 8 In this model, x is the observed variable and y is the labeled variable. The modeling formula is to calculate the conditional probability P(y|x) of y given x.

[0165]

[0166] Among them, s l (y i (x, i) represents the state feature function, which is independent of the preceding and following sequence labels and depends only on x. l represents the total number of state feature functions defined at node x, and i represents the position of node x in the sequence. μ l For s l Weighting coefficients;

[0167] t k (y i-1 ,y i (x, i) represents the transition feature function, k represents the total number of transition feature functions defined at node y, and i represents the position of node y in the sequence. y is related to x and the labels of the preceding and following sequences. λ k For t k Weighting coefficients;

[0168] Z(x) is a factor used for probability normalization.

[0169] In this embodiment, the state feature function and transition feature function are replaced with matrix parameters composed of conditional probabilities by a linear chain CRF layer. The linear chain CRF layer can perform logical corrections on two adjacent predicted label values, so that the i-th input x is related not only to its own position i in the sequence, but also to the observed variable x at the adjacent position. i x i-1 This technology allows for labeling a single character in an input text sequence by considering its context, effectively improving the accuracy of labeling.

[0170] In this embodiment of the disclosure, steps S2011 and S2013 are not sequentially executed.

[0171] In this embodiment, sequence labels obtained by training a dataset using two or more deep learning models are input into an error-correcting CRF layer. The CRF layer can be followed by multiple deep learning models or combinations of deep learning models, which improves the versatility and utilization of the CRF layer when training neural networks. This allows the error-correcting CRF layer, which significantly improves CRF computational efficiency, to be used in a large number of neural network training and applications.

[0172] In conjunction with the above embodiments, this disclosure provides an example of a labeling method:

[0173] Example 1:

[0174] This disclosure belongs to the field of natural language processing, and specifically relates to sequence labeling algorithms for natural language, such as word segmentation, part-of-speech tagging, and named entity recognition. It further relates to a text sequence labeling algorithm using a CRF layer with error correction rules.

[0175] Human knowledge and information are mostly recorded in the form of languages ​​and scripts created by humans, and computers can conveniently and quickly store and record text. However, computers can only transmit and store information; they cannot directly recognize, understand, or use language and scripts. Natural language processing (NLP) is an algorithmic technology for processing human natural language text.

[0176] Sequence labeling is a crucial model processing task in the field of natural language processing. Specifically, sequence labeling tasks mainly involve word segmentation, part-of-speech tagging (POS tagging), named entity recognition, event trigger word extraction, event element extraction, keyphrase extraction, and many other specific model tasks that can be transformed into sequence labeling implementations.

[0177] The essence of sequence labeling is to select a segment of N-grams from text as the result and output it in the form of sequence label tags. Therefore, this invention selects three basic natural language processing tasks—word segmentation, part-of-speech tagging, and entity recognition—as specific tasks for experimentation and demonstration to illustrate the invention solution, which is representative and the conclusions are universal.

[0178] The specific modeling forms for sequence labeling tasks, including word segmentation, part-of-speech tagging, and entity recognition, are as follows:

[0179] 1) Word segmentation: This process divides a sentence (a sequence of characters) into a sequence of words. For example, the sentence "Secretary Zhang is currently employed by the Water Resources Bureau." is broken down into "Secretary Zhang", "currently", "employed", "in", "Water Resources Bureau", and ".".

[0180] 2) Part-of-speech tagging: Assign a part of speech to each word. Such as noun, verb, adjective, etc. In the previous example, "Secretary Zhang" and "Water Resources Bureau" are nouns, "currently" is an adverb, "in office" is a verb, "at" is a preposition, and "." is a punctuation mark.

[0181] 3) Named entity recognition: Extracting specific types of lexical entities from the text. For example, "Secretary Zhang" (type: person's name) and "Water Resources Bureau" (type: organization) in the text, while the rest of the text is not an entity.

[0182] Word segmentation, part-of-speech tagging, and named entity recognition can all be transformed into sequence labeling problems. Sequence labeling achieves segmentation and classification by assigning a label to each unit (character or word, referred to as a token). The most commonly used labeling methods include (BIEOS) tags, such as:

[0183] 1) Word segmentation: “Zhang (B) Shu (I) Ji (E) Mu (B) Qian (E) Ren (B) Zhi (E) Yu (S) Shui (B) Li (I) Ju (E). (B)”, where B represents Begin, I represents Inside, E represents End, and S represents Single. The places where B or S appears are the boundaries. By identifying the boundaries marked by B, S, and E, the words can be extracted: “Zhang Xiaochuan”, “currently”, “in office”, “Yu”, “Shui Li Ju”, “.”.

[0184] 2) Part-of-speech tagging: “Zhang (B-NR) Xiao (I-NR) Chuan (E-NR) currently (B-ADV) serves (BV) at (S-PREP) Water Conservancy (I-NG) Bureau (E-NG). (B-WJ)”, where B in B-NR represents the boundary and NR represents the category, here indicating a noun or personal name. This method distinguishes both the boundary and the category, i.e., the part of speech.

[0185] 3) Entity Recognition: The tag “Zhang Xiaochuan (E-Person) currently (O) works (O) at (O) the Water Resources Bureau (E-Organization)” is similar to part-of-speech tagging, except it includes an additional tag “O” to indicate “Outside,” meaning it's not a category relevant to the task. By processing the tag, the entities “Zhang Xiaochuan” (type: Person) and “Water Resources Bureau” (type: Time) can be extracted.

[0186] In addition, sequence labeling tasks include various other task types such as event subject identification, feature identification, key phrase extraction, and text error detection. The modeling methods for labeling these tasks are highly similar to those for entity recognition. For example, in event recognition, the sentence "Zhang Xiaochuan is currently employed at the Water Resources Bureau." is similar to entity recognition. In this example, the subject, action, and object can be extracted.

[0187] With the development of technology, the number of parameters in deep learning model algorithms has become increasingly large, ranging from millions to hundreds of millions. The direct problem brought about by the huge models is that the efficiency of use is very low. Algorithm researchers and application developers often focus on stacking model structures and measuring parameters, and rarely pay attention to how to add control logic to control the computation time of the model and improve the computation efficiency of the model.

[0188] For example, in the classic Bi-LSTM-CRF model for handling sequence labeling, the LSTM layer is widely criticized for its inability to process sequences directly in parallel due to the autoregressive nature of time-series modeling. Furthermore, the CRF layer, based on the Viterbi algorithm, also requires computation on each sequence individually; it's worth noting that the computational cost of a single CRF layer is significantly higher than that of a single LSTM layer. Therefore, optimizing the CRF layer can lead to significant performance improvements.

[0189] To discuss the relationship between the computational efficiency of deep learning models and the resulting improvement in the F1 score, the following prediction efficiency statistics were performed using a named entity recognition task. This task was trained using 50 million characters of entity recognition annotation data, including three categories: Person, Organization, and Company. The maximum sequence length the model could accept was 100 tokens. During the prediction phase, the batch size was 1000, and the LSTM hidden layer size was 128. The time percentage of each layer in the prediction efficiency (inference time) is shown in Table 1.

[0190]

[0191] Table 1

[0192] As shown in Table 1, firstly, in the Bi-LSTM-CRF model, the main feature fitting occurs in the LSTM layer, while the CRF layer only improves the model's F1 score by 3.9%. Therefore, it can be concluded that in the vast majority of sequence predictions, the CRF layer does not actually modify the sequence labeling results. During the model prediction stage, statistics show that the percentage of sentence sequence labeling results changed after passing through the CRF layer was 8.7%, while the percentage of results remaining completely unchanged after passing through the CRF layer was 91.3%.

[0193] Second, the CRF layer accounts for the largest proportion of time consumption during model prediction (model inference). It should also be noted that while the preprocessing time is exactly the same for each model in the table, their proportions in the overall model prediction time differ.

[0194] Therefore, it can be concluded that while the CRF layer considers global optimization, its computational cost is very high, but its actual improvement on the model's F1 score is very limited. Thus, optimizing the computational efficiency of the CRF layer is a highly valuable task in engineering applications, as it can significantly improve the model's computational efficiency and save time and space resources.

[0195] The labeling method provided in this disclosure includes:

[0196] 1) Analyze the characteristics and process of CRF optimization for input data, and propose label error statistics for CRF layers;

[0197] 2) For data to be input into the CRF layer, determine in advance whether it should be input into the CRF layer.

[0198] The computational effect and function of the CRF layer

[0199] The CRF layer calculation is primarily performed using the label transition probability matrix obtained during model training. The calculation process employs the Viterbi algorithm, performing the calculation one label at a time. An example is shown below:

[0200] The model input sequence is: Zhang Xiaochuan currently works at the Water Resources Bureau.

[0201] After passing through the BI-LSTM layer, the vector matrix corresponding to the sequence is obtained. Probability selection is performed according to Softmax, resulting in the sequence:

[0202] (B-Person)(I-Person)(O)(O)(O)(O)(O)(O)(B-Organiztion)(I-Organiztion)(E-Organiztion)(O)

[0203] The final sequence is obtained after passing through a CRF layer:

[0204] (B-Person)(I-Person)(E-Person)(O)(O)(O)(O)(O)(B-Organiztion)(I-Organiztion)(E-Organiztion)(O)

[0205] In this example, the CRF layer acts as a label corrector, meaning that a B label must be followed by I or E, an I label must be followed by E, and an O label must not be followed by I, etc. Ultimately, this determines the two entity results: "Zhang Xiaochuan" and "Water Resources Bureau".

[0206] From a communication perspective, sequence labeling can be viewed as a communication process. The deep learning model (such as RNN, CNN, Transformer, etc.) can be seen as a channel transmission of the correct sequence label results, which inevitably produces some errors. The CRF layer is equivalent to a channel error checking code, correcting some erroneous codes according to global optimality.

[0207] The most popular algorithm for sequence labeling tasks is currently "deep learning model + CRF":

[0208] Deep learning model section:

[0209] In sequence labeling, the most classic BI-LSTM-CRF model currently uses BI-LSTM, which stands for Bidirectional LSTM Recurrent Neural Network. This is a deep learning model whose task is to assign a score to each unit (taking a character as an example) according to its category, essentially performing a classification task for each character. Figure 1As shown, the character "Zhang" corresponds to a 7-dimensional vector, where the score of the "B-Person" dimension is the highest, indicating that the probability of this character belonging to this category is higher. That is, the character "Zhang" is classified into the "B-Person" category. The Bi-LSTM can be replaced by other models, such as Bi-GRU, multi-layer CNN, multi-layer Bi-LSTM, Transformer and other common feature extractors.

[0210] With the development of technology, Google developed the more powerful BERT model (https: / / arxiv.org / pdf / 1810.04805) in 2018, the Albert model in 2019, and the GPT model developed by OpenAI, etc. Such models use Transformer as the feature extractor, with very deep model layers, strong feature expression capabilities, and much improved problem-solving abilities compared to RNN-based models.

[0211] For the CRF part, specifically referring to the Linear Chain CRF, it is a probabilistic graphical model.

[0212] The probabilistic graphical structure of the linear chain conditional random field is as Figure 8 shown.

[0213] In Figure 8 , x is the observed variable and y is the labeled variable. The modeling formula of this model is to find the conditional probability P(y|x) of y given x. [[ID=第十七]]

[0214]

[0215] Among them, s l (y i , x, i) is the state feature function, which is independent of the labels of the previous and subsequent sequences and only related to x. l represents the total number of state feature functions defined at the x node, i represents the position of the x node in the sequence, and μ l is the weight coefficient of s l ;

[0216] t k (y i-1 , y i , x, i) is the transition feature function, k represents the total number of transition feature functions defined at the y node, i represents the position of the y node in the sequence, y is related to both x and the labels of the previous and subsequent sequences, and λ<00"is the weight coefficient of t k ;

[0217] Z(x) is the factor for probability normalization,

[0218] For deep learning-based models, the CRF layer mainly replaces the feature function with feature vectors.

[0219] In sequence labeling tasks, the main function of the CRF layer is to optimize the relationships between labels and find the label sequence with the highest probability (generally using the Viterbi algorithm for decoding). For example, a B-Person label is unlikely to be followed by an I-Orgnization label or an O label, but it is more likely to be followed by an I-Person label or an E-Person label. After optimization by the CRF layer, the accuracy of sequence labeling is improved.

[0220] As can be seen, the essential function of the CRF layer applied to deep learning models is to perform logical corrections on two adjacent predicted label values. This logic is obtained through statistical training with data.

[0221] The above-mentioned logical correction is essential in the Bi-LSTM-CRF model because logical errors in adjacent labels account for a very large proportion of the total erroneous labels.

[0222] It's worth noting that deep learning models like BERT, which have emerged in the last two years, can largely mitigate logical errors between adjacent labels, such as the error of a B-Person label being followed by an I-Orgnization label. However, these models cannot completely overcome connection errors between sequence labels in specific sequence labeling tasks. Therefore, using CRF layers followed by models like BERT, Albert, and GPT still has value.

[0223] The main process of sequence labeling model is as follows: Figure 9 As shown (taking word segmentation as an example, the same applies to other tasks).

[0224] Step S501: Input a natural language text sequence, such as "Secretary Zhang is currently employed by the Water Resources Bureau", into a model built by various feature extractors to obtain sequence labels and corresponding sequence labels. Feature extractors include, but are not limited to, RNN, Transformer, and CNN. Models built based on feature extractors include, but are not limited to, Bi-LSTM, Bert, Albert, and GPT.

[0225] Step S502: Input sequence labels and corresponding sequence labels into the CRF layer to obtain corrected sequence labels, for example: (B)(I)(E)(B)(E)(B)(E)(S)(B)(I)(E)(B).

[0226] Statistical metrics for model training in sequence labeling tasks primarily use the F1 score for evaluation, which balances prediction precision and recall. These will not be described in detail here.

[0227] CRF layer label logic rules:

[0228] The sequence labeling results of the CRF layer are generally executed according to the BIOES labeling system. For example, let a label be B-Person, where B is the position label, indicating that it is the first token of the person entity, and Person is the category label, indicating that the entity is a person's name. The sequence labeling logic can be briefly summarized as follows:

[0229] B+nI+E is the most common positional label structure, where n is a non-negative integer. Category labels within the same positional label structure must be identical.

[0230] Given that there is no S tag in the sequence result (i.e., one token represents one entity), the category tag cannot appear in an isolated sequence tag; there must be the same type tag before and after it.

[0231] The S tag and the O tag form a separate positional structure.

[0232] The S-structure, O-structure, and BIE-structure appear in sequence.

[0233] Error statistics and examples without CRF layer

[0234] For sequence labeling models without CRF layers, a Bi-LSTM model is used to train a named entity recognition model. The differences between the labeling results and the real labels are statistically analyzed, and the following are the main typical label logic errors. The left side is the real label labeled by humans, and the right side is the label predicted by the model training.

[0235] First, the BIE structure is damaged, such as:

[0236]

[0237]

[0238] as well as:

[0239]

[0240] Second, in the BIE structure, inconsistent type tags are not allowed. For example, in the following example, there is a separate "Company" type tag, which is not permitted:

[0241]

[0242] The following table shows the percentages of all logical errors, including the two types mentioned above, as well as errors that are logically correct but inconsistent with the annotation results:

[0243]

[0244]

[0245] Table 2

[0246] For entity recognition, the data statistics in the table match the F1 score obtained from model training. If the logical errors can be successfully corrected using a CRF layer, the F1 score can be improved from 91.4% to 95.3%. For word segmentation, since there are no category labels, the problem can be further simplified. For part-of-speech tagging, there are 4.9% other errors, specifically referring to joint errors caused by the combined effect of position labels and categories, which will be discussed later.

[0247] It should be noted that, apart from the three sequence labeling tasks mentioned above, other sequence labeling tasks such as event element extraction have similar statistical characteristics and results to entity recognition, and will not be elaborated here.

[0248] CRF layer error detection method

[0249] After the model passes through the Bi-LSTM layer, the softmax layer outputs the sequence label results. These are then processed using the following error detection method: if errors are found, the CRF layer is used for sequence correction; otherwise, the result is directly output as the final result. The process flow is as follows: Figure 10 As shown.

[0250] Step S601: Input a natural language text sequence, such as "Secretary Zhang is currently employed by the Water Resources Bureau", into a model built by various feature extractors to obtain sequence labels and corresponding sequence labels. Feature extractors include, but are not limited to, RNN, Transformer, and CNN. Models built based on feature extractors include, but are not limited to, Bi-LSTM, Bert, Albert, and GPT.

[0251] Step S602: Determine whether the sequence label conforms to the logic of the CRF layer according to logic rules 1, 2, 3 or 5, 4; for sequence labels that conform to the CRF layer, output them directly; for sequence labels that do not conform to the CRF layer, input them to the CRF layer.

[0252] Step S603: Input sequence labels and corresponding sequence labels into the CRF layer to obtain corrected sequence labels, for example: (B)(I)(E)(B)(E)(B)(E)(S)(B)(I)(E)(B).

[0253] in, Figure 10 The judgment logic and corresponding CRF layer in this solution are referred to as the "error correction CRF layer". The specific rules mainly include:

[0254] Rule 1: Set O = 0, I = 0, B = 1, E = -1. Sum the sequence values. If the sum is 0, the sequence is valid; if it is not 0, it is definitely incorrect. This rule can detect errors in B or E within the BIE structure.

[0255] Rule 2: Let B = 0, O = -1, E = 0, I = 1. Calculate the sequence x(n) + x(n-1). If any element in the sequence is 0, then the sequence is definitely incorrect. This rule can detect errors in the I parameter of the BIE structure.

[0256] Rule 3: If there are M type labels in a sequence, assign values ​​from 1 to M to each of the M labels. Filter out the S labels in the sequence. If any remaining sequence contains labels whose values ​​before and after the value m are not m, then the sequence is definitely incorrect. This rule can detect inconsistencies in category labels.

[0257] Rule 4: The above three rules must be performed in sequence for testing.

[0258] Note: Rules 1 to 4 above cannot guarantee 100% error detection, for example, the following part-of-speech tagging sequence:

[0259]

[0260] This shows that the labeling of the word "interconnected" is incorrect. While the location and category labels are correct, the combination of these two labels reveals a problem. This type of issue is more common in part-of-speech tagging tasks, appearing as "other types of errors" in the error type statistics table, but is almost nonexistent in entity recognition tasks. Therefore, in part-of-speech tagging tasks, rule 3 needs to be replaced with rule 5.

[0261] Rule 5: Based on Rules 1 and 2, assuming the position labels are correct, segment the sequence labels according to three structures: BIE, S, and O (O labels are not used in part-of-speech tagging tasks). For each structure, if the number of category labels is greater than 1, the sequence is definitely incorrect. This rule can detect inconsistent category labels. While slower than Rule 3, it is particularly suitable for part-of-speech tagging tasks.

[0262] It should be noted that the rules that conform to CRF tag logic are not limited to the above five rules. Any rule that aims to satisfy CRF tag logic rules falls within the framework of this invention.

[0263] In conjunction with the above embodiments and examples, this disclosure also provides an example 2 of a training method.

[0264] Example 2:

[0265] Step 1: Train the model using a deep learning model + CRF structure (such as Bi-LSTM-CRF) to obtain the transition probability matrix parameters of the CRF layer;

[0266] Step 2: Remove the CRF layer and retrain the model (e.g., Bi-LSTM) using labeled data to obtain the Bi-LSTM parameters;

[0267] Step 3: Fill the CRF layer parameters obtained in Step 1 and the Bi-LSTM parameters obtained in Step 2 into the model of the de-corrected CRF layer. Based on this, add the aforementioned CRF layer encoding logic error detection algorithm (5 rules for verification).

[0268] Step 4: In the prediction and inference stage, if no logical error in the sequence label is detected, the result is output directly; otherwise, the transition probability matrix obtained in the first step is used to calculate the CRF layer to obtain the error-corrected sequence result.

[0269] The application effects of Examples 1 and 2:

[0270] The preset rules of Example 1 and Example 2 were applied to the model prediction stage of named entity recognition. The F1 score and the percentage of computation time are shown in Table 3 below:

[0271]

[0272]

[0273] Table 3

[0274] As shown in Table 3, the processing time for preprocessing is consistent across all models, but the proportions differ. The processing speed using the error-correcting CRF layer is significantly faster than that of the non-error-correcting CRF layer, with a speed improvement of approximately 181% for the single-layer Bi-LSTM model. It should also be noted that because this set of rules uses rule 3 but not rule 5 to overcome the minimal error caused by the combined effect of position and category labels, the F1 score is approximately 0.2% lower than that of the model without the error-correcting CRF layer.

[0275] The preset rules of Example 1 and Example 2 were applied to the model prediction stage of the word segmentation task. The F1 score and the percentage of computation time are shown in Table 4 below:

[0276]

[0277] Table 4

[0278] The preset rules of Example 1 and Example 2 were applied to the model prediction stage of the part-of-speech tagging task. The F1 score and computation time percentage are shown in Table 5 below:

[0279]

[0280]

[0281] Table 5

[0282] The deep learning model is the BERT model, combined with the CRF model for entity recognition tasks. The F1 score and computation time percentage during the prediction phase are shown in Table 6 below.

[0283]

[0284] Table 6

[0285] The data analysis in the table above shows that, in the computation of different models and tasks, the improved CRF layer computation using this method significantly enhances the prediction (inference) speed of the model while maintaining consistent accuracy; simultaneously, it saves computational resources and improves computational power. This invention demonstrates a very significant improvement in model processing performance in practical engineering applications.

[0286] Furthermore, sequence labeling tasks are not limited to word segmentation, part-of-speech tagging, and entity recognition; they also include various other task types such as event subject identification, feature identification, key phrase extraction, and text error detection. Analysis shows that the labeling logic rules for all current sequence labeling tasks can be reduced to the aforementioned CRF layer labeling logic rules. These natural language processing tasks have similar processing flows to entity recognition, and therefore will not be illustrated here. Thus, this method is universally applicable to various natural language processing sequence labeling tasks.

[0287] This disclosure proposes a judgment method for input data of the CRF layer, namely an error-correcting CRF model. This model fully considers the trade-off and comparison between the computational efficiency and effectiveness of the CRF layer, ensuring that data that already conforms to the output pattern of the CRF layer does not need to enter the CRF layer for computation, thus consuming computational resources and significantly accelerating computational efficiency.

[0288] This disclosure introduces a pattern checking and error correction method conforming to label standards to determine whether to proceed to the CRF layer calculation. This method considers most cases of label logic errors, enabling logical errors to largely cover factual errors and maintaining the consistency of model metrics. However, it should be noted that the error correction rules described in this invention are merely one description of CRF layer label logic rules; any rules that are transformed but still fall under the CRF layer label logic description are still applicable to the scope of this invention.

[0289] In this embodiment of the disclosure, combined with Figure 11 As shown, a labeling device 300 is provided, the device comprising:

[0290] Natural language processing module 301 is used to process natural language based on a deep learning model and assign sequence labels to each word unit; wherein, the word unit includes: at least one character or at least one word;

[0291] The determining module 302 is used to determine whether the sequence label conforms to the sequence label rules based on preset rules;

[0292] The output module 303 is used to send the sequence label into a conditional random field (CRF) for correction if the sequence label does not meet the sequence label rules.

[0293] In this embodiment of the disclosure, combined with Figure 12 As shown, a neural network training device 400 is provided, the device 400 comprising:

[0294] The first training module 401 is used to train the dataset by combining the first deep learning model with the CRF layer model, to obtain the parameters of the first deep learning model and the parameters of the CRF model, discard the obtained first deep learning model parameters, and retain the obtained CRF model parameters.

[0295] The second training module 403 is used to train a separate second deep learning model to obtain the parameters of the second deep learning model.

[0296] The error-correcting CRF model determination module 405 is used to obtain a trained CRF model with error correction by using the obtained CRF layer model parameters and the second deep learning model parameters, and adding the preset rules in the aforementioned sequence labeling method; wherein, the error-correcting CRF model includes: determining whether the sequence label conforms to the sequence labeling rules based on the preset rules in the aforementioned sequence labeling method; if the sequence label does not conform to the sequence labeling rules, then correcting it according to the CRF layer and outputting the corrected sequence label;

[0297] Module 407 is used to input the text to be labeled into a sequence labeling model with error correction CRF that has been trained, and to obtain the sequence results.

[0298] In this embodiment of the disclosure, an electronic device is provided, the electronic device comprising:

[0299] processor;

[0300] Memory used to store processor-executable instructions;

[0301] When the processor is used to run the computer service, it implements the steps in the feedback method described above.

[0302] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0303] In this embodiment of the disclosure, a storage medium is provided, the storage medium having computer-executable instructions, which are executed by a processor to implement the steps in the feedback method described above.

[0304] Alternatively, if the integrated units described above in the embodiments of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0305] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A sequence labeling method, characterized in that, The method includes: Natural language is processed using a deep learning model, and a sequence label is assigned to each word unit; wherein, the word unit includes: at least one character or at least one word; the sequence label includes: a position label, indicating the position of the components in the word; Based on preset rules, determine whether the sequence label conforms to the sequence label rules; If the sequence label does not conform to the sequence label rule, the sequence label is sent to a linear conditional random field (CRF) for correction. The step of determining whether the sequence label conforms to the sequence label rules based on preset rules includes: According to the preset rules, the position tags are detected to determine whether there are any erroneous position tags in the sequence tags; if the sequence tags contain at least one erroneous position tag, it is determined that the sequence tags do not conform to the sequence tag rules. The step of detecting whether there is an erroneous position label in the sequence label obtained by the position label according to the preset rule includes: According to the first rule of integrity verification in the preset rules, the B tag indicating the entity head is assigned the value N, the E tag indicating the entity tail is assigned the value -N, the I tag indicating the entity middle position is assigned the value 0, and the O tag that is irrelevant to the task is assigned the value 0; wherein, the entity refers to the task extraction target in the sequence labeling task. After assigning values, perform addition operations to obtain the result. If the result of the operation is not 0, then the B label or the E label in the location label is incorrect.

2. The sequence labeling method according to claim 1, characterized in that, The sequence tag also includes: Type label, indicating the target type label for the sequence labeling task.

3. The sequence labeling method according to claim 2, characterized in that, The step of determining whether the sequence label conforms to the sequence label rules based on preset rules further includes: According to the preset rules, the type label is detected to determine whether there is an erroneous type label in the sequence label; If the sequence label contains at least one erroneous position label and / or at least one erroneous type label, the sequence label is determined to be inconsistent with the sequence label rules.

4. The sequence labeling method according to claim 1, characterized in that, The step of detecting whether there is an erroneous position label in the sequence label obtained by the position label according to the preset rule includes: According to the second rule of integrity correction in the preset rules, the B tag indicating the entity head is assigned a value of 0, the E tag indicating the entity tail is assigned a value of 0, the I tag indicating the middle position of the entity is assigned a value of M, and the O tag that is not of interest to the task is assigned a value of -M. After assigning values, summing any two adjacent terms yields multiple results. If any of the multiple operation results is 0, then the sequence label includes the error location label.

5. The sequence labeling method according to claim 3, characterized in that, The step of detecting whether there is an erroneous type label in the sequence label obtained by the type label according to the preset rule includes: According to the third rule of category consistency correction in the preset rules, the S tags corresponding to single words in the sequence tags are filtered to obtain the filtered sequence tags; In the sequence tags after filtering S tags, if the type tag of the Kth sequence tag is different from the type tags of the (K-1)th and (K+1)th sequence tags, then it is determined that there is an erroneous type tag in the sequence tags.

6. The sequence labeling method according to claim 3, characterized in that, The step of detecting whether there is an erroneous type label in the sequence label obtained by the type label according to the preset rule includes: According to the fourth rule of category consistency correction in the preset rules, the tags in the sequence tags are divided into BIE structures indicating entities, S tag structures indicating single words, and O tag structures indicating task-independent information. Determine the number of types of type tags in each segmented structure of the sequence tag. If the number of types of type tags in a structure is greater than 1, then it is determined that there are erroneous type tags in the sequence tag.

7. A training method for sequence labeling, characterized in that, The method includes: The dataset is trained by combining the first deep learning model with the CRF layer model to obtain the parameters of the first deep learning model and the CRF model. The obtained parameters of the first deep learning model are discarded, and the obtained parameters of the CRF model are retained. The parameters of the second deep learning model are obtained by training it using a separate second deep learning model. Using the obtained CRF layer model parameters and the second deep learning model parameters, and adding the preset rules of any one of claims 1 to 6, a trained CRF model with error correction is obtained; wherein, the error correction CRF model includes: determining whether the sequence label conforms to the sequence label rules based on the preset rules of any one of claims 1 to 6; if the sequence label does not conform to the sequence label rules, then correcting it according to the CRF layer and outputting the corrected sequence label; The text to be labeled is input into a sequence labeling model with error correction CRF trained to obtain the sequence result.

8. A sequence labeling apparatus, characterized in that, The device includes: The natural language processing module is used to process natural language based on a deep learning model and assign sequence labels to each word unit; wherein, the word unit includes: at least one character or at least one word; the sequence label includes: a position label, indicating the position of the components in the word; The determination module is used to determine whether the sequence label conforms to the sequence label rules based on preset rules; The correction module is used to send the sequence label to a Conditional Random Field (CRF) for correction if the sequence label does not conform to the sequence label rule. The determining module is further configured to detect whether there is an erroneous position label in the sequence label obtained by the position label according to the preset rule; and to determine that the sequence label does not conform to the sequence label rule when the sequence label contains at least one erroneous position label. The determining module is further configured to, according to the first rule of integrity verification in the preset rules, assign the value N to the B tag indicating the entity head, assign the value -N to the E tag indicating the entity tail, assign the value 0 to the I tag indicating the entity middle position, and assign the value 0 to the O tag that is irrelevant to the task; wherein, the entity refers to the task extraction target in the sequence labeling task; after assigning values, a summation operation is performed to obtain the operation result; if the operation result is not 0, then the B tag or the E tag in the position label is incorrect.

9. A sequence labeling training device, characterized in that, The device includes: The first training module is used to train the dataset by combining the first deep learning model with the CRF layer model to obtain the parameters of the first deep learning model and the CRF model, discard the obtained first deep learning model parameters, and retain the obtained CRF model parameters. The second training module is used to train a separate second deep learning model to obtain the parameters of the second deep learning model. The error-correcting CRF model determination module is used to utilize the obtained CRF layer model parameters and the second deep learning model parameters, and add the preset rules of any one of claims 1 to 6 to obtain a trained CRF model with error correction; wherein, the error-correcting CRF model includes: determining whether the sequence label conforms to the sequence label rules based on the preset rules of any one of claims 1 to 6; if the sequence label does not conform to the sequence label rules, then correcting it according to the CRF layer and outputting the corrected sequence label; The module is used to input the text that needs to be labeled into a sequence labeling model with error correction CRF that has been trained, and to obtain the sequence results.

10. An electronic device, characterized in that, The electronic device includes: Memory; A processor, connected to the memory, for executing computer instructions stored in the memory, capable of implementing the method according to any one of claims 1 to 6 or 7.

11. A computer storage medium, characterized in that, The medium stores computer-executable instructions; when executed by a processor, the computer-executable instructions can implement the method described in any one of claims 1 to 6 or 7.

Images