Sequence tagging implementation method and apparatus in natural language processing, and storage medium

Abstract

Provided in the embodiments of the present application are a sequence tagging implementation method and apparatus in natural language processing, and a computer-readable storage medium. The method comprises: for each sub-tag in sequence tagging, using a current first weighting coefficient and a current first offset coefficient, which correspond to the sub-tag, to perform feature transformation on a vector sequence, so as to obtain a first depth feature vector corresponding to the sub-tag; using a current second weighting coefficient and a current second offset coefficient, which correspond to the sub-tag, to perform feature transformation on a coding vector sequence, so as to obtain a second depth feature vector corresponding to the sub-tag; and for each sub-tag, on the basis of the first depth feature vector of the sub-tag and the second depth feature vector of the sub-tag, performing calculation to obtain an initial sequence tagging score of the sub-tag. By means of the technical solution in the embodiments of the present application, sequence tagging can be implemented.

Description

Method, device and storage medium for implementing sequence tagging in natural language processing Technical Field

[0001] The present application relates to the field of industrial technology, and in particular to a method, device, and computer-readable storage medium for implementing sequence annotation in natural language processing.

[0002] Background of the Invention

[0003] In the field of natural language processing (NLP), many tasks can be solved by converting input language sequences into labeled sequences. Examples include named entity recognition, part-of-speech tagging, and punctuation tagging. Named entity recognition involves identifying names of people, places, and organizations within a sentence. Part-of-speech tagging identifies the parts of speech of words within a given input sequence. Punctuation tagging identifies the position and type of punctuation marks within a sentence.

[0004] The sequence labeling problem can be considered a generalization of the classification problem, or a simpler form of the more complex structure prediction problem. The input of a sequence labeling problem is an observation sequence, and the output is a labeled sequence or state sequence. The goal of the problem is to learn a model that can predict a labeled sequence for the observation sequence.

[0005] Currently, pre-training + fine-tuning is the basic paradigm for NLP tasks and is widely used in various tasks, including the sequence labeling task mentioned above. Specifically, this basic paradigm is to pre-train on a general corpus and fine-tune on a corpus that is highly relevant to the downstream task, so that the pre-trained language model (PLM) can quickly adapt to the downstream task.

[0006] Among them, the pre-training process is relatively mature, but different fine-tuning processes will have different effects on the final sequence labeling accuracy. Therefore, technicians in this field are still committed to finding various implementation solutions for sequence labeling.

[0007] Summary of the Invention

[0008] In view of this, the embodiments of the present application provide, on the one hand, a method for implementing sequence labeling in natural language processing, and on the other hand, a device for implementing sequence labeling in natural language processing and a computer-readable storage medium, which can implement sequence labeling in natural language processing.

[0009] To solve the above technical problems, the technical solution of this application is implemented as follows:

[0010] A method for implementing sequence labeling in natural language processing, comprising: generating a plurality of sentence training corpora from unprocessed downstream task corpora; for a current sentence training corpus, inputting the current sentence training corpus into a pre-training model to encode the sentence training corpus and obtain a vector sequence; the pre-training model is obtained by pre-training in a general corpus; for each sub-label in the sequence labeling, performing feature transformation on the vector sequence using the current first weighting coefficient and the current first offset coefficient corresponding to the sub-label to obtain a first deep feature vector corresponding to the sub-label; performing feature transformation on the encoded vector sequence using the current second weighting coefficient and the current second offset coefficient corresponding to the sub-label to obtain a second deep feature vector corresponding to the sub-label; for For each sub-tag, based on the first deep feature vector of the sub-tag and the second deep feature vector of the sub-tag, an initial sequence labeling score of the sub-tag is calculated; the initial sequence labeling scores of each sub-tag of each recognition unit of the sentence training corpus are normalized to obtain the training probability distribution of each sub-tag; a loss value is calculated according to the true probability distribution of each sub-tag of each recognition unit of the sentence training corpus and the training probability distribution of each sub-tag; before the training end condition is reached, the parameters of the pre-training model and the first weighting coefficient, the first offset coefficient, the second weighting coefficient and the second offset coefficient of each sub-tag are adjusted according to the loss value, and the operation of inputting the current sentence training corpus into a pre-training model is returned to be executed for the current sentence training corpus.

[0011] In one embodiment, the sequence labeling is identified based on a single word in a sentence; for each sub-tag, the initial score of the sequence labeling of the sub-tag is calculated based on the first deep feature vector of the sub-tag and the second deep feature vector of the sub-tag, including: for each sub-tag, the inner product of the first deep feature and the second deep feature of each word is used as the initial score of the sub-tag of the word.

[0012] In one embodiment, the sequence annotations identified based on individual words in a sentence include punctuation mark annotations.

[0013] In one embodiment, the sequence labeling is identified in units of fields in a sentence; for each sub-tag, the initial score of the sequence labeling of the sub-tag is calculated based on the first deep feature vector of the sub-tag and the second deep feature vector of the sub-tag, including: for each sub-tag, the inner product of the first deep feature of the first word in each field and the second deep feature of the last word in the field is used as the initial score of the sub-tag of the field.

[0014] In one embodiment, the sequence tagging that is performed based on fields in a sentence includes named entity recognition.

[0015] In one embodiment, the method further includes: using the currently acquired limited downstream task corpus to train an anomaly detection model; extracting the current sentence prediction corpus from the general corpus; using the current sentence prediction corpus as the input of the anomaly detection model to obtain the anomaly score output by the anomaly detection model; when the anomaly score is less than a set threshold, using the current sentence prediction corpus as the downstream task corpus; when the number of downstream task corpora reaches the requirement, executing the operation of generating multiple sentence training corpora from the unprocessed downstream task corpus; otherwise, returning to executing the operation of extracting the current sentence prediction corpus from the general corpus.

[0016] In one embodiment, the loss value is calculated according to the following loss function:

[0017] Among them, C represents the number of labels, m is the number of units that need to be labeled in the sentence training corpus; i represents the i-th unit that needs to be labeled in the sentence training corpus; y and They are all vectors, y represents the true probability distribution of the sub-labels of the sentence training corpus, Represents the training probability distribution of sub-labels of the sentence training corpus.

[0018] A sequence tagging implementation device in natural language processing includes: a first module for generating multiple sentence training corpora from unprocessed downstream task corpora; a second module for inputting the current sentence training corpora obtained into a pre-training model to encode the sentence training corpora and obtain a vector sequence; the pre-training model is obtained by pre-training in a general corpus; a third module for performing feature transformation on the vector sequence for each sub-label in the sequence tagging using the current first weighting coefficient and the current first offset coefficient corresponding to the sub-label to obtain a first deep feature vector corresponding to the sub-label; and performing feature transformation on the encoded vector sequence using the current second weighting coefficient and the current second offset coefficient corresponding to the sub-label to obtain a vector sequence corresponding to the sub-label. a second deep feature vector of the label; a fourth module, for calculating, for each sub-label, the initial sequence labeling score of the sub-label based on the first deep feature vector of the sub-label and the second deep feature vector of the sub-label; a fifth module, for normalizing the initial sequence labeling scores of the sub-labels of each recognition unit of the sentence training corpus to obtain the training probability distribution of each sub-label; a sixth module, for calculating a loss value according to the true probability distribution of each sub-label of each recognition unit of the sentence training corpus and the training probability distribution of each sub-label, and before reaching the end condition of training, adjusting the parameters of the pre-training model and the first weighting coefficient, the first offset coefficient, the second weighting coefficient and the second offset coefficient of each sub-label according to the loss value, and instructing the second module to continue to execute the operation.

[0019] In one embodiment, the sequence tagging is performed based on a single word in a sentence; and the fourth module uses, for each sub-tag, the inner product of the first deep feature and the second deep feature of each word as the initial score of the sub-tag of the word.

[0020] In one embodiment, the sequence labeling is identified in units of fields in a sentence; and the fourth module uses, for each sub-label, the inner product of the first deep feature of the first word in each field and the second deep feature of the last word in the field as the initial score of the sub-label of the field.

[0021] In one embodiment, it further includes: a seventh module, which is used to train an anomaly detection model using the currently acquired limited downstream task corpus; an eighth module, which is used to extract the current sentence prediction corpus from the general corpus; a ninth module, which is used to use the current sentence prediction corpus as the input of the anomaly detection model to obtain the anomaly score output by the anomaly detection model; a tenth module, which is used to use the current sentence prediction corpus as the downstream task corpus when the anomaly score is less than a set threshold; when the number of downstream task corpora reaches the requirement, notify the first module to perform the operation; otherwise, notify the eighth module to perform the operation.

[0022] A sequence tagging implementation device in natural language processing includes at least one memory and at least one processor, wherein: the at least one memory is used to store a computer program; the at least one processor is used to call the computer program stored in the at least one memory to execute the sequence tagging implementation method in natural language processing as described above.

[0023] A computer-readable storage medium having a computer program stored thereon; the computer program can be executed by a processor and implement the sequence tagging implementation method in natural language processing as described above.

[0024] It can be seen from the above technical solution that in the embodiment of the present application, since the probability distribution of the label is calculated based on the deep feature value of each label of each prediction unit in the sentence training corpus, the prediction accuracy of each prediction unit, especially the prediction unit with a combination relationship, can be obtained by performing a combination operation on the deep feature value of each label, thereby better realizing sequence labeling.

[0025] In addition, by using the currently acquired limited downstream task corpus to train an anomaly detection model, we can further obtain corpus belonging to the downstream task corpus in the general corpus, which can expand the number of downstream task corpus and further improve the prediction accuracy of sequence labeling.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to better understand the present application, the embodiments of the present application will be described in detail below with reference to the accompanying drawings so that those skilled in the art can better understand the above and other features and advantages of the present application. In the accompanying drawings:

[0028] FIG1 is a schematic diagram of a method for implementing sequence labeling in an application.

[0029] FIG2 is an exemplary flowchart of a method for implementing sequence labeling in natural language processing in an embodiment of the present application.

[0030] FIG3 is an exemplary flowchart of a method for acquiring downstream task corpus in an embodiment of the present application.

[0031] FIG4 is an exemplary structural diagram of a sequence tagging implementation device in natural language processing according to an embodiment of the present application.

[0032] FIG5 is an exemplary structural diagram of another apparatus for implementing sequence labeling in natural language processing according to an embodiment of the present application.

[0033] The accompanying drawings are numerals as follows:

[0034] Methods of implementing this application

[0035] Figure 1 shows a schematic diagram of a method for implementing sequence labeling in an application. As shown in Figure 1, the method includes the following processing steps:

[0036] Step S11: Generate multiple sentence training corpora from the unprocessed downstream task corpora.

[0037] Step S12: For the current sentence training corpus, the current sentence training corpus is input into a pre-training model to encode the sentence training corpus and obtain a vector sequence. The pre-training model is obtained by pre-training in a general corpus.

[0038] Step S13: After processing the vector sequence based on a dense layer, an initial sequence annotation score of each sub-label of the sentence training corpus is obtained.

[0039] Step S14: normalize the initial sequence annotation scores of the sub-tags to obtain the training probability distribution of the sub-tags of the current sentence training corpus.

[0040] Step S15, calculate a loss value based on the true probability distribution of each sub-label of each recognition unit of the sentence training corpus and the training probability distribution of each sub-label. Before the training end condition is reached, adjust the parameters of the pre-training model and the dense layer parameters according to the loss value, and return to execute step S12.

[0041] The above process has no advantage in the accuracy of sequence annotation for identifying fields in a sentence. Therefore, in order to improve the accuracy of sequence annotation for identifying fields in a sentence, a method for calculating label scores based on deep feature values ​​is considered in the embodiment of the present application.

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the technical solutions of this application are described in detail below with reference to the accompanying drawings and examples.

[0043] FIG2 is an exemplary flow chart of a method for implementing sequence tagging in natural language processing according to an embodiment of the present application. As shown in FIG2 , the method may include the following steps:

[0044] Step S21: Generate multiple sentence training corpora from the unprocessed downstream task corpora.

[0045] Step S22: training the current sentence corpus, for example, a sentence training corpus of length n [x1, x2, ..., x n ], input the current sentence training corpus into a pre-training model to encode the sentence training corpus and obtain a vector sequence [h1,h2,...,h n The pre-training model is obtained by pre-training in a general corpus.

[0046] Step S23: for each sub-tag t in the sequence tag, t=1, 2, ..., m, use the current first weighting coefficient W corresponding to the sub-tag qt and the current first offset coefficient b qt Perform feature transformation q on the vector sequence it =W qt h i +b qt , get the first depth feature vector q corresponding to the sub-label it ; Using the current second weighting coefficient W corresponding to the sub-tag kt and the current second offset coefficient b kt Perform feature transformation k on the encoding vector sequence it =W kt h i +b kt , get the second depth feature vector k corresponding to the sub-label it .

[0047] Step S24: for each sub-label t, the first depth feature vector q of the sub-label is calculated based on the first depth feature vector q of the sub-label. it and the second deep feature vector k of the sub-label it , calculate and obtain the initial score of the sequence annotation of the sub-label t.

[0048] In this step, the first depth feature vector q based on the sub-label it and the second deep feature vector k of the sub-label it There are many specific implementations for calculating the initial score of the sequence annotation of the sub-tag. Two of them are briefly introduced below.

[0049] The first type: If the sequence annotation is performed based on a single word in a sentence, then for each sub-label t, t = 1, 2, ..., m, the first depth feature q of each word i is it and the second depth feature k it The inner product s i =q it k it As the initial score of the sub-tag of the word, the initial score of the sequence annotation of the sub-tag of the sentence training corpus is obtained.

[0050] The second type: If the sequence annotation is performed based on the field in the sentence, then for each sub-tag t, t = 1, 2, ..., m, the first depth feature q of the first word i in each field is it and the second deep feature k of the last word j in the field jt The inner product s ij =q it k jt As the initial score of the sub-tag of the field, the initial score of the sequence annotation of the sub-tag of the sentence training corpus is obtained.

[0051] Of course, there are other ways to implement sequence tagging based on the fields in a sentence, such as using the second deep feature k of the first word i in each field as the it and the first deep feature q of the last word j in the field jt The inner product s ji =q jt k it As the initial score of the subtag of the field.

[0052] Step S25 , normalizing the initial sequence annotation scores of each sub-label of each recognition unit of the sentence training corpus to obtain a training probability distribution of each sub-label.

[0053] In this step, the initial sequence annotation scores of each sub-label can be normalized using a softmax function or the like.

[0054] Step S26, calculate a loss value based on the true probability distribution of each sub-label of each recognition unit of the sentence training corpus and the training probability distribution of each sub-label. Before the training end condition is reached, adjust the parameters of the pre-training model and the first weighting coefficient, first offset coefficient, second weighting coefficient and second offset coefficient of each sub-label according to the loss value, and return to execute step S22.

[0055] In this step, the loss value can be calculated according to the following loss function (1):

[0056] Among them, C represents the number of labels, m is the number of units that need to be labeled in the sentence training corpus; i represents the i-th unit that needs to be labeled in the sentence training corpus; y and They are all vectors, y represents the true probability distribution of the sub-labels of the sentence training corpus, Represents the training probability distribution of sub-labels of the sentence training corpus.

[0057] Furthermore, parameter adjustments can be performed through backpropagation with the help of optimizers and gradient descent. Furthermore, training can end when a preset number of training cycles has been reached, when the loss value meets the required value, or when the model's performance is constantly monitored and terminated early when the required value is met.

[0058] Sentence boundary disambiguation (SBD), also known as sentence boundary detection or punctuation prediction, is a rule-based sentence boundary detection technique and a key foundational task in natural language processing (NLP). In many NLP pipelines, SBD results often serve as the primary input for downstream NLP tasks, such as sentence similarity, sentence classification, machine translation, named entity recognition, and entity linking. Therefore, a good SBD directly improves the performance of the final task. However, in real-world scenarios, text often comes in a variety of input formats. For example, text in formats like HTML and PDF often lacks clear sentence boundaries after parsing. Furthermore, human-edited text, such as Weibo posts and e-commerce reviews, often contains numerous punctuation errors. This significantly impacts all NLP tasks that rely on sentences as input. Furthermore, automatic speech recognition (ASR) results lack punctuation. SBD is essential for subsequent processing based on ASR results.

[0059] To this end, in this embodiment, the sequence annotation for identifying individual words in a sentence is taken as an example, and the solution in this application is verified by taking punctuation mark annotation as an example.

[0060] This experiment collected 500 million sentences from the Chinese Wikipedia corpus and generated sentences ranging in length from 30 to 120. These sentences were then automatically labeled based on the presence of commas, periods, and question marks in the original sentences. The sequence annotations now have four labels indicating the probability distribution of "comma," "period," "question mark," and "none of the three." A SBD comparison experiment was conducted, and the results are shown in Table 1 below. P represents precision, and R represents recall. P, R, and F1 are performance metrics.

[0061] Table 1

[0062] As shown in the table above, the performance of this solution applied to SBD is generally acceptable. Some prediction errors for commas and periods are primarily due to long English sentences mixed with Chinese and consecutive numbers. The worst performance, for question mark prediction, is due to too few training samples. Incorporating sample imbalance techniques should alleviate this problem.

[0063] The solution in this application was verified using named entity recognition as an example of sequence annotation for identifying fields within a sentence. The sequence annotation can have seven labels indicating the probability distribution of "place name starting character," "place name ending character," "person name starting character," "person name ending character," "organization name starting character," "organization name ending character," and "none of these."

[0064] This experiment compared the technical solution in this application with the BERT+Softmax solution on the China People's Daily corpus. The results are shown in Table 2 below.

[0065] Table 2

[0066] The results show that the technical solution in this application outperforms the Bert+softmax method in all indicators. This is because, firstly, the technical solution in this application can treat entity fragments as a whole to calculate the loss, which is consistent with the entity as the prediction unit during prediction. Secondly, for sentence training corpus with nested entities. For example, "Peking University" contains two entities, "Beijing" and "Peking University", the technical solution in this application can identify these two entities, while Bert+softmax can only identify one entity.

[0067] Furthermore, under the pre-training and fine-tuning paradigm, considering that the corpus highly relevant to the downstream task may be too small to support good adjustments of the PLM, in order to make our solution more perfect and domain-adaptive, in the embodiment of this application, before generating multiple sentence training corpora from the unprocessed downstream task corpus, the downstream task corpus acquisition method shown in FIG3 is further performed. This method may include the following processing:

[0068] S31, using the currently acquired limited downstream task corpus to train an anomaly detection model.

[0069] S32, extracting the current sentence prediction corpus from the general corpus.

[0070] S33: Using the current sentence prediction corpus as the input of the anomaly detection model to obtain an anomaly score output by the anomaly detection model.

[0071] S34, when the abnormal score is less than the set threshold, the current sentence prediction corpus is used as the downstream task corpus; when the number of downstream task corpora reaches the requirement, the operation of generating multiple sentence training corpora from the unprocessed downstream task corpus in step S21 is executed; otherwise, return to execute step S32.

[0072] The above describes in detail the sequence labeling method for natural language processing according to an embodiment of the present invention. The following describes in detail the apparatus for implementing sequence labeling for natural language processing according to an embodiment of the present invention. The apparatus for implementing sequence labeling for natural language processing according to an embodiment of the present invention can be used to implement the sequence labeling method for natural language processing according to an embodiment of the present invention. For details not disclosed in the apparatus embodiment of the present invention, please refer to the corresponding description of the method embodiment of the present invention.

[0073] Figure 4 is an exemplary structural diagram of a sequence tagging implementation device for natural language processing in an embodiment of the present application. As shown in Figure 4, the device may include: a first module 401, a second module 402, a third module 403, a fourth module 404, a fifth module 405, and a sixth module 406.

[0074] The first module 401 is used to generate multiple sentence training corpora from unprocessed downstream task corpora.

[0075] The second module 402 is used to input the acquired current sentence training corpus into a pre-training model to encode the sentence training corpus and obtain a vector sequence; the pre-training model is obtained by pre-training in a general corpus.

[0076] The third module 403 is used to perform feature transformation on the vector sequence for each sub-label in the sequence labeling using the current first weighting coefficient and the current first offset coefficient corresponding to the sub-label to obtain a first depth feature vector corresponding to the sub-label; and perform feature transformation on the encoding vector sequence using the current second weighting coefficient and the current second offset coefficient corresponding to the sub-label to obtain a second depth feature vector corresponding to the sub-label.

[0077] The fourth module 404 is configured to calculate, for each sub-tag, a sequence labeling initial score of the sub-tag based on the first deep feature vector of the sub-tag and the second deep feature vector of the sub-tag.

[0078] The fifth module 405 is used to normalize the initial sequence annotation scores of each sub-label of each recognition unit of the sentence training corpus to obtain the training probability distribution of each sub-label.

[0079] The sixth module 406 is used to calculate a loss value based on the true probability distribution of each sub-label of each recognition unit of the sentence training corpus and the training probability distribution of each sub-label. Before the training end condition is reached, the parameters of the pre-training model and the first weighting coefficient, first offset coefficient, second weighting coefficient and second offset coefficient of each sub-label are adjusted according to the loss value, and the second module 402 is instructed to continue to perform the operation.

[0080] If the sequence labeling is performed on a per-character basis in a sentence, the fourth module uses, for each sub-label, the inner product of the first deep feature and the second deep feature of each character as the initial score of the sub-label for the character. If the sequence labeling is performed on a per-field basis in a sentence, the fourth module uses, for each sub-label, the inner product of the first deep feature of the first character in each field and the second deep feature of the last character in the field as the initial score of the sub-label for the field.

[0081] In one embodiment, the apparatus may further include: a seventh module 407 , an eighth module 408 , a ninth module 409 and a tenth module 410 .

[0082] The seventh module 407 is used to train an anomaly detection model using the currently acquired limited downstream task corpus.

[0083] The eighth module 408 is used to extract the current sentence prediction corpus from the general corpus.

[0084] The ninth module 409 is used to use the current sentence prediction corpus as the input of the anomaly detection model to obtain the anomaly score output by the anomaly detection model.

[0085] The tenth module 410 is configured to use the current sentence prediction corpus as the downstream task corpus when the anomaly score is less than a set threshold. When the number of downstream task corpora reaches the required amount, the operation of the first module 401 can be started; otherwise, the eighth module 408 is notified to perform the operation.

[0086] In fact, the engineering template creation device provided in this embodiment of the present application can be implemented in various ways. For example, the eco-design assessment device can be compiled into a plug-in installed in a smart terminal using an application programming interface that complies with specific rules, or it can be packaged into an application for user download and use.

[0087] When compiled as a plug-in, the project template creation device can be implemented in various plug-in formats, such as ocx, dll, and cab. The eco-design assessment device provided by this implementation of the present application can also be implemented using specific technologies, such as Flash plug-in technology, RealPlayer plug-in technology, MMS plug-in technology, MIDI plug-in technology, or ActiveX plug-in technology.

[0088] The project template creation method provided by this implementation of the present application can be stored in various storage media in the form of instruction storage or instruction set storage. These storage media include, but are not limited to, floppy disks, optical disks, DVDs, hard disks, flash memory, USB flash memory, CF cards, SD cards, SDHC cards, MMC cards, SM cards, memory sticks, and xD cards.

[0089] It should be clear that the operating system operating in the computer can implement the functions of any of the above embodiments not only by executing the program code read by the computer from the storage medium, but also by using instructions based on the program code to implement part or all of the actual operations.

[0090] For example, Figure 5 is an exemplary structural diagram of another apparatus for implementing sequence labeling in natural language processing according to an embodiment of the present application. This apparatus can be used to perform the method shown in Figure 2 or to implement the apparatus shown in Figure 4. As shown in Figure 5, the apparatus may include at least one memory 51 and at least one processor 52. In addition, it may include other components such as communication ports, input / output controllers, and network communication interfaces. These components communicate via a bus 53, etc.

[0091] At least one memory 51 is used to store computer programs. In one example, the computer program can be understood to include the various modules of the apparatus shown in FIG4 . Additionally, at least one memory 51 can store an operating system, etc. Operating systems include, but are not limited to, Android operating system, Symbian operating system, Windows operating system, Linux operating system, etc.

[0092] At least one processor 52 is configured to invoke a computer program stored in at least one memory 51 to execute the sequence tagging method for natural language processing described in the examples of this application. The processor 52 may be a CPU, a processing unit / module, an ASIC, a logic module, or a programmable gate array, and may receive and send data via a communication port.

[0093] It should be understood that "and / or" as used herein is intended to include any and all possible combinations of one or more of the associated listed items.

[0094] The number of the embodiments of the present application is only used for description and does not represent the advantages of the embodiments.

[0095] It can be seen from the above technical solution that in the embodiment of the present application, since the probability distribution of the label is calculated based on the deep feature value of each label of each prediction unit in the sentence training corpus, the prediction accuracy of each prediction unit, especially the prediction unit with a combination relationship, can be obtained by performing a combination operation on the deep feature value of each label, thereby better realizing sequence labeling.

[0096] In addition, by using the currently acquired limited downstream task corpus to train an anomaly detection model, we can further obtain corpus belonging to the downstream task corpus in the general corpus, which can expand the number of downstream task corpus and further improve the prediction accuracy of sequence labeling.

[0097] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for implementing sequence labeling in natural language processing, characterized in that: include: Generate multiple sentence training corpora from the unprocessed downstream task corpus; For the current sentence training corpus, the current sentence training corpus is input into a pre-training model to encode the sentence training corpus and obtain a vector sequence; the pre-training model is obtained by pre-training in a general corpus; For each sub-label in the sequence annotation, the vector sequence is transformed by using the current first weighting coefficient and the current first offset coefficient corresponding to the sub-label to obtain a first deep feature vector corresponding to the sub-label; the encoding vector sequence is transformed by using the current second weighting coefficient and the current second offset coefficient corresponding to the sub-label to obtain a second deep feature vector corresponding to the sub-label; For each sub-label, based on the first deep feature vector of the sub-label and the second deep feature vector of the sub-label, calculate a sequence labeling initial score of the sub-label; Normalizing the initial scores of sequence annotations of each sub-label of each recognition unit of the sentence training corpus to obtain a training probability distribution of each sub-label; A loss value is calculated according to the true probability distribution of each sub-label of each recognition unit of the sentence training corpus and the training probability distribution of each sub-label. Before the training end condition is reached, the parameters of the pre-training model and the first weighting coefficient, the first offset coefficient, the second weighting coefficient and the second offset coefficient of each sub-label are adjusted according to the loss value, and the operation of inputting the current sentence training corpus into a pre-training model is returned to be executed for the current sentence training corpus.

2. The method according to claim 1, characterized in that The sequence labeling is performed based on a single word in a sentence; for each sub-label, an initial score of the sequence labeling of the sub-label is calculated based on the first deep feature vector of the sub-label and the second deep feature vector of the sub-label, including: for each sub-label, the inner product of the first deep feature and the second deep feature of each word is used as the initial score of the sub-label of the word.

3. The method according to claim 2, characterized in that The sequence marking for identifying a single word in a sentence includes: punctuation mark marking.

4. The method according to claim 1, characterized in that: The sequence labeling is identified based on the field in the sentence; for each sub-tag, based on the first deep feature vector of the sub-tag and the second deep feature vector of the sub-tag, the initial score of the sequence labeling of the sub-tag is calculated, including: for each sub-tag, the inner product of the first deep feature of the first word in each field and the second deep feature of the last word in the field is used as the initial score of the sub-tag of the field.

5. The method according to claim 4, characterized in that The sequence tagging that is performed based on the fields in the sentence includes: named entity recognition.

6. The method according to any one of claims 1 to 5, characterized in that Further including: An anomaly detection model is trained using the currently acquired limited downstream task corpus; Extract the current sentence prediction corpus from the general corpus; Using the current sentence prediction corpus as the input of the anomaly detection model to obtain an anomaly score output by the anomaly detection model; When the abnormal score is less than a set threshold, the current sentence prediction corpus is used as the downstream task corpus; When the amount of downstream task corpora reaches the requirement, performing the operation of generating multiple sentence training corpora from the unprocessed downstream task corpora; Otherwise, return to execute the operation of extracting the prediction corpus of the current sentence from the general corpus.

7. The method according to any one of claims 1 to 5, characterized in that The loss value is calculated according to the following loss function: Where C represents the number of labels, m is the number of units that need to be labeled in the sentence training corpus; i represents the i-th unit that needs to be labeled in the sentence training corpus; y and are all vectors, y represents the true probability distribution of the sub-labels of the sentence training corpus, Represents the training probability distribution of sub-labels of the sentence training corpus.

8. A sequence tagging implementation device in natural language processing, characterized in that: include: The first module is used to generate multiple sentence training corpora from unprocessed downstream task corpora; The second module is used for inputting the acquired current sentence training corpus into a pre-training model to encode the sentence training corpus and obtain a vector sequence; the pre-training model is obtained by pre-training in a general corpus; The third module is used to perform feature transformation on the vector sequence for each sub-label in the sequence annotation using the current first weighting coefficient and the current first offset coefficient corresponding to the sub-label to obtain a first depth feature vector corresponding to the sub-label; and to perform feature transformation on the vector sequence using the current second weighting coefficient and the current second offset coefficient corresponding to the sub-label to obtain a first depth feature vector corresponding to the sub-label. The coefficient performs feature transformation on the encoding vector sequence to obtain a second deep feature vector corresponding to the sub-label; A fourth module is used to calculate, for each sub-label, a sequence labeling initial score of the sub-label based on the first deep feature vector of the sub-label and the second deep feature vector of the sub-label; The fifth module is used to normalize the initial scores of the sequence annotations of each sub-label of each recognition unit of the sentence training corpus to obtain the training probability distribution of each sub-label; The sixth module is used to calculate a loss value based on the true probability distribution of each sub-label of each recognition unit of the sentence training corpus and the training probability distribution of each sub-label. Before the training end condition is reached, the parameters of the pre-training model and the first weighting coefficient, the first offset coefficient, the second weighting coefficient and the second offset coefficient of each sub-label are adjusted according to the loss value, and the second module is instructed to continue to perform the operation.

9. The device according to claim 8, characterized in that The sequence tagging is performed based on a single word in a sentence; the fourth module uses, for each sub-tag, the inner product of the first deep feature and the second deep feature of each word as the initial score of the sub-tag of the word.

10. The device according to claim 8, characterized in that The sequence labeling is identified in units of fields in a sentence; the fourth module uses, for each sub-label, the inner product of the first deep feature of the first word in each field and the second deep feature of the last word in the field as the initial score of the sub-label of the field.

11. The device according to any one of claims 8 to 11, characterized in that Further including: The seventh module is used to train an anomaly detection model using the currently acquired limited downstream task corpus; The eighth module is used to extract the current sentence prediction corpus from the general corpus; A ninth module is used to use the current sentence prediction corpus as the input of the anomaly detection model to obtain an anomaly score output by the anomaly detection model; A tenth module is used for using the current sentence prediction corpus as downstream task corpus when the abnormal score is less than a set threshold; When the amount of downstream task corpus reaches the requirement, notify the first module to perform the operation; Otherwise, notify the eighth module to perform the operation.

12. A sequence tagging implementation device in natural language processing, characterized in that: The invention comprises at least one memory (51) and at least one processor (52), wherein: The at least one memory (51) is used to store a computer program; The at least one processor (52) is used to call the computer program stored in the at least one memory (51) to execute the sequence tagging implementation method in natural language processing according to any one of claims 1 to 7. Law.

13. A computer-readable storage medium having a computer program stored thereon; characterized in that: The computer program can be executed by a processor and implements the sequence tagging implementation method in natural language processing as described in any one of claims 1 to 7.

Images