Method for named entity recognition and related products

Abstract

Embodiments of the present application provide a method for named entity recognition and related products. The method comprises: obtaining an input text to be processed, wherein the input text comprises entities to be recognized; processing the entities to be recognized in the input text in parallel using a pre-trained non-autoregressive generation model; and determining the entity sequence of all entities in the input text based on the processing result of the non-autoregressive generation model. Through the technical scheme of the present application, the entities to be recognized in the input text can be processed in parallel with the aid of the pre-selected non-autoregressive generation model, which not only breaks through the inert thinking of traditional sequential processing, but also effectively improves the processing efficiency without losing accuracy.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of information processing technology, and more specifically, to a method for named entity recognition, as well as an electronic device and a computer-readable storage medium for performing the aforementioned method. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention set forth in the claims. The description herein may include concepts that may be explored, but not necessarily concepts that have been previously conceived or explored. Therefore, unless otherwise stated, what is described in this section is not prior art for the purposes of this application's specification and claims, and is not acknowledged as prior art simply by virtue of its inclusion in this section.

[0003] Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), applicable to many scenarios such as dialogue systems, document analysis, and search engines. Currently, NER tasks can be divided into three subtasks: planar NER, nested NER, and discontinuous NER. Typically, these three subtasks are fine-tuned individually during NER implementation, a tedious process that impacts the overall system accuracy. To address this, related technologies employ unified fine-tuning for all three tasks, but this requires generating each word sequentially within the sentence; that is, the generation of each subsequent word depends on the preceding word, which also slows down the inference process. Summary of the Invention

[0004] The known performance of named entity recognition is unsatisfactory, which is a very frustrating process.

[0005] Therefore, there is a great need for an improved scheme for named entity recognition that can effectively improve processing efficiency without sacrificing accuracy.

[0006] In this context, embodiments of the present invention aim to provide a method and related products for named entity recognition.

[0007] In a first aspect of the present invention, a method for named entity recognition is proposed, comprising: acquiring input text to be processed, wherein the input text includes entities to be identified; processing the entities to be identified in the input text in parallel using a pre-trained non-autoregressive generative model; and determining an entity sequence of all entities in the input text based on the processing result of the non-autoregressive generative model.

[0008] In one embodiment of the present invention, parallel processing of entities to be identified in the input text using a pre-trained non-autoregressive generative model includes performing the following steps based on the non-autoregressive generative model: obtaining word representation vectors and position vectors of entities to be identified in the input text; determining entity prediction input sequences of entities to be identified based on the word representation vectors and position vectors of entities to be identified; and performing prediction processing on all entity prediction input sequences in parallel.

[0009] In another embodiment of the present invention, wherein the non-autoregressive generative model includes an encoder and a decoder, and the method includes: encoding a text sequence of the input text based on the encoder to obtain a word representation vector of the entity to be identified; performing entity initial classification processing on the text sequence of the input text based on the encoder to obtain a position vector of the entity to be identified; determining an entity prediction input sequence of the entity to be identified based on the processing result of the encoder; and performing prediction processing on all the entity prediction input sequences in parallel based on the decoder.

[0010] In another embodiment of the present invention, encoding the text sequence of the input text based on the encoder includes: during the encoding process of the text sequence based on the encoder, using the tag index of each word in the text sequence or the tag index of the first sub-word of each word as the word representation vector of each word.

[0011] In another embodiment of the present invention, entity start classification processing of the text sequence of the input text based on the encoder includes: predicting the starting position of the entity start word in the text sequence and the number of entities starting from the starting position based on the encoder.

[0012] In one embodiment of the invention, the encoder performs entity initial classification processing according to the following formula: p(ne j ) = Softmax(υ j W ne +b ne ),in, R represents the vector space, and Omax represents the maximum number of entities that start at the same word. υ j =H[start] j ], vj represents the word representation vector of the j-th word in the text sequence, and startj is the tag index of the j-th word in the text sequence or the tag index of the first sub-word of the j-th word; d represents the hidden layer size of the non-autoregressive generative model, L represents the length of the text sequence; Wne represents the trainable matrix, and bne represents the hyperparameters.

[0013] In another embodiment of the present invention, determining the entity prediction input sequence of the entity to be identified includes: superimposing the word representation vector and the position vector of the entity to be identified; and determining the entity prediction input sequence of the entity to be identified based on the superposition result of the word representation vector and the position vector of the entity to be identified.

[0014] In another embodiment of the present invention, determining the entity prediction input sequence of the entity to be identified specifically includes: constructing an input sequence with a fixed length using a specified mask marker, as the entity prediction input sequence; wherein the length of the entity prediction input sequence is the maximum entity length lmax+3 of the superposition result of the word representation vector and the position vector of the entity to be identified.

[0015] In another embodiment of the present invention, the parallel prediction processing of all entity prediction input sequences based on the decoder includes: during the training phase of the non-autoregressive generative model, processing each word in each entity prediction input sequence in parallel based on the decoder to output an entity sequence for each word; and during the inference phase of the non-autoregressive generative model, processing the entity starting word in each entity prediction input sequence in parallel based on the decoder to output an entity sequence for each entity starting word.

[0016] In one embodiment of the present invention, the parallel processing of entity start words in each entity prediction input sequence based on the decoder specifically includes: for each word j in the index set belonging to the entity start word, and each word j being less than the number of entities ne j For each entity, a fixed-length entity prediction input sequence consisting of a specified mask identifier is constructed, and an entity sequence is generated based on the constructed entity prediction input sequence; where S start ={j|p(ne j =0)<τ s}, S start This represents the set of indices of the starting words of the detected entities, with the number of entities determined by `ne`. j ←arg maxne j p(ne j ) obtain, τ s This represents the threshold for the initial classification of an entity, where 0 < τ. s <1.

[0017] In another embodiment of the invention, the method further includes: in response to the fact that the output sequence of the decoder does not contain an end marker, determining all target output markers in the output sequence as an entity sequence, wherein the target output markers include all output markers starting from the third position of the output sequence.

[0018] In another embodiment of the present invention, the method further includes: during the training of the non-autoregressive generative model, obtaining a first loss generated in the entity initial classification processing stage and a second loss generated in the entity sequence generation stage; summing the first loss and the second loss to obtain the total loss of the non-autoregressive generative model; and fine-tuning the non-autoregressive generative model based on the total loss to achieve optimized training of the non-autoregressive generative model.

[0019] In another embodiment of the present invention, obtaining the input text to be processed includes: performing word segmentation on the original text to obtain the input text.

[0020] In a second aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing computer instructions for named entity identification, wherein when the computer instructions are executed by the processor, the electronic device causes the electronic device to perform the methods described in accordance with the foregoing and the following embodiments.

[0021] In a third aspect of the present invention, a computer-readable storage medium is provided, comprising program instructions for named entity identification, which, when executed by a processor, cause the methods described according to the foregoing and the following embodiments to be implemented.

[0022] The method and related products for named entity recognition according to embodiments of the present invention can utilize a trained non-autoregressive generative model to process entities to be recognized in input text in parallel to obtain an entity sequence of all entities. It can be seen that the solution of the present invention, considering that entities are largely independent of each other during the named entity recognition process, can process each entity in parallel, making the processing of entities independent of each other. Therefore, it not only breaks through the inertia of traditional sequential processing but also effectively improves processing efficiency without sacrificing accuracy. Attached Figure Description

[0023] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0024] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0025] Figure 2 A schematic flowchart of a method for named entity recognition according to an embodiment of the present invention is shown.

[0026] Figure 3 A schematic flowchart of a method for named entity recognition according to another embodiment of the present invention is shown.

[0027] Figure 4 A schematic flowchart of a method for named entity recognition according to another embodiment of the present invention is shown.

[0028] Figure 5 A schematic diagram illustrating the architecture of a non-autoregressive generative model according to an embodiment of the present invention is shown; and

[0029] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown.

[0030] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0031] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0032] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention is shown. For example... Figure 1 As shown, the computing system 100 may include: a central processing unit (CPU) 101, random access memory (RAM) 102, read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that... Figure 1The structural block diagrams described are for illustrative purposes only and are not intended to limit the scope of the invention. In some cases, certain devices may be added or removed depending on the specific circumstances.

[0033] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," "unit," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0034] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0035] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0036] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0037] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0038] Embodiments of the present invention will now be described with reference to flowchart illustrations and block diagrams of apparatus (or systems) according to embodiments of the invention. It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the computer or other programmable data processing apparatus, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0039] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0040] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0041] According to embodiments of the present invention, a method for named entity recognition and related products are proposed. Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinguishing purposes only and has no limiting meaning.

[0042] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview

[0043] The inventors discovered that current Named Entity Recognition (NER) methods are inefficient. Specifically, current NER processing models include various types, such as fragment extraction-based models, data structure-based models, and sequence-to-sequence models (Seq2Seq). Among these, the Seq2Seq model can handle various NER tasks and ensures state-of-the-art (SOTA) accuracy, making it a hot research topic in the NER field in recent years. However, the inventors found that the Seq2Seq model, in its implementation, requires generating sentences sequentially from left to right, meaning each word in the sentence must be generated one by one, with the generation of each subsequent word depending on the previous one. This demonstrates that the Seq2Seq model has low processing efficiency and cannot meet the needs of practical applications.

[0044] Further analysis revealed that in NER tasks, entities extracted from a sentence (such as names of people and places) are largely independent of each other. Based on this, the inventors discovered a way to break away from the traditional approach of sequentially generating entities one by one, processing each entity in parallel, thus ensuring that the processing of entities is independent. This effectively improves processing efficiency without sacrificing accuracy.

[0045] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below. Exemplary methods

[0046] The following is for reference. Figure 2 This section describes a method for named entity recognition according to an exemplary embodiment of the present invention. It should be noted that the embodiments of the present invention can be applied to any applicable scenario.

[0047] Figure 2 A schematic flowchart of a method 200 for named entity recognition according to an embodiment of the present invention is shown.

[0048] like Figure 2 As shown, in step S201, the input text to be processed can be obtained. This input text may include entities to be identified, such as names, place names, numbers, dates, or any other entities identified by names.

[0049] The input text here can be understood as text requiring named entity recognition in various NER tasks (such as planar NER, nested NER, and discontinuous NER). It should be noted that the present invention does not limit the specific type of input text; for example, it can be Chinese, English, or other languages.

[0050] In practical applications, the input text to be processed can be obtained in various ways. For example, in some embodiments, the original text can be obtained, and the input text to be processed can be obtained by segmenting the original text into words. In other embodiments, if the original text has already been segmented into words, it can be directly used as the input text to be processed. It should be noted that the description of the process of obtaining the input text to be processed here is only an illustrative example, and the solution of the present invention is not limited thereto. It can be adjusted according to the application design requirements and the state of the original text.

[0051] Next, in step S202, the entities to be identified in the input text can be processed in parallel using a pre-trained non-autoregressive generative model.

[0052] Traditional Natural Language Processing (NLP) techniques (including Named Entity Recognition) typically employ autoregressive models to generate sentences sequentially from left to right, relying on the dependency of each subsequent word on the preceding word to ensure accuracy. Non-autoregressive generative models, however, differ from autoregressive models by assuming that words in the generated sentence are independent, with each subsequent word generated without dependence on the preceding one. Due to the strong independence assumption of non-autoregressive generative models, the dependency on the preceding word is lost during sentence generation. Therefore, non-autoregressive generative models are generally considered to be faster than autoregressive models, but with lower quality. However, in NLP tasks, entities such as names of people and places extracted from sentences are independent, meaning that generating a subsequent entity can be independent of the preceding one. Therefore, this invention breaks with traditional thinking, creatively applying non-autoregressive generative models to NLP tasks. Furthermore, because entities are independent, there is no need to rely on the dependency of each subsequent word on the preceding one to ensure accuracy, ensuring that the quality of entity generation using non-autoregressive models is not reduced. Therefore, processing efficiency can be effectively improved without sacrificing accuracy.

[0053] In practical applications, the non-autoregressive generative model can be pre-trained to enable it to process entities to be identified in parallel. After obtaining the input text to be processed, the pre-trained non-autoregressive generative model can be used to process the entities to be identified in the input text in parallel. It can be seen that the parallel processing method of this invention using a non-autoregressive generative model is significantly better than the traditional sequential processing method using an autoregressive model.

[0054] Finally, in step S203, the entity sequence of all entities in the input text can be determined based on the processing results of the non-autoregressive generative model. After completing the parallel processing of the entities to be identified based on the trained non-autoregressive generative model, the entity sequence of all entities can be obtained based on the parallel processing results. Thus, in the named entity recognition process, the entity sequence of all entities can be generated quickly, rather than generating entities sequentially one by one.

[0055] By creatively applying non-autoregressive generative models, which have advantages such as fast processing speed and strong independence assumptions, to the NER task, the pre-trained non-autoregressive generative models can be used to process the entities to be identified in the input text in parallel. This not only breaks through the inertia of traditional sequential processing, but also effectively improves processing efficiency without sacrificing accuracy.

[0056] Figure 3 A schematic flowchart of a method 300 for named entity recognition according to another embodiment of the present invention is shown. It will be understood that method 300 is a... Figure 2 Further limitations and / or extensions of Chinese method 200. Therefore, the foregoing is combined with Figure 2 The relevant detailed descriptions also apply to the following text.

[0057] like Figure 3 As shown, in step S301, the input text to be processed can be obtained. As mentioned above, the input text may include names of people, places, or other entities to be identified by names, and the input text can be specifically obtained by performing word segmentation or other methods on the original text.

[0058] After obtaining the input text, steps S302 to S304 can be executed based on the trained non-autoregressive generative model.

[0059] Specifically, in step S302, the word representation vector and position vector of the entity to be identified in the input text can be obtained. The word representation vector of the entity to be identified can be understood as being determined based on the context representation of the entity, while the position vector can be understood as being determined based on the starting position of the entity and the number of entities starting at that position. In some embodiments, the text sequence of the input text can be encoded using a pre-trained encoder that supports context representation to obtain the word representation vector of the entity to be identified. The text sequence of the input text can be subjected to entity initial classification processing using a pre-trained encoder that supports entity initial classification to obtain the position vector of the entity to be identified. It should be noted that the description of the process for obtaining the word representation vector and position vector here is merely illustrative.

[0060] Next, in step S303, the entity prediction input sequence of the entity to be identified can be determined based on the word representation vector and position vector of the entity to be identified.

[0061] In some embodiments, the word representation vectors and position vectors of the entity to be identified can be superimposed first. For example, superposition can be achieved using word representation vectors + position vectors. Then, based on the superposition result of the word representation vectors and position vectors of the entity to be identified, the entity prediction input sequence is determined. Specifically, a fixed-length input sequence can be constructed using specified mask markers as the entity prediction input sequence. In traditional techniques, because the length of the output sentence is uncertain (it may be longer or shorter than the input sentence), it is necessary to predict the sentence length in advance using encoder output information. However, in NER tasks, entities are extracted from the input sentence, so the length of the output sentence is usually no longer than the length of the input sentence. Therefore, the solution of this invention does not predict the length of the input sentence but directly uses a fixed-length entity prediction input sequence. This effectively simplifies the entire processing and further improves processing efficiency.

[0062] In some embodiments, the length of the entity prediction input sequence is the maximum entity length lmax + 3 resulting from the superposition of the word representation vector and position vector of the entity to be identified. The length value 3 may include one entity category label (e.g., <adr>(marker) and 1 sequence start marker (e.g.) <s> (marker) and 1 sequence end marker (e.g.)< / s> The length occupied by the marker. It should be noted that the specific representation of each marker is not limited here; it is merely an example. <s>、< / s> Etc.

[0063] Then, in step S304, all entity prediction input sequences can be processed in parallel based on the non-autoregressive generative model. As mentioned above, the pre-trained non-autoregressive generative model follows the principle of mutual independence between entities during entity processing, thus enabling parallel prediction processing of entity prediction input sequences without loss of accuracy.

[0064] Finally, in step S305, the entity sequence of all entities in the input text can be determined based on the processing results of the non-autoregressive generative model. This completes the processing of all entities in the input text, improving processing speed while maintaining accuracy.

[0065] Figure 4 A flowchart illustrating a method 400 for named entity recognition according to another embodiment of the present invention is shown schematically. It will be understood that method 400 is a... Figure 2 Further limitations and / or extensions of methods 200 and 300. Therefore, the foregoing is combined with Figure 2 and Figure 3 The relevant detailed descriptions also apply to the following text.

[0066] like Figure 4 As shown, in step S401, the original text can be segmented to obtain the input text to be processed. The original text can be Chinese, English, or other languages. In some embodiments, a general word segmentation technique combined with a word segmentation library (e.g., a pre-trained neural network model supporting word segmentation) can be used to segment the original text. It should be noted that the description of the input text acquisition process here is merely illustrative and can be customized according to application requirements. For example, if the original text has already been pre-segmented, further word segmentation is unnecessary, and the original text can be directly used as the input text.

[0067] After obtaining the input text, a pre-trained non-autoregressive generative model can be used to process it. Specifically, the non-autoregressive generative model can include an encoder and a decoder, which can be used to process the input text.

[0068] Specifically, in step S402, the text sequence of the input text can be encoded based on the encoder to obtain the word representation vector of the entity to be identified. Specifically, in some embodiments, the encoder can be pre-trained to enable it to determine the word representation vector using contextual representation. During the encoding of the text sequence based on the encoder, the tag index of each word in the text sequence or the tag index of the first sub-word of each word can be used as the word representation vector of each word. For example, the original text is "...calf / toe cramps...", which is processed by word segmentation to obtain the input text. Here, "calf" is a single word in the text sequence of the input text, while "cramps" in the original text is divided into two sub-words, "cr" and "amps". Therefore, during the encoder encoding process, the tag index of each word in the text sequence can be used as the word representation vector of that word, or the tag index of the first sub-word (e.g., "cr") of a word (e.g., "cramps") can be used as the word representation vector of that word.

[0069] In step S403, entity initiation classification can be performed on the text sequence of the input text based on the encoder to obtain the position vector of the entity to be identified. Specifically, in some embodiments, the encoder can be pre-trained so that it not only has the function of determining word representation vectors using context representation, but also has the function of entity initiation classification. Then, the trained encoder is used to predict the starting position of the entity initiation word in the text sequence and the number of entities starting from the starting position.

[0070] For example, if the original text is "The doctor found that I have swollen and sore muscles", and the text sequence of the input text contains "The doctor found that I have swollen and sore muscles", after the above entity initial classification processing, the starting position of the disease type entity will be at the position where "muscle" is located, and the number of entities starting at this position will be 2 (swelling, soreness).

[0071] Furthermore, in some implementation scenarios, the encoder can perform initial entity classification processing based on the following formula: p(ne j ) = Softmax(υ j W ne +b ne ),in, R represents a vector space, O max This indicates the maximum number of entities that begin at the same word; υ j =H[start] j ], v j Let start represent the word representation vector of the j-th word in the text sequence. j It is either the tag index of the j-th word in the text sequence or the tag index of the first sub-word of the j-th word; d represents the hidden layer size of the non-autoregressive generative model, L represents the length of the text sequence; W ne Let b represent a trainable matrix. ne This represents hyperparameters.

[0072] After obtaining the word representation vector and position vector of the entity to be identified using the encoder, the entity prediction input sequence of the entity to be identified can be determined based on the processing result of the encoder. Specifically, in step S404, the word representation vector and position vector of the entity to be identified can be superimposed. Then, in step S405, the entity prediction input sequence of the entity to be identified can be determined using the superposition result of the word representation vector and position vector of the entity to be identified. This entity prediction input sequence is obtained by constructing a fixed-length input sequence using a specified mask marker, and the length of this entity prediction input sequence is the maximum entity length l of the superposition result of the word representation vector and position vector of the entity to be identified. max +3. Where the value 3 represents the length occupied by the three markers: entity category marker, sequence start marker, and sequence end marker.

[0073] Finally, in step S406, prediction processing can be performed on all entity prediction input sequences in parallel based on the decoder. Specifically, during the training phase of the non-autoregressive generative model, each word in the entity prediction input sequence can be processed in parallel based on the decoder to output an entity sequence for each word. This improves the performance of the decoder. During the inference phase of the non-autoregressive generative model, the starting word of each entity prediction input sequence can be processed in parallel based on the decoder to output an entity sequence for each starting word. This optimizes the inference speed of the decoder.

[0074] The following combination Figure 5 The solution of the present invention will be further explained. Wherein, Figure 5 This can be understood as a specific implementation architecture of the non-autoregressive generative model used in methods 200 to 400. Therefore, the preceding text, combined with... Figures 2-4 The relevant details described in the text also apply to the following text.

[0075] like Figure 5 As shown, this non-autoregressive generative model can include an encoder and a decoder. Figure 5 In the diagram, the left side shows the encoder's functionality, while the right side shows the encoder's functionality during the training and inference phases, respectively.

[0076] In this embodiment, the task of recognizing flat, nested, and discontinuous entities can be uniformly defined as an entity sequence generation problem based on autoregressive generation (NAG). Then, a pre-trained non-autoregressive generation model (NAG model) consisting of an encoder and a decoder is used as the backbone of the entire model. The input to this NAG model can be an input sequence S (i.e., the text sequence of the input text mentioned earlier), which can be understood as a specific representation of the tokenized input sentence: S = [w0, ..., w0]. L-1 ], where L represents the length of the input sequence S. The set of entity sequences output by this NAG model can be represented as: Where M represents the number of entities, ES i Indicated by l i The i-th entity composed of t tags, T i It is the type label for entity i.

[0077] like Figure 5 As shown, the encoder-side tasks may include acquiring a high-quality contextual representation of the input text and initial entity classification. For example, for the original text "...calf / toe cramps...", the corresponding text sequence has a length of 10 ( Figure 5 The tagged text sequence shown above is marked with "0-9". In entity start classification, it can be determined that entity ("calf") starts from position 2 of the text sequence, and there is a total of 1 entity at position 2; and ("cramps") starts from position 4 of the text sequence, and there is a total of 1 entity at position 4. It should be noted that this is only an example of the results of entity start classification, and there is no limit to the number of entities starting at the entity start position. For example, it can be 0, 1, 2 or other values.

[0078] In some embodiments, the pre-trained NAG encoder encodes the input sequence S and outputs a context representation: H = Encoder(S). d represents the hidden layer size of the non-autoregressive generative model, and L represents the length of the text sequence (i.e., the input sequence S). The encoder (e.g., a Transformer-based encoder code) provides contextual encoding for the tokens. Therefore, the token index of the first sub-word of each word can be used as the word representation vector for that word. It should be noted that after word segmentation, some words may remain complete words (e.g., "calf"), while others (e.g., "cramps") will be divided into multiple sub-words (e.g., "cr", "amps"). Therefore, the first sub-word of each word mentioned above can also be understood as the word itself (e.g., the first sub-word of "calf" is "calf"), or the first sub-word among multiple sub-words (e.g., the first sub-word of "cramps" is "cr").

[0079] In some embodiments, the word representation vector can be specifically represented as: v j =H[start] j ]. Where, v j Let start represent the word representation vector of the j-th word in the text sequence. j It is either the tag index of the j-th word in the text sequence or the tag index of the first sub-word of the j-th word.

[0080] Furthermore, to incentivize the encoder to gain a deeper understanding of the input sentence and provide information to the decoder, the encoder needs to detect where entities will begin and how many entities will begin at that position. In practical applications, this task can be transformed into a multi-class classification task for each word j, predicting the number of entities ne starting with word j. j :p(ne j ) = Softmax(υ j W ne +b ne ),in, R represents a vector space, O max This indicates the maximum number of entities that begin at the same word. Note that if ne... j =0 indicates that there is no entity start at word j.

[0081] During training, the decoder generates an entity sequence for each input word j. To accelerate autoregressive generation, this embodiment proposes a non-autoregressive generation model (NAG model). This NAG model eliminates the order dependency between target tags Y and can simultaneously generate the target sentence tag y for a given input X. t :

[0082] Where l represents the length of the target sentence, θ represents the vector space, and y t This represents the decoder's prediction result for X.

[0083] Additionally, the decoder needs to know the length of the target sentence before generating the entity sequence. Traditionally, the length prediction task is treated as a classification task, and the length prediction is performed using information from the encoder output. This example discards this length prediction task, uses the same output sequence length, and uses the first generated end marker or symbol (e.g.) as the end signal for the entity sequence.

[0084] Specifically, the maximum entity length (in units of subwords) can be represented as l. max ∈Z + During training, ne can be trained according to its span length (i.e., the span length covering the entity). j Sort the entities in ascending order. Specifically, for each word j and each n = 0, 1, 2, ..., ne... j -1, the decoder can generate entities (ES). j,n T j,n Without named entities starting with the word 'j', ES j,n It will be a sub-tag of the word j, T j,n It will be a non-entity marker <o>Therefore, the decoder's input and target output are:

[0085] Decoder input: length l max +3 is marked by a mask. <m>The resulting sequence is fed into the decoder. For example... Figure 5 As shown, length l max The value is 3, marked by a mask. <m>The entity prediction input sequence of length 6 (3+3) is used as the input to the decoder.

[0086] The decoder's target output: The target output sequence is in the form of <s>AND j,n ES j,n < / s> Among them, ES j,n It is the sub-tag sequence of this entity, ET j,n It is a tag added to the NAG vocabulary, representing the entity class tag T. j,n The non-entity marker O corresponds to a special marker. <o>.

[0087] Furthermore, in some embodiments, since the lengths of solids and non-solids differ, padding markers (e.g., ...) can be used as needed. <pad>Fill the target output sequence with a length of l max +3. For example... Figure 5 As shown, for non-entity " / ", the final generated target output sequence is of length 6: <s> <o> / <pad> <pad> <pad>< / pad> < / pad> < / pad> < / o> < / s> .

[0088] like Figure 5 As shown, the decoder does not need to generate an entity sequence for every word during inference. Specifically, it can distinguish which words are likely to be the starting words for named entities. For example, a threshold τ for when entity classification begins can be used. s This is used to differentiate them. Specifically, for each word j in the index set that belongs to the starting word of an entity, and each word is less than the number of entities ne... j For each entity, construct a fixed-length entity prediction input sequence consisting of a specified mask identifier, and generate an entity sequence based on the constructed entity prediction input sequence.

[0089] Among them, S start ={j|p(ne j =0)<τ s }, S start This represents the set of indices of the starting words of the detected entities, with the number of entities determined by `ne`. j ←arg maxne j p(ne j ) obtain, τ s This represents the threshold for the initial classification of an entity, where 0 < τ. s <1.

[0090] In practical applications, after the decoder generates the output sequence, entity labels are used to mark ET. j,n This can be obtained as the second generated token; the token sequence from the third position to the first token is the generated entity token sequence. In response to the decoder's output sequence not containing an end token, all target output tokens in that output sequence are determined as an entity sequence, where the target output tokens include all output tokens starting from the third position of the output sequence. For example, if there are no tokens in the output sequence, then all output tokens starting from the third position will be considered as the generated entity token sequence.

[0091] Furthermore, for the decoder, its reception is only determined by the mask markings. <m>The input sequence (i.e., the entity prediction input sequence mentioned earlier) does not know where the entities begin or which contexts should be given more attention during generation. Furthermore, the decoder needs to be informed of the number of entities starting from the same word so that it can generate entities of varying lengths instead of the same entities. Therefore, two positional embeddings (word start position embedding and entity number position embedding) can be introduced into the decoder's embedding layer. For the word start position embedding (WSPE), all mask tokens in the decoder input can share the same word start position index, e.g., the token index startj of the first subword of word j. Moreover, the word start position embedding vector is obtained by looking up the positional embedding layer of the encoder.

[0092] For Entity Quantity Location Embedding (NEPE), for each n <ne j Let n represent the nth entity, and specifically the nth shortest entity starting from word j. n can be mapped to a randomly initialized learnable embedding vector NEPE. n This positional embedding is also shared by all tags input to the decoder.

[0093] Furthermore, during the training of the non-autoregressive generative model, the first loss generated in the initial entity classification stage and the second loss generated in the entity sequence generation stage can be obtained. Then, the first and second losses can be summed to obtain the total loss of the non-autoregressive generative model. Finally, the non-autoregressive generative model can be fine-tuned based on the total loss to achieve optimized training.

[0094] In some embodiments, the task of detecting where entities will begin and how many entities will begin at that location can be transformed into a multi-class classification task for each word j, predicting the number of entities ne starting with word j. j Cross-entropy loss is used during this task. (i.e., the first loss mentioned above) is optimized.

[0095] In some embodiments, for a given sequence of input and target output to the decoder, the generation loss of the NAG model can be calculated according to the following formula. (i.e., the second loss mentioned above):

[0096] Among them, generation loss This is the average cross-entropy loss per label. In some embodiments, this can be discarded. <pad>Loss of the label.

[0097] During the fine-tuning of the NAG model, the entire framework of the non-autoregressive generative model was optimized end-to-end, including the total loss for entity initial classification and entity sequence generation. Of which, total loss Based on this total loss Fine-tuning of the NAG model.

[0098] Therefore, this embodiment constructs a non-autoregressive generative framework for named entity recognition (NAG-NER). Given an input sentence, this framework first encodes the sentence and detects how many entities will begin at each input sentence marker. Then, it requires the decoder to generate a set of target entities accordingly. Experiments have shown that this non-autoregressive generative framework achieves significant speedup without sacrificing accuracy, and outperforms traditional NER models.

[0099] It should be noted that the solution of this invention is not limited to applications such as data preprocessing for machine translation and syntactic parsing for writing correction, but can also be applied to all NER-based tasks (such as dialogue systems, document analysis, and search engines). The solution of this invention can improve inference speed without reducing accuracy, thereby providing users with a better experience. Exemplary device

[0100] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 6 Related products for the method for named entity recognition according to exemplary embodiments of the present invention will be described.

[0101] Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of the present invention is shown. Figure 6 As shown, the electronic device 600 may include a processor 601 and a memory 602. The memory 602 stores computer instructions for named entity recognition. When these computer instructions are executed by the processor 601, they cause the electronic device 600 to perform actions according to the foregoing description. Figures 2 to 4 The described method. For example, in some embodiments, the electronic device 600 can analyze the original text, obtain input text containing entities to be identified, pre-train a non-autoregressive generative model, use the trained non-autoregressive generative model to process the entities to be identified in the input text in parallel, and generate an entity sequence of all entities in the input text, etc. Based on this, the electronic device 600 can use a pre-trained non-autoregressive generative model to process the entities to be identified in the input text in parallel, which not only breaks through the inertia of traditional sequential processing, but also effectively improves processing efficiency without sacrificing accuracy.

[0102] It should be noted that although several devices or sub-devices of the device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0103] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0104] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.< / pad> < / m> < / pad> < / o> < / m> < / m> < / o> < / adr>

Claims

1. A method for named entity recognition, characterized in that, include: Obtain the input text to be processed, wherein the input text includes entities to be identified; The input text is processed in parallel using a pre-trained non-autoregressive generative model, which includes the following steps based on the non-autoregressive generative model: obtaining the word representation vector and position vector of the entity to be identified in the input text; and determining the entity prediction input sequence of the entity to be identified based on the word representation vector and position vector of the entity to be identified. The method involves parallel prediction processing of all the entity prediction input sequences, wherein determining the entity prediction input sequence of the entity to be identified includes: superimposing the word representation vector and position vector of the entity to be identified; and determining the entity prediction input sequence of the entity to be identified based on the superposition result of the word representation vector and position vector of the entity to be identified; and The entity sequence of all entities in the input text is determined based on the processing results of the non-autoregressive generative model.

2. The method according to claim 1, characterized in that, The non-autoregressive generative model includes an encoder and a decoder, and the method includes: The encoder encodes the text sequence of the input text to obtain the word representation vector of the entity to be identified. The encoder performs entity initialization classification on the text sequence of the input text to obtain the position vector of the entity to be identified. Based on the processing results of the encoder, the entity prediction input sequence of the entity to be identified is determined; and The decoder performs prediction processing on all the entity prediction input sequences in parallel.

3. The method according to claim 2, characterized in that, Encoding the text sequence of the input text based on the encoder includes: During the encoding of the text sequence based on the encoder, the tag index of each word in the text sequence or the tag index of the first sub-word of each word is used as the word representation vector of each word.

4. The method according to claim 2, characterized in that, The entity initialization classification process of the text sequence of the input text based on the encoder includes: The encoder predicts the starting position of the first word of an entity in the text sequence and the number of entities starting from that starting position.

5. The method according to claim 4, characterized in that, The encoder performs initial entity classification processing according to the following formula: in, R represents the vector space, O max This indicates the maximum number of entities that begin at the same word. v j Let start represent the word representation vector of the j-th word in the text sequence. j It is either the tag index of the j-th word in the text sequence or the tag index of the first sub-word of the j-th word; d represents the hidden layer size of the non-autoregressive generative model, L represents the length of the text sequence; W ne Let b represent a trainable matrix. ne This represents hyperparameters.

6. The method according to claim 1, characterized in that, The specific steps for determining the entity prediction input sequence of the entity to be identified include: Construct a fixed-length input sequence using a specified mask marker to serve as the input sequence for entity prediction; Wherein, the length of the entity prediction input sequence is the maximum entity length l resulting from the superposition of the word representation vector and position vector of the entity to be identified. max +3.

7. The method according to claim 2, characterized in that, The parallel prediction processing of all entity prediction input sequences based on the decoder includes: During the training phase of the non-autoregressive generative model, each word in the input sequence of each entity prediction is processed in parallel based on the decoder to output an entity sequence for each word; During the inference phase of the non-autoregressive generative model, the decoder processes the entity start word in the input sequence of each entity prediction in parallel to output an entity sequence for each entity start word.

8. The method according to claim 7, characterized in that, The parallel processing of the entity prediction input sequence for each entity based on the decoder specifically includes: For each word j in the index set that belongs to the entity's starting word, and for each entity whose number is less than the entity number nej, construct a fixed-length entity prediction input sequence consisting of a specified mask identifier, and generate an entity sequence based on the constructed entity prediction input sequence; in, Sstart = {j |p(ne j = 0)<τ s }, Sstart This represents the set of indices of the starting words of the detected entities, with the number of entities determined by `ne`. j ← arg maxne j p(ne j ) obtain, τ s This represents the threshold for the initial classification of an entity, where 0 < τ. s <1.

9. The method according to claim 7, characterized in that, The method further includes: In response to the fact that the output sequence of the decoder does not contain an end marker, all target output markers in the output sequence are determined as an entity sequence, wherein the target output markers include all output markers starting from the third position of the output sequence.

10. The method according to claim 2, characterized in that, The method further includes: During the training of the non-autoregressive generative model, the first loss generated in the initial entity classification stage and the second loss generated in the entity sequence generation stage are obtained. The first loss and the second loss are summed to obtain the total loss of the non-autoregressive generative model; and The non-autoregressive generative model is fine-tuned based on the total loss to achieve optimized training of the non-autoregressive generative model.

11. The method according to any one of claims 1 to 10, characterized in that, The input text to be processed includes: The original text is segmented into words to obtain the input text.

12. An electronic device, characterized in that, include: processor; as well as A memory storing computer instructions for named entity identification, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, It includes program instructions for named entity recognition, which, when executed by a processor, cause the method according to any one of claims 1-11 to be implemented.

Images