Transform-based unified information extraction method

Abstract

The invention discloses a unified information extraction method based on Transform, belongs to the technical field of natural language processing, and realizes unification of information extraction tasks through a sequence-to-sequence modeling mode. According to the method, parameter redundancy among the independent models is effectively reduced, and accurate multi-dimensional information extraction of entities, relationships, classification and the like can still be realized in a low-resource scene through the pre-training language model. Meanwhile, the finite-state machine is adopted to restrain the generation process, accurate structuring of the output result under the complex text condition is further ensured, and the overall extraction effect and the flexibility of system application are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly relates to a unified information extraction method based on the Transformer architecture for realizing joint modeling and efficient execution of entity recognition, relation extraction, and text classification tasks. Background Art

[0002] In the field of natural language processing, information extraction technology, as the core task of structured text understanding, mainly includes subtasks such as entity recognition, relation extraction, event detection, and text classification. The current mainstream methods are mainly divided into traditional methods based on rule templates and independent task modeling methods based on deep learning, but there are still significant technical bottlenecks in actual application scenarios.

[0003] Traditional information extraction methods mostly rely on manually constructed rule systems or statistical machine learning models (such as conditional random field CRF). Such methods show high accuracy in structured text of limited domains, but have serious limitations in processing unstructured text: First, it is difficult for rule templates to cover complex language expression variants, especially in scenarios such as entity nesting and cross-sentence relation reasoning, where the recall rate drops sharply; Second, entity recognition and relation extraction often adopt a cascaded pipeline design, resulting in the accumulation of upstream task errors downstream. For example, incorrect entity boundary recognition will directly affect the accuracy of relation classification; Third, traditional methods lack the ability to model global semantics and are difficult to handle complex language phenomena such as anaphora resolution and semantic ambiguity.

[0004] With the development of deep learning technology, end-to-end models based on recurrent neural networks (RNN) and convolutional neural networks (CNN) have significantly improved information extraction performance. However, existing deep learning methods still have inherent defects: 1) RNN-based models are limited by the sequence order encoding mechanism and have bottlenecks in capturing long-distance dependencies and parallel processing, resulting in difficulties in balancing efficiency and effect when processing discourse-level text; 2) Existing methods usually decouple different extraction tasks into independent models (such as separately training entity recognition and relation extraction models). This separated architecture not only generates a large amount of redundant computation but also severs the potential semantic associations between tasks. For example, the constraint effect of relation labels on entity types is not effectively utilized; 3) Highly dependent on large-scale labeled data for independent training, and in real-world scenarios, labeled data such as entities, relations, and events are often unevenly distributed, resulting in significant performance degradation for low-resource tasks. Summary of the Invention

[0005] 1. Technical Problems to be Solved: Aiming at the problems existing in the prior art, the purpose of the present invention is to provide a unified information extraction method based on Transformer, which realizes the unification of information extraction tasks through a sequence-to-sequence modeling method. This method effectively reduces the parameter redundancy between independent models, and can still achieve accurate multi-dimensional information extraction of entities, relationships, classifications, etc. in low-resource scenarios through pre-trained language models. At the same time, a finite state machine is used to constrain the generation process, further ensuring the precise structuring of the output results under complex text conditions, and significantly improving the overall extraction effect and the flexibility of system applications.

[0006] 2. Technical solution: Aiming at the problems of fragmentation and complex processes among entity recognition, relationship extraction, and text classification tasks in traditional information extraction methods, this technical solution designs a unified generative information extraction model based on the Transformer architecture. Through an end-to-end sequence generation mechanism, the original text is directly converted into structured JSON data to achieve unified multi-task modeling. The specific technical solution is divided into the following steps: (1) Data unification construction: Structurally reorganize the original annotated data, and uniformly encode heterogeneous information such as entities, relationships, and events into a predefined JSON format. By introducing special identifiers such as, etc., a serialized annotation template is constructed to form standardized training data containing type tags and association relationships, while retaining the complete relevance of text context semantics.

[0007] (2) Model architecture design: Construct a multi-layer encoder-decoder structure based on Transformer. At the encoder end, a bidirectional attention mechanism is used to extract deep text features, and a multiple function coupling of entity extraction, relationship derivation, and structure generation is realized through a shared parameter mechanism to ensure the collaborative optimization among multiple tasks.

[0008] (3) Decoding optimization: Introduce a syntax constraint module in the sequence generation process, use a finite state automaton to real-time verify the JSON syntax structure, and combine the beam search algorithm to screen candidate sequences that conform to a predetermined pattern to ensure the format correctness of the generated results and significantly reduce the generation probability of invalid outputs.

[0009] (4) Progressive training strategy: Adopt a two-stage training mechanism. In the initial stage, semantic understanding pre-training is carried out based on general corpora, and in the later stage, domain expertise is injected through multi-task incremental learning.

[0010] 3. Beneficial effects: Adopting the technical solution provided by the present invention, compared with the prior art, it has the following beneficial effects: (1) Unified information extraction: Traditional information extraction methods often design independent models for tasks such as entities, relationships, and classification respectively, which easily leads to parameter redundancy between models and coordination problems between tasks. This solution proposes a generative model based on Transformer. Through sequence-to-sequence modeling, multi-task information extraction is unified in one model, effectively solving the information island problem caused by task dispersion and reducing the complexity of model deployment and maintenance.

[0011] (2) Efficient pre-training and low resource dependence: Existing methods usually require a large amount of domain-annotated data to train specific task models, resulting in high data acquisition costs and insufficient generalization ability. Based on pre-trained language models, this solution realizes the accurate extraction of information such as entities, relationships, and classification in text with less domain-annotated data, thus significantly reducing the dependence on large-scale annotated data and improving the application flexibility of the model.

[0012] (3) Precise structured generation: Traditional extraction algorithms are prone to information loss or incorrect matching when dealing with long texts or texts with complex structures. This solution uses a finite state machine to effectively constrain the generation process, enabling the Transformer generative model to better capture the internal structural information of the text during sequence generation, ensuring that the extracted results are more accurate and consistent in multiple dimensions such as entities, relationships, and classification, thus significantly improving the overall effect of information extraction.

[0013] It should be noted that the structures not introduced in this invention are the same as the prior art or can be implemented using the prior art because they do not involve the design key points and improvement directions of this invention, and will not be elaborated here. Detailed implementation manners

[0014] To facilitate the understanding of this invention, the following will provide a more comprehensive description of this invention and give several embodiments of this invention. However, this invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of this invention more thorough and comprehensive.

[0015] The unified information extraction method based on Transformer proposed in this patent adopts a generative architecture, mainly including four core modules: a pre-trained language encoder module, a sequence decoding module, a structured generation constraint module, and a result parsing module. The length of the text sequence input to the model is L, and the maximum supported length is 512 characters.

[0016] 1. Pre-trained language encoder module: This module uses the AlBert model pre-trained on a large-scale Chinese corpus as the encoder, and its main function is to extract deep semantic features from the input text.

[0017] (1)Input processing: After the text sequence is tokenized by WordPiece, it is added with a word embedding layer (dimension 768) and a positional encoding (sinusoidal positional encoding, dimension 768), and the output dimension is L×768 (2)Core structure: Stacked by 12 layers of ALBERT-Encoder, each layer contains: a. Multi-head self-attention sub-layer Number of heads: 12 heads for parallel computing; Dimension of a single head: 768 / 12 = 64; Query / Key / Value linear transformation: Input is 768 dimensions, output is 64×12 = 768 dimensions; Attention scaling factor: 1 / √64; Attention mask: Full visible mode.

[0018] b. Feed-forward neural network sub-layer First linear layer: Input 768 → dimension expansion to 3072; Activation function: GELU (Gaussian Error Linear Unit); Second linear layer: 3072 → dimension compression to 768.

[0019] c. Regularization mechanism Layer normalization: Executed before the residual connection, epsilon = 1e-12; Residual connection: The output of each sub-layer is added to the input; Dropout: Both the attention weights and activation values are discarded with a probability of 0.1.

[0020] Parameter sharing: All Encoder layers share the weight matrix, significantly reducing the number of parameters.

[0021] 2. Sequence decoding module: The decoder is composed of 2 layers of standard Transformer Decoder stacked.

[0022] (1)Initial input: Special symbol <cls>Start decoding with an embedding (learnable vector, dimension 512).

[0023] (2) Core structure: Stacked with 2 layers of standard Decoder, each layer contains: a. Masked multi-head self-attention sub-layer Number of heads: 8 heads for parallel computing; Dimension per head: 512 / 8 = 64; Causal mask: Prevent information leakage, only allowing attention to the current position and previous positions; Scaled dot-product attention calculation: Scaling factor 1 / √64.

[0024] b. Encoder-decoder attention sub-layer Source of keys / values: Output of the last layer of Encoder (dimension L×768); Query transformation: Project the decoder hidden state to 768 dimensions through a linear layer; Multi-head calculation: 8 heads in parallel, using scaled dot-product for attention score calculation.

[0025] c. Feed-forward neural network sub-layer First linear layer: Expand the dimension from 512 to 2048; Activation function: ReLU; Second linear layer: Compress the dimension from 2048 to 512.

[0026] d. Regularization mechanism Layer normalization: Performed before the residual connection, epsilon = 1e-12; Residual connection: Add the output of each sub-layer to the input; Dropout: Discard the output of each sub-layer with a probability of 0.1.

[0027] Output processing: The final hidden state passes through a linear layer (dimension 512 → vocabulary size) and Softmax to generate a probability distribution.

[0028] 3. Structured generation constraint module: This module innovatively introduces a finite state machine (FSM) mechanism to structurally constrain the generation process. The specific implementation includes: (1) Define JSON syntax state transition rules: Maintain the set of characters allowed at the current generation position (such as quotation marks, parentheses, key-value separators, etc.); (2) Real-time syntax verification: At each generation step, perform an intersection operation on the top-k candidate characters output by the model and the FSM allowed character set; (3) Dynamic mask mechanism: Apply an attention mask with negative infinity to characters that do not conform to the current JSON syntax; (4) Error rollback mechanism: When it is unable to generate legal characters for n consecutive times (the default value of n is 3), it rolls back to the nearest valid state node.

[0029] 4. Result parsing module: This module converts the generated target sequence into structured data through regular expression matching and syntax analysis: (1) JSON validity verification: Check basic grammars such as bracket matching and quote closing; (2) Type coercion: Convert numerical values in string form into corresponding data types; (3) Structure normalization: Unify the case of key names and remove redundant whitespace; (4) Error handling: Locally discard the fields that fail to be parsed and record exception logs.

[0030] 5. Loss function: Loss function for model training: (1) Cross-entropy loss: Calculate the character-level difference between the generated sequence and the labeled sequence; (2) The final output of the model is normalized structured data that conforms to the JSON Schema definition, including entity lists, relationship triples, and text classification information, which can be directly used for downstream application system integration.

[0031] Example: Text information extraction in the medical field 1. Data preprocessing: The original medical text is labeled in JSON format, including entities (such as diseases, symptoms) and relationships (such as "causes", "accompanies").

[0032] 2. Model training: a. Pre-training stage: Use the Chinese Wikipedia corpus to train the ALBERT encoder; b. Incremental learning stage: Load medical field data (the labeled amount is only 10% of the general data), and jointly optimize entity, relationship, and classification tasks.

[0033] 3. Generation and parsing: a. Input text: "The patient has fever and cough, and is diagnosed with pneumonia." b. Output JSON: {"entities": [{"type": "symptom", "value": "fever"}, {"type": "disease", "value": "pneumonia"} "relations": [{"subject": "fever", "object": "pneumonia", "type": "accompanies"}]}

[0034] The embodiments described above merely represent certain implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.< / cls>

Claims

1. A unified information extraction method based on Transformer, characterized in that, It includes the following steps: S1. Data unification construction: Encode the entity, relationship, and classification information in the original text into a predefined JSON format, generate a serialized annotation template by introducing special identifiers, and form standardized training data containing type tags and associated relationships; S2. Model architecture design: Build a multi-layer encoder-decoder structure based on Transformer. The encoder uses a bidirectional attention mechanism to extract deep semantic features of the text, and the decoder couples entity recognition, relationship extraction, and text classification tasks through a shared parameter mechanism to achieve multi-task collaborative optimization; S3. Decoding optimization: Introduce a finite state machine (FSM) during the sequence generation process to perform real-time verification of the JSON syntax structure, and combine the beam search algorithm to filter candidate sequences that conform to the predefined pattern to ensure the structural correctness of the generated results; S4. Progressive training strategy: Through a two-stage training mechanism, first perform semantic understanding pre-training based on general corpus, and then inject domain expertise through multi-task incremental learning.

2. The unified information extraction method based on Transformer according to claim 1, wherein, The encoder in step S2 uses a pre-trained ALBERT model, which includes 12 stacked Transformer encoding layers. Each layer includes a multi-head self-attention sublayer, a feed-forward neural network sublayer, and a residual connection and layer normalization mechanism, and all encoding layers share weight parameters.

3. A unified information extraction method based on Transformer according to claim 1, characterized in that, The finite state machine (FSM) in step S3 specifically includes: (a) Define JSON syntax state transition rules to constrain the set of legal characters during the generation process; (b) During decoding, perform an intersection operation on the Top-k candidate characters and the FSM allowed character set, and apply a negative infinity mask to illegal characters; (c) When legal characters cannot be generated for a continuous preset number of times, backtrack to the nearest valid state node.

4. The unified information extraction method based on Transformer according to claim 1, characterized in that The two-stage training in step S4 includes: (a) Pre-training stage: Optimize the semantic representation ability of the encoder based on general corpus; (b) Incremental learning stage: Jointly optimize the decoder parameters of entity recognition, relationship extraction, and text classification tasks through a multi-task loss function, and the loss weights of each task are dynamically adjusted according to the annotation data distribution.

5. The unified information extraction method based on Transformer according to claim 1, wherein The method further includes a result parsing module, and the specific steps are: (a) Verify the JSON syntax validity of the generated sequence through regular expressions; (b) Force-convert numerical values in string form to the corresponding data type; (c) Unify the key name format and remove redundant whitespace; (d) Locally discard fields that fail to be parsed and record exception logs.

6. The unified information extraction method based on Transformer according to claim 1, characterized in that The decoder is composed of 2 stacked Transformer decoding layers. Each layer includes a masked multi-head self-attention sublayer, an encoder-decoder attention sublayer, a feed-forward neural network sublayer, and a residual connection and layer normalization mechanism, and the masked multi-head self-attention sublayer uses a causal masking mechanism to prevent information leakage.

7. A unified information extraction method based on Transformer according to claim 3, characterized in that, The finite state machine (FSM) dynamically updates the state transition rules during decoding, specifically including: expanding or contracting the allowed character set according to the context semantics of the current generation position.

8. A unified information extraction method based on Transformer according to claim 1, characterized in that The beam width of the beam search algorithm is set to 3 - 5, and the beam width is dynamically adjusted according to the confidence of the candidate sequence during the generation process. When the confidence is lower than the preset threshold, the beam width is reduced to improve the generation efficiency.

9. A unified information extraction method based on Transformer according to claim 1, characterized in that, The learning rate used in the pre-training stage is from 1e-5 to 5e-5, the batch size is 16 - 32, and the parameters of the first 6 layers of the encoder are frozen during the training process to improve the training stability.