Relationship extraction method for precast concrete component quality standard field
By using the ALBERT-BiGRU-SA model, combined with the ALBERT pre-trained model, bidirectional gated recurrent units, and self-attention mechanism, the problem of identifying long-distance dependencies and complex relationships in the field of quality standards for precast concrete components was solved, achieving efficient and accurate relationship extraction.
Patent Information
- Application Number
- CN202411036970.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the field of quality standards for precast concrete components, existing technologies struggle to effectively handle long-distance dependencies and complex technical terms, resulting in low accuracy and efficiency in relation extraction.
The ALBERT-BiGRU-SA model is adopted, which combines the ALBERT pre-trained model, bidirectional gated recurrent units and self-attention mechanism to improve the ability to capture long-distance dependencies and understand complex relationships.
It improves the accuracy and speed of relation identification, reduces computational resource consumption, and is suitable for large-scale data processing in resource-constrained environments.
Smart Images

Figure CN119005134A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a new relationship identification method specifically for the field of quality standards of prefabricated concrete components. Background Art
[0002] As a pillar industry of the national economy, the construction industry is an important part of the modern industry. Leading the construction of a modern industrial system with scientific and technological innovation and taking intelligent construction as a starting point, we will vigorously promote the digital transformation and high-quality development of the construction industry. As the basic material of prefabricated concrete buildings, the quality control of prefabricated concrete components is critical to the overall quality of the building. Therefore, scientific and technological innovation research in the field of prefabricated concrete components is of great significance.
[0003] With the development of the industry, the informatization of the construction industry has made remarkable achievements, gradually developing from the original job level and department level to the enterprise level and social level. Among them, project intelligence has gradually become the main means of quality control and cost control. However, in the field of prefabricated buildings, the application of intelligence is mostly limited to the three-dimensional structural design stage of concrete structures, and there is a lack of intelligent means in the field of quality control of prefabricated concrete components. The number of national standards for prefabricated concrete components in the public sector has increased year by year. The content of the standards is large and complicated, and most companies have their own standards according to the specific types of production components. In view of the many standards on the market, it is crucial to study an intelligent and professional standard knowledge organization and utilization method for the development of the industry. Knowledge graph happens to be an extremely excellent knowledge organization and utilization tool. In previous studies, other industries have explored and practiced the use of knowledge graphs to integrate and apply knowledge, and then developed decision-making software based on knowledge graphs, and achieved remarkable results in the process. One of the core tasks of building knowledge graphs is relationship extraction. Relationship extraction is a core area of information extraction. Its main goal is to identify specific relationships between entities from text. In this field, relationships between entities can be extracted. Over time, relation extraction technology has gone through three main stages: rule-based methods, traditional machine learning methods, and deep learning-based methods. In the field of machine learning, it is further divided into supervised learning, unsupervised learning, and semi-supervised learning. In the deep learning stage, the focus is on supervised learning and remote supervised learning.
[0004] Rule-based relation extraction methods mainly rely on manually written rules to identify and extract relations from text. Such methods are usually divided into trigger word-based methods and dependency analysis-based methods. Its advantage is that it can achieve high accuracy in specific scenarios, but it has disadvantages: low recall rate, time-consuming and labor-intensive rule writing and difficult maintenance, and poor adaptability and robustness in different environments. Machine learning-based methods can be divided into supervised learning, semi-supervised learning, and unsupervised learning. Supervised learning has the advantage of high accuracy, but it relies on a large amount of labeled data, and the labeling process is time-consuming and costly. Semi-supervised learning combines a small amount of labeled data with a large amount of unlabeled data to reduce the need for labeling, but the performance of the model depends on the quality of the labeled data. Unsupervised learning discovers patterns through the intrinsic structure of the data without the need for data labeling, but its accuracy and reliability are usually lower than those of supervised learning methods. In the field of deep learning, relation extraction mainly uses the pipeline method and joint extraction method of supervised learning. The pipeline method divides entity recognition and relationship classification into two independent steps, each of which can be optimized independently, which is convenient for error analysis and debugging. The joint extraction method is end-to-end, processing entity recognition and relationship classification at the same time, improving overall performance, but with high complexity and resource requirements. In 2013, Socher et al. proposed a relation extraction method based on RNN, which assigns a vector and a matrix to each node of the analysis tree, effectively solving the limitations of the word vector space model in long phrase parsing. The RNN model has advantages in processing sequence information, but faces gradient explosion and long-term dependency problems. CNN improves efficiency and performance through feature extraction layer and mapping layer as well as weight sharing mechanism. After the above-mentioned early deep learning model cycle, people began to consider how to deal with long dependency problems, and two variants based on RNN, LSTM and GRU, were proposed. LSTM effectively learns and retains context through its unique gating mechanism and cell state. GRU and its variant BiGRU capture context information by introducing a gating mechanism. Compared with LSTM, GRU has a simpler structure, faster running speed and less resource consumption. In addition, Zhou and Zhang et al., Lu Y et al. studied BiGRU in combination with attention mechanism in the fields of finance and military respectively, and found that the relation extraction performance is excellent. In summary, BiGRU's advantage in the field of relation extraction lies in its ability to comprehensively capture and analyze long-distance dependencies in text data, while providing richer and more accurate contextual understanding by integrating forward and reverse information. In addition, when combined with the attention mechanism, BiGRU can further improve the accuracy and efficiency of relation extraction. These features make BiGRU a powerful tool for handling complex relation extraction tasks, especially in application scenarios that require high accuracy and in-depth analysis.
[0005] In the process of constructing the knowledge graph in the field of quality standards for precast concrete components, after completing named entity recognition, it is necessary to extract the relationships between entities. The accuracy and efficiency of relationship extraction directly affect the integrity and practicality of the knowledge graph. However, relationship extraction in this field faces the problem of long-distance dependency caused by the long distance between different entity relationships in a sentence, as well as the challenges brought to traditional relationship extraction algorithms by the complexity of professional vocabulary. These factors jointly affect the performance of traditional algorithms in terms of accuracy and computational efficiency.
[0006] In view of this, the present invention provides a relationship extraction method for the field of quality standards of precast concrete components to solve the technical problems existing in the above-mentioned prior art. Summary of the invention
[0007] In view of the above-mentioned technical deficiencies, the present invention provides a new relationship identification method ALBERT-BiGRU-SA specifically for the field of quality standards for precast concrete components; first, the ALBERT pre-trained model is introduced. The ALBERT model not only reduces the resource consumption of the model but also maintains efficient performance through its lightweight design, namely factorized word embedding and cross-layer parameter sharing. In addition, the method further integrates the architecture based on the bidirectional gated recurrent unit and the self-attention mechanism to enhance the model's ability to capture long-distance dependencies and understand complex relationship types. The introduction of the self-attention mechanism is particularly aimed at improving the ability to recognize relationships between entities that are far apart in a sentence, thereby effectively processing relationship extraction in long sentences. Combined with BiGRU, this method not only improves the accuracy of relationship recognition, but also improves the processing speed through the parallel computing characteristics of the self-attention mechanism.
[0008] In order to solve the technical problems mentioned above, the present invention proposes a new relationship identification method ALBERT-BiGRU-SA specifically for the field of quality standards of precast concrete components. The specific steps are as follows:
[0009] Step 1: In the text preprocessing phase, the input text data is processed and converted into a formatted sequence. This includes synchronously marking the entity locations, relationship types, and text itself in the text, ensuring that each sentence S is accurately parsed. This step is critical because it provides the model with the necessary input format, including sentences, entity tags, and their relationships.
[0010] Step 2: Perform word segmentation and word embedding. Each sentence is segmented and embedded through the ALBERT model to generate a word vector matrix E. The ALBERT model is used at this stage to convert text sequences into high-dimensional word vectors, which can effectively capture the semantic features and contextual information of words.
[0011] ALBERT not only maintains semantic representation capabilities close to BERT through factored word embedding, cross-layer parameter sharing, and sentence order prediction (SOP) technology, but also significantly reduces resource consumption during training and reasoning. In the field of quality standards for precast concrete components, the Chinese version of ALBERT provides effective solutions to the following challenges:
[0012] (1) Improved resource efficiency: ALBERT’s high efficiency is particularly important when processing large amounts of technical specification documents, allowing it to efficiently process large amounts of data even in resource-constrained environments.
[0013] (2) Long-distance dependency processing: Through SOP technology, ALBERT enhances the ability to recognize complex engineering terms in long sentences.
[0014] (3) Improved accuracy: Combined with the BMES annotation method, the Chinese version of ALBERT provides more fine-grained Chinese text analysis capabilities and improves recognition accuracy.
[0015] ALBERT word embedding first has input text T{w1,w2,…,w n}, after tokenization and optional word segmentation, the text is broken down into smaller units (such as words or subwords). Each word or subword is converted into a corresponding word vector. This conversion involves looking up a pre-trained word embedding table and using parameter sharing technology to reduce storage and computing requirements. The word embedding process is shown in formula (1).
[0016] E = ALBERT_Embedding(T) (1)
[0017] Finally, we get the output word vector matrix E, which contains word vectors, sentence vectors, and position vectors. We pass the matrix E to other parts of the model for processing.
[0018] Step 3: Perform sequence modeling and apply self-attention mechanism. The word embedding matrix E is then input to the BiGRU layer, which processes the sequence using a bidirectional gated recurrent unit to capture forward and backward contextual dependencies and outputs a hidden state matrix H. Then, the self-attention mechanism is applied to H to optimize the relationship representation between entities and generate the final result matrix A .
[0019] Bidirectional Gated Recurrent Unit (BiGRU) is a variant of Recurrent Neural Network (RNN) that combines the features of GRU and bidirectional RNN. BiGRU consists of two independent GRUs, which process the forward and reverse information of the sequence respectively, and finally integrate the information from both directions to form a comprehensive contextual understanding of the current input point. For an input sequence X = {x1, ..., x n-1 , x t}, the forward and reverse GRU of the BiGRU model are calculated as follows:
[0020] Forward GRU: For time t, the forward hidden state The calculation is shown in formula (2).
[0021]
[0022] Reverse GRU: Reverse hidden state for time t The calculation is shown in formula (3).
[0023]
[0024] Finally, for the total hidden state h at time t t It consists of the concatenation of the forward and reverse hidden states, and its calculation process is shown in formula (4).
[0025]
[0026] The bidirectional sequence structure of BiGRU is similar to that of BiLSTM. In the practice of this method, there are fewer long-distance dependencies, most of which are sentence-level dependencies, and BiGRU is not inferior to BiLSTM in parsing long-distance dependencies, so BiGRU with fewer parameters is selected as the main body. Compared with BiLSTM, BiGRU has fewer parameters, which makes BiGRU have advantages in model size and computational efficiency. Due to fewer parameters, BiGRU tends to converge faster during training, which is especially beneficial for tasks with limited training time or large data scale. Although BiLSTM can capture more complex information in some cases, BiGRU can also effectively learn the characteristics of sequence data through a simpler structure, and sometimes even provide better generalization ability. The update gate and reset gate mechanism of GRU simplifies the calculation of hidden states, which helps to quickly update and retain key information in relation extraction.
[0027] In summary, BiGRU combines the parameter efficiency of GRU and the comprehensive context capture capability of bidirectional RNN, making it a highly competitive choice in relation extraction tasks. Especially when faced with the need to balance model performance and computing resources, BiGRU provides an effective alternative. In the field of precast concrete component quality standards, this feature makes BiGRU particularly suitable for extracting complex entity relationships in professional texts of precast concrete component quality standards.
[0028] The self-attention mechanism can enhance the ability to capture key information at the sentence level. The self-attention mechanism is a technique that enables the model to focus on the internal relationships between all words in the input sentence. It obtains a comprehensive representation of the sentence by calculating the relationship between each word and all other words in the sentence. First, the W sequence {w1, w2, ..., w n}, the self-attention mechanism generates a vector representation for the sentence through the following steps:
[0029] First, the vector is linearly transformed, that is, for each word w in the sentence i , respectively generate query (Query, Q), key (Key, K) and value (Value, V) vectors. i =W Q w i , K i =W K w i , V i =W V w i .W Q , W K , W V is a learnable weight matrix. Then the attention calculation is performed for each pair of words w i and w j , calculate its attention score, and the score calculation is shown in formula (5).
[0030]
[0031] Obtain the attention weight and use the Softmax function to obtain the standardized attention weight, as shown in formula (6).
[0032] α ij =softmax(Score(Q i , K j )) (6)
[0033] Finally, the sentence vector representation is generated, that is, the representation of the sentence is calculated using the attention weight and value vector, as shown in formula (7).
[0034]
[0035] At this time, the matrix obtained in the previous layer is combined with S i The concatenation is passed to the hidden state matrix H and then passed to the next level of the model.
[0036] When processing text in the field of quality standards for precast concrete components, the self-attention mechanism demonstrates its unique advantages as follows:
[0037] Capturing complex entity relationships. The quality standard documents for precast concrete components are full of professional terms and entities, and there are often complex relationships between these entities. The self-attention mechanism effectively captures the complex contextual relationships between these entities through in-depth analysis of the text, significantly improving the accuracy of entity recognition.
[0038] Effective processing of long sentences. One of the characteristics of documents in this field is the widespread existence of long sentences. The self-attention mechanism provides a more comprehensive sentence-level representation by comprehensively analyzing the association between each word in the sentence and other words, thereby effectively processing long sentences.
[0039] Reduce information loss. Compared with traditional sequence models, the self-attention mechanism retains more information when processing long-distance dependencies, which is crucial for understanding texts with long sentences and complex sentence structures. This is especially important for accurately analyzing information in quality standard documents for precast concrete components.
[0040] By integrating the self-attention mechanism into the model, it is possible to more effectively process complex text in the field of precast concrete component quality standards. This approach not only improves the accuracy of the model's entity recognition, but also enhances its ability to understand long and complex sentences. Based on the above characteristics, the model can ensure high-precision sentence-level vector representation of precast concrete component quality standard text while ensuring processing efficiency, providing strong support for in-depth analysis and knowledge extraction.
[0041] Step 4: Probability calculation and decoding. Process A through the softmax function to obtain the probability distribution P of each possible relationship. Combining P to obtain the relationship sequence R and the output of this step of relationship recognition is to select the most likely relationship type based on the highest probability.
[0042] The softmax decoding layer is introduced to convert the output of the model into a probability distribution, so as to classify each possible relationship category. In the relationship extraction task, the goal of the model is to identify the relationship between entity pairs from the text. The softmax layer is usually located at the last stage of the neural network and is responsible for converting the output of the previous layer, that is, the feature representation of the entity pair, into the probability of predicting the relationship category. Given a feature vector z of an entity pair, the softmax layer calculates the probability P(c|z) of each possible relationship category c as shown in formula (8).
[0043]
[0044] Among them, z c is the component of the feature vector z corresponding to category c, Yes c The exponent, denominator is the sum of the exponentials of all possible classes, ensuring that the sum of all class probabilities is 1.
[0045] In the context of relation extraction, the input of the Softmax layer is the feature vectors extracted by other context encoding models, which encode the information of entity pairs and their contexts. The output of the Softmax layer is a probability distribution representing the probability that a given entity pair belongs to each relation category. Combined with other layers of the neural network, the Softmax layer can be trained in an end-to-end manner, allowing gradients to be directly back-propagated from the loss function, optimizing the overall performance of the model.
[0046] In summary, in the relation extraction task, the Softmax decoding layer is a key component that enables the model to learn how to distinguish different relation categories from complex text features. By converting the high-dimensional feature vectors of deep neural networks into probability distributions, the Softmax layer provides a way to perform multi-category classification and provides interpretable and easy-to-operate results for the relation extraction task.
[0047] Finally, the output of the relation extraction model is:
[0048]
[0049] in Represents a set of predefined relationship types, and S represents an entity set.
[0050] The beneficial effect of the present invention is that in the field of quality standards for precast concrete components, which is professional knowledge-intensive and full of complex professional terms and technical specifications, the ALBERT-BiGRU-SA model first explores the effect of introducing the ALBERT pre-trained model. The ALBERT model not only reduces the resource consumption of the model but also maintains efficient performance through its lightweight design, namely factorized word embedding and cross-layer parameter sharing. In addition, the method further integrates the architecture based on bidirectional gated recurrent units and the self-attention mechanism to enhance the model's ability to capture long-distance dependencies and understand complex relationship types. The introduction of the self-attention mechanism is particularly aimed at improving the ability to recognize relationships between entities that are far apart in a sentence, thereby effectively processing relationship extraction in long sentences. Combined with BiGRU, this method not only improves the accuracy of relationship recognition, but also improves the processing speed through the parallel computing characteristics of the self-attention mechanism. The overall architecture of the model is designed to improve the accuracy and processing efficiency of relationship extraction in the field of quality standards for precast concrete components. At the same time, while improving processing accuracy and efficiency, it is also committed to reducing the consumption of computing resources, providing a more efficient and resource-saving solution for processing complex text analysis in the professional field of quality standards for precast concrete components. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1The overall architecture diagram of the pre-trained model for text representation enhancement provided by the present invention;
[0052] Figure 2 A schematic diagram of an experimental process of the relationship extraction model provided by the present invention.
[0053] Figure 3 ALBERT-BiGRU-SA recognition performance statistics DETAILED DESCRIPTION
[0054] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0055] A relation extraction method for the field of quality standards for precast concrete components. In the text preprocessing stage, the input text data is processed and converted into a formatted sequence. This includes synchronously marking the entity positions, relation types, and text itself in the text to ensure that each sentence S is accurately parsed. This step is critical because it provides the necessary input format for the model, including sentences, entity tags, and their relationships. When performing word segmentation and word embedding, each sentence is segmented and embedded through the ALBERT model to generate a word vector matrix E. The ALBERT model is used in this stage to convert text sequences into high-dimensional word vectors that can effectively capture the semantic features of words and contextual information. When performing sequence modeling and applying the self-attention mechanism. The word vector matrix E is then input to the BiGRU layer, which processes the sequence using a bidirectional gated recurrent unit to capture forward and backward contextual dependencies and outputs a hidden state matrix H. Then, the self-attention mechanism is applied to H to optimize the relationship representation between entities and generate the final result matrix A. Probability calculation and decoding are then performed. A is processed through the softmax function to obtain the probability distribution P of each possible relationship. Combining P to obtain the relationship sequence R and the output of this step of relationship identification, the most likely relationship type is selected based on the highest probability. The specific steps are as follows:
[0056] Step 1: Data Preprocessing
[0057] Since the experimental data involves a new field and there is no known public data set, the data sources used in the experiment are all self-built local data sets. The data sources are all obtained by sorting and extracting the quality standard files of precast concrete components within China Construction Third Engineering Bureau. Since the quality standard data is relatively complex, the inspection standards and precautions are selected and converted into recognizable TXT files through OCR recognition and Python scripting.
[0058] Then the data was preliminarily cleaned. The general principle was that the text in a chapter was displayed in one line as one piece of data, which was convenient for later segmentation and annotation of data by chapter entries. Subsequently, a script was written and manually removed the garbled characters recognized by the OCR script. Finally, the data entries (one per chapter) were obtained by classifying the chapters and processing the character materials. These data included data on building construction production, pipe gallery production, and pipe segment production. In order to make the data available, each piece of data was collected in a TXT document, and each piece of data was separated by a line break for subsequent experimental use. Some examples of the collected data are shown in Table 1.
[0059] Table 1 Text data examples
[0060]
[0061]
[0062] Step 2: Data Annotation
[0063] Data annotation requires entity annotation and relationship annotation. Entity recognition must be completed before relationship extraction. Since the present invention describes a method for relationship extraction and entity recognition has been completed before, only a brief description is given here. Before annotating the entity, the entity needs to be defined. According to the guidance of a dedicated person and the specific document structure of the quality standard for precast concrete components, the entity can be divided into four categories: project steps (XMBZ), operation subjects (CZZT), attribute step requirements (SXBZYQ), and operation step requirements (CZBZYQ).
[0064] Then the entities are annotated. This article intends to use the "BMES" annotation method, that is, for each annotated entity, "B" represents the beginning of the entity, "M" represents the middle of the entity, "E" represents the end of the entity, "S" represents the single character of the entity, and "O" represents that the character is not an entity. For example, the annotation result of the operation subject "Binding Rebar Bend Hook" is "Binding B-CZZT Rebar M-CZZT Bend M-CZZT Hook E-CZZT". The complete annotation rules are shown in Table 2.
[0065] Table 2 Entity annotation rules
[0066]
[0067] After determining the text annotation method, select the Elf Annotation Assistant as the annotation tool. As an AI-related data annotation tool, it can annotate entities and relationships within a text segment. During the annotation process, first import the text content in TXT format into the Elf Annotation, and then the Elf Annotation will automatically split the text into 212 independent sub-annotation tasks based on the text.
[0068] Users browse the text content on the page and perform operations according to the predetermined requirements to select entities in the text, assign labels to these entities, and establish relationships between entities. Repeat this series of operations until all documents are annotated. After the annotation is completed, the annotation assistant will automatically organize and export the data in JSON format. In the exported JSON file, "E" represents entity, "R" represents relationship, and "content" is the text content. These key nodes are the elements that must be processed in the entity recognition process. By writing a script, you can read the content in "content" and the corresponding "E" label, and automatically convert them into the "BMES" annotation format. The entire dataset is divided into three parts according to the standard "training-validation-testing", with a ratio of 8:1:1. After the division is completed, the three types of datasets are saved in the BMES format for subsequent experiments. This process ensures accurate annotation and efficient management of data, providing structured and high-quality data support for subsequent model training and verification.
[0069] In summary, in the process of data acquisition and annotation in this study, the automated tool "Elf Annotation Assistant" and Python scripts were used to process text data in the field of quality standards for precast concrete components. This method not only improves the efficiency of annotation, but also ensures the high quality of the annotation results. Through this automated tool, this paper can quickly and accurately annotate a large amount of data, which is crucial for building accurate named entity recognition and relationship extraction models. In general, this method improves work efficiency while maintaining annotation quality, providing a solid data foundation for subsequent model training and analysis.
[0070] Step 3: Text preprocessing. In the text preprocessing phase, the input text data is processed and converted into a formatted sequence. This includes synchronously marking the entity locations, relationship types, and text itself in the text, ensuring that each sentence S is accurately parsed. This step is critical because it provides the model with the necessary input format, including sentences, entity tags, and their relationships.
[0071] Step 4: Perform word segmentation and word embedding. Each sentence is segmented and embedded through the ALBERT model to generate a word vector matrix E. The ALBERT model is used at this stage to convert text sequences into high-dimensional word vectors, which can effectively capture the semantic features and contextual information of words.
[0072] ALBERT not only maintains semantic representation capabilities close to BERT through factored word embedding, cross-layer parameter sharing, and sentence order prediction (SOP) technology, but also significantly reduces resource consumption during training and reasoning. In the field of quality standards for precast concrete components, the Chinese version of ALBERT provides effective solutions to the following challenges:
[0073] (1) Improved resource efficiency: ALBERT’s high efficiency is particularly important when processing large amounts of technical specification documents, allowing it to efficiently process large amounts of data even in resource-constrained environments.
[0074] (2) Long-distance dependency processing: Through SOP technology, ALBERT enhances the ability to recognize complex engineering terms in long sentences.
[0075] (3) Improved accuracy: Combined with the BMES annotation method, the Chinese version of ALBERT provides more fine-grained Chinese text analysis capabilities and improves recognition accuracy.
[0076] ALBERT word embedding first has input text T{w1,w2,…,w n}, after tokenization and optional word segmentation, the text is broken down into smaller units (such as words or subwords). Each word or subword is converted into a corresponding word vector. This conversion involves looking up a pre-trained word embedding table and using parameter sharing technology to reduce storage and computing requirements. The word embedding process is shown in formula (1).
[0077] E = ALBERT_Embedding(T) (1)
[0078] Finally, we get the output word vector matrix E, which contains word vectors, sentence vectors, and position vectors. We pass the matrix E to other parts of the model for processing.
[0079] Integrating the ALBERT layer not only effectively improves the accuracy of recognition, but also demonstrates advantages in terms of resource consumption, ensuring the high standards of the model's technical performance and optimized efficiency. This comprehensive consideration provides strong technical support for the construction and application of knowledge graphs in the field of quality standards for precast concrete components, and demonstrates the important value of ALBERT in processing text analysis tasks in professional fields.
[0080] Step 5: Perform sequence modeling and apply self-attention mechanism. The word embedding matrix E is then input to the BiGRU layer, which processes the sequence using a bidirectional gated recurrent unit to capture forward and backward contextual dependencies and outputs a hidden state matrix H. Then, the self-attention mechanism is applied to H to optimize the relationship representation between entities and generate the final result matrix A.
[0081] Bidirectional Gated Recurrent Unit (BiGRU) is a variant of Recurrent Neural Network (RNN) that combines the features of GRU and bidirectional RNN. BiGRU consists of two independent GRUs, which process the forward and reverse information of the sequence respectively, and finally integrate the information from both directions to form a comprehensive contextual understanding of the current input point. For an input sequence X = {x1, ..., xn-1 , x t}, the forward and reverse GRU of the BiGRU model are calculated as follows:
[0082] Forward GRU: For time t, the forward hidden state The calculation is shown in formula (2).
[0083]
[0084] Reverse GRU: For the time-wide reverse hidden state The calculation is shown in formula (3).
[0085]
[0086] Finally, for the total hidden state h at time t t It consists of the concatenation of the forward and reverse hidden states, and its calculation process is shown in formula (4).
[0087]
[0088] The bidirectional sequence structure of BiGRU is similar to that of BiLSTM. In the practice of this method, there are fewer long-distance dependencies, most of which are sentence-level dependencies, and BiGRU is not inferior to BiLSTM in parsing long-distance dependencies, so BiGRU with fewer parameters is selected as the main body. Compared with BiLSTM, BiGRU has fewer parameters, which makes BiGRU have advantages in model size and computational efficiency. Due to fewer parameters, BiGRU tends to converge faster during training, which is especially beneficial for tasks with limited training time or large data scale. Although BiLSTM can capture more complex information in some cases, BiGRU can also effectively learn the characteristics of sequence data through a simpler structure, and sometimes even provide better generalization ability. The update gate and reset gate mechanism of GRU simplifies the calculation of hidden states, which helps to quickly update and retain key information in relation extraction.
[0089] In summary, BiGRU combines the parameter efficiency of GRU and the comprehensive context capture capability of bidirectional RNN, making it a highly competitive choice in relation extraction tasks. Especially when faced with the need to balance model performance and computing resources, BiGRU provides an effective alternative. In the field of precast concrete component quality standards, this feature makes BiGRU particularly suitable for extracting complex entity relationships in professional texts of precast concrete component quality standards.
[0090] The self-attention mechanism can enhance the ability to capture key information at the sentence level. The self-attention mechanism is a technique that enables the model to focus on the internal relationships between all words in the input sentence. It obtains a comprehensive representation of the sentence by calculating the relationship between each word and all other words in the sentence. First, the W sequence {w1, w2, ..., w n}, the self-attention mechanism generates a vector representation for the sentence through the following steps:
[0091] First, the vector is linearly transformed, that is, for each word w in the sentence i , respectively generate query (Query, Q), key (Key, K) and value (Value, V) vectors. i =W Q w i , K i =W K w i , V i =W V w i .W Q , W K , W V is a learnable weight matrix. Then the attention calculation is performed for each pair of words w i and w j , calculate its attention score, and the score calculation is shown in formula (5).
[0092]
[0093] Obtain the attention weight and use the Softmax function to obtain the standardized attention weight, as shown in formula (6).
[0094] α ij =softmax(Score(Q i , K j )) (6)
[0095] Finally, the sentence vector representation is generated, that is, the representation of the sentence is calculated using the attention weight and value vector, as shown in formula (7).
[0096]
[0097] At this time, the matrix obtained in the previous layer is concatenated with Si to form a hidden state matrix H and passed to the next level of the model.
[0098] When processing text in the field of quality standards for precast concrete components, the self-attention mechanism demonstrates its unique advantages as follows:
[0099] (1) Capturing complex entity relationships. The quality standard documents for precast concrete components are full of professional terms and entities, and there are often complex relationships between these entities. The self-attention mechanism effectively captures the complex contextual relationships between these entities through deep analysis of the text, significantly improving the accuracy of entity recognition.
[0100] (2) Effective processing of long sentences. One of the characteristics of documents in this field is the widespread existence of long sentences. The self-attention mechanism provides a more comprehensive sentence-level representation by comprehensively analyzing the relationship between each word in the sentence and other words, thereby effectively processing long sentences.
[0101] (3) Reduce information loss. Compared with traditional sequence models, the self-attention mechanism retains more information when processing long-distance dependencies, which is crucial for understanding texts with long sentences and complex sentence structures. This is especially important for accurately analyzing information in quality standard documents for precast concrete components.
[0102] By integrating the self-attention mechanism into the model, it is possible to more effectively process complex text in the field of precast concrete component quality standards. This approach not only improves the accuracy of the model's entity recognition, but also enhances its ability to understand long and complex sentences. Based on the above characteristics, the model can ensure high-precision sentence-level vector representation of precast concrete component quality standard text while ensuring processing efficiency, providing strong support for in-depth analysis and knowledge extraction.
[0103] Step 6: Probability calculation and decoding. Process A through the softmax function to obtain the probability distribution P of each possible relationship. Combining P to obtain the relationship sequence R and the output of this step of relationship recognition is to select the most likely relationship type based on the highest probability.
[0104] The softmax decoding layer is introduced to convert the output of the model into a probability distribution, so as to classify each possible relationship category. In the relationship extraction task, the goal of the model is to identify the relationship between entity pairs from the text. The softmax layer is usually located at the last stage of the neural network and is responsible for converting the output of the previous layer, that is, the feature representation of the entity pair, into the probability of predicting the relationship category. Given a feature vector z of an entity pair, the softmax layer calculates the probability P(c|z) of each possible relationship category c as shown in formula (8).
[0105]
[0106] Among them, z c is the component of the feature vector z corresponding to category c, Yes c The exponent, denominator is the sum of the exponentials of all possible classes, ensuring that the sum of all class probabilities is 1.
[0107] In the context of relation extraction, the input of the Softmax layer is the feature vectors extracted by other context encoding models, which encode the information of entity pairs and their contexts. The output of the Softmax layer is a probability distribution representing the probability that a given entity pair belongs to each relation category. Combined with other layers of the neural network, the Softmax layer can be trained in an end-to-end manner, allowing gradients to be directly back-propagated from the loss function, optimizing the overall performance of the model.
[0108] In summary, in the relation extraction task, the Softmax decoding layer is a key component that enables the model to learn how to distinguish different relation categories from complex text features. By converting the high-dimensional feature vectors of deep neural networks into probability distributions, the Softmax layer provides a way to perform multi-category classification and provides interpretable and easy-to-operate results for the relation extraction task.
[0109] Finally, the output of the relation extraction model is:
[0110]
[0111] in Represents a set of predefined relationship types, and S represents an entity set.
[0112] Experimental analysis:
[0113] (1) Comparative ablation experiment
[0114] In order to verify the effectiveness of the ALBERT-BiGRU-SA model in relational recognition in this field and the role of each part of the model, this model sets up comparative experiments and ablation experiments. The comparative experiment settings are shown in Table 3. Three commonly used models are set up to compare with this model on the data set in this field. The ablation experiment is shown in Table 4. The experimental comparison verifies the influence of the ALBERT word embedding module and the self-attention mechanism module on the parameters.
[0115] Table 3 Comparative experimental design table
[0116]
[0117] Table 4 Ablation experiment design table
[0118]
[0119] Table 5 Comparison of ablation experiment results
[0120]
[0121] First, the comparative experiment is shown in Table 3. According to the experimental results, the performance of the model gradually improves from BiLSTM to ALBERT-BiGRU-SA, which is specifically reflected in the significant improvement in Precision, Recall and F1 Score. Comparing BiLSTM with BiGRU, BiGRU has a slight improvement in all evaluation indicators compared to BiLSTM, indicating that BiGRU has better performance in relation extraction tasks when processing professional data than BiLSTM, probably due to its more efficient parameter update and lower computational complexity. BiGRU has a small number of units and a higher overall computational efficiency of the model, so choosing BiGRU as the main body of the relation extraction model can improve the overall computational efficiency.
[0122] To explore the impact of introducing the self-attention mechanism, when comparing BiLSTM and BiLSTM-SA, it can be clearly seen that the introduction of the self-attention mechanism significantly improves the performance of the model, especially in Recall and F1 Score. This shows that the self-attention mechanism can effectively capture long-distance dependencies at the sentence level and enhance the model's understanding and use of contextual information.
[0123] Compared with other models, ALBERT-BiGRU-SA showed the best performance, especially in terms of precision and recall, both reaching 88.67% and 88.63%, indicating that the introduction of the ALBERT pre-training model and the combination of the self-attention mechanism have a significant effect in improving the model's ability to handle complex relationships and contextual information in the quality standards of precast concrete components. Compared with the traditional BiGRU model, Precision, Recall, and F1 Score increased by 8.56%, 8.78%, and 8.67%, respectively.
[0124] Next is the ablation experiment, as shown in Table 5. The ablation experiment shows the contribution of each component to the model performance by gradually adding ALBERT and self-attention mechanism modules. The contribution of the self-attention mechanism can be shown in the comparison between BiGRU and BiGRU-SA. The model performance has been significantly improved, especially in precision, recall and F1 score, which further confirms the effectiveness of the self-attention mechanism in enhancing the model's ability to capture long-distance dependencies in relation extraction tasks.
[0125] The contribution of the ALBERT module is that ALBERT-BiGRU has improved all evaluation indicators compared to pure BiGRU. Although the improvement is not as significant as the self-attention mechanism, it also verifies the effectiveness of the ALBERT pre-trained model in improving the model's understanding and processing of relations in the field of quality standards for precast concrete components. After combining ALBERT with the self-attention mechanism, ALBERT-BiGRU-SA showed the best performance in the ablation experiment, further proving that ALBERT and the self-attention mechanism can have a significant impact on model performance when working together, especially when dealing with complex relationship extraction tasks, which can be understood and predicted more accurately.
[0126] Taking all the above factors into consideration, the ALBERT-BiGRU-SA model can more accurately identify entities and their relationships in text when dealing with relation extraction tasks in the field of quality standards for precast concrete components, significantly improving recognition accuracy. At the same time, due to the architectural advantages of BiGRU, the computational efficiency of extraction can be improved. This shows that by combining advanced pre-training models, self-attention mechanisms, and efficient sequence processing structures, the problem of low relation extraction recognition accuracy and computational efficiency in this field can be effectively solved.
[0127] (2) Model performance experiment
[0128] The statistical bar chart of the recognition performance of local precast concrete component quality standard data using the ALBERT-BiGRU-SA relation extraction algorithm is shown in Figure 3.
[0129] After conducting experimental research on relation extraction in this field, this paper found that the "No Relation" (NoRel) category has the highest recognition rate. The main reason for this phenomenon is that most sentences labeled as NoRel contain only a single named entity, making the recognition process relatively simple. In contrast, other relation categories involving double entity annotations are more difficult to recognize. In particular, the recognition rate of the "Involve" relation category is also relatively high, thanks to the ability of ALBERT word embedding vectors in capturing rich information and the advantage of BiGRU in processing long-distance relation extraction. However, for the two types of relations "Operation Requirement" (ORequire) and "Attribute Requirement" (ARequire), the recognition rate is relatively low. This is mainly because the entity boundaries involved in these relationship types are not obvious, and the two relationships themselves are relatively close, which leads to confusion in the recognition process. Nevertheless, thanks to the sentence-level feature capture capability provided by the self-attention mechanism, the recognition rate of these relations is still maintained above an acceptable level, thus ensuring a high level of overall recognition effect.
[0130] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for extracting relations in the field of quality standards for precast concrete components, characterized in that: The specific steps are as follows: Step 1: In the text preprocessing phase, the input text data is processed and converted into a formatted sequence; this includes synchronously marking the entity locations, relationship types, and text itself in the text, ensuring that each sentence S is accurately parsed; this step is critical because it provides the model with the necessary input format, including sentences, entity tags, and their relationships; Step 2: Perform word segmentation and word embedding. Each sentence is segmented and embedded through the ALBERT model to generate a word vector matrix E. The ALBERT model is used to convert text sequences into high-dimensional word vectors at this stage. These vectors can effectively capture the semantic features and contextual information of words. ALBERT word embedding first has input text T{w1,w2,…,w n }, after tokenization and optional word segmentation, the text is decomposed into smaller units (such as words or subwords), and each word or subword is converted into a corresponding word vector. This conversion involves looking up a pre-trained word embedding table and using parameter sharing technology to reduce storage and computing requirements; the word embedding process is shown in formula (1): E = ALBERT_Embedding(T) (1) Finally, we get the output word vector matrix E, which contains word vectors, sentence vectors, and position vectors. We pass the matrix E to other parts of the model for processing. Step 3: Perform sequence modeling and apply self-attention mechanism; the word embedding matrix E is then input to the BiGRU layer, which processes the sequence using a bidirectional gated recurrent unit to capture forward and backward contextual dependencies and outputs a hidden state matrix H; then, the self-attention mechanism is applied to H to optimize the relationship representation between entities and generate the final result matrix A; For an input sequence X = {x1, x n-1 , w t }, the forward and reverse GRU of the BiGRU model are calculated as follows: Forward GRU: Forward hidden state at time t The calculation is shown in formula (2): Reverse GRU: Reverse hidden state for time t The calculation is shown in formula (3): Finally, for the total hidden state h at time t t It consists of the concatenation of the forward and reverse hidden states, and its calculation process is shown in formula (4): BiGRU combines the parameter efficiency of GRU and the comprehensive context capture capability of bidirectional RNN, making it a highly competitive choice for relation extraction tasks; Especially when faced with the need to balance model performance and computing resources, BiGRU provides an effective alternative; in the field of precast concrete component quality standards, this feature makes BiGRU particularly suitable for extracting complex entity relationships in professional texts of precast concrete component quality standards; The self-attention mechanism can enhance the ability to capture key information at the sentence level. First, it receives the W sequence {w1, w2, ..., w n }, the self-attention mechanism generates a vector representation for the sentence through the following steps; First, the vector is linearly transformed, that is, for each word w in the sentence i , respectively generate query (Query, Q), key (Key, K) and value (Value, V) vectors; Q i =W Q w i , K i =W K w i , V i =W V w i ; W Q , W K , W V is a learnable weight matrix; attention is then calculated for each pair of words w i and w j , calculate its attention score, the score calculation is shown in formula (5): Get the attention weight and use the Softmax function to get the standardized attention weight, as shown in formula (6): α ij =softmax(Score(Q i ,K j )) (6) Finally, the sentence vector representation is generated, that is, the representation of the sentence is calculated using the attention weight and value vector; as shown in formula (7): At this time, the matrix obtained in the previous layer is combined with S i Concatenate into hidden state matrix H and pass it to the next level of the model; Step 4: Probability calculation and decoding; process A through the softmax function to obtain the probability distribution P of each possible relationship; combine P to obtain the relationship sequence R and the output of this step of relationship recognition to select the most likely relationship type based on the highest probability; The softmax decoding layer is introduced to convert the output of the model into a probability distribution, so as to classify each possible relationship category. In the relationship extraction task, the goal of the model is to identify the relationship between entity pairs from the text. The softmax layer is usually located at the last stage of the neural network and is responsible for converting the output of the previous layer, that is, the feature representation of the entity pair, into the probability of predicting the relationship category. Given a feature vector z of an entity pair, the softmax layer calculates the probability P(c|z) of each possible relationship category c as shown in formula (8): Among them, z c is the component of the feature vector z corresponding to category c, Yes c The exponent, denominator is the sum of the indices of all possible categories, which ensures that the sum of all category probabilities is 1; In the context of relation extraction, the input of the Softmax layer is the feature vectors extracted by other context encoding models, which encode the information of entity pairs and their contexts; In summary, in the relation extraction task, the Softmax decoding layer is a key component that enables the model to learn how to distinguish different relation categories from complex text features; by converting the high-dimensional feature vectors of deep neural networks into probability distributions, the Softmax layer provides a way to perform multi-category classification and provides interpretable and easy-to-operate results for the relation extraction task; Finally, the output of the relation extraction model is: in Represents a set of predefined relationship types, and S represents an entity set.
2. A method for extracting relations in the field of quality standards for precast concrete components according to claim 1, characterized in that: ALBERT described in step 2 not only maintains semantic representation capabilities close to BERT through factorized word embedding, cross-layer parameter sharing, and sentence order prediction (SOP) technology, but also significantly reduces resource consumption during training and inference.
3. The method for extracting relations in the field of quality standards for precast concrete components according to claim 1, characterized in that: The bidirectional gated recurrent unit (BiGRU) described in step 3 is a variant of the recurrent neural network (RNN). It combines the characteristics of the GRU and the bidirectional RNN. The BiGRU consists of two independent GRUs, which process the forward and reverse information of the sequence respectively, and finally integrate the information in the two directions to form a comprehensive contextual understanding of the current input point.
4. The method for extracting relations in the field of quality standards for precast concrete components according to claim 1, characterized in that: The self-attention mechanism described in step 3 is a technique that enables the model to focus on the internal relationships between all words in the input sentence; it obtains a comprehensive representation of the sentence by calculating the relationship between each word and all other words in the sentence.
5. The method for extracting relations in the field of quality standards for precast concrete components according to claim 1, characterized in that: In step 3, the self-attention mechanism is integrated into the model to process complex texts in the field of precast concrete component quality standards; the accuracy of the model in entity recognition is ensured, and the ability to understand long sentences and complex sentences is enhanced; while ensuring processing efficiency, high-precision sentence-level vector representation of precast concrete component quality standard text is ensured, providing strong support for in-depth analysis and knowledge extraction.
6. The method for extracting relations in the field of quality standards for precast concrete components according to claim 1, characterized in that: The output of the Softmax layer described in step 4 is a probability distribution representing the probability that a given entity pair belongs to each relation category; combined with other layers of the neural network, the Softmax layer can be trained in an end-to-end manner, allowing gradients to be directly back-propagated from the loss function to optimize the overall performance of the model.
7. A method for extracting relations in the field of quality standards for precast concrete components according to any one of claims 1 to 6, characterized in that: The text data in step 1 is obtained by sorting and extracting the quality standard documents of precast concrete components. Since the quality standard data is relatively complicated, the inspection standards and precautions are selected and converted into a recognizable TXT file through OCR recognition and Python scripting; Then the data is preliminarily cleaned. The general principle is to display the text in one chapter in one line as one piece of data, so as to facilitate the later segmentation and annotation of data by chapter entries. Then, a script is written and the garbled characters recognized by the OCR script are manually removed. Finally, we classified the data into chapters and processed the character materials to obtain data entries (one entry for each chapter). To make the data available, we collected each data entry in a TXT document and separated each data entry with a line break for easy use in subsequent experiments.
8. The method for extracting relations in the field of quality standards for precast concrete components according to claim 7, characterized in that: After the text data is processed in step 1, the data needs to be annotated. The data annotation includes entity annotation and relationship annotation: Entity labeling: Entity recognition should be completed before relationship extraction. Before labeling, entities need to be defined. According to the guidance of dedicated personnel and the document structure of specific precast concrete component quality standards, entities are divided into four categories: project steps (XMBZ), operation subjects (CZZT), attribute step requirements (SXBZYQ), and operation step requirements (CZBZYQ); Relationship annotation: Users browse the text content on the page and perform operations according to predetermined requirements to select entities in the text, assign labels to these entities, and establish relationships between entities. This series of operations is repeated until all documents are annotated; After the annotation is completed, the annotation assistant will automatically organize and export the data in JSON format. In the exported JSON file, "E" represents entity, "R" represents relationship, and "content" is text content. By writing a script, the content in "content" and the corresponding "E" tag are read and automatically converted into the "BMES" annotation format; The entire data set is divided into three parts according to the "training-validation-testing" standard, with a ratio of 8:1:
1. After the division is completed, the three types of data sets are saved in the BMES format.
Citation Information
Patent Citations
Text relation extraction method based on double-layer attention mechanism and bidirectional GRU
CN110502749A
Entity relationship extraction method for wind tunnel fault text knowledge
CN113901178A
Relationship extraction method and system based on pointer network and fragment arrangement fusion decoding
CN117408256A
