Railway construction quality element entity recognition method based on semantic completeness

Abstract

The application discloses a railway construction quality element entity identification method based on semantic completeness, and specifically comprises the following steps: step 1, randomly sampling and selecting quality detection related specifications from railway specifications to construct a corpus; step 2, constructing an RSC-NER named entity model; step 3, pre-training a Transformer encoder by using RSim-WWM; step 4, training the RSC-NER named entity model; and step 5, identifying railway construction quality element entities by using the RSC-NER model trained in step 4 to generate a classification result. The railway construction quality element entities are identified by using the application, so that the accuracy and effectiveness of quality element entity identification can be improved, the efficiency of construction quality detection and monitoring can be improved, and the labor cost can be reduced.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology and relates to a method for recognizing entity elements of railway construction quality based on semantic completeness. Background Technology

[0002] As a vital pillar industry in my country, the quality of railway construction directly impacts industry development, industrial upgrading, and national welfare. The continuous development of natural language processing technology has accelerated the digitalization and intelligentization of railway construction in my country, particularly improving quality monitoring and inspection technologies. However, due to the characteristics of railway engineering projects—large scale, high investment, long construction period, complex technology, numerous participating units, high risks, and difficulties in organization and implementation—quality control and inspection still face challenges such as massive amounts of information, slow data processing speed, and lengthy inspection cycles, and are currently in the early stages of development.

[0003] In intelligent railway construction, the digitalization and intelligentization process involves numerous objects, broadly categorized into four main types: bridges, roadbeds, tunnels, and tracks. Quality inspection of these objects is further subdivided into the inspection of multiple sub-objects; for example, bridge construction should include the inspection of foundation pile construction. Due to the complexity and tediousness of the quality inspection process, resulting in a vast amount of information, we need to extract key inspection objects from specifications and existing project construction logs as our construction quality elements. This allows for subsequent entity alignment and attribute comparison, improving the accuracy and efficiency of construction quality monitoring and reducing labor costs. Summary of the Invention

[0004] The purpose of this invention is to provide a method for identifying railway construction quality element entities based on semantic completeness. Using this method to identify railway construction quality element entities can improve the accuracy and effectiveness of quality element entity identification, increase the efficiency of construction quality inspection and monitoring, and reduce labor costs.

[0005] The technical solution adopted in this invention is a method for recognizing entities of railway construction quality elements based on semantic completeness, which specifically includes the following steps:

[0006] Step 1: Construct a corpus by randomly sampling quality inspection-related standards from railway specifications;

[0007] Step 2, construct the RSC-NER named entity model;

[0008] Step 3: Pre-train the Transformer encoder using RSim-WWM;

[0009] Step 4: Train the RSC-NER named entity model;

[0010] Step 5: Use the RSC-NER model trained in Step 4 to identify the entities of railway construction quality elements and generate classification results.

[0011] The invention is further characterized by:

[0012] In step 2, the RSC-NER model includes: text input, Transformer encoder, word embedding, introduction of semantic completeness features, vector concatenation, and softmax classification.

[0013] The RSC-NER model processes text sequences as follows:

[0014] 1) Import text sequences;

[0015] 2) The word embedding sequence vector is obtained through the Transformer encoder;

[0016] 3) Generate subsequence vectors by arranging fragments;

[0017] 4) Perform entity checks on each subsequence sequentially and then perform max pooling;

[0018] 5) Perform span embedding on the word embedding sequence;

[0019] 6) Copy the word embedding sequence and mask it to introduce semantic completeness features;

[0020] 7) Vector join;

[0021] 8) Softmax classification;

[0022] 9) Return the entity recognition and classification results.

[0023] The specific process of step 3 is as follows:

[0024] Step 3.1: Input the text sequence and encode it, representing the encoded text sequence as a vector A = (a1, a2, ..., a...). n );

[0025] Step 3.2, alternately mask the vector representation obtained in step 3.1, replace the masked vector with MASK, and the overall vector representation of the masked sentence is B. The semantic similarity between each masked sentence and the original sentence is calculated using the following formula (1):

[0026]

[0027] Step 3.3: Sort the semantic similarity of the masked sentences obtained after alternating masking in Step 3.2 from small to large, and select the word codes corresponding to the top 15% as the masked words for training.

[0028] Step 3.4, the masking step, the masked words selected in 3.3, the masked vector is embedded into the Transformer encoder in the bert-base-cased model for subsequent training and prediction of masked words;

[0029] Step 3.5: Repeat steps 3.1-3.4 above until the training accuracy reaches 50% or more, which is to obtain the trained Transformer encoder.

[0030] The specific process of step 4 is as follows:

[0031] Step 4.1: Input the Transformer encoder trained in Step 3 into the Named Entity Recognition (RSC-NER) model, and then input the text sequence (s1, s2, ..., s...) into the RSC-NER model. n ), text sequence (s1, s2, ..., s n After step 3, the Transformer encoder obtains the word embedding sequence (e1, e2, ..., e...). n );

[0032] Step 4.2: Construct subsequences of the word embedding sequence generated in Step 4.1 using the fragment construction method, and perform entity judgment on the constructed subsequences in turn;

[0033] Step 4.3: Perform max pooling on the word embedding subsequence from which entity detection is being performed in Step 4.2. The max-pooled representation vector is E. Concatenate vector E with the overall representation vector T of the text sequence to obtain a new vector e(c). The formula for calculating vector e(c) is:

[0034] e(c) = E·T (2);

[0035] Step 4.4: Use vector x to represent the span of the subsequence, and concatenate the span embedding x with the representation vector e(c) to obtain the vector w used for entity classification. t The calculation formula is as follows:

[0036] w t =e(c)·x (3);

[0037] Step 4.5, copy the word embedding sequence (e1, e2, ..., e) obtained in Step 4.1. n The process involves sequentially masking the subsequence currently being used for entity detection, and then replacing the masked subsequence with a MASK.

[0038] Step 4.6: Perform average pooling on the original sequence from Step 4.1 and the masked sequence from Step 4.5 respectively to obtain the text representation vector o of the original sequence and the text representation vector p of the masked sequence. Calculate the difference between the two vectors bit by bit to obtain the vector |op|, and find the intersection |o∩p| and the union |o∪p| of the two vectors. Concatenate the above vectors to obtain a new vector u. The formula for calculating u is as follows:

[0039] u=o·p·|op|·|o∩p|·|o∪p| (4);

[0040] Step 4.7: The vector obtained in step 4.6 is nonlinearly mapped to a shorter vector C using a feedforward neural network (FNN). i f L Let V represent a feedforward neural network with L layers, where L is greater than 0, and concatenate it with the vector obtained in step 4.4 to obtain vector V, C. i The calculation formula is as follows:

[0041] C i =Linear(f L (u)) (5);

[0042] The formula for calculating vector V is as follows:

[0043] V = w t ·C i (6);

[0044] Step 4.8: Use a softmax classifier to obtain the entity recognition and classification results. The calculation formula is as follows:

[0045] y r =softmax(W e *V+b e (7);

[0046] Among them, W e It is the weight matrix, b e It is the offset.

[0047] The beneficial effects of this invention are as follows: This invention proposes a task-adaptive pre-training scheme, RSim-WWM, for the task of recognizing quality element entities. It constructs a corpus suitable for quality element entity recognition using 150 railway specifications. The RSim-WWM pre-training scheme is then pre-trained on a BERT model instance, bert-base-cased, to enhance model performance. The trained Transformer encoder is fed into the Named Entity Recognition (RSC-NER) model. The model is trained and used on one of three publicly available Chinese datasets: MSRA-NER, Taobao e-commerce NER, or Weibo-NER. Finally, the trained model is used to identify and classify railway construction quality element entities. The railway construction quality element recognition method based on pre-training and semantic completeness of this invention can effectively improve the accuracy of quality element entity recognition during railway construction quality inspection and monitoring, enhance semantic representation capabilities, and focus on the overall structural information of the text sequence. Attached Figure Description

[0048] Figure 1 This is the overall flowchart of the railway construction quality element entity recognition method based on semantic completeness of the present invention;

[0049] Figure 2 This is a schematic diagram of the overall framework of the RSC-NER model, which incorporates semantic completeness features in the railway construction quality element entity recognition method based on semantic completeness of the present invention.

[0050] Figure 3 This is a schematic diagram of the overall structure of RSim-WWM, the comprehensive pre-training scheme in the railway construction quality element entity recognition method based on semantic completeness of this invention. Detailed Implementation

[0051] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] This invention relates to a method for recognizing entities of railway construction quality elements based on semantic completeness. The overall flowchart is as follows: Figure 1As shown, the dataset was first prepared, and three selected Chinese datasets were labeled to adapt to the training of the technology of this invention. Semantic completeness features were used, which are defined as "the semantic representation of the original text sequence by a masked text sequence". Hyperparameters were set for the RSim-WWM task adaptation pre-training and the training of the RSC-NER model. A corpus suitable for this invention was constructed by randomly sampling 150 railway specifications. Using the above corpus, the comprehensive pre-training scheme RSim-WWM was pre-trained on a BERT model instance bert-base-cased to enhance the model's performance. The trained Transformer encoder was put into the named entity recognition model RSC-NER and trained and recognized using one of the three publicly available Chinese datasets: MSRA-NER, Taobao e-commerce NER, or Weibo-NER. Finally, the trained RSC-NER model was used to identify and classify railway construction quality element entities.

[0054] Example 2

[0055] This invention relates to a method for recognizing entities of railway construction quality elements based on semantic completeness, specifically including the following steps:

[0056] Step 1: Randomly sample n railway specifications from 150 railway standards to construct a corpus related to quality inspection for pre-training the Transformer encoder. For example, select Article 4.4.2 from the railway standard GBJ12-87 "Standard Gauge Railway Design Specification for Industrial Enterprises": The width of the top surface of the track bed should meet the requirements of 2.9m for connecting lines for shunting operation, and 0.1m widening on the outside of curves with a radius of 400m and below.

[0057] Step 2: Train the RSC-NER model using one of the three publicly available Chinese datasets: MSRA-NER, Taobao e-commerce NER, or Weibo-NER. Annotate the aforementioned publicly available Chinese datasets using a method that adds entity tail annotations to the BIO dataset. For example, using the Weibo-NER dataset, the annotation results for "Women who are willing to look others in the eye are more beautiful when they are confident" are shown in Table 1.

[0058] Table 1

[0059]

[0060] Step 3: Introduce semantic completeness features into the named entity recognition model to propose the RSC-NER named entity model. The specific model framework diagram is as follows: Figure 2As shown, the RSC-NER model includes the following modules: text input, Transformer encoder, word embedding, introduction of semantic completeness features, vector concatenation, and softmax classification. The model processing procedure is shown in the table below:

[0061]

[0062]

[0063] Step 4: Set hyperparameters for RSC-NER model training. The hyperparameters for model training are set. Due to the large number of hyperparameters in the experimental design, only the parameters for the number of iterations, learning rate, and semantic completeness embedding size are shown below. See Table 2:

[0064] Table 2

[0065]

[0066] Step 5: To improve the model's performance in quality element entity recognition, a new integrated pre-training scheme, RSim-WWM, is proposed. The RSim-WWM scheme is used to pre-train the Transformer encoder on a BERT model instance, bert-base-cased. The general process of the RSim-WWM pre-training scheme is as follows: Figure 3 As shown. The RSim-WWM solution process is as follows:

[0067] Step 5.1: Input the text sequence and encode it into a series of vector representations (a1, a2, ..., a...). n The overall vector representation is A, and the vectors a1, a2, ..., a3 are... n Each of these represents a word in the input text sequence.

[0068] Step 5.2: Alternately mask the vector representations obtained in Step 5.1, replacing the masked vectors with MASK. For example, masking a1 yields the masked sentence ([MASK], a2, ..., a...). n Let B be the overall vector representation of the masked sentence. Calculate the semantic similarity between each masked sentence and the original sentence. The formula for calculating semantic similarity is as follows:

[0069]

[0070] Step 5.3: Sort the semantic similarity of the masked sentences obtained after alternating masking in step 5.2 from smallest to largest, and select the word codes corresponding to the top 15% as the masked words for training.

[0071] Step 5.4: Mask the masked words selected in step 5.3, and embed the masked vector into the Transformer encoder in the bert-base-cased model for subsequent training and prediction of masked words.

[0072] Step 5.5: Repeat steps 5.1-5.4 above until the training accuracy reaches 50% or more, then stop training to obtain the trained Transformer encoder we need.

[0073] Step 6: Input the Transformer encoder trained in Step 5 into the Named Entity Recognition (RSC-NER) model, and train the model using one of three Chinese datasets: MSRA-NER, Taobao e-commerce NER, or Weibo-NER. The model training process is described in detail below. Figure 2 The specific details are as follows:

[0074] Step 6.1, input the text sequence (s1, s2, ..., s...) into the RSC-NER model. n ).

[0075] Step 6.2: Obtain the word embedding sequence (e1, e2, ..., e3) through the Transformer encoder. n ).

[0076] Step 6.3: Use the fragment construction method to construct subsequences of the word embedding sequence generated in step 6.2. The subsequences constructed by the fragment method are all continuous subsequences with a span less than a certain value k, such as (e1), (e1, e2), etc., and perform entity judgment on the constructed subsequences in turn.

[0077] Step 6.4: Perform max pooling on the word embedding subsequence from Step 6.3 where entity detection is being performed. For example, if the subsequence where entity detection is being performed is (e1, e2), the max-pooled representation vector is E. Concatenate this E with the overall representation vector T of the text sequence to obtain a new vector e(c). The formula for calculating E is as follows:

[0078] E = maxpooling(e1, e2) (2)

[0079] The formula for calculating the new vector e(c) is:

[0080] e(c)=E·T (3)

[0081] Step 6.5: An embedding x of dimension 25 is used to represent the span of the subsequence. x is a vector that can be learned through backpropagation and is initialized but not locked during training. Concatenating the span embedding x with the representation vector e(c) yields the vector w used for entity classification. tThe calculation formula is as follows:

[0082] w t =e(c)·x (4)

[0083] Step 6.6, copy the word embedding sequence (e1, e2, ..., e) obtained in Step 6.2. n The process involves sequentially masking the subsequence currently being used for entity detection, and then replacing the masked subsequence with a mask. For example, if the subsequence currently being used for entity detection is (e1, e2), the masked sequence would be ([MASK], [MASK], ..., e2). n The specific method of masking is to set all elements in the masked vector to zero.

[0084] Step 6.7: Perform average pooling on the original sequence from Step 6.1 and the masked sequence from Step 6.5 respectively to obtain the text representation vector o of the original sequence and the text representation vector p of the masked sequence. Calculate the difference between the two vectors bit by bit to obtain the vector |op|, and find the intersection |o∩p| and the union |o∪p| of the two vectors. Concatenate the above five vectors to obtain a new vector u. The formula for calculating u is as follows:

[0085] u=o·p·|op|·|o∩p|·|o∪p| (5)

[0086] Step 6.8: The vector obtained in step 6.7 is nonlinearly mapped to a shorter vector C using a feedforward neural network (FNN). i f L Let V represent a feedforward neural network with L layers, where L is greater than 0, and concatenate it with the vector obtained in step 6.5 to obtain vector V, C. i The calculation formula is as follows:

[0087] C i =Linear(f L (u)) (6)

[0088] The formula for calculating vector V is as follows:

[0089] V = w t ·C i (7)

[0090] Step 6.9: Use a softmax classifier to obtain the entity recognition and classification result. The calculation formula is as follows:

[0091] y r =softmax(W e *V+b e (8)

[0092] Among them, W e It is the weight matrix, be It is the offset.

[0093] Step 7: Input railway construction specification information data and use the RSC-NER model to identify the entities of railway construction quality elements and generate classification results. For example, as shown in Figure 3 below:

[0094] Table 3

[0095]

[0096]

[0097] Example 3

[0098] The following example illustrates the training process of the RSC-NER model. The input text sequence is "The rails cannot fall off," represented as (s1, s2, s3, s4, s5, s6). After passing through the Transformer encoder, the word embedding sequence is obtained as (e1, e2, e3, e4, e5, e6), and the overall vector is represented as T. Subsequences of the word embedding sequence are constructed sequentially using the fragment construction method, such as (e1, e2). Entity judgment is then performed on each of the constructed subsequences. The subsequences undergoing entity judgment are max-pooled using formula (2), and the resulting vector is represented as E. T and E are concatenated using formula (3) to generate a new vector e(c). Then, a vector x with dimension 25 is embedded to represent the span of the subsequence. e(c) and x are concatenated using formula (4) to generate a vector w. t Copy (e1, e2, e3, e4, e5, e6) and cover the subsequences that are currently being judged for entities in turn. Replace the covered subsequences with MASK, for example, cover (e1, e2) to get the covered sequence ([MASK], [MASK], e3, e4, e5, e6). Perform average pooling on the original sequence (e1, e2, e3, e4, e5, e6) and the covered sequence respectively to get vector o and vector p. Calculate the difference, intersection and union of the above two vectors to get vectors |op|, |o∩p|, |o∪p|. Connect the above vectors using formula (5) to get a new vector u. Nonlinearly map u to a shorter vector C using a feedforward neural network FNN using formula (6). i Using formula (7) to calculate w t With C i The vector V used for classification is obtained by connecting the vectors. Finally, the entity recognition classification result is obtained by using the softmax classifier and formula (8).

Claims

1. A method for recognizing entities of railway construction quality elements based on semantic completeness, characterized in that: Specifically, the steps include the following: Step 1: Construct a corpus by randomly sampling quality inspection-related standards from railway specifications; Step 2, construct the RSC-NER named entity model; Step 3: Pre-train the Transformer encoder using RSim-WWM; The specific process of step 3 is as follows: Step 3.1: Input the text sequence and encode it, representing the encoded text sequence as a vector A = ; Step 3.2: Alternately mask the vector representations obtained in Step 3.1, replace the masked vectors with MASK, and let the overall vector representation of the masked sentence be B. Calculate the semantic similarity between each masked sentence and the original sentence using the following formula (1). : （1）； Step 3.3: Sort the semantic similarity of the masked sentences obtained after alternating masking in Step 3.2 from small to large, and select the word codes corresponding to the top 15% as the masked words for training. Step 3.4, the masking step, the masked words selected in 3.3, the masked vectors are embedded into the Transformer encoder in the bert-base-cased model for subsequent training and prediction of masked words; Step 3.5: Repeat steps 3.1-3.4 above until the training result accuracy reaches more than 50%, which means you get the trained Transformer encoder. Step 4: Train the RSC-NER named entity model; the specific process of step 4 is as follows: Step 4.1: Input the Transformer encoder trained in Step 3 into the Named Entity Recognition (RSC-NER) model, and then input the text sequence into the RSC-NER model. Text sequence The word embedding sequence is obtained after step 3 using the Transformer encoder. ; Step 4.2: Construct subsequences of the word embedding sequence generated in Step 4.1 using the fragment construction method, and perform entity judgment on the constructed subsequences in turn; Step 4.3: Perform max pooling on the word embedding subsequence from which entity detection is being performed in Step 4.

2. The max-pooled representation vector is E. Concatenate vector E with the overall representation vector T of the text sequence to obtain a new vector e(c). The formula for calculating vector e(c) is: （2）； Step 4.4: Use vector x to represent the span of the subsequence, and concatenate the span embedding x with the representation vector e(c) to obtain the vector used for entity classification. The calculation formula is as follows: （3）； Step 4.5, copy the word embedding sequence obtained in Step 4.

1. The subsequence currently being used for entity detection is masked in turn, and the masked subsequence is replaced with a MASK. Step 4.6: Perform average pooling on the original sequence from Step 4.1 and the masked sequence from Step 4.5 respectively to obtain the original sequence text representation vector o and the masked sequence text representation vector p. Calculate the difference between the two vectors bit by bit to obtain the vector |op|, and find the intersection |o| of the two vectors. p| and union|o p|, concatenating the above vectors yields a new vector u, calculated using the following formula: | o p | | o p |（4）； Step 4.7: The vector obtained in step 4.6 is nonlinearly mapped to a shorter vector using a feedforward neural network (FNN). , Let V represent a feedforward neural network with L layers, where L is greater than 0, and concatenate it with the vector obtained in step 4.4 to obtain vector V. The calculation formula is as follows: （5）； The formula for calculating vector V is as follows: V= （6）； Step 4.8: Use a softmax classifier to obtain the entity recognition and classification results. The calculation formula is as follows: （7）； in, It is a weight matrix. It is the offset; Step 5: Use the RSC-NER model trained in Step 4 to identify the entities of railway construction quality elements and generate classification results.

2. The railway construction quality element entity recognition method based on semantic completeness according to claim 1, characterized in that: In step 2, the RSC-NER model includes: text input, Transformer encoder, word embedding, introduction of semantic completeness features, vector connection, and softmax classification.

3. The railway construction quality element entity recognition method based on semantic completeness according to claim 2, characterized in that: The RSC-NER model processes text sequences as follows: 1) Import text sequence; 2) The word embedding sequence vector is obtained through the Transformer encoder; 3) Generate subsequence vectors by arranging fragments; 4) Perform entity checks on each subsequence sequentially and then perform max pooling; 5) Perform span embedding on the word embedding sequence; 6) Copy the word embedding sequence and mask it to introduce semantic completeness features; 7) Vector join; 8) Softmax classification; 9) Return the entity recognition and classification results.

Images