An entity recognition method based on pre-trained language model

Abstract

The present invention discloses an entity recognition method based on a pre-trained language model. The present invention relates to deep learning, pre-trained language models in natural language processing tasks and entity recognition subtasks in the field of data integration. The present invention adds attention to the unique needs of the entity recognition task itself, specifically proposing a joint entity recognition model that focuses on similar fragments and digital information fragments. Through an encoder that perceives similar fragments and an encoder that perceives digital fragments, the problem of pre-trained language model's attention distraction on a small training set can be effectively handled, so that the model can better handle entity recognition tasks. The present invention can significantly improve on small training sets compared to the existing latest methods. Most of the current methods require a large amount of training data, which means that a large amount of manual participation in labeling data is required. The present invention can effectively save the cost of manual labeling.

Description

Technical Field

[0001] The present invention relates to deep learning, pre-trained language models in natural language processing tasks, and entity recognition subtasks in the field of data integration. The invention adds attention to the unique requirements of the entity recognition task itself, specifically proposing a joint entity recognition model that focuses on similar fragments and digital information fragments. Through an encoder that perceives similar fragments and an encoder that perceives digital fragments, the problem of pre-trained language model's attention being distracted on a small training set can be effectively handled, so that the model can better handle entity recognition tasks. Background Art

[0002] Entity recognition (also known as entity resolution) is a task that aims to identify whether two records refer to the same real-world entity. Traditional entity recognition methods focus on structured data, which are mostly atomic, that is, with short attribute values. In recent years, the increasing demand for matching textual entity data in the big data era, such as matching long text product descriptions, has posed a challenge to traditional entity recognition methods.

[0003] In the field of natural language processing (NLP), text classification and similarity problems are closely related to the problem of text data. Text classification is the task of predicting one or more category labels for a given text for different scenarios (such as sentiment analysis, topic detection, and question-answering type detection). Recently, some studies have used a transformer-based BERT structure to design a linear classifier at the top level of the model, so that downstream tasks such as text classification and sentence question answering can be handled.

[0004] Thanks to the research and development of pre-trained language models and their mature applications, two entity recognition methods based on pre-trained language models have been proposed, namely the DITTO method and the JointBERT method, both of which have been proven to achieve better performance results. DITTO can leverage domain knowledge by injecting unique tags during preprocessing, and JointBERT is a dual-objective training method based on pre-trained language models that can achieve better performance under large data sets.

[0005] However, these methods directly use pre-trained language models and separate entity recognition from the characteristics of the entity itself as a natural language processing (NLP) task, without considering the unique needs of entity recognition tasks. DITTO's domain prior knowledge is not always available and effective, which leads to poor generalization of the entire model; and JointBERT, with limited training sets, will have a negative impact on the original pre-trained language model, reducing the recognition accuracy of the model.

[0006] Therefore, the present invention aims to address the problems of attention distraction in deep learning models and the inability to focus on the characteristics of the entity data itself under small training sets, and proposes an entity recognition method that can pay more attention to similar fragments and digital information. Summary of the invention

[0007] In view of the above problems, the present invention proposes an entity recognition method that does not require manual injection of unique tags and prior knowledge, and enhances the focus on key digital information and similar fragments in text data.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] Step 1: Data preprocessing. Generate all entity candidate pairs in advance according to the processing method based on text similarity. For each entity candidate pair (record u, record v), first generate sequence I and sequence II for the entity text descriptions of record u and record v respectively, and then use a special identifier to connect sequence I and sequence II to obtain a comprehensive sequence III.

[0010] Step 2: Use the pre-trained language model to segment the pre-processed generated sequence I, generated sequence II, and comprehensive generated sequence III, extract word embedding vector features, and align them according to different text lengths. Finally, the word vectors E corresponding to the generated sequence I, generated sequence II, and comprehensive generated sequence III are obtained respectively. u 、Word vector E v And the combined word vector E t .

[0011] Step 3: Extracted word vector E u 、Word vector E v And word vector E t The similar segment encoder of the present invention is input to capture the similar segments between the two records of each entity candidate pair (record u, record v), thereby increasing the attention paid to important segments.

[0012] Input: record u and record v are word vectors E extracted by the pre-trained language model u and E v , and the entire word vector E after combining the entity candidate pair (record u, record v) t .

[0013] 3-1. Extract background semantics and transform the word vector E of the entire entity candidate pair (record u, record v) t As background semantics, we choose E t [CLS] is used to represent the entire background semantics. The word vector of “[CLS]” summarizes the word vector E tAll token word vectors.

[0014] 3-2. Background semantic fusion, respectively in the word vector E of the two records u And word vector E v A background semantic fusion operator is performed on E t [CLS] contains the overall background semantics of the record pair, and adopting this fusion operator will help the attention interaction between record u and record v.

[0015] 3-3. Interactive attention between two records, in order to allow record u and record v to influence each other's attention, the present invention simultaneously proposes a mechanism for calculating the attention from record v to record u and a mechanism for calculating the attention from record u to record v. Combined with the overall background semantic expression, an embedding vector containing similar fragment perception under the overall background semantics can be obtained.

[0016] 3-4. Feature compression and extraction, using convolutional neural network CNN for feature compression, input word vector after interactive attention mechanism and The concatenation vector Convolve it to obtain information about similar fragments. Define a convolution matrix where d a is the output vector dimension of CNN, l is the convolution kernel size, and d is the word vector dimension. Convolution operations are performed using convolution kernels of different sizes, and further maximum pooling operations are performed to obtain output features.

[0017] Step 4: At the same time, the digital information fragment encoder is also used to encode the preprocessed input, and the digital information fragments are filtered according to the self-attention weight of the pre-trained language model on the number itself to give a vector passed through the digital encoder.

[0018] Input: The input of the digital information fragment encoder is slightly different from the previous encoder. It only needs the entire word vector E after the entity candidate pair (record u, record v) is combined. t .

[0019] 4-1. Calculate the edit distance of all entity candidate pairs. First, calculate the edit distance of the character substring containing numbers for each entity candidate pair (s u ,s v ), where s u and v are character substrings containing numbers from record u and record v respectively. In this way, similar combinations of numbers can be selected from records u and v.

[0020] 4-2. Substring filtering: remove character substring combinations that contain substring relationships.u is another character substring s v A substring of s, or a character substring s v is another character substring s u When two character substrings are compared, the edit distance should be changed to 0 because two character substrings may describe the same information in different formats.

[0021] 4-3. Quantifying the similarity of character substrings. Simply using the length of the edit distance cannot describe all records, because different entity texts describe their matching character substrings differently, and will produce results of different magnitudes based on the number of character substrings and the length of the character substrings themselves. Therefore, there is no way to quantify this standard using the edit distance alone, so the edit distance ratio (Levenshtein ratio) is used to describe the difference, adding a ratio based on the calculated edit distance.

[0022] 4-4. Other distance metrics, one thing needs to be noted, the main consideration of the present invention is that in the fragment containing numbers, its value is regarded as the character substring of the product model, because in practical applications, such as product ID, the character substring of the numerical model type is quite common. However, in the fragment containing numbers, its numerical type can also be regarded as the size description of the number itself, and the distance calculation at this time can use other metrics such as the square function.

[0023] 4-5. Important character substring selection: introduce the self-attention weight of the pre-trained language model to filter out character substrings whose self-attention weight is lower than the set threshold.

[0024] 4-6. Comprehensive edit distance score calculation, after obtaining all important character substrings, generate digital-aware word vectors and and their respective edit distance scores

[0025] 4-7. Feature extraction based on convolutional neural network CNN, similar to the CNN module in the similar perceptual encoder above, also uses CNN to extract First, convolution operations of different lengths are performed, and then the output features are obtained through the same maximum pooling operation.

[0026] Step 5: Concatenate the similarity segment encoder and the digital segment encoder output vectors and The joint feature vector is passed through a linear layer and a softmax layer to output the classification probability. According to the label data, the loss function is calculated, and then back propagation is performed to continuously fine-tune and optimize the model parameters. New model input is obtained from step 2 again, and the training is stopped when the loss function converges. The final language model and encoder parameters are obtained.

[0027] The beneficial effects of the present invention are as follows:

[0028] In view of the shortcomings of existing studies that directly use natural language processing methods and ignore the unique requirements of entity recognition tasks to solve text data entity recognition tasks, an entity recognition method that can combine the characteristics of real-world entities is proposed. The present invention includes a similarity segment encoder and a digital segment encoder, which can effectively handle the problem that text data is interfered by redundant fields and is insensitive to key digital model information in entity recognition.

[0029] Secondly, compared with the use of named entity recognition, data enhancement and other technical means, the present invention does not require too much design, that is, it can achieve a good improvement in recognition accuracy without the need for prior knowledge. At the same time, the present invention also improves time efficiency to a certain extent.

[0030] The present invention can significantly improve on existing state-of-the-art methods on small training sets. Currently, most methods require a large amount of training data, which means that a large amount of manual participation in data labeling is required. The present invention can effectively save the cost of manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a similar segment encoder of the present invention;

[0032] Figure 2 It is a flow chart of extracting key model information by the digital information segment encoder of the present invention;

[0033] Figure 3 It is a flow chart of the overall implementation of the method of the present invention;

[0034] Figure 4 It is an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings and examples.

[0036] like Figure 4 As shown in , the data input of this paper is the text description of two entities, and special identifiers are added to process them into sequences. The overall process is as follows Figure 3As shown, the input passes through a similarity segment encoder module and a digital segment encoder module respectively, and finally the outputs of the two encoders are combined to give the classification result. The specific steps are as follows:

[0037] Step 1: Text data preprocessing

[0038] A large number of entity text description records that need to be identified are grouped into record pairs. The present invention uses a block method to generate candidate pairs, thereby reducing the identification search space.

[0039] The formula for concatenating text data is as follows:

[0040] serialize(u,v)::=[CLS]serialize(u)[SEP]serialize(v)[SEP] (1)

[0041] in,

[0042] serialize(x)::=[COL]attr1[VAL]val1…[COL]attr k [VAL]val k (2)

[0043] Indicates that in a text data, there are multiple attribute values ​​to describe an entity. Use "COL" to highlight the attribute name, and "VAL" to highlight the attribute value. serialize(x) means simply generating a sequence of records. The ::= symbol indicates that the serialization method is defined as the right-hand side. Figure 4 After a record is generated, it will get "COL title VALMEMORYIA...". attr1 and val1 represent the attribute name and attribute value input of the record respectively, such as Figure 4 , records are generally in the form of two tuples (attr k ,val k ), k represents the kth tuple, and a tuple in record u is "title" and "MEMORIA...". serialize(u,v) concatenates the generated sequences of the two records with the special identifiers "[CLS]" and "[SEP]", such as Figure 4 , get "[CLS]COL title VAL MEMORIA…[SEP]COL title VALCrucial…[SEP]".

[0044] The original input of each record u and record v is serialized by the serialization method to obtain the independent input of each record [CLS]serialize(u)[SEP] and [CLS]serialize(v)[SEP], which is obtained by formula (1). At the same time, the joint input of records u and v is also required, which is obtained by formula (2). Among them, "[SEP]", "[COL]", and "[VAL]" are special tags that retain the input structure.

[0045] Step 2: Extract word vectors

[0046] The pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) is used to segment the pre-processed data, extract word embedding vector features, and align them according to different text lengths. i and val i All need to be tokenized to alleviate the problem of insufficient corpus during training. For example, "8GB" in record u1 will be split into "8" and "#GB", and a "#" symbol is added to indicate that it is part of the previous word. Finally, the vector E of each record itself is obtained. u and E v , record the entire word vector E after combining (u,v) t .

[0047] Step 3: Similar segment encoder reallocates attention

[0048] Figure 1 The pipeline of the similarity segment encoder is shown.

[0049] Input: The input of the similarity segment encoder consists of three parts: the word vector E extracted by the pre-trained language model of record u and record v u and E v , and the entire word vector E after combining the entity candidate pair (record u, record v) t . u and l v represents the length of the sequence of u and v after token serialization, and d represents the dimension of each token word vector. Similarly, this paper identifies the joint expression of record u and v as word vector E t , here l t = l u +l v -1, to ensure the consistency of input length. In this way, the model obtains three input word vectors E t 、E u 、E v .

[0050] 3-1. Extract background semantics and transform the word vector E of the entire entity candidate pair (record u, record v) t As the background semantics for attention interaction, we choose E t [CLS] is used to represent the entire semantics. The expression vector of “[CLS]” summarizes E t All token embedding expressions of word vectors. When using the BERT-base model, the word vector dimension of each token is 768. Here, if all the information of the entire background semantics is output, it is l t ×768 dimensions, but the vector is accurate to the features of each token. On the one hand, it is not suitable as a broad background semantics, and on the other hand, considering vector alignment, “CLS” is used to summarize all semantics.

[0051] 3-2. Background semantic fusion, respectively in the word vector E of the two records u and E v A background semantic fusion operator is performed on

[0052] E′ u =E u +E′ t [CLS] (3)

[0053] E′ v =E v +E′ t [CLS] (4)

[0054] Where E t ′[CLS] means expanding E in one dimension t [CLS] to align E u and E v A vector of shape and size. t [CLS] contains the overall background semantics of the entity candidate pair (record u, record v). Using this fusion operator will be very helpful for the attention interaction between records u and v.

[0055] 3-3. Interactive attention between records. In order to allow records u and v to influence each other's attention, the present invention simultaneously proposes an attention calculation mechanism from record v to record u and an attention calculation mechanism from record u to record v.

[0056] For the attention calculation from record v to record u, consider its attention distribution matrix A:

[0057] A=B / repeat(B j ,l v )(5)

[0058] in and The repeat operation is to sum the columns of the unnormalized attention distribution matrix B to get the column sum B j , and perform vector expansion along the first dimension (row dimension) to facilitate subsequent matrix operations.

[0059] Then E′ v Fuse it with the attention distribution matrix A and add the result back to E u Go up and finally get the word vector C after interaction u :

[0060] C u =E u +A T E′ v (6)

[0061] Similarly, the word vector C after interaction v It can also be calculated by the same interactive attention method from u to v. Finally, C u and E t , C v and E t , and obtain an embedding vector that contains similar fragment perception under the overall background semantics.

[0062]

[0063]

[0064] Since splicing is required, here is the explanation The meaning of C u The output dimension is l u ×d, and compare it with l t ×d word vector E t When splicing, only the front l u Length d-dimensional vector, here 1:l u Represents the word vector from column 1 to column l u List all d-dimensional vectors. It should be noted that the word vector C after interaction u and C v The token embedding vectors of "[CLS]" (in the first column of the word vector) are all separately compared with E t The token "[CLS]" in and

[0065] 3-4. Feature compression and extraction, using convolutional neural network CNN for feature compression, input and The concatenation vector Convolve it to obtain information about similar fragments. Define a convolution matrix where d a is the output vector dimension of CNN, and the convolution kernel size l is set to 3, 4, and 5 respectively. Convolution operations are performed using convolution kernels of different sizes, and further maximum pooling operations are performed to obtain output features.

[0066] Step 4: Digital information segment encoder extracts key model information

[0067] like Figure 2 As shown, the digital information fragment encoder obtains key digital information through attention weight and edit distance.

[0068] Input: The input of the digital information fragment encoder is slightly different from the previous encoder. It only needs to record the entire word vector E after combining (u,v) t .

[0069] 4-1. Calculation of edit distance of all candidate pairs. This paper first calculates the edit distance (s u ,s v ), where s u and v are character substrings containing numbers from record u and record v respectively. In this way, similar combinations of numbers can be selected from text records u and v.

[0070] Specifically, the following is how to calculate two character substrings s u and v The edit distance is:

[0071]

[0072] Here i and j represent s respectively. u and v In real data sets, the character substrings of different models often differ by only 1-2 replaceable letters or numbers, such as "1060" and "1070" only differ by "6" and "7". In order to reflect this difference, this paper changes the cost of "replacement" in the editing operation to 2 based on the original edit distance calculation formula, so as to amplify such tiny character differences.

[0073] 4-2. Substring filtering: character substring combinations containing substring relationships are removed. When a character substring s u is another character substring s v A substring of s, or a character substring s v is another character substring s uWhen two character substrings are used, the edit distance should be changed to 0. This is because two character substrings may describe the same information in different formats. For example, both "1x8gb" and "8gb" describe the same specification information.

[0074] 4-3. Quantifying the similarity of character substrings. Simply using the length of the edit distance cannot describe all records, because different texts describe their matching substring pairs differently, and different magnitudes of results will be produced based on the number of substring pairs and the length of the substrings themselves. Therefore, simply using the edit distance itself cannot quantify this standard, so the edit distance ratio is used to describe the difference. A ratio is added based on the calculated edit distance. The calculation formula is as follows:

[0075]

[0076] Among them l sum Yes u plus s v The edit distance length is set to a threshold when r(s u ,s v )≥0.5, then it is considered that (s u ,s v ) are similar character substrings.

[0077] 4-4. Other distance metrics, one thing needs to be noted, the main consideration of the present invention is that in the fragment containing numbers, its value is regarded as the character substring of the product model, because in practical applications, such as product ID, the character substring of the numerical model type is quite common. However, in the fragment containing numbers, its numerical type can also be regarded as the size description of the number itself, and the distance calculation at this time can use other metrics such as the square function.

[0078] 4-5. Important character substring pair selection, introduce the self-attention weight of the pre-trained language model to filter out these irrelevant character substring pairs. Specifically, use and To express s u and v The self-attention weight. If the self-attention or The weight exceeds the set threshold 1 / l t , then this pair of character substrings can be regarded as a pair of character substrings containing important information, and additional attention can be added during the model training process.

[0079] 4-6. Comprehensive edit distance score calculation, after obtaining all important character substring pairs, generate digitally aware expression vectors and and their respective edit distance scores That is and It will be calculated by averaging all character substrings s u and v The token embedding expression vectors in u and v are obtained, and the edit distance score It is calculated by the following formula:

[0080]

[0081] in, represents a set of character substring pairs, Represents the number of character substring pairs.

[0082] 4-7. Feature extraction based on convolutional neural network CNN, similar to the CNN module in the similar perceptual encoder above, also uses CNN to extract The convolution here is defined as the concatenated vector and matrix Operations between a is the output vector dimension of CNN, and the sliding window size of the convolution kernel is 1 and 2. First, the convolution operation of different convolution kernels is performed, and then the output feature is obtained by the maximum pooling operation in the same way.

[0083] Step 5: Combine Outputs

[0084] like Figure 3 As shown, the splicing similar segment encoder and digital segment encoder output vector and The joint feature vector is passed through a linear layer and a softmax layer to output the classification probability. According to the label data, the loss function between the predicted sample and the true value sample is calculated:

[0085]

[0086] Among them, y * represents the real result in the training data, which is composed of 0 and 1 to indicate match or mismatch; y represents the 0 and 1 prediction results finally output by the present invention; N represents the number of training sets, and each batch size of the present invention is 32, so N is 32.

[0087] Then back propagation is performed to continuously fine-tune and optimize the model parameters, and new model input is obtained from step 2 again, and training is stopped when the loss function converges. The final language model and encoder parameters are obtained.

Claims

1. An entity recognition method based on a pre-trained language model, characterized in that The steps include: Step 1: Data preprocessing: Generate all entity candidate pairs in advance according to the text similarity-based processing method. For each entity candidate pair (record u, record v), first generate sequence I and sequence II for the entity text descriptions of record u and record v respectively, and then use a special identifier to connect sequence I and sequence II to obtain a comprehensive sequence III. Step 2: Use the pre-trained language model to segment the pre-processed generated sequence I, generated sequence II, and comprehensive generated sequence III, extract word embedding vector features, and align them according to different text lengths. Finally, the word vectors E corresponding to the generated sequence I, generated sequence II, and comprehensive generated sequence III are obtained respectively. u 、Word vector E v And the combined word vector E t ; Step 3: Extracted word vector E u 、Word vector E v And word vector E t Input similarity segment encoder, which captures the similarity segments between the two records of each entity candidate pair (record u, record v); Step 4: At the same time, the digital information fragment encoder is also used to encode the preprocessed input, and the digital information fragments are screened according to the self-attention weight of the pre-trained language model on the number itself, and the vector passed through the digital encoder is given; Step 5: Concatenate the similarity segment encoder and the digital segment encoder output vectors and Pass the joint feature vector through a linear layer and a softmax layer to output the classification probability; calculate the loss function based on the label data, and then perform back propagation to continuously fine-tune and optimize the model parameters, and get new model input from step 2 again, iterate repeatedly, and stop training when the loss function converges; get the final language model and encoder parameters; Step 3 is implemented as follows: Input: record u and record v are word vectors E extracted by the pre-trained language model u and E v , and the entire word vector E after combining the entity candidate pair (record u, record v) t ; 3-1. Extract background semantics and transform the word vector E of the entire entity candidate pair (record u, record v) t As background semantics, we choose E t [CLS] is used to represent the entire background semantics. The word vector of "[CLS]" summarizes the word vector E t All token word vectors; 3-2. Background semantic fusion, respectively in the word vector E of the two records u And word vector E v A background semantic fusion operator is performed on E t [CLS] contains the overall background semantics of the record pair. Using this fusion operator will help the attention interaction between record u and record v; 3-3. Interactive attention between two records: a mechanism for calculating the attention from record v to record u and a mechanism for calculating the attention from record u to record v are proposed; combined with the overall background semantic expression, an embedding vector containing similar fragment perception under the overall background semantics is obtained; 3-4. Feature compression and extraction, using convolutional neural network CNN for feature compression, input word vector after interactive attention mechanism and The concatenation vector Convolve it to obtain information of similar fragments; define a convolution matrix where d a is the output vector dimension of CNN, l is the convolution kernel size, and d is the word vector dimension; convolution operations are performed using convolution kernels of different sizes, and further maximum pooling operations are performed to obtain output features. Step 4 is implemented as follows: Input: The input of the digital information fragment encoder is slightly different from the previous encoder. It only needs the entire word vector E after the entity candidate pair (record u, record v) is combined. t ; 4-1. Calculate the edit distance of all entity candidate pairs. First, calculate the edit distance of the character substring containing numbers for each entity candidate pair (s u ,s v ), where s u and v are character substrings containing numbers from record u and record v, respectively, so as to facilitate the selection of similar digital model combinations from records u and v; 4-2. Substring filtering: remove character substring combinations that contain substring relationships. u is another character substring s v A substring of s, or a character substring s v is another character substring s u When the substring of a character is different, the edit distance should be changed to 0, because two character substrings may describe the same information in different formats; 4-3. Quantify the similarity of character substrings and add an edit distance ratio based on the calculated edit distance; 4-4. Important character substring selection: introduce the self-attention weight of the pre-trained language model to filter out character substrings whose self-attention weight is lower than the set threshold; 4-5. Comprehensive edit distance score calculation, after obtaining all important character substrings, generate digital-aware word vectors and and their respective edit distance scores 4-6. Feature extraction based on convolutional neural network CNN, also using CNN to extract The joint features of; first perform convolution operations with different convolution kernels, and then obtain the output features through the same maximum pooling operation 2. The entity recognition method based on a pre-trained language model according to claim 1, characterized in that Step 1 is implemented as follows: The text description records of the entities to be identified are combined into record pairs, and the candidate pairs are generated using the block method. The formula for splicing text data is as follows: serialize(u,v)::=[CLS]serialize(u)[SEP]serialize(v)[SEP](1) in, serialize(x)::=[COL]attr1[VAL]val1…[COL]attr k [VAL]val k (2) Indicates that in a text data, there are multiple attribute values ​​to describe an entity. Use "COL" to highlight the attribute name, and "VAL" to highlight the attribute value; serialize(x) means simply generating a sequence of records. The ::= symbol represents that the serialization method is defined as the right-hand side. Attr1 and val1 represent the attribute name and attribute value input of the record respectively. Records are generally in the form of a two-tuple (attr k ,val k ), k represents the k-th tuple, and serialize(u,v) concatenates the generated sequences of the two records using the special identifiers "[CLS]" and "[SEP]"; The original input of each record u and record v is serialized through the serialization method to obtain the independent input of each record [CLS]serialize(u)[SEP] and [CLS]serialize(v)[SEP], which is obtained by formula (1). At the same time, the joint input of records u and v is also required, which is obtained by formula (2); among them, "[SEP]", "[COL]", and "[VAL]" are special tags that retain the input structure.

3. The entity recognition method based on a pre-trained language model according to claim 2 is characterized in that Step 2 is implemented as follows: Use the pre-trained language model BERT to segment the pre-processed data, extract word embedding vector features, align them according to different text lengths, and in the actual application encoding, each attr i and val i All need to be tokenized, and finally we get the vector E that records u and v itself. u and E v , record the entire word vector E after combining (u,v) t .

4. The entity recognition method based on a pre-trained language model according to claim 1, characterized in that Step 3-2 background semantic fusion is specifically implemented as follows: The word vector E between the two records u and E v A background semantic fusion operator is performed on AND' u =And u +E′ t [CLS](3) AND' v =And v +E′ t [CLS](4) Where E t ′[CLS] means expanding E in one dimension t [CLS] to align E u and E v A vector of shape and size; because E t [CLS] contains the overall background semantics of the entity candidate pair (record u, record v). Using this fusion operator will be very helpful for the attention interaction between records u and v.

5. The entity recognition method based on a pre-trained language model according to claim 1 or 4, characterized in that Step 3-3 proposes an attention calculation mechanism from record v to record u and an attention calculation mechanism from record u to record v, which are specifically implemented as follows: For the attention calculation from record v to record u, consider its attention distribution matrix A: A=B / repeat(B j ,l v )(5) in and The repeat operation is to sum the columns of the unnormalized attention distribution matrix B to get the column sum B j , and perform vector expansion along the first dimension; Then E′ v Fuse it with the attention distribution matrix A and add the result back to E u Go up and finally get the word vector C after interaction u : C u =And u +A T AND' v (6) Similarly, the word vector C after interaction v It is also calculated by the same interactive attention method from u to v; finally, C u and E t , C v and E t , get the embedding vector containing similar fragment perception under the overall background semantics; Since splicing is required, additional instructions are given The meaning of C u The output dimension is l u ×d, and compare it with l t ×d word vector E t When splicing, only the front l u Length d-dimensional vector, here 1:l u Represents the word vector from column 1 to column l u List all d-dimensional vectors, the word vector C after interaction u and C v The token embedding vectors of "[CLS]" in E t The token "[CLS]" in and 6. The entity recognition method based on a pre-trained language model according to claim 1, characterized in that The edit distance calculation of all candidate pairs described in step 4-1 is as follows: First, the edit distance (s) of each candidate pair of character substrings containing numbers is calculated. u ,s v ), where s u and v are character substrings containing numbers from record u and record v respectively; specifically, the following gives how to calculate the two character substrings s u and v The edit distance is: Here i and j represent s respectively. u and v character substring subscript; in real data sets, the difference between character substrings of different models is often only 1-2 replaceable letters or numbers. Therefore, based on the original edit distance calculation formula, the cost of "replacement" in the edit operation is changed to 2, so that such tiny character differences can be amplified.

7. The entity recognition method based on a pre-trained language model according to claim 6, characterized in that The specific edit distance ratio used in the quantified character substring similarity described in step 4-3 is as follows: Among them l sum Yes u plus s v The edit distance length is set to a threshold when r(s u ,s v )≥0.5, then it is considered that (s u ,s v ) are similar character substrings.

8. The entity recognition method based on a pre-trained language model according to claim 7, characterized in that The comprehensive edit distance score calculation described in step 4-5 generates a digitally-aware expression vector after obtaining all important character substring pairs. and and their respective edit distance scores That is and It will be calculated by averaging all character substrings s u and v The token embedding expression vectors in u and v are obtained, and the edit distance score It is calculated by the following formula: in, represents a set of character substring pairs, Represents the number of character substring pairs.

Images