A named entity recognition method for intelligent question answering system

By using sliding window to convert Q&A data in the intelligent question and answer system, combined with forward timing and reverse timing operations of the Bert model, combined with multi-scale convolutional operations, the problem of insufficient accuracy and efficiency of naming entity recognition is solved, and the intelligence is achieved The high accuracy and efficiency of naming entity recognition in the Q&A system make full use of the performance of the machine reading comprehension model.

CN113947085BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202111235560.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-22
Publication Date
2024-11-08
Estimated Expiration
2041-10-22

AI Technical Summary

Technical Problem

Traditional naming entity recognition methods are not accurate enough in intelligent question-and-answer systems, and do not fully utilize the performance of machine reading comprehension models, especially in introducing artificial prior knowledge.

Method used

The dialogue data of the Q&A system is converted into machine reading comprehension data by sliding window. The word vector feature representation is performed through the Bert pre-trained model, and local feature extraction is performed by combining forward timing and reverse timing operations. Multi-scale convolution operations are used. Get global features, and finally splice the feature matrix in equal weights, for starting and ending position prediction, realizing named entity recognition.

Benefits of technology

It improves the accuracy and efficiency of naming entity recognition, has good practicality in question-and-answer systems with high accuracy requirements, can achieve excellent results on public data, and make full use of the performance of pre-trained models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113947085B_ABST
    Figure CN113947085B_ABST
Patent Text Reader

Abstract

A named entity recognition method for an intelligent question-answering system includes the following steps: splicing context for each sample in a sliding window manner, and converting ordinary text data into machine reading comprehension data; generating word vector features through a Bert pre-training model; encoding the word vector features using multi-scale convolution and bidirectional temporal operations, respectively, to obtain global features and local features of the sample; splicing global features and local features for word vector features in an equal-weighted manner to enhance word vector features, to obtain context features; and using multiple linear networks to identify named entities in text for context features. The present invention can make full use of the context information of sentences input by users in a question-answering system to accurately extract named entities in user questions; can achieve excellent results on public data for named entity recognition, and has good practicality in intelligent question-answering systems with high accuracy requirements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and artificial intelligence technology, and in particular to a named entity recognition method for an intelligent question-answering system. Background Art

[0002] Named entity recognition refers to an algorithm that extracts predefined entities from a given text. How to make a computer accurately encode the semantic information of a sentence is a key problem that needs to be solved by an intelligent question-answering system. A complete intelligent question-answering system generally includes four basic tasks: information extraction, lexical analysis, information retrieval, and text generation. In the field of information extraction, named entity recognition, as an important branch, has the task of labeling entities in sentences, which is of great significance. As the basic unit of text, it contains a lot of semantic information and plays an important role in encoding the semantic information of customer questions.

[0003] When solving named entity recognition with traditional supervised learning methods based on feature engineering, humans are required to manually extract features, which wastes a lot of manpower. In addition, people's experience may lead to errors in the feature extraction process, resulting in the propagation of errors, and the effectiveness of the method is limited.

[0004] With the development of deep learning in the field of natural language processing, more and more named entity recognition methods using deep learning have been proposed. In these methods, sentences need to be converted into word vectors and some operations are performed through deep learning models. The named entity recognition task can be converted into a token classification problem. Recent progress in natural language representation has made it possible to migrate the internal state of upstream trained models to downstream tasks. In order to better utilize pre-trained models, converting the named entity recognition task into a machine reading comprehension task has received widespread attention. However, the traditional method of using sequence labeling to solve the named entity recognition task cannot introduce artificial prior knowledge, resulting in insufficient accuracy of named entity recognition. Summary of the invention

[0005] In order to overcome the problems of insufficient accuracy, insufficient efficiency and under-utilization of machine reading comprehension model performance in the prior art named entity recognition methods, a named entity recognition method for an intelligent question-answering system is provided, which can accurately identify entities in sentences input by users in the question-answering system, can achieve excellent results on public data of named entity recognition, and has good practicality in question-answering systems with high accuracy requirements.

[0006] Technical solution: To achieve the above purpose, the present invention provides a named entity recognition method for an intelligent question answering system, comprising the following steps:

[0007] A named entity recognition method for an intelligent question answering system comprises the following steps:

[0008] S1: Splice the context for each target sample in a sliding window manner, and treat the conversation data in the question-answering system as ordinary text data, convert it into machine reading comprehension data, and splice a natural language question for each sample to obtain the sample's initial vector s;

[0009] S2: Pass the vector s as input to the Bert pre-trained model to obtain the word vector feature representation matrix E bert ;

[0010] S3: Transform the matrix E bert Through the forward time series operation and the reverse time series operation respectively, and vertically splicing the results of the two operations, the local feature representation matrix E is obtained. Local ;

[0011] S4: Transform the matrix E bert Through convolution and pooling operations, we get vector l and expand l to be the same as E local The same shape is obtained, and the global feature representation matrix E is obtained. global ; E bert 、E local 、E global The context feature representation matrix E is obtained by splicing in the horizontal direction in an equal-weighted manner context ;

[0012] S5: Using two matrices: Start with the linear transformation matrix T start , the linear change matrix of the end position T end , respectively for E context Perform linear transformation to obtain the corresponding starting position prediction probability matrix I start And the end position prediction probability matrix I end ; and use the argmax function on the two probability representation matrices to get the result of whether each token can be used as the start index or the end index

[0013] S6: For each token at the start position in the target text, its feature representation is horizontally concatenated with the feature representation of each token at the end position in the target text to obtain a vector m i ; All vectors m i Vertical splicing to get matrix E match ; Use the linear transformation matrix T match With E matchMultiply them, and then use the Sigmoid function to normalize the result; a group of tokens with a probability greater than or equal to the threshold are taken as the start and end of an entity extracted from the sample, and the extracted entity is the recognition result.

[0014] Furthermore, the forward sequential operation in step S3 is performed on the matrix E bert =[k <1> ;k <2> ;…;k <n>< / n> Each column vector of ] is calculated in the following order from left to right:

[0015]

[0016]

[0017]

[0018]

[0019]

[0020] c t =f t ·c t-1 +g t ·i t

[0021]

[0022]

[0023] Among them, k <t>< / t> is the matrix E bert The column vector of the tth column will get two vectors c of recorded information after each calculation is completed. t , These two vectors will participate in the next calculation. In each calculation, four intermediate vectors i will be obtained. t 、f t , o t , g t , when calculating the intermediate vector, E i 、E f 、E o 、E g They are respectively the input vector k when generating the intermediate vector <t>< / t> The weight transfer matrix, U i , U f , U o , U g They are the vector c obtained from the previous operation when generating the intermediate vector. t-1 The weight transfer matrix, bi 、b f 、b o 、b g are the bias vectors added when calculating the intermediate vectors, σ and ta are activation functions, exp represents the exponential function, and · represents the operation of multiplying the elements at the same position in the participating vectors to obtain a vector of the same dimension. Sorting them in sequence will result in the matrix of forward sequential operation.

[0024] Furthermore, the reverse timing operation in step S3 requires that E bert The column vectors in [k <n>< / n> ;k <n-1>< / n-1> ;…;k <1> ] in reverse order, and then operate the column vectors in the reversed matrix from left to right to obtain the output vector.

[0025] Furthermore, in step S4, the matrix E bert After obtaining the vector l through convolution operations and pooling operations at multiple scales, the matrix E is obtained. global With E context The specific operation process is:

[0026] A=Conv(E bert )

[0027] A=Relu(A)

[0028] l=MaxPooling(A)

[0029] E global = expand(l)

[0030] E context =concat(E bert ,E local ,E global )

[0031]

[0032] First, E bert The matrix A is obtained by using three-dimensional convolution kernels of three scales, where the lengths of the convolution kernels of the three scales are 2, 3, and 4 respectively, and the width is E. bert The width of the convolution kernel is , and the three convolution kernels need to perform Relu activation and MaxPooling pooling operations respectively after convolution; finally, the vectors obtained from the three scales are horizontally spliced ​​to obtain the vector l; l is expanded to the same size as E by expansion. local The same shape is obtained to obtain the global feature representation matrix E global, use horizontal splicing to splice the three feature representation matrices E bert 、E local 、E global Get the context feature representation matrix E context , concat is the horizontal splicing operation.

[0033] Furthermore, the specific process of predicting the start position and the end position of each token in step S5 is as follows:

[0034] I start =softmax(E context T start

[0035] I end =softmax(E context T end

[0036]

[0037]

[0038] First, use the matrix T start and the context feature representation matrix E context Multiply them together and perform a softmax operation to get the probability matrix I of all tokens at the beginning of the entity. start , where softmax is an operation to normalize the vector; using the matrix T end R context Perform the same operation to obtain the probability matrix I of all tokens at the end position of the entity end ; respectively for I start ,I end Use argmax to get the result of each token being at the start or end position

[0039] Further, in step S6, Token at each start position in the target text i , and its characteristics are expressed as Token at each end position in the target text j The feature representation is horizontally concatenated to obtain the vector m i ; Vertically concatenate all vectors m to get the matrix E match ; Use the linear transformation matrix T match With E match Multiply them together, and then use the Sigmoid function to normalize the result; each pair of matching tokens is the starting token of an extracted named entity.

[0040] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0041] 1. Splice the context for each sample in a sliding window manner and convert the ordinary text data into machine reading comprehension data. The spliced ​​samples are passed as input to the Bert pre-training model to encode the information and obtain word vector features. Such features contain more complete context information.

[0042] 2. The present invention uses forward time series operations and reverse time series operations to further encode word vector features, obtain local features of each token, and use local features to enhance Bert features.

[0043] 3. The present invention uses multi-scale convolution operations to encode the Bert output matrix to obtain the feature representation of the entire sentence and use it as the global feature of each token. The three features are fused in a horizontal splicing method with equal weights, so that each token can take into account both the local information and the global information of the text when performing subsequent classification.

[0044] 4. The present invention fully extracts the information in the sentences input by the user before and after in the question-answering system, can achieve excellent results on the public data of named entity recognition, and has good practicality in the question-answering system with high accuracy requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of a model of the identification method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, the present invention provides a named entity recognition method for an intelligent question answering system, and the specific steps are as follows:

[0048] S1: Splice the context for each sample in a sliding window manner, and convert the ordinary text data into machine reading comprehension data, and use double pointers to mark the start and end positions of the target text in the sliding window. Here we take the continuous dialogue data in the question-answering system: s1 = "If I organize it again now, I can do better", s2 = "Then you organize a trip to Cambodia", s3 = "A trip to Thailand can be organized" as an example. The goal at this time is to extract the entities in the text s2. Splice the context for s2 in a sliding window manner, and combine the text in a triple form, that is, in the form of {question, previous text, target text, following text}, and convert it into machine reading comprehension data. And the sample vector obtained by mapping the combined data according to the vocabulary is represented as follows:

[0049] [101,2824,1140,...,5303,5300,5303]

[0050] S2: The vector s obtained in step 1 is passed as input to the Bert pre-trained model. After word embedding, position embedding, and weight calculation operations, the word vector feature representation matrix E is obtained. bert :

[0051] [[1.5219e-01,1.2128e+00,1.5422e+00,...,7.6726e-01,7.7842e-01,5.3879e-01],

[0052] [-1.3302e-01,8.8468e-01,5.6150e-01,...,-9.7261e-02,2.0107e-01,4.4926e-01],

[0053] [-1.5161e-01,7.5818e-01,8.5848e-01,...,-4.9229e-02,3.6858e-01,8.9574e-01],

[0054] [-2.9381e-02,5.2613e-01,5.4271e-01,...,5.8694e-01,5.1136e-01,4.9406e-01],

[0055] [2.9093e-01,6.1812e-01,4.2058e-01,...,5.0125e-01,5.8357e-01,4.0746e-01],

[0056] [1.1721e-02,6.3548e-01,4.1624e-01,...,5.3428e-01,5.8405e-01,4.5776e-01]]

[0057] S3: The matrix E obtained in step S2 bert The forward timing operation and the reverse timing operation are respectively performed, and the results of the forward timing operation and the reverse timing operation are spliced ​​to obtain the final result. The forward timing operation is to perform the matrix E of the result of step S2. bert =[k <1> ;k <2> ;…;k <n>< / n> Each column vector in ] is calculated as follows, k <t>< / t> is the matrix E bertThe column vector of the tth column will get two vectors c of recorded information after each calculation is completed. t , These two vectors will participate in the next calculation. In each calculation, four intermediate vectors i will be obtained. t 、f t , o t , g t , when calculating the intermediate vector, E i 、E f 、E o 、E g are the weight transfer matrices of the input vector when generating the intermediate vector, U i , U f , U o , U g are the vector c obtained from the previous operation when generating the intermediate vector t-1 The weight transfer matrix, b i 、b f 、b o 、b g are the bias vectors added when calculating the intermediate vectors, σ and ta are activation functions, exp represents the exponential function, and · represents the operation of multiplying the elements at the same position in the participating vectors to obtain a vector of the same dimension. Sorting them in sequence will result in the matrix of forward sequential operation.

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] c t =f t ·c t-1 +g t ·i t

[0064]

[0065]

[0066] For reverse timing operation, E bert The column vectors in [k <n>< / n> ;k <n-1>< / n-1> ;…;k <1>] in reverse order, and then the column vectors in the reversed matrix are operated in order from left to right. Each calculation process is the same as the forward sequential operation. Finally, the input k <t>< / t> The corresponding output in the forward and reverse timing operations Splice and get h t . All h t Press the corresponding k <t>< / t> In E bert Arrange in order to get the operation result matrix E of this step. local :

[0067] [[1.3548,1.3259,0.4744,...,0.1502,0.5139,1.6822],

[0068] [0.0096,0.5778,0.2842,...,0.1450,-0.0977,0.6557],

[0069] [0.0222,0.8222,0.8878,...,0.6338,-0.1823,0.5913],

[0070] [0.2443,0.6035,0.2297,...,0.1787,0.5568,0.9126],

[0071] [0.2571,0.7812,0.1626,...,0.0790,0.1188,1.0229],

[0072] [0.2927,0.8151,0.2736,...,0.1522,0.4337,0.6115]]

[0073] S4: Transform the matrix E bert Through convolution operation, regularization operation and pooling operation respectively, we get vector l, and expand l into the same vector as E Local The same shape, we get the matrix E global First, E bert The matrix A is obtained by using three-dimensional convolution kernels of three scales, where the lengths of the convolution kernels of the three scales are 2, 3, and 4 respectively, and the width is E. bert The width of the convolution kernel is , and the three convolution kernels need to be activated and pooled after convolution. Finally, the vectors obtained from the three scales are horizontally spliced ​​to obtain the vector l. By expanding the vector, l is expanded to the same size as E. local The same shape is obtained to obtain the global feature representation matrix E global , horizontally concatenate the three feature representation matrices E using equal weightsbert 、E local 、E global Get the context feature representation matrix E context :

[0074] [[1.5219e-01,1.2128e+00,1.5422e+00,...,0.1502,0.5139,1.6822],

[0075] [-1.3302e-01,8.8468e-01,5.6150e-01,...,0.1450,-0.0977,0.6557],

[0076] [-1.5161e-01,7.5818e-01,8.5848e-01,...,0.6338,-0.1823,0.5913],

[0077] [-2.9381e-02,5.2613e-01,5.4271e-01,...,0.1787,0.5568,0.9126],

[0078] [2.9093e-01,6.1812e-01,4.2058e-01,...,0.0790,0.1188,1.0229],

[0079] [1.1721e-02,6.3548e-01,4.1624e-01,...,0.1522,0.4337,0.6115]]

[0080] S5: Using two matrices: Start with the linear transformation matrix T start , the linear change matrix of the end position T end , respectively for E context Perform linear transformation to obtain the corresponding starting position prediction probability matrix I start And the end position prediction probability matrix I end Where T start With T end is a randomly initialized matrix. During model training, its data is gradually fixed. And its shape is the same as E context The transposed shape is consistent. And use the argmax function on the two probabilities to get the result of whether each token can be used as the start index or the end index

[0081] [0, 0, 0, ... 0, 0, 0]

[0082] [0, 0, 0, ... 0, 0, 0]

[0083] S6: For step S5 Token at each start position in the target text i , and its characteristics are expressed as Token at each end position in the target text j The feature representation is horizontally concatenated to obtain the vector m i . All vectors m i Vertical splicing to get matrix E match . Using the linear transformation matrix T match With E match Multiply, where T match is a randomly initialized matrix, and the data in it is gradually fixed during model training. And its shape is the same as E match The shapes after transposition are consistent. Then use the Sigmoid function to normalize the results, get the probability of each pair of start and end positions matching, and select the token pairs with a probability greater than 0.5 as the result.

[0084] In this embodiment, the following vector is obtained. Since there is only one token pair at the start position and the end position in the target text, there is only one value in the vector. The matching probability of the first group of token pairs is greater than the threshold value 0.5, so the first group of token pairs can be used as the start and end tokens of a named entity respectively, and a named entity result "Cambodia" can be obtained according to the start position and the end position.

[0085] [0.9433].

[0086] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A named entity recognition method for an intelligent question answering system, characterized by: The steps include: S1: Splice the context for each target sample in a sliding window manner, and treat the conversation data in the question-answering system as ordinary text data, convert it into machine reading comprehension data, and splice a natural language question for each sample to obtain the sample's initial vector s; S2: Pass the vector s as input to the Bert pre-trained model to obtain the word vector feature representation matrix E bert ; S3: Transform the matrix E bert Through the forward time series operation and the reverse time series operation respectively, and vertically splicing the results of the two operations, the local feature representation matrix E is obtained. Local ; The forward sequential operation in step S3 is to perform a matrix E bert =[k <1> ;k <2> ;…;k <n>< / n> Each column vector of ] is calculated in the following order from left to right: c t =f t ·c t-1 +g t ·i t Among them, k <t>< / t> is the matrix E bert The column vector of the tth column will get two vectors c of recorded information after each calculation is completed. t , These two vectors will participate in the next calculation. In each calculation, four intermediate vectors i will be obtained. t 、f t , o t , g t , when calculating the intermediate vector, E i 、E f 、E o 、E g They are respectively the input vector k when generating the intermediate vector <t>< / t> The weight transfer matrix, U i , U f , U o , U g They are the vector c obtained from the previous operation when generating the intermediate vector. t-1 The weight transfer matrix, b i 、b f 、b o 、b g are the bias vectors added when calculating the intermediate vectors, σ and ta are activation functions, exp represents the exponential function, and · represents the operation of multiplying the elements at the same position in the participating vectors to obtain a vector of the same dimension. Sorting them in sequence will result in the matrix of forward sequential operation; The reverse timing operation in step S3 requires E bert The column vectors in [k <n>< / n> ;k <n-1>< / n-1> ;…;k <1> ] are arranged in reverse order, and then the column vectors in the reversely arranged matrix are operated in sequence from left to right to obtain the output vector; S4: Transform the matrix E bert Through convolution and pooling operations, we get vector l and expand l to be the same as E Local The same shape is obtained, and the global feature representation matrix E is obtained. global ; E bert 、E local 、E global The context feature representation matrix E is obtained by splicing in the horizontal direction in an equal-weighted manner contex t; S5: Using two matrices: Start with the linear transformation matrix T start , the linear change matrix of the end position T end , respectively for E context Perform linear transformation to obtain the corresponding starting position prediction probability matrix I start And the end position prediction probability matrix I end ; and use the argmax function on the two probability representation matrices to get the result of whether each token can be used as the start index or the end index S6: For each token at the start position in the target text, its feature representation is horizontally concatenated with the feature representation of each token at the end position in the target text to obtain a vector m i ; All vectors m i Vertical splicing to get matrix E match ; Use the linear transformation matrix T match With E match Multiply them, and then use the Sigmoid function to normalize the result; a group of tokens with a probability greater than or equal to the threshold are taken as the start and end of an entity extracted from the sample, and the extracted entity is the recognition result.

2. According to claim 1, a method for named entity recognition for an intelligent question answering system is characterized in that: In step S4, the matrix E bert The vectors are obtained through convolution operations and pooling operations at multiple scales, and then the matrix E is obtained. global With E context The specific operation process is: A=Conv(E bert ) A=Relu(A) l=MaxPooling(A) It is global =expand(l) AND context =concat(E bert ,AND local ,AND global ) First, E bert The matrix A is obtained by using three-dimensional convolution kernels of three scales, where the lengths of the convolution kernels of the three scales are 2, 3, and 4 respectively, and the width is E. bert The width of the convolution kernel is , and the three convolution kernels need to perform Relu activation and MaxPooling pooling operations respectively after convolution; finally, the vectors obtained from the three scales are horizontally spliced ​​to obtain the vector l; l is expanded to the same size as E by expansion. local The same shape is obtained to obtain the global feature representation matrix E global , use horizontal splicing to splice the three feature representation matrices E bert 、E local 、E global Get the context feature representation matrix E context , concat is the horizontal splicing operation.

3. The method for named entity recognition for an intelligent question answering system according to claim 1, characterized in that: The specific process of predicting the start position and the end position of each token in step S5 is as follows: I start =softmax(E context T start ) I end =softmax(E context T end ) First, use the matrix T start and the context feature representation matrix E context Multiply them together and perform a softmax operation to get the probability matrix I of all tokens at the beginning of the entity. start , where softmax is an operation to normalize the vector; using the matrix T end For E context Perform the same operation to obtain the probability matrix I of all tokens at the end position of the entity end ; respectively for I start ,I end Use argmax to get the result of each token being at the start or end position 4. The method for named entity recognition for an intelligent question answering system according to claim 1, characterized in that: In step S6, Token at each start position in the target text i , and its characteristics are expressed as Token at each end position in the target text j The feature representation is horizontally concatenated to obtain the vector m i ; Vertically concatenate all vectors m to get the matrix E match ; Use the linear transformation matrix T match With E match Multiply them together, and then use the Sigmoid function to normalize the result; each pair of matching tokens is the starting token of an extracted named entity.

Citation Information

Patent Citations

  • A control method and a control device for realizing medical semantic normalization based on LSTM

    CN109670177A

  • Natural language reasoning method for intelligent question-answering system

    CN111475620A

  • Named entity recognition method and apparatus, terminal device and storage medium

    WO2021159718A1