Named entity recognition method for man-machine conversation system

Abstract

A named entity recognition method oriented to a man-machine dialogue system comprises the steps that firstly, causal masks of a large language model are removed to fully mine the bidirectional semantic embedding representation capacity of the large language model so that the large language model can be more suitable for an accurate token classification task; then calculating the semantic similarity of the semantic embedding of the large language model, and fusing the semantic similarity as new supervision information into the semantic embedding of the large language model to improve the problem that the label supervision capability is insufficient when the mask-removed large language model recovers bidirectional attention, so that the model obtains better semantic embedding to calculate a corresponding entity score; and finally, respectively calculating entity and type scores to identify named entities. According to the method, the bidirectional semantic understanding information and the semantic similarity information of the large language model are considered, and the named entity recognition accuracy of the large language model in the man-machine conversation system is improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a named entity recognition method for a human-computer dialogue system. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, human-computer dialogue systems have been widely used in intelligent customer service, voice assistants, online education and other fields. As one of the core technologies of human-computer dialogue systems, named entity recognition plays an important role in text understanding, user intent recognition and contextual semantic parsing. By identifying entities such as names, place names, and organization names in user input text, the system can understand user needs more accurately and provide services that are more in line with the context. However, with the increasing complexity of human-computer dialogue systems, how to efficiently identify named entities in human-computer dialogue sentences and more accurately understand user needs has become a key problem that human-computer dialogue systems need to solve. Technology based on large language models can provide accurate and efficient dialogue capabilities and high generalization for human-computer dialogue systems.

[0003] Traditional named entity recognition methods are mainly based on rules or statistical learning models, which rely on manually designed features or a large amount of annotated data in a specific field and have limited adaptability. In recent years, deep learning models such as BiLSTM-CRF and Transformer have made significant progress in named entity recognition tasks. These models achieve efficient prediction of entity boundaries and types by leveraging contextual information and powerful feature learning capabilities. However, they may still face bottlenecks in generalization capabilities when faced with diverse language expressions and cross-domain application scenarios.

[0004] The rise of large language models has brought new opportunities for the development of named entity recognition and human-computer dialogue technology. In particular, generative large language models based on causal mask modeling and autoregressive modeling have demonstrated powerful context understanding capabilities. However, in practical applications, models based on causal mask modeling may have the problem of insufficient semantic information. Each token is only visible to the token behind it, but not to the previous token, which limits the ability to model the global context and performs poorly in strict token classification tasks. This may lead to performance degradation when performing named entity recognition on complex syntactic structures or when context dependency is required. In order to more accurately identify named entities, the present invention proposes a named entity recognition method for human-computer dialogue systems. By eliminating the limitations of the masking mechanism and combining semantic similarity calculations, the accuracy and generalization ability of named entity recognition are improved. Summary of the invention

[0005] In order to overcome the shortcomings of low accuracy and poor generalization of named entity recognition in current human-computer dialogue systems, and to address the problems of complex syntactic structures and insufficient context understanding in human-computer dialogues, the present invention proposes a named entity recognition method for human-computer dialogue systems, which uses a demasked large language model to obtain rich contextual semantic information, and integrates semantic similarity to enable the model to better understand the semantic information in human-computer dialogues.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A named entity recognition method for human-computer dialogue systems first removes the causal mask of a large language model to fully exploit the bidirectional semantic embedding representation capability of the large language model, making it more suitable for accurate token classification tasks; then the semantic similarity of the semantic embedding of the large language model is calculated and fused into the semantic embedding of the large language model as new supervision information to improve the problem of insufficient label supervision capability of the de-masked large language model when restoring bidirectional attention, so that the model can obtain better semantic embedding to calculate the corresponding entity score; finally, the entity and type scores are calculated respectively to identify named entities.

[0008] Further, the method comprises the following steps:

[0009] Step 1: A human-computer dialogue statement is represented as C = {c1,…c i ,…,c L}, where c1 represents the first character in the human-computer dialogue sentence, c i Indicates the i-th character in the human-computer dialogue sentence, c L Indicates the Lth character in the human-computer dialogue sentence, where L represents the number of characters;

[0010] Step 2: Use the large language model to obtain the token input sequence of the human-computer dialogue sentence C Where n represents the length of the sequence after C is converted into tokens;

[0011] Step 3: Use the large language model to transform the input sequence X into the initial embedding vector H = {h1,…h i ,…,h n},in d is the dimension of the hidden vector in the large language model;

[0012] Step 4: Calculate the attention matrix of a single attention head in the large language model Where p is the number of heads in the multi-head attention mechanism, the causal mask Mask of the large language model is set to all zeros, and W Q , W K , W V They are the query matrix, key matrix and value matrix of the large language model respectively;

[0013] Step 5: Merge multiple single-head attention matrices to obtain the final multi-head attention matrix MultiHead(H) = Concat(head1,…head i ,…,head p )W O , where W O is the output matrix of the large language model;

[0014] Step 6: Calculate HW+HBA, where the W weight parameter matrix contains the query matrix W Q , the bond matrix W K , value matrix W V And the output matrix W O , B and A are two learnable low-rank matrices, which together form the LoRA layer;

[0015] Step 7: H is calculated through multiple stacked large language model blocks to obtain the hidden representation sequence H of the last layer;

[0016] Step 8: In H = {h1,…h i ,…,h n}, calculate each h i Calculate the cosine similarity with the entire H sequence in

[0017] Step 9: Calculate whether the predicted token should be the predicted label sequence of the entity in W s is a learnable matrix,

[0018] Step 10: Calculate Y 1 Cross entropy loss function with Y'

[0019]

[0020] in y i ' is the label data used to indicate whether the token is a named entity, y i 'The first dimension value is 1, indicating that the token is a named entity, y i 'The second dimension value is 1, indicating that the token is not a named entity;

[0021] Step 11: Calculate the predicted label sequence of the entity type of the predicted token Where W l is a learnable weight matrix, m is the number of named entity types that need to be recognized;

[0022] Step 12: Calculate Y 2 The cross entropy loss function with Y

[0023]

[0024] in y i It is the label data used to indicate what type of named entity the token is;

[0025] Step 13: Calculate the total loss function Where λ1 and λ2 are hyperparameters;

[0026] Step 14: Repeat steps 2 to 13. When the loss is less than the specified minimum value, the calculation ends;

[0027] Step 15: Calculate the predicted named entity recognition result in the human-computer dialogue system pred = argmax(softmax(Y 2 )).

[0028] The technical concept of the present invention is as follows: In the named entity recognition method in the human-computer dialogue system, the two-way semantic understanding ability of the large language model is fully exploited, and the semantic similarity information is integrated into the semantic understanding of the large language model as auxiliary information to improve the accuracy of the large language model for strict token classification tasks.

[0029] The beneficial effects of the present invention are: comprehensive consideration of the bidirectional semantic understanding information and semantic similarity information of the large language model, thereby improving the accuracy of named entity recognition of the large language model in the human-computer dialogue system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The figure is a flowchart of a named entity recognition method for a human-computer dialogue system. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings.

[0032] Reference Figure 1 , a named entity recognition method for a human-computer dialogue system, comprising the following steps:

[0033] Step 1: A human-computer dialogue statement is represented as C = {c1,…c i ,…,c L}, where c1 represents the first character in the human-computer dialogue sentence, c i Indicates the i-th character in the human-computer dialogue sentence, c L Indicates the Lth character in the human-computer dialogue sentence. L indicates the number of characters;

[0034] Step 2: Use the large language model to obtain the token input sequence of the human-computer dialogue sentence C Where n represents the length of the sequence after C is converted into tokens;

[0035] Step 3: Use the large language model to transform the input sequence X into the initial embedding vector H = {h 1, …h i ,…,h n},in d is the dimension of the hidden vector in the large language model;

[0036] Step 4: Calculate the attention matrix of a single attention head in the large language model Where p is the number of heads in the multi-head attention mechanism, the causal mask Mask of the large language model is set to all zeros, and W Q , W K , W V They are the query matrix, key matrix and value matrix of the large language model respectively;

[0037] Step 5: Merge multiple single-head attention matrices to obtain the final multi-head attention matrix MultiHead(H) = Concat(head 1, …head i ,…,head p )W O , where W O is the output matrix of the large language model;

[0038] Step 6: Calculate HW+HBA, where the W weight parameter matrix contains the query matrix W Q , the bond matrix W K , value matrix W V And the output matrix W O , B and A are two learnable low-rank matrices, which together form the LoRA layer;

[0039] Step 7: H is calculated through multiple stacked large language model blocks to obtain the hidden representation sequence H of the last layer;

[0040] Step 8: At H = {h 1, …h i ,…,h n}, calculate each h i Calculate the cosine similarity with the entire H sequence in

[0041] Step 9: Calculate whether the predicted token should be the predicted label sequence of the entity in W sis a learnable matrix,

[0042] Step 10: Calculate Y 1 Cross entropy loss function with Y'

[0043]

[0044] in y i ' is the label data used to indicate whether the token is a named entity, y i 'The first dimension value is 1, indicating that the token is a named entity, y i 'The second dimension value is 1, indicating that the token is not a named entity;

[0045] Step 11: Calculate the predicted label sequence of the entity type of the predicted token Where W l is a learnable weight matrix, m is the number of named entity types that need to be recognized;

[0046] Step 12: Calculate Y 2 The cross entropy loss function with Y

[0047]

[0048] in y i It is the label data used to indicate what type of named entity the token is;

[0049] Step 13: Calculate the total loss function Where λ1 and λ2 are hyperparameters;

[0050] Step 14: Repeat steps 2 to 13. When the loss is less than the specified minimum value, the calculation ends;

[0051] Step 15: Calculate the predicted named entity recognition result in the human-computer dialogue system pred = argmax(softmax(Y 2 )).

[0052] In this example, for the natural language sentence "Which county in Ningbo ranks higher in the list of top 100 counties", the human-computer dialogue system needs to identify the entity of the place type. After passing through the word segmenter of the large language model, we can get: <s>In Ningbo, which county ranks higher in the list of top 100 counties? When calculating the attention matrix, the original causal mask become This allows the large language model to obtain rich bidirectional semantic information, making it more suitable for accurate token classification tasks. It calculates the similarity scores between the last layer of hidden representations of multiple stacked blocks of the large language model, and calculates the similarity loss function through learnable weight parameters. In order to improve the problem of insufficient label supervision ability of the demasked large language model when restoring bidirectional attention, the last layer of hidden representation is calculated by a linear classifier and the loss function of the label The loss is used to reversely update the LoRA parameters of the large language model, so that after the hidden representation from the large language model passes through the linear classifier, the named entity recognition result of the natural language sentence can be obtained as Ningbo City.

[0053] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.< / s>

Claims

1. A named entity recognition method for a human-computer dialogue system, characterized in that: First, the causal mask of the large language model is removed to fully tap the bidirectional semantic embedding representation capability of the large language model, making it more suitable for accurate token classification tasks; then the semantic similarity of the semantic embedding of the large language model is calculated and fused into the semantic embedding of the large language model as new supervision information to improve the problem of insufficient label supervision ability of the de-masked large language model when restoring bidirectional attention, so that the model can obtain better semantic embedding to calculate the corresponding entity scores; finally, the entity and type scores are calculated separately to identify named entities.

2. A method for named entity recognition for a human-computer dialogue system as claimed in claim 1, characterized in that: The method comprises the following steps: Step 1: A human-computer dialogue statement is represented as C = {c 1, …c i ,…,c L }, where c1 represents the first character in the human-computer dialogue sentence, c i Indicates the i-th character in the human-computer dialogue sentence, c L Indicates the Lth character in the human-computer dialogue sentence, where L represents the number of characters; Step 2: Use the large language model to obtain the token input sequence X of the human-computer dialogue sentence C = {x 1, …x i ,x n }, Where n represents the length of the sequence after C is converted into tokens; Step 3: Use the large language model to transform the input sequence X into the initial embedding vector H = {h 1, …h i ,…,h n },in d is the dimension of the hidden vector in the large language model; Step 4: Calculate the attention matrix of a single attention head in the large language model Where p is the number of heads in the multi-head attention mechanism, the causal mask Mask of the large language model is set to all zeros, and W Q , W K , W V They are the query matrix, key matrix and value matrix of the large language model respectively; Step 5: Combine multiple single-head attention matrices to obtain the final multi-head attention matrix MultiHead(H) = Concat(head 1, …head i ,…,head p )W O , where W O is the output matrix of the large language model; Step 6: Calculate HW+HBA, where the W weight parameter matrix contains the query matrix W Q , the bond matrix W K , value matrix W V And the output matrix W O , B and A are two learnable low-rank matrices, which together form the LoRA layer; Step 7: H is calculated through multiple stacked large language model blocks to obtain the hidden representation sequence H of the last layer; Step 8: At H = {h 1, …h i ,…,h n }, calculate each h i Calculate the cosine similarity with the entire H sequence in Step 9: Calculate whether the predicted token should be the predicted label sequence of the entity in W s is a learnable matrix, Step 10: Calculate Y 1 Cross entropy loss function with Y' Where Y'={y' 1, …y i ',…,y' N }, y i ' is the label data used to indicate whether the token is a named entity, y i 'The first dimension value is 1, indicating that the token is a named entity, y i 'The second dimension value is 1, indicating that the token is not a named entity; Step 11: Calculate the predicted label sequence of the entity type of the predicted token Where W l is a learnable weight matrix, m is the number of named entity types that need to be recognized; Step 12: Calculate Y 2 The cross entropy loss function with Y Where Y = {y 1, …y i, …y n }, y i It is the label data used to indicate what type of named entity the token is; Step 13: Calculate the total loss function Where λ1 and λ2 are hyperparameters; Step 14: Repeat steps 2 to 13. When the loss is less than the specified minimum value, the calculation ends; Step 15: Calculate the predicted named entity recognition result in the human-computer dialogue system pred = argmax(softmax(Y 2 )).

Images