A named entity recognition method for human-computer dialogue system

Abstract

A named entity recognition method for human-computer dialogue systems first removes the causal mask of a large language model to fully exploit the bidirectional semantic embedding representation capability of the large language model, making it more suitable for accurate token classification tasks; then, the semantic similarity of the semantic embedding of the large language model is calculated and fused into the semantic embedding of the large language model as new supervisory information to improve the insufficient label supervision capability of the demasked large language model when restoring bidirectional attention, thereby enabling the model to obtain a better semantic embedding to calculate the corresponding entity score; finally, the entity and type scores are calculated separately to identify named entities. This invention takes into account the bidirectional semantic understanding information and semantic similarity information of the large language model, improving the accuracy of named entity recognition of the large language model in human-computer dialogue systems.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a named entity recognition method for a human-computer dialogue system. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, human-computer dialogue systems have been widely used in fields such as intelligent customer service, voice assistants, and online education. As one of the core technologies of human-computer dialogue systems, named entity recognition plays a vital role in text understanding, identifying user intent, and contextual semantic parsing. By identifying entities such as names of people, places, and organizations in user input text, the system can more accurately understand user needs and provide more contextually appropriate services. However, as the complexity of human-computer dialogue systems increases, how to efficiently identify named entities in human-computer dialogue sentences and thus more accurately understand user needs has become a key issue that human-computer dialogue systems need to address. Technologies based on large language models can provide human-computer dialogue systems with accurate and efficient dialogue capabilities and high generalization.

[0003] Traditional named entity recognition methods are primarily based on rule-based or statistical learning models. These methods rely on manually designed features or large amounts of labeled data from specific domains, resulting in limited adaptability. In recent years, deep learning models such as BiLSTM-CRF and Transformer have made significant progress in named entity recognition. By leveraging contextual information and powerful feature learning capabilities, these models achieve efficient prediction of entity boundaries and types. However, they may still face generalization bottlenecks when faced with diverse linguistic expressions and cross-domain application scenarios.

[0004] The rise of large language models has brought new opportunities for the development of named entity recognition and human-computer dialogue technology. In particular, generative large language models based on causal mask modeling and autoregressive modeling have demonstrated powerful context understanding capabilities. However, in practical applications, models based on causal mask modeling may have the problem of insufficient semantic information. Each token is only visible to the token behind it, but not to the previous token, which limits the ability to model the global context and performs poorly in strict token classification tasks. This may lead to performance degradation when performing named entity recognition on complex syntactic structures or when context dependency is required. In order to more accurately identify named entities, the present invention proposes a named entity recognition method for human-computer dialogue systems. By eliminating the limitations of the masking mechanism and combining semantic similarity calculation, the accuracy and generalization ability of named entity recognition are improved. Summary of the Invention

[0005] In order to overcome the shortcomings of low accuracy and poor generalization of named entity recognition in current human-computer dialogue systems, and to address the problems of complex syntactic structures and insufficient context understanding in human-computer dialogues, the present invention proposes a named entity recognition method for human-computer dialogue systems. It uses a demasked large language model to obtain rich contextual semantic information, and by fusing semantic similarity, the model can better understand the semantic information in human-computer dialogues.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A named entity recognition method for human-computer dialogue systems first removes the causal mask of a large language model to fully exploit the bidirectional semantic embedding representation capability of the large language model, making it more suitable for accurate token classification tasks. Then, the semantic similarity of the semantic embedding of the large language model is calculated and fused into the semantic embedding of the large language model as new supervision information to improve the insufficient label supervision capability of the demasked large language model when restoring bidirectional attention, thereby enabling the model to obtain better semantic embeddings to calculate the corresponding entity scores. Finally, the entity and type scores are calculated separately to identify named entities.

[0008] Furthermore, the method comprises the following steps:

[0009] Step 1: A human-computer dialogue statement is represented as C = {c1,…c i ,…,c L}, where c1 represents the first character in the human-computer dialogue sentence, c i Indicates the i-th character in the human-computer dialogue sentence, c L Indicates the Lth character in the human-computer dialogue sentence, where L represents the number of characters;

[0010] Step 2: Use the large language model to obtain the token input sequence of the human-computer dialogue sentence C Where n represents the length of the sequence after converting C into tokens;

[0011] Step 3: Use the large language model to convert the input sequence X into the initial embedding vector H = {h1,…h i ,…,h n},in d is the dimension of the hidden vector in the large language model;

[0012] Step 4: Calculate the attention matrix of a single attention head in the large language model Where p is the number of heads in the multi-head attention mechanism, the causal mask Mask of the large language model is set to all zeros, and W Q , W K , W V They are the query matrix, key matrix and value matrix of the large language model respectively;

[0013] Step 5: Merge multiple single-head attention matrices to obtain the final multi-head attention matrix MultiHead(H) = Concat(head1,…head i ,…,head p )W O , where W O is the output matrix of the large language model;

[0014] Step 6: Calculate HW+HBA, where the W weight parameter matrix contains the query matrix W Q , bond matrix W K , value matrix W V And the output matrix W O , B and A are two learnable low-rank matrices, which together form the LoRA layer;

[0015] Step 7: H is calculated through multiple stacked large language model blocks to obtain the hidden representation sequence H of the last layer;

[0016] Step 8: In H={h1,…h i ,…,h n}, calculate each h i Calculate the cosine similarity with the entire H sequence in

[0017] Step 9: Calculate whether the predicted token should be the predicted label sequence of the entity in W s is a learnable matrix,

[0018] Step 10: Calculate Y 1 Cross entropy loss function with Y'

[0019]

[0020] in y i ' is the label data used to indicate whether the token is a named entity, y i 'The first dimension value is 1, indicating that the token is a named entity, y i 'The second dimension value is 1, indicating that the token is not a named entity;

[0021] Step 11: Calculate the predicted label sequence of the entity type of the predicted token Where W l is a learnable weight matrix, m is the number of named entity types that need to be recognized;

[0022] Step 12: Calculate Y 2 Cross entropy loss function with Y

[0023]

[0024] in y i It is the label data used to indicate what type of named entity the token is;

[0025] Step 13: Calculate the total loss function Where λ1 and λ2 are hyperparameters;

[0026] Step 14: Repeat steps 2 to 13. When the loss is less than the specified minimum value, the calculation ends;

[0027] Step 15: Calculate the predicted named entity recognition result in the human-computer dialogue system pred = argmax(softmax(Y 2 )).

[0028] The technical concept of the present invention is as follows: In the named entity recognition method in the human-computer dialogue system, the bidirectional semantic understanding ability of the large language model is fully exploited, and by integrating the semantic similarity information as auxiliary information into the semantic understanding of the large language model, the accuracy of the large language model for strict token classification tasks is improved.

[0029] The beneficial effects of the present invention are: comprehensive consideration of the bidirectional semantic understanding information and semantic similarity information of the large language model, thereby improving the accuracy of named entity recognition of the large language model in the human-computer dialogue system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The figure is a flowchart of a named entity recognition method for a human-computer dialogue system. DETAILED DESCRIPTION

[0031] The present invention will be further described below with reference to the accompanying drawings.

[0032] Reference Figure 1 , a named entity recognition method for a human-computer dialogue system, comprising the following steps:

[0033] Step 1: A human-computer dialogue statement is represented as C = {c1,…c i ,…,c L}, where c1 represents the first character in the human-computer dialogue sentence, c i Indicates the i-th character in the human-computer dialogue sentence, c L Indicates the Lth character in the human-computer dialogue sentence. L indicates the number of characters.

[0034] Step 2: Use the large language model to obtain the token input sequence of the human-computer dialogue sentence C Where n represents the length of the sequence after converting C into tokens;

[0035] Step 3: Use the large language model to convert the input sequence X into the initial embedding vector H = {h 1, …h i ,…,h n},in d is the dimension of the hidden vector in the large language model;

[0036] Step 4: Calculate the attention matrix of a single attention head in the large language model Where p is the number of heads in the multi-head attention mechanism, the causal mask Mask of the large language model is set to all zeros, and W Q , W K , W V They are the query matrix, key matrix and value matrix of the large language model respectively;

[0037] Step 5: Combine multiple single-head attention matrices to obtain the final multi-head attention matrix MultiHead(H) = Concat(head 1, …head i ,…,head p )W O , where W O is the output matrix of the large language model;

[0038] Step 6: Calculate HW+HBA, where the W weight parameter matrix contains the query matrix W Q , bond matrix W K , value matrix W V And the output matrix W O , B and A are two learnable low-rank matrices, which together form the LoRA layer;

[0039] Step 7: H is calculated through multiple stacked large language model blocks to obtain the hidden representation sequence H of the last layer;

[0040] Step 8: At H = {h 1, …h i ,…,h n}, calculate each h i Calculate the cosine similarity with the entire H sequence in

[0041] Step 9: Calculate whether the predicted token should be the predicted label sequence of the entity in W sis a learnable matrix,

[0042] Step 10: Calculate Y 1 Cross entropy loss function with Y'

[0043]

[0044] in y i ' is the label data used to indicate whether the token is a named entity, y i 'The first dimension value is 1, indicating that the token is a named entity, y i 'The second dimension value is 1, indicating that the token is not a named entity;

[0045] Step 11: Calculate the predicted label sequence of the entity type of the predicted token Where W l is a learnable weight matrix, m is the number of named entity types that need to be recognized;

[0046] Step 12: Calculate Y 2 Cross entropy loss function with Y

[0047]

[0048] in y i It is the label data used to indicate what type of named entity the token is;

[0049] Step 13: Calculate the total loss function Where λ1 and λ2 are hyperparameters;

[0050] Step 14: Repeat steps 2 to 13. When the loss is less than the specified minimum value, the calculation ends;

[0051] Step 15: Calculate the predicted named entity recognition result in the human-computer dialogue system pred = argmax(softmax(Y 2 )).

[0052] In this example, for the natural language sentence "Which county in Ningbo City ranks higher in the list of top 100 counties?", the human-computer dialogue system needs to identify the location type entity. After the word segmentation of the large language model, the following is obtained: <s>In Ningbo City, which county ranks higher in the list of top 100 counties? When calculating the attention matrix, the original causal mask become This allows the large language model to obtain rich bidirectional semantic information, making it more suitable for accurate token classification tasks. It calculates the similarity scores between the last layer of hidden representations of multiple stacked blocks of the large language model and calculates the similarity loss function through learnable weight parameters. In order to improve the problem of insufficient label supervision ability of the demasked large language model when restoring bidirectional attention, the last layer of hidden representation is calculated by a linear classifier and the loss function of the label The loss is used to reversely update the LoRA parameters of the large language model, so that after the hidden representation from the large language model passes through the linear classifier, the named entity recognition result of the natural language sentence can be obtained as Ningbo City.

[0053] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.< / s>

Claims

1. A named entity recognition method for a human-computer dialogue system, characterized in that: The method comprises the following steps: Step 1: A human-computer dialogue statement is expressed as ,in Indicates the first character in the human-computer dialogue sentence. Indicates the first i characters, Indicates the first characters, Indicates the number of characters; Step 2: Use a large language model to obtain human-computer dialogue sentences The token input sequence ,in Indicates that The length of the sequence after conversion to tokens; Step 3: Use a large language model to transform the input sequence Converted to the initial embedding vector ,in , is the dimension of the hidden vector in the large language model; Step 4: Calculate the attention matrix of a single attention head in the large language model ,in is the number of heads in the multi-head attention mechanism, the causal mask of the large language model is set to all zeros, , , They are the query matrix, key matrix and value matrix of the large language model respectively; Step 5: Merge multiple single-head attention matrices to obtain the final multi-head attention matrix ,in is the output matrix of the large language model; Step 6. Calculation ,in The weight parameter matrix contains the query matrix , bond matrix , value matrix and the output matrix , and are two learnable low-rank matrices that form the LoRA layer; Step 7 After calculation of multiple stacked large language model blocks, the hidden representation sequence of the last layer is obtained ; Step 8: In each All with the whole Calculate cosine similarity of sequences , ,in ; Step 9: Calculate whether the predicted token should be the predicted label sequence of the entity ,in , is a learnable matrix, ; Step 10: Calculation and The cross entropy loss function ， in , It is the label data used to indicate whether the token is a named entity. The first dimension value is 1, which means the token is a named entity. A value of 1 in the second dimension indicates that the token is not a named entity; Step 11: Calculate the predicted label sequence of the entity type of the predicted token ,in is a learnable weight matrix, , is the number of named entity types that need to be recognized; Step 12: Calculation and The cross entropy loss function ， in , It is the label data used to indicate what type of named entity the token is; Step 13: Calculate the total loss function ,in and It is a super parameter; Step 14: Repeat steps 2 to 13. When the loss is less than the specified minimum value, the calculation ends; Step 15: Calculate the predicted named entity recognition results in the human-computer dialogue system .

Images