Intelligent customer service response generation method and system

Abstract

The invention discloses an intelligent customer service response generation method and system, and relates to the technical field of intelligent customer service response generation. According to the method, firstly, partial entities in a statement are shielded, a model is forced to more depend on other vocabularies in a context to deduce the significance of the shielded entities instead of excessively depending on the appearance of a specific entity to generate a response, and the model learns to extract information from the overall context; therefore, the ability of the model to understand and utilize the context information is enhanced, and the model can still make reasonable inferences when encountering problems and containing unknown entities. And secondly, the model can pay more attention to contextual information related to the occluded entity by amplifying the weight of the entity having the relationship with the occluded entity, and even if a plurality of entities exist, the model can generate an answer according to the information related to the occluded entity, so that the generated answer is more related and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent customer service response generation, and in particular to an intelligent customer service response generation method and system. Background Art

[0002] After-sales customers receive a large number of inquiries from users every day. Therefore, establishing an intelligent customer service response generation system to quickly respond to users' common questions can reduce the workload of customer service personnel and improve service efficiency, and ensure that the information obtained by all users is unified and accurate, avoiding misunderstandings and problems caused by inconsistent information transmission; the question and answer system can also collect user problem data to understand the issues that users are most concerned about and the pain points of services.

[0003] In the existing technology, Transform can be used as the basic architecture of the question-answering model. The encoder in Transform encodes user questions based on a multi-head attention mechanism and a feedforward neural network to obtain the semantic vector of the user question; the decoder in Transform decodes the semantic vector of the user question through a multi-head attention mechanism, a feedforward neural network and a masked multi-head self-attention mechanism, and outputs the corresponding answer.

[0004] However, when the user's question contains multiple entities that need to be paid attention to, such as when the question includes both the electricity bill and the electricity meter, the transform's attention mechanism may deviate when allocating attention weights, resulting in the obtained contextual meaning and the focus of the user's question deviating, and the generated answer is not highly relevant to the user's question. Summary of the invention

[0005] Based on this, in order to solve the technical problem in the prior art that when a user question contains multiple entities, the generated answer is not highly relevant to the question, the present invention provides an intelligent customer service response generation method and system.

[0006] The present invention provides a method for generating an intelligent customer service response, comprising:

[0007] Input user questions into the question-answering model and output the customer service answers corresponding to the user questions;

[0008] The question-answering model includes an entity masking module, an encoder module, a decoder module and an output module connected in sequence; the encoder module includes a first multi-head attention mechanism module and a masked entity attention mechanism module connected in sequence; the entity masking module is used to mask some entities in the user question; the encoder module is used to encode the masked user question to obtain a first question semantic vector; the first multi-head attention mechanism module in the decoder module is used to weight the first question semantic vector to obtain a first answer semantic vector; the masked entity attention mechanism module in the decoder module is used to increase the weight of the entity or event in the first answer semantic vector that has a relationship with the masked entity to obtain a second answer semantic vector; the output module is used to obtain the corresponding customer service answer based on the second answer semantic vector.

[0009] Furthermore, the entity mask module specifically includes:

[0010] The entity recognition layer is used to perform entity recognition on user questions and obtain all entities in the user questions;

[0011] The entity relationship extraction layer is used to extract entities and events that have relationships in user questions, as well as entities that have relationships with each other;

[0012] The entity selection layer is used to randomly select k entities from all entities and label them as

[0013] The entity mask generation layer is used to generate an entity mask vector with the same length as the sequence of user questions to be answered, where The mask value of the corresponding position in the entity mask vector is set to 0; the mask value of the remaining positions is set to 1;

[0014] The relation mask layer is used to generate a relation mask vector with the same length as the sequence of user questions to be answered, where The mask value corresponding to the entity or event with a relationship in the relationship mask vector is L i Indicates; the mask values ​​of the remaining positions are set to 0;

[0015] Entity mask layer; used to multiply the entity mask vector by the user question to obtain the masked user question to be answered;

[0016] The relationship mask layer is used to multiply the relationship mask vector by the user question to be answered to obtain entities or events that have a relationship with the masked entity.

[0017] Furthermore, the masked entity attention mechanism module specifically includes:

[0018] Masked entity attention calculation layer, used to generate masked entity enhancement matrix

[0019]

[0020] in, is the mask entity enhancement matrix The element value of the i-th row and j-th column in ; represents the i-th masked entity; n Li Indicates the number of tokens corresponding to entities or events that have a relationship with the i-th masked entity; e represents a natural number;

[0021] A masked entity weighted layer that multiplies the matrix with the attention score matrix output by the multi-head attention mechanism module:

[0022]

[0023] in, Represents the output of the masked entity attention mechanism module; I represents the all-1 matrix; Attentionoutput masked The attention score matrix representing the output of the multi-head attention module in the encoder module.

[0024] Furthermore, the decoder module further includes:

[0025] a masked multi-head self-attention mechanism module connected to the input end of the first multi-head attention mechanism module, configured to mask all positions after the output position being generated when calculating the self-attention weight;

[0026] The first feedforward neural network module connected to the output end of the masked entity attention mechanism module is used to perform nonlinear operations on the second answer semantic vector.

[0027] Furthermore, the encoder module specifically includes: a second multi-head attention mechanism module and a second feedforward neural network module connected in sequence;

[0028] The second multi-head attention mechanism module calculates the attention weights of any two positions in the user question to capture the long-distance dependencies in the user question;

[0029] The second feedforward neural network module is used to perform nonlinear transformation on the output of the second feedforward neural network module to obtain a first question semantic vector.

[0030] Furthermore, the output module specifically includes: a linear transformation layer, a Softmax activation function layer and a greedy decoding layer connected in sequence;

[0031] The linear transformation layer is used to perform a linear transformation on the second answer semantic vector;

[0032] The Softmax activation function layer is used to convert the output of the linear transformation layer into a probability distribution;

[0033] The greedy decoding layer is used to use a greedy decoding strategy to select the word with the highest probability as output according to the probability distribution output by the Softmax activation function layer.

[0034] The present invention provides an intelligent customer service response generation system, comprising:

[0035] The response generation module is used to input user questions into the question-answering model and output the customer service answers corresponding to the user questions;

[0036] The question-answering model includes an entity masking module, an encoder module, a decoder module and an output module connected in sequence; the encoder module includes a first multi-head attention mechanism module and a masked entity attention mechanism module connected in sequence; the entity masking module is used to mask some entities in the user question; the encoder module is used to encode the masked user question to obtain a first question semantic vector; the first multi-head attention mechanism module in the decoder module is used to weight the first question semantic vector to obtain a first answer semantic vector; the masked entity attention mechanism module in the decoder module is used to increase the weight of the entity or event in the first answer semantic vector that has a relationship with the masked entity to obtain a second answer semantic vector; the output module is used to obtain the corresponding customer service answer based on the second answer semantic vector.

[0037] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0038] In the intelligent customer service response generation method provided by the present invention, by blocking some entities in the user's question, the model is forced to rely more on other words in the context to infer the meaning of the blocked entity. In this way, the model will not rely too much on the appearance of specific entities to generate responses, but learns to obtain overall context information from the relationship between entities and the relationship between entities and events, thereby enhancing the model's ability to understand and utilize contextual information, and helping the model to more accurately understand the focus of user questions in multi-entity situations; and in order to compensate for the information loss caused by blocking entities during the encoding process, the weight of entities that have a relationship with the blocked entity is amplified during the encoder decoding process, so that when generating answers, it can focus on information related to the blocked entity, thereby generating answers that are more relevant to the focus of the question when the information is blocked. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 A schematic diagram of the network structure of the original Transformer provided by the present invention;

[0041] Figure 2 A schematic diagram of a flow chart of an intelligent customer service response generation method provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the network structure of the improved Transformer provided by the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Currently, Transformer can be used as the basic architecture of the question-answering model to process user queries and generate corresponding intelligent customer service responses. The original Transformer network structure is as follows: Figure 1As shown in the figure, it includes an encoder composed of N encoder layers, a decoder composed of N decoder layers, and an output layer. The encoder layer first embeds the tokens in the input text (Input Emebedding) and adds positional encoding (Positional Encoding); then the embedding operation and the token sequence after adding positional encoding are processed through multiple stacked encoder layers. Each encoder layer contains a multi-head attention mechanism layer (Multi-HeadAttention) and a feedforward neural network layer (FeedForward). Each sublayer uses a residual connection and is then normalized (Add&Norm). The multi-head attention mechanism layer allows the Transformer to pay attention to other tokens when processing each token, thereby capturing rich contextual information; the feedforward neural network layer enhances the nonlinear modeling ability of the model, thereby generating a vector representing the semantics of the question. Then, the vector representing the semantics of the question is decoded through multiple stacked decoder layers. In addition to the multi-head attention mechanism layer and the feedforward neural network layer, each decoder layer also contains a masked multi-head self-attention mechanism layer (Masked Multi-Head Attention). The masked multi-head self-attention mechanism ensures that in the process of generating a sequence, the current time step only focuses on the previous time step and does not see future information. The output of each time step of the decoder is a vector, which represents the probability distribution of the next most likely word in the sequence generated so far. This output vector is converted into the probability distribution of each word in the vocabulary through a linear layer (Linear) plus a softmax function. The word with the highest probability is selected as the output word of this time step and then sent to the next time step of the decoder. This cycle continues until an end symbol (EOS, End of Sequence) is generated or the preset maximum length is reached.

[0045] When the Transformer model generates intelligent customer service responses, it may not be able to focus on the key points of the question when faced with questions with multiple entities, resulting in inappropriate answers. This is mainly because the core of the Transformer model is to assign a weight to each element in the sequence through the self-attention mechanism and aggregate information based on these weights. Since entities are more important in the input sequence, the weights corresponding to entities account for a larger proportion. However, when the question contains multiple entities and the importance of these entities in the question is unevenly distributed, the self-attention mechanism may find it difficult to accurately distinguish which entities are more relevant to the question, resulting in the generated response failing to focus on the core point of the question.

[0046] In this context, the present invention provides an intelligent customer service response generation method to solve the above-mentioned defects. By blocking some entities in the user's question, the model is forced to rely more on other words in the context to infer the meaning of the blocked entity. In this way, the model will not rely too much on the appearance of specific entities to understand the problem, but learns to obtain overall context information from the relationship between entities and the relationship between entities and events, thereby enhancing the model's ability to understand and utilize contextual information, and helping the model to more accurately understand the focus of user questions in multi-entity situations; and in order to compensate for the information loss caused by blocking entities during the encoding process, a masked entity attention mechanism module is introduced into the multi-head attention mechanism module of the decoder layer in the original Transformer to amplify the weights of entities or events that have a relationship with the blocked entity, so that when generating an answer, it can pay attention to information related to the blocked entity, thereby generating an answer that is more relevant to the focus of the question when the information is blocked.

[0047] Figure 2 The process of the intelligent customer service response generation method of this embodiment is shown. Figure 2 The intelligent customer service response generation method is described in detail, and specifically includes the following steps:

[0048] S1: Build and improve Transformer.

[0049] The improved Transformer includes an entity mask module, an encoder module, a decoder module, and an output module connected in sequence. The decoder module is the result of adding a mask entity attention mechanism module to the output end of the multi-head attention mechanism module of the decoder layer in the original Transformer. The structure of the improved Transformer is as follows: Figure 3As shown. The encoder module is the same as the encoder of the original transform, including a second multi-head attention mechanism module and a second feedforward neural network module connected in sequence; the second multi-head attention mechanism module calculates the attention weights of any two positions in the user question to capture the long-distance dependency in the user question; the second feedforward neural network module is used to perform a nonlinear transformation on the output of the second feedforward neural network module to obtain the first question semantic vector. The decoder module includes a masked multi-head self-attention mechanism module, a first multi-head attention mechanism module, a masked entity attention mechanism module and a first feedforward neural network module. The output module includes a linear transformation layer, a Softmax activation function layer and a greedy decoding layer connected in sequence; the linear transformation layer is used to perform a linear transformation on the output of the encoding module; the Softmax activation function layer is used to convert the output of the linear transformation layer into a probability distribution; the greedy decoding layer is used to use a greedy decoding strategy according to the probability distribution output by the Softmax activation function layer to select the word with the highest probability as the output.

[0050] Among them, the entity mask module specifically includes:

[0051] The entity recognition layer is used to perform entity recognition on user questions and obtain all entities in the user questions.

[0052] The entity relationship extraction layer is used to extract entities and events that have relationships in user questions, as well as entities that have relationships with each other.

[0053] The entity selection layer is used to randomly select k entities from all entities and label them as

[0054] The entity mask generation layer is used to generate an entity mask vector with the same length as the sequence of user questions to be answered, where The mask value at the corresponding position in the entity mask vector is set to 0; the mask values ​​of the remaining positions are set to 1.

[0055] The relation mask layer is used to generate a relation mask vector with the same length as the sequence of user questions to be answered, where The mask value corresponding to the entity or event with a relationship in the relationship mask vector is L i Indicates; the mask values ​​of the remaining positions are set to 0.

[0056] Entity mask layer; used to multiply the entity mask vector by the user question to obtain the masked user question to be answered.

[0057] The relationship mask layer is used to multiply the relationship mask vector by the user question to be answered to obtain entities or events that have a relationship with the masked entity.

[0058] Among them, the masked entity attention mechanism module specifically includes:

[0059] The masked entity attention calculation layer is used to generate the masked entity enhancement matrix based on the relation mask vector.

[0060]

[0061] in, n is the number of tokens obtained after segmenting the user's question, and is also the dimension of the entity mask vector; is the mask entity enhancement matrix The element value of the i-th row and j-th column in ; represents the i-th masked entity; when the masked entity enhances the matrix When the token corresponding to the diagonal element has a relationship with the token corresponding to the masked entity, the element value of the corresponding position in the masked entity enhancement matrix is Among them, n Li Indicates the number of entities or events (tokens) that have a relationship with the i-th masked entity (token); The more other entities or events (tokens) that have a relationship with a masked entity (token), the greater the weight of the masked entity (token); masked entity enhancement matrix The element values ​​at other positions are 0.

[0062] Masked entity weighting layer, used to enhance the masked entity matrix Multiply it with the attention score matrix output by the multi-head attention mechanism module in the encoder module:

[0063]

[0064] Among them, I is a matrix of all 1s; Attention output masked is the attention score matrix output by the multi-head attention mechanism module in the encoder layer.

[0065] Each head in the multi-head attention module in the decoder layer first calculates the query Q masked , key K masked Sum value V masked Matrix: Q masked =K masked =V masked =output+position; where output is the output sequence of the previous time step, and position is the position code corresponding to the sequence. Then create a mask M (the mask is generally a lower triangular matrix) to block the information of the future time step; then based on Q masked , key Kmasked And M calculates the attention weight: where d model Indicates Q masked and key K masked The superscript T indicates transposition. Finally, based on the attention weight, V masked Weighted: Attentionoutput masked =AttentionWeight masked ·V masked The output of the multi-head attention mechanism module is based on the masked user questions. Therefore, in order to compensate for the information loss caused by the masking of some entities, the masked entity weighted layer enhances the attention output based on the masked entity matrix. masked The weights of entities or events that are related to the masked entity are enhanced, and the weights of entities that are related to the masked entity in the semantic vector output by the encoder layer are amplified through the masked entity attention mechanism module. In this way, even if some entity information is blocked during the decoding process, the model can still pay attention to the information related to the occluded entity, thereby generating answers that are more relevant to the focus of the question.

[0066] S2: Collect user questions and customer service answer pairs to build a knowledge base; train the improved Transformer based on the knowledge base to obtain a large question-answering model for generating answers.

[0067] S201: Classify and label user questions and customer service answers.

[0068] First, a large number of user questions and customer service answers are collected from customer service records. Then, the data is cleaned to remove invalid, erroneous or duplicate records. Then, according to the nature of the question, the question category is defined, such as bill inquiry, fault repair, business consultation, etc. Finally, the data is labeled according to the question type. For example, if the user's question is: "Why is my electricity bill so high?", the customer service answer is: "Your electricity bill may increase due to frequent use of air conditioning in summer. It is recommended to check the electricity usage details." Then the data is classified and labeled as bill inquiry.

[0069] S202: Perform entity recognition, relationship extraction, and event extraction on different types of user questions and customer service answers.

[0070] First, use a pre-trained NLP model (such as BERT or spaCy, etc.) to identify entities in the text, including but not limited to: user name, address, meter number, electricity fee, and fault type. Then, based on dependency syntactic analysis or remote supervision methods, identify the relationship between entities and define the relationship type: for example, there is an association between the meter numbered A12345 and Li Si's user name. Then define event types: such as "abnormal electricity fee" and "equipment failure", and use event extraction models to identify events and related elements in the text. For example, entity recognition is: identifying "electricity fee" as an entity from the question, relationship extraction is: identifying the relationship between "electricity fee" and "air conditioning use" as "leading to", and event extraction is: extracting the "abnormal electricity fee" event from the question-answer pair.

[0071] S203: Construct a knowledge graph using the extracted entities, relationships, and events to obtain a knowledge base.

[0072] First, choose a knowledge graph storage method, such as Neo4j and Apache Jena. Then, use entities or events as nodes and relationships as edges to build a knowledge graph. Then, integrate the extracted entities, relationships, and events into the knowledge graph, and optimize and verify the knowledge graph. For example, create nodes in the knowledge graph: "electricity bill" and "air conditioning use"; create a relationship edge: "air conditioning use" - [results in] -> "electricity bill".

[0073] S204: Train the improved Transformer based on the knowledge base.

[0074] First, the knowledge base dataset is preprocessed, including word segmentation, tokenization, and stop word removal, and the dataset is expanded through data enhancement techniques such as back translation, synonym replacement, and sentence reorganization. Based on the expanded dataset, the Adam optimization algorithm is used to optimize the weight coefficients in the improved transform with the loss function as the optimization target.

[0075] S3: Input the user's question into the question-answering model and output the customer service answer corresponding to the user's question.

[0076] S301: Classify user questions to obtain the user's query intention.

[0077] First, the user question data is preprocessed by word segmentation and stop word removal, and then the features of the user question are extracted using TF-IDF, Word2Vec or BERT, and classified using SVM, Random Forest, neural network, etc. For example, if the user question is: "How to apply for electricity discount?", the query intent is classified as: service application.

[0078] S302: Input the query statement into the question-answering model to generate an answer.

[0079] Taking the answer to the question “Why is my electricity bill so expensive?” as an example, the Q&A model needs to go through the following steps to generate an answer:

[0080] S3021: Before inputting the question into the question-answering model, the text needs to be preprocessed. This includes:

[0081] a. Word segmentation: Split the question text into words or subword units (tokens): ["Excuse me", "my home", "electricity bill", "why", "so", "expensive"].

[0082] b. Encoding: Convert the tokenized text into a numerical representation that the model can understand, usually using pre-trained word embeddings or a lookup table to map each word to a numerical vector: [t1, t2, t3, t4, t5, t6].

[0083] c. Add special markers: Add special markers such as [CLS] and [SEP] at the beginning and end of the sequence to indicate the beginning and end of the sequence: [CLS],[t1,t2,t3,t4,t5,t6,[SEP].

[0084] S3022: Input the entity mask module and perform random masking.

[0085] S3023: Input encoder, input the masked sequence to the encoder part. Including:

[0086] a. Position encoding: Adding position encoding to word embedding enables the model to understand the position information of words.

[0087] b. Second multi-head attention mechanism: The self-attention layer in the encoder calculates the relationship between each word and all other words in the sequence from different representation subspaces and generates a contextual representation of each word.

[0088] d. Second feedforward neural network: After the self-attention layer, the model further processes the representation of each position through a feedforward neural network.

[0089] S3024: Generate an answer using the decoder. After the encoder has processed the question, the decoder starts generating an answer word by word. This includes:

[0090] a. Input to the decoder: The decoder receives the output of the encoder and a previously generated answer sequence (initially [CLS]).

[0091] b. Masked Multi-Head Self-Attention Mechanism Module: In the self-attention layer of the decoder, a mask is used to prevent the decoder from paying attention to future positions.

[0092] c. First multi-head attention mechanism: The encoder-decoder attention layer in the decoder allows the decoder to focus on relevant parts of the encoder output.

[0093] d. Masked Entity Attention Mechanism: Weight the tokens in the output of the first multi-head attention mechanism that are related to the masked entity token.

[0094] e. Second feedforward neural network: performs nonlinear operations.

[0095] S3025: Use the output module to obtain output. Including:

[0096] a. Generate the next word: The decoder outputs the probability distribution of the next word in the sequence, which is usually normalized using the softmax function.

[0097] f. Greedy decoding: Sample or select the word with the highest probability from the probability distribution as the next word and add it to the output sequence.

[0098] S3026: Repeat the decoding steps until a complete response is generated, usually when a [SEP] marker is encountered or a predetermined maximum length is reached.

[0099] In summary, the user questions to be answered are input into the question-answering model, the entity masking module is used to mask some entities in the user questions; the encoder module is used to encode the masked user questions to obtain the first question semantic vector; the first multi-head attention mechanism module in the decoder module is used to weightedly process the first question semantic vector to obtain the first answer semantic vector; the masked entity attention mechanism module in the decoder module is used to increase the weight of the entities or events in the first answer semantic vector that have a relationship with the masked entities to obtain the second answer semantic vector; the output module is used to obtain the corresponding customer service answer based on the second answer semantic vector.

[0100] The above intelligent customer service response generation method first blocks some entities in the sentence before the self-attention mechanism of the encoder layer, forcing the model to rely more on other words in the context to infer the meaning of the blocked entity, rather than relying too much on the appearance of a specific entity to generate a response. Instead, it learns to extract information from the overall context, thereby enhancing the model's ability to understand and utilize contextual information, which helps the model to make reasonable inferences when encountering unknown entities in the question. Secondly, by amplifying the weight of entities that have a relationship with the blocked entity, the model can pay more attention to contextual information related to the blocked entity, such as the dependency between entities, the role of the entity in the question, etc. In this way, even if there are multiple entities, the model can generate answers based on information related to the blocked entity, making the generated answers more relevant and accurate. For example, in the question and answer of the power system, if an entity in the question is blocked, the model can focus on contextual information related to the blocked entity, such as electricity charges, electricity consumption, etc., by amplifying the weights of other entities that have a relationship with the blocked entity, thereby generating answers that are more relevant to the focus of the question. In general, by amplifying the weights of entities that are related to the occluded entities in the encoder, the information loss caused by the occluded entities can be compensated, so that the model can still generate answers that are more relevant to the focus of the question even when the information is occluded.

[0101] The above is an intelligent customer service response generation method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding intelligent customer service response generation system, including:

[0102] The response generation module is used to input user questions into the question-answering model and output the customer service answers corresponding to the user questions.

[0103] The large question-answering model includes an entity masking module, an encoder module, a decoder module and an output module connected in sequence; the encoder module includes a first multi-head attention mechanism module and a masked entity attention mechanism module connected in sequence; the entity masking module is used to mask some entities in the user question; the encoder module is used to encode the masked user question to obtain a first question semantic vector; the first multi-head attention mechanism module in the decoder module is used to weight the first question semantic vector to obtain a first answer semantic vector; the masked entity attention mechanism module in the decoder module is used to increase the weight of the entity or event in the first answer semantic vector that has a relationship with the masked entity to obtain a second answer semantic vector; the output module is used to obtain the corresponding customer service answer based on the second answer semantic vector.

[0104] For the specific definition of the intelligent customer service response generation system, please refer to the definition of the intelligent customer service response generation method above, which will not be repeated here. Each module in the above-mentioned intelligent customer service response generation system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0106] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for generating intelligent customer service response, characterized in that: include: Input user questions into the question-answering model and output the customer service answers corresponding to the user questions; The question-answering model includes an entity masking module, an encoder module, a decoder module and an output module connected in sequence; the encoder module includes a first multi-head attention mechanism module and a masked entity attention mechanism module connected in sequence; the entity masking module is used to mask some entities in the user question; the encoder module is used to encode the masked user question to obtain a first question semantic vector; the first multi-head attention mechanism module in the decoder module is used to weight the first question semantic vector to obtain a first answer semantic vector; the masked entity attention mechanism module in the decoder module is used to increase the weight of the entity or event in the first answer semantic vector that has a relationship with the masked entity to obtain a second answer semantic vector; the output module is used to obtain the corresponding customer service answer based on the second answer semantic vector.

2. The intelligent customer service response generation method according to claim 1, characterized in that: The entity mask module specifically includes: The entity recognition layer is used to perform entity recognition on user questions and obtain all entities in the user questions; The entity relationship extraction layer is used to extract entities and events that have relationships in user questions, as well as entities that have relationships with each other; The entity selection layer is used to randomly select k entities from all entities and label them as The entity mask generation layer is used to generate an entity mask vector with the same length as the sequence of user questions to be answered, where The mask value of the corresponding position in the entity mask vector is set to 0; the mask values ​​of the remaining positions are set to 1; The relation mask layer is used to generate a relation mask vector with the same length as the sequence of user questions to be answered, where The mask value corresponding to the entity or event with a relationship in the relationship mask vector is L i Indicates; the mask values ​​of the remaining positions are set to 0; Entity mask layer; used to multiply the entity mask vector by the user question to obtain the masked user question to be answered; The relationship mask layer is used to multiply the relationship mask vector by the user question to be answered to obtain entities or events that have a relationship with the masked entity.

3. The intelligent customer service response generation method according to claim 2, characterized in that: The masked entity attention mechanism module specifically includes: Masked entity attention calculation layer, used to generate masked entity enhancement matrix in, is the mask entity enhancement matrix The element value of the i-th row and j-th column in ; represents the i-th masked entity; n Li Indicates the number of tokens corresponding to entities or events that have a relationship with the i-th masked entity; e represents a natural number; A masked entity weighted layer that multiplies the matrix with the attention score matrix output by the multi-head attention mechanism module: in, Represents the output of the masked entity attention mechanism module; I represents the all-1 matrix; Attention output masked The attention score matrix representing the output of the multi-head attention module in the encoder module.

4. The intelligent customer service response generation method according to claim 1, characterized in that: The decoder module further comprises: a masked multi-head self-attention mechanism module connected to the input end of the first multi-head attention mechanism module, configured to mask all positions after the output position being generated when calculating the self-attention weight; The first feedforward neural network module connected to the output end of the masked entity attention mechanism module is used to perform nonlinear operations on the second answer semantic vector.

5. The intelligent customer service response generation method according to claim 1, characterized in that: The encoder module specifically includes: a second multi-head attention mechanism module and a second feedforward neural network module connected in sequence; The second multi-head attention mechanism module calculates the attention weights of any two positions in the user question to capture the long-distance dependencies in the user question; The second feedforward neural network module is used to perform nonlinear transformation on the output of the second feedforward neural network module to obtain a first question semantic vector.

6. The intelligent customer service response generation method according to claim 1, characterized in that: The output module specifically includes: a linear transformation layer, a Softmax activation function layer and a greedy decoding layer connected in sequence; The linear transformation layer is used to perform a linear transformation on the second answer semantic vector; The Softmax activation function layer is used to convert the output of the linear transformation layer into a probability distribution; The greedy decoding layer is used to use a greedy decoding strategy to select the word with the highest probability as output according to the probability distribution output by the Softmax activation function layer.

7. An intelligent customer service response generation system, characterized in that: include: The response generation module is used to input user questions into the question-answering model and output the customer service answers corresponding to the user questions; The question-answering model includes an entity masking module, an encoder module, a decoder module and an output module connected in sequence; the encoder module includes a first multi-head attention mechanism module and a masked entity attention mechanism module connected in sequence; the entity masking module is used to mask some entities in the user question; the encoder module is used to encode the masked user question to obtain a first question semantic vector; the first multi-head attention mechanism module in the decoder module is used to weight the first question semantic vector to obtain a first answer semantic vector; the masked entity attention mechanism module in the decoder module is used to increase the weight of the entity or event in the first answer semantic vector that has a relationship with the masked entity to obtain a second answer semantic vector; the output module is used to obtain the corresponding customer service answer based on the second answer semantic vector.

Images