Intelligent customer service method and system based on user intention recognition

Abstract

The application discloses an intelligent customer service method and system based on user intention recognition, and belongs to the technical field of intelligent customer service, which comprises the following steps: S1, receiving multi-modal dialogue data of a user; S2, analyzing the multi-modal dialogue data based on a user intention recognition model to obtain a user intention recognition result; S3, a response module of the intelligent customer service determines a response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feeds back the response content to the user according to the determined response strategy. The application can realize interactive guidance of user intention recognition and emotion recognition, and improves the accuracy of user intention recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent customer service, and particularly relates to an intelligent customer service method and system based on user intent recognition. BACKGROUND

[0002] With the vigorous development of artificial intelligence technology, intelligent customer service has become an important link between users and services. They gradually replace the traditional manual customer service mode with powerful automated processing capabilities and efficient service response speed. Intelligent customer service systems can receive user inquiries in real time, quickly analyze user intent and provide corresponding answers or services through advanced natural language processing technology and machine learning algorithms. Although existing intelligent customer service solutions have achieved certain results in intelligent response, they still have many defects in actual application. For example, most intelligent customer service systems still mainly rely on single text data when processing user inquiries, and do not make full use of other modal data such as voice and image, resulting in insufficient understanding of user intent.

[0003] Emotion analysis is one of the important research directions of intelligent customer service. The emotional state of the user may affect the acceptance and satisfaction of the user to the content of the intelligent customer service response. Through emotion analysis, the intelligent customer service system can identify the emotional state of the user in real time, such as anger, satisfaction, anxiety, etc., and adjust the service strategy according to these emotional states in combination with the recognition result of the user intent to provide more accurate, considerate and personalized services. In fact, the user's intent and emotion are closely related, and the user's intent usually reflects the user's emotion, and accurate grasp of the user's emotion can correctly understand the user's true intent. However, in the existing intelligent customer service system, the intent recognition and emotion analysis technology are usually realized separately, and the effective fusion and mutual guidance of emotion analysis and intent analysis are not considered, which is particularly insufficient when dealing with complex and emotionally rich user inquiries. Especially when facing inquiries with ambiguous user intent, complex emotions or various expression methods, the existing intelligent customer service system often fails to accurately capture the user's true needs, resulting in inappropriate response strategies or inaccurate answers, which seriously affects the user experience and service quality.

[0004] Therefore, it has become a current urgent problem to provide an intelligent customer service method and system that can fuse multi-modal data, realize emotion and intent interactive recognition, and improve the accuracy of user intent and emotion recognition. SUMMARY

[0005] In view of the defects in the prior art, the present application provides an intelligent customer service method based on user intent recognition, comprising the following steps:

[0006] S1: receiving multi-modal dialogue data of a user, the multi-modal dialogue data comprising text data;

[0007] S2: analyzing the multi-modal dialogue data based on a user intention recognition model to obtain a user intention recognition result; the user intention recognition model comprises a shared bottom layer module, an emotion recognition module, and an intention recognition module;

[0008] The shared bottom layer module is configured to analyze the multi-modal dialogue data to obtain multi-modal features, and input the multi-modal features into the emotion recognition module and the intention recognition module respectively.

[0009] The emotion recognition module uses an attention mechanism to realize emotion recognition guided by the intention based on the multi-modal features, and outputs a user emotion recognition result, and inputs the emotion recognition result into the intention recognition module.

[0010] The intention recognition module realizes emotion-driven intention recognition based on the multi-modal features and the emotion recognition result, and inputs the user intention recognition result into the emotion recognition module.

[0011] The emotion recognition module and the intention recognition module are iterated multiple times, and when the number of iterations reaches a preset iteration threshold, or when the change of a joint loss function of the emotion recognition module and the intention recognition module is less than a preset threshold, the user intention recognition result currently output by the intention recognition module is taken as a final user intention recognition result, and the user emotion recognition result currently output by the emotion recognition module is taken as a final user emotion recognition result.

[0012] S3: The response module of the intelligent customer service determines a response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feeds back the response content to the user according to the determined response strategy.

[0013] Further, in step S1, the multi-modal dialogue data further comprises audio data and video data; in step S2, the shared bottom layer module is configured to analyze the multi-modal dialogue data to obtain multi-modal features, and input the multi-modal features into the emotion recognition module and the intention recognition module, comprising:

[0014] A Bert model is used as a text encoder to extract text features of the text data;

[0015] A wav2vec model is used as an acoustic encoder to extract acoustic features of the audio data;

[0016] A Faster RCNN model is used as a visual encoder to extract visual features of the video data;

[0017] aligning the text feature, the acoustic feature and the visual feature by using a time sequence alignment module, the time sequence alignment module being composed of an LSTM module and a Softmax function;

[0018] connecting the aligned text feature and the acoustic feature to obtain an acoustic enhanced feature;

[0019] connecting the aligned text feature and the visual feature to obtain a visual enhanced feature;

[0020] fusing the acoustic enhanced feature and the visual enhanced feature to obtain a non-text feature;

[0021] calculating a fusion weight between the text feature and the non-text feature, and obtaining the multi-modal feature based on the fusion weight;

[0022] sending the multi-modal feature to the emotion recognition module and the intention recognition module.

[0023] Further, the emotion recognition module comprises an attention module and an emotion recognition model;

[0024] In step S2, the emotion recognition module realizes intention-guided emotion recognition by using an attention mechanism based on the multi-modal feature, outputs an emotion recognition result of the user, and inputs the emotion category to the intention recognition module, comprising:

[0025] In the first iteration, the emotion recognition module performs emotion recognition according to the multi-modal feature to obtain a first emotion recognition result, and inputs the first emotion recognition result to the intention recognition module;

[0026] In the kth iteration, the emotion recognition module realizes intention-guided emotion recognition by using an attention mechanism based on the multi-modal feature and the (k-1)th user intention recognition result, obtains the kth user emotion recognition result, and inputs the kth emotion recognition result to the intention recognition module, wherein 2≤k≤K, and K is a preset iteration threshold;

[0027] The realization of the intention-guided emotion recognition by using the attention mechanism comprises:

[0028] The attention module receives the multi-modal feature and the user intention recognition result as input, obtains an attention weight of each element in the multi-modal feature according to the relevance of each element in the multi-modal feature and the user intention recognition result, inputs the multi-modal feature and the attention weight into the emotion recognition model, and obtains the kth user emotion recognition result.

[0029] Further, the intention recognition module comprises an intention recognition model implemented based on a Transformer encoder; in step S2, the intention recognition module implements emotion-driven intention recognition based on the multi-modal feature and the emotion recognition result, and inputs the user intention recognition result to the emotion recognition module, comprising:

[0030] In the tth iteration, the intention recognition model takes the multi-modal feature and the tth emotion recognition result as input to obtain the tth user intention recognition result, 1≤t≤K;

[0031] When t

[0032] When t=K, or when the change of the joint loss function is less than the preset threshold, the tth user intention recognition result is taken as the final user intention recognition result, and the tth user emotion recognition result is taken as the final user emotion recognition result.

[0033] Further, in step S3, the response module of the intelligent customer service determines a response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feeds back the response content to the user according to the determined response strategy, comprising:

[0034] The response module determines a response strategy based on the final user emotion recognition result, and determines a response content based on the final user intention recognition result, and feeds back the response content to the user according to the response strategy.

[0035] The application also provides an intelligent customer service system based on user intention recognition, comprising:

[0036] An input module for receiving multi-modal dialogue data of a user, the multi-modal dialogue data comprising text data;

[0037] A user intention recognition module for analyzing the multi-modal dialogue data based on a user intention recognition model to obtain a user intention recognition result; the user intention recognition model comprising a shared bottom layer module, an emotion recognition module and an intention recognition module;

[0038] The shared bottom layer module is used to analyze the multi-modal dialogue data to obtain a multi-modal feature, and input the multi-modal feature to the emotion recognition module and the intention recognition module respectively;

[0039] The emotion recognition module realizes emotion recognition guided by the intention based on the multi-modal features by using an attention mechanism, outputs an emotion recognition result of the user, and inputs the emotion recognition result into the intention recognition module;

[0040] The intention recognition module realizes emotion-driven intention recognition based on the multi-modal features and the emotion recognition result, and inputs the user intention recognition result into the emotion recognition module;

[0041] The emotion recognition module and the intention recognition module are iterated for multiple times, and when the number of iterations reaches a preset iteration threshold, or when the change of a joint loss function of the emotion recognition module and the intention recognition module is less than a preset threshold, the user intention recognition result currently output by the intention recognition module is taken as a final user intention recognition result, and the user emotion recognition result currently output by the emotion recognition module is taken as a final user emotion recognition result.

[0042] The response module is configured to determine a response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feed back the response content to the user according to the determined response strategy.

[0043] The application further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the intelligent customer service method based on user intention recognition.

[0044] The application can more accurately understand the real intention of the user by receiving and fusing multi-modal data of the user, and can avoid information loss or misunderstanding caused by single modal data, and improve the accuracy of user intention recognition. In the application, the emotion recognition module realizes emotion recognition guided by the intention by using an attention mechanism, and the emotion recognition result is used as input information to assist the intention recognition module to make more accurate intention judgment. The recognition mechanism of interactive guidance of intention recognition and emotion recognition enables the system to more deeply understand the needs and emotions of the user, and can accurately capture the real needs of the user when facing the user's intention and complex emotions, and provide a more user-expected response interaction result. BRIEF DESCRIPTION OF DRAWINGS

[0045] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0046] Figure 1is a flow chart showing a smart customer service method based on user intention recognition according to an embodiment of the present application.

[0047] Figure 2 is an architecture of a user intention recognition model according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the present application in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all the other embodiments obtained by those skilled in the art without creative work fall into the protection scope of the present application.

[0049] The terms used in the embodiments of the present application are only for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two.

[0050] It should be understood that, although the terms first, second, third, etc. can be used in the embodiments of the present application to describe …, these … should not be limited to these terms. These terms are only used to distinguish … from each other. For example, without departing from the scope of the embodiments of the present application, the first … can also be referred to as the second …, and similarly, the second … can also be referred to as the first ….

[0051] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0052] Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0053] It is also to be noted that the terms "comprising", "comprises" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a product or process that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such product or process. An element proceeded by "comprises a" does not, without further restriction, exclude the presence of additional identical elements in the process or product that comprises the element.

[0054] As shown in Figure 1 Embodiments of the present application disclose a smart customer service method based on user intention recognition, which comprises the following steps:

[0055] S1: receiving multi-modal dialogue data of a user, wherein the multi-modal dialogue data comprises text data;

[0056] S2: analyzing the multi-modal dialogue data based on a user intention recognition model to obtain a user intention recognition result; the user intention recognition model comprises a shared bottom layer module, an emotion recognition module and an intention recognition module;

[0057] The shared bottom layer module is configured to analyze the multi-modal dialogue data to obtain multi-modal features, and input the multi-modal features into the emotion recognition module and the intention recognition module, respectively.

[0058] The emotion recognition module is configured to realize emotion recognition guided by intention based on the multi-modal features by using an attention mechanism, output an emotion recognition result of the user, and input the emotion recognition result into the intention recognition module.

[0059] The intention recognition module is configured to realize emotion-driven intention recognition based on the multi-modal features and the emotion recognition result, and input the user intention recognition result into the emotion recognition module.

[0060] The emotion recognition module and the intention recognition module are configured to iterate multiple times, and when the number of iterations reaches a preset iteration threshold, or when a change of a joint loss function of the emotion recognition module and the intention recognition module is less than a preset threshold, the user intention recognition result currently output by the intention recognition module is taken as a final user intention recognition result, and the user emotion recognition result currently output by the emotion recognition module is taken as a final user emotion recognition result.

[0061] S3: a response module of the smart customer service determines a response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feeds back the response content to the user according to the determined response strategy.

[0062] When the intelligent customer service is implemented, the recognition mechanism of intention recognition and emotion recognition interactive guidance is adopted, the emotion recognition module utilizes the attention mechanism to realize the emotion recognition of intention guidance, meanwhile, the emotion recognition result is taken as input information to assist the intention recognition module to make more accurate intention judgment, so that the system can more deeply understand the demand and emotion of the user, when the user's intention is ambiguous and the emotion is complex, the real demand of the user can be accurately captured, and the response interaction result more in line with the user's expectation can be provided.

[0063] In the step S1, the multi-modal dialogue data further includes audio data and video data; in the step S2, the shared bottom layer module is used for analyzing the multi-modal dialogue data to obtain multi-modal features, and the multi-modal features are input into the emotion recognition module and the intention recognition module respectively, including:

[0064] The Bert model is used as a text encoder to extract text features of the text data;

[0065] The wav2vec model is used as an acoustic encoder to extract acoustic features of the audio data;

[0066] The Faster RCNN model is used as a visual encoder to extract visual features of the video data;

[0067] The text features, the acoustic features and the visual features are aligned by using a time alignment module, and the time alignment module is composed of an LSTM module and a Softmax function;

[0068] The aligned text features and the acoustic features are connected to obtain acoustic enhanced features;

[0069] The aligned text features and the visual features are connected to obtain visual enhanced features;

[0070] The acoustic enhanced features and the visual enhanced features are fused to obtain non-text features;

[0071] The fusion weight between the text features and the non-text features is calculated, and the multi-modal features are obtained based on the fusion weight;

[0072] The multi-modal features are sent to the emotion recognition module and the intention recognition module.

[0073] In the present example, the Bert language model with outstanding performance in the field of natural language processing is used to extract the text features of the text data. i The text features are obtained from the last hidden layer of the Bert encoder, as follows:

[0074]

[0075] where TextEncoder denotes a Bert encoder, denotes an input text sentence t i denotes a text feature obtained after passing through the Bert encoder, l S denotes a length of the text sentence t i denotes a feature dimension of the text sentence t i .

[0076] For audio data, a wav2vec model is used to extract acoustic features of the audio data. The wav2vec model is a self-supervised pre-training acoustic recognition model implemented by a convolutional neural network. As shown below:

[0077]

[0078] where AcousticEncoder denotes a wav2vec model, denotes an input audio segment a i denotes an acoustic feature obtained after passing through the wav2vec model, l A denotes a length of the audio segment a i denotes an acoustic feature dimension of the audio segment a A .

[0079] For video data, a Faster RCNN is used to extract visual features of the video data.

[0080] where Faster RCNN is a typical representative of a two-stage target detection model, as shown below:

[0081]

[0082] where VisualEncoder denotes a Faster RCNN model, denotes an input video segment v i denotes a visual feature obtained after passing through the Faster RCNN model, l V denotes a length of the video segment v i denotes a visual feature dimension of the audio segment a V .

[0083] After obtaining the text feature, acoustic feature and visual feature, the features need to be aligned. In the embodiment of the present application, a time alignment module is used to align the text feature, acoustic feature and visual feature. The time alignment module is composed of an LSTM network and a Softmax function. The alignment operation is represented as follows:

[0084]

[0085] wherein, represents the aligned features of the text modal data, audio modal data and video modal data respectively. It can be seen that the alignment operation is actually realized by taking the text as a reference benchmark to align the audio and video.

[0086] After alignment, the text feature is connected with the acoustic feature and the visual feature respectively to obtain acoustic enhanced features and visual enhanced features:

[0087]

[0088] wherein, and represent the acoustic enhanced features and visual enhanced features respectively, ReLU represents an activation function, f * represents a linear layer, and || represents connection.

[0089] Then, the acoustic enhanced features and visual enhanced features are fused to obtain a non-text feature h i , as shown below:

[0090]

[0091] wherein l represents the length of the non-text feature h i , and f * represents a linear layer. Finally, the fusion weight between the text feature and the non-text feature is calculated to fuse the text feature and the non-text feature to obtain a multi-modal feature:

[0092]

[0093] wherein, || ||2 represents an L2 norm, ε is a hyperparameter, and f represents a normalization block containing a dropout layer.

[0094] In this way, the multi-modal feature of the user input data can be obtained, information loss or misunderstanding caused by single modal data can be avoided, and the accuracy of user intent recognition is improved. Moreover, in the embodiment of the present application, the unique way of obtaining the multi-modal feature can more accurately understand the context information and better capture the real intention of the user compared with the existing method of obtaining the multi-modal feature. After obtaining the multi-modal feature, the multi-modal feature needs to be sent to the emotion recognition module and the intent recognition module respectively to recognize the user emotion and the user intent.

[0095] In the embodiment of the present application, the emotion recognition module includes an attention module and an emotion recognition model. There are various implementation schemes for the emotion recognition model in the field, such as an emotion recognition model EF-LSTM, LF-LSTM, etc. based on LSTM, M-BERT, etc. based on BERT. In the embodiment of the present application, the EF-LSTM emotion recognition model is taken as an example to describe how to realize the intent-guided emotion recognition. Since the user intent recognition is not performed when the emotion recognition is performed for the first time, the emotion recognition needs to be divided into two stages, i.e. the first emotion recognition and the multiple emotion recognitions after the first emotion recognition.

[0096] Specifically, in step S2, the emotion recognition module realizes the intent-guided emotion recognition based on the multi-modal feature by using the attention mechanism, outputs the emotion recognition result of the user, and inputs the emotion category to the intent recognition module, including:

[0097] In the first iteration, the emotion recognition module performs emotion recognition according to the multi-modal feature to obtain the first emotion recognition result, and inputs the first emotion recognition result to the intent recognition module. At this time, since the user intent recognition is not performed, the multi-modal feature is directly taken as the input of the EF-LSTM emotion recognition model, and the output of the EF-LSTM emotion recognition model is the first emotion recognition result.

[0098] In the kth iteration, the emotion recognition module realizes the intent-guided emotion recognition by using the attention mechanism according to the multi-modal feature and the k-1th user intent recognition result, obtains the kth user emotion recognition result, and inputs the kth emotion recognition result to the intent recognition module, where 2≤k≤K, and K is the preset iteration threshold. At this time, in order to realize the intent-guided emotion recognition, the fusion of the multi-modal feature and the k-1th intent recognition result is realized by using the attention mechanism, the fusion result is taken as the input of the EF-LSTM emotion recognition model, and then the kth emotion recognition result is obtained.

[0099] The emotion recognition guided by the intention by using the attention mechanism includes: the attention module receives the multi-modal feature and the user intention recognition result as input, obtains an attention weight of each element in the multi-modal feature according to the relevance of each element in the multi-modal feature and the user intention recognition result, inputs the multi-modal feature and the attention weight into the emotion recognition model, and obtains the user emotion recognition result for the kth time. The detailed process is as follows:

[0100] The multi-modal feature is represented as Wherein z n The multi-modal feature is represented as Contains the n th element. Then, the attention weight calculation is performed: w i = Attention (z n , I), wherein Attention () is used to calculate the relevance between the element z n And the intention recognition result I, and outputs a weight. Here, Attention () can be realized by using, for example, cosine similarity. Through the above calculation, the attention weight vector W = [w1, w2,..., w n ] can be obtained. Therefore, the weighted multi-modal feature is represented as Finally, the weighted multi-modal feature z' is input into the EF-LSTM emotion recognition model, so as to obtain the emotion recognition result guided by the intention.

[0101] As can be seen from the above, in the embodiment of the application, the importance of each element in the multi-modal feature is adjusted by the attention mechanism, so as to pay more attention to the features related to the user intention, make full use of the emotional information embodied by the user intention, and improve the accuracy of emotion recognition.

[0102] In the embodiment of the application, the intention recognition module includes an intention recognition model based on a Transformer encoder. The user intention recognition realized by using the Transformer encoder is a known technology in the art, and will not be described here.

[0103] In step S2, the intention recognition module realizes emotion-driven intention recognition based on the multi-modal feature and the emotion recognition result, and inputs the user intention recognition result into the emotion recognition module, including:

[0104] In the t th iteration, the intention recognition model inputs the multi-modal feature and the emotion recognition result for the t th time as input, obtains the user intention recognition result for the t th time, and 1≤t≤K. The detailed process is described as follows:

[0105] First, the multi-modal feature​ and the emotion recognition result e t The fusion can be implemented by splicing or weighted fusion to form a unified input representation The self-attention mechanism in the Transformer encoder and the feed-forward neural network are used for calculation: let the input of the lth layer be V t (l-1) For the first layer, V t (D) =X t The output of the lth layer is:

[0106] V t (l) =FFN(V t (l-1) +Attention(V t (l-1) ))

[0107] Where Attention represents the self-attention mechanism, and FFN represents the feed-forward neural network. V t (L) is mapped to the label space of user intent recognition through a fully connected layer to obtain the predicted user intent y t =W O V t (L) Where W O is a learnable weight matrix, L is the number of encoder layers, and V t (L) is the final output of the encoder.

[0108] As can be seen from this process, the emotion recognition result e t will affect the calculation of the attention score, thereby affecting the generation of the final output representation of the encoder. For example, if the emotion recognition result indicates that the user is currently in an angry state, the model may pay more attention to features related to anger when processing the multi-modal data. In this way, the model can better capture the emotional component in the user intent, thereby improving the accuracy of user intent recognition.

[0109] In the embodiments of the present application, user intent recognition and user emotion recognition are interactive guidance, therefore, the stopping iteration problem needs to be considered jointly. On the one hand, a maximum number of iterations K is set for the user intent recognition model, by limiting the number of iterations, the overfitting phenomenon of the model is avoided and the stability of the model is ensured. On the other hand, during the iteration process, the loss function value after each iteration is calculated. If the difference between the results of two consecutive iterations is very small, less than a preset threshold, it can be considered that the model has converged, that is, the change of the loss function value tends to be stable, at which time the iteration can be stopped. The iteration process of the intent recognition module is as follows:

[0110] when t < K, and the change of the joint loss function is not less than a preset threshold, inputting the tth user intention recognition result to the emotion recognition module;

[0111] when t = K, or when the change of the joint loss function is less than a preset threshold, taking the tth user intention recognition result as a final user intention recognition result, and taking the tth user emotion recognition result as a final user emotion recognition result.

[0112] wherein the joint loss function is a loss function of the emotion recognition module and the intention recognition module. As follows:

[0113]

[0114] wherein y t is a sample category of intention recognition, is a predicted category of intention recognition, e t is a sample category of emotion recognition, is a predicted category of emotion recognition, and t is an iteration number.

[0115] In the embodiment of the present application, after obtaining the final user intention recognition result and the user emotion recognition result, the response module of the intelligent customer service determines a response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feeds back the response content to the user according to the determined response strategy. Specifically, it includes:

[0116] The response module determines a response strategy based on the final user emotion recognition result, and determines a response content based on the final user intention recognition result, and feeds back the response content to the user according to the response strategy. For example, when the user is identified to be in different emotions such as anger and positivity, the response content corresponding to the user's intention is presented to the user. Alternatively, different processing methods are selected according to different degrees of user emotion, such as directly transferring to a manual customer service. The related content is all known calculations in the art, and will not be repeated here.

[0117] According to another embodiment of the present application, the present application also provides an intelligent customer service system based on user intention recognition, which comprises:

[0118] An input module for receiving multi-modal dialogue data of a user, wherein the multi-modal dialogue data comprises text data;

[0119] The user intention recognition module is configured to analyze the multi-modal dialogue data based on a user intention recognition model to obtain a user intention recognition result, wherein the user intention recognition model comprises a shared bottom layer module, an emotion recognition module, and an intention recognition module.

[0120] The shared bottom layer module is configured to analyze the multi-modal dialogue data to obtain multi-modal features, and input the multi-modal features into the emotion recognition module and the intention recognition module, respectively.

[0121] The emotion recognition module is configured to implement intention-guided emotion recognition by using an attention mechanism based on the multi-modal features, output a user emotion recognition result, and input the user emotion recognition result into the intention recognition module.

[0122] The intention recognition module is configured to implement emotion-driven intention recognition based on the multi-modal features and the user emotion recognition result, and input the user intention recognition result into the emotion recognition module.

[0123] The emotion recognition module and the intention recognition module are configured to iterate multiple times, and when the number of iterations reaches a preset iteration threshold, or when a change in a joint loss function of the emotion recognition module and the intention recognition module is less than a preset threshold, the user intention recognition result currently output by the intention recognition module is taken as a final user intention recognition result, and the user emotion recognition result currently output by the emotion recognition module is taken as a final user emotion recognition result.

[0124] The response module is configured to determine a response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feed back the response content to the user according to the determined response strategy.

[0125] According to another embodiment of the present application, the present application further provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the intelligent customer service method based on user intention recognition.

[0126] Note that the computer readable medium described above can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer readable signal medium can include a computer readable program code propagated on or through a computer readable medium, in baseband or as part of a carrier wave. The computer readable signal medium can take a variety of forms, including but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. The computer readable signal medium can be any computer readable medium that can be used to carry or store computer readable program code for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted or received over any suitable medium, including but not limited to, wire, cable, fiber optic, RF (radio frequency), or any suitable combination of the foregoing.

[0127] The computer readable medium described above can be included in the electronic device described above; alternatively, the computer readable medium can exist as a separate entity in which the electronic device is enclosed.

[0128] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0129] The computer program product of the present disclosure can be a computer program embodied on a non-transitory computer readable medium. When the program runs on a computer, the flowchart and / or block diagram in the flowchart and / or block diagram can be implemented.

[0130] The units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0131] The above describes the preferred embodiments of the present disclosure, which aims to make the spirit of the present disclosure clearer and easier to understand, and is not intended to limit the present disclosure. Any modification, replacement, improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the appended claims of the present disclosure.

Claims

1. An intelligent customer service method based on user intent recognition, characterized in that, Includes the following steps: S1: Receive multimodal dialogue data from the user, the multimodal dialogue data including text data; S2: Analyze the multimodal dialogue data based on the user intent recognition model to obtain the user intent recognition result; the user intent recognition model includes a shared underlying module, an emotion recognition module, and an intent recognition module; The shared underlying module is used to analyze the multimodal dialogue data to obtain multimodal features, and input the multimodal features into the emotion recognition module and the intent recognition module respectively. The emotion recognition module includes an attention module and an emotion recognition model; based on the multimodal features, the emotion recognition module uses an attention mechanism to achieve intention-guided emotion recognition, outputs the user's emotion recognition result, and inputs the emotion recognition result into the intention recognition module, including: In the first iteration, the emotion recognition module performs emotion recognition based on the multimodal features to obtain the first emotion recognition result, and inputs the first emotion recognition result into the intention recognition module; In the k-th iteration, the emotion recognition module uses the attention mechanism to achieve intention-guided emotion recognition based on the multimodal features and the user intention recognition result of the (k-1)-th iteration, and obtains the user emotion recognition result of the k-th iteration. The k-th emotion recognition result is then input to the intention recognition module, where 2≤k≤K, and K is a preset iteration number threshold. The aforementioned intention-guided emotion recognition using attention mechanisms includes: The attention module receives the multimodal features and the user intent recognition result as input, obtains the attention weight of each element based on the correlation between each element in the multimodal features and the user intent recognition result, and inputs the multimodal features and the attention weight into the emotion recognition model to obtain the k-th user emotion recognition result. The intent recognition module realizes emotion-driven intent recognition based on the multimodal features and the emotion recognition result, and inputs the user intent recognition result into the emotion recognition module; The emotion recognition module and the intent recognition module perform multiple iterations. When the number of iterations reaches a preset iteration threshold, or when the change of the joint loss function of the emotion recognition module and the intent recognition module is less than a preset threshold, the user intent recognition result currently output by the intent recognition module is taken as the final user intent recognition result, and the user emotion recognition result currently output by the emotion recognition module is taken as the final user emotion recognition result. S3: The intelligent customer service response module determines the response strategy and response content based on the final user intent recognition result and the final user emotion recognition result, and feeds back the response content to the user according to the determined response strategy.

2. The intelligent customer service method based on user intent recognition according to claim 1, characterized in that, In step S1, the multimodal dialogue data further includes audio data and video data; in step S2, the shared underlying module is used to analyze the multimodal dialogue data to obtain multimodal features, and input the multimodal features into the emotion recognition module and the intention recognition module respectively, including: Using the Bert model as the text encoder to extract the text features of the text data; Using the wav2vec model as the acoustic encoder to extract the acoustic features of the audio data; Using the Faster RCNN model as the visual encoder to extract the visual features of the video data; Using the temporal alignment module to align the text features, the acoustic features and the visual features, and the temporal alignment module is composed of an LSTM module and a Softmax function; Connecting the aligned text features and the acoustic features to obtain enhanced acoustic features; Connecting the aligned text features and the visual features to obtain enhanced visual features; Fusing the enhanced acoustic features and the enhanced visual features to obtain non-text features; Calculating the fusion weight between the text features and the non-text features, and obtaining the multimodal features based on the fusion weight; Sending the multimodal features to the emotion recognition module and the intention recognition module.

3. The intelligent customer service method based on user intent recognition according to claim 2, characterized in that, The intention recognition module includes an intention recognition model implemented based on a Transformer encoder; in step S2, the intention recognition module realizes emotion-driven intention recognition based on the multimodal features and the emotion recognition result, and inputs the user intention recognition result into the emotion recognition module, including: At the t-th iteration, the intention recognition model takes the multimodal features and the t-th emotion recognition result as inputs to obtain the t-th user intention recognition result, where 1 ≤ t ≤ K; When t < K and the change of the joint loss function is not less than a preset threshold, input the t-th user intention recognition result into the emotion recognition module; When t = K, or when the change of the joint loss function is less than the preset threshold, take the t-th user intention recognition result as the final user intention recognition result, and take the t-th user emotion recognition result as the final user emotion recognition result.

4. The intelligent customer service method based on user intent recognition according to claim 1, characterized in that, In step S3, the response module of the intelligent customer service determines the response strategy and response content based on the final user intention recognition result and the final user emotion recognition result, and feeds back the response content to the user according to the determined response strategy, including: The response module determines the response strategy based on the final user emotion recognition result, determines the response content based on the final user intention recognition result, and feeds back the response content to the user according to the response strategy.

5. An intelligent customer service system based on user intent recognition, characterized in that, The system includes: An input module for receiving the multimodal dialogue data of the user, and the multimodal dialogue data includes text data; The user intent recognition module is used to analyze the multimodal dialogue data based on the user intent recognition model to obtain the user intent recognition result; the user intent recognition model includes a shared underlying module, an emotion recognition module, and an intent recognition module; The shared underlying module is used to analyze the multimodal dialogue data to obtain multimodal features, and input the multimodal features into the emotion recognition module and the intent recognition module respectively. The emotion recognition module includes an attention module and an emotion recognition model; based on the multimodal features, the emotion recognition module uses an attention mechanism to achieve intention-guided emotion recognition, outputs the user's emotion recognition result, and inputs the emotion recognition result into the intention recognition module, including: In the first iteration, the emotion recognition module performs emotion recognition based on the multimodal features to obtain the first emotion recognition result, and inputs the first emotion recognition result into the intention recognition module; In the k-th iteration, the emotion recognition module uses the attention mechanism to achieve intention-guided emotion recognition based on the multimodal features and the user intention recognition result of the (k-1)-th iteration, and obtains the user emotion recognition result of the k-th iteration. The k-th emotion recognition result is then input to the intention recognition module, where 2≤k≤K, and K is a preset iteration number threshold. The aforementioned intention-guided emotion recognition using attention mechanisms includes: The attention module receives the multimodal features and the user intent recognition result as input, obtains the attention weight of each element based on the correlation between each element in the multimodal features and the user intent recognition result, and inputs the multimodal features and the attention weight into the emotion recognition model to obtain the k-th user emotion recognition result. The intent recognition module realizes emotion-driven intent recognition based on the multimodal features and the emotion recognition result, and inputs the user intent recognition result into the emotion recognition module; The emotion recognition module and the intent recognition module perform multiple iterations. When the number of iterations reaches a preset iteration threshold, or when the change of the joint loss function of the emotion recognition module and the intent recognition module is less than a preset threshold, the user intent recognition result currently output by the intent recognition module is taken as the final user intent recognition result, and the user emotion recognition result currently output by the emotion recognition module is taken as the final user emotion recognition result. The response module is used to determine the response strategy and response content based on the final user intent recognition result and the final user emotion recognition result, and to feed back the response content to the user according to the determined response strategy.

6. The intelligent customer service system based on user intent recognition according to claim 5, characterized in that, The multimodal dialogue data also includes audio data and video data; the shared underlying module is used to analyze the multimodal dialogue data to obtain multimodal features, and input the multimodal features into the emotion recognition module and the intent recognition module respectively, including: The BERT model is used as a text encoder to extract text features from the text data. Use the wav2vec model as the acoustic encoder to extract the acoustic features of the audio data; Use the Faster RCNN model as the visual encoder to extract the visual features of the video data; Use the temporal alignment module to align the text features, the acoustic features and the visual features. The temporal alignment module consists of an LSTM module and a Softmax function; Connect the aligned text features and the acoustic features to obtain enhanced acoustic features; Connect the aligned text features and the visual features to obtain enhanced visual features; Fuse the enhanced acoustic features and the enhanced visual features to obtain non-text features; Calculate the fusion weight between the text features and the non-text features, and obtain the multimodal features based on the fusion weight; Send the multimodal features to the emotion recognition module and the intention recognition module.

7. The intelligent customer service system based on user intent recognition according to claim 6, characterized in that, The intention recognition module includes an intention recognition model implemented based on a Transformer encoder; the intention recognition module realizes emotion-driven intention recognition based on the multimodal features and the emotion recognition result, and inputs the user intention recognition result into the emotion recognition module, including: At the t-th iteration, the intention recognition model takes the multimodal features and the t-th emotion recognition result as inputs to obtain the t-th user intention recognition result, where 1 ≤ t ≤ K; When t < K and the change in the joint loss function is not less than a preset threshold, input the t-th user intention recognition result into the emotion recognition module; When t = K, or when the change in the joint loss function is less than the preset threshold, take the t-th user intention recognition result as the final user intention recognition result, and take the t-th user emotion recognition result as the final user emotion recognition result.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements an intelligent customer service method based on user intention recognition according to any one of claims 1 to 4.

Images