Multi-dimensional customer emotion recognition method and device and medium

Abstract

The invention discloses a multi-dimensional customer emotion recognition method and device and a medium, and the method comprises the steps: obtaining real-time emotion data, and carrying out the principal component analysis of the real-time emotion data, so as to obtain to-be-recognized emotion parameter data; performing emotional feature extraction on the to-be-recognized emotional parameter data to determine emotional feature data; performing emotional parameter feature connection on the emotional feature data, and inputting the emotional feature data after the emotional parameter feature connection into a preset emotional classification module to obtain first emotional category data; based on the first emotion category data, determining second emotion category data through emotion relevancy judgment; and according to the second emotion category data, determining emotion change data of the bank customer through real-time emotion change judgment. According to the method, the technical problems of poor recognition capability, low emotion analysis accuracy and simplification of emotion analysis on emotion expressions in other forms such as voice and face images of clients in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent emotion recognition technology, and in particular to a multi-dimensional customer emotion recognition method, device and medium. Background Art

[0002] In the process of serving customers, positive emotions of customers represent their trust and satisfaction, while negative emotions may indicate that customers will churn or complain. Therefore, it is crucial for service providers to understand customers' emotional state in real time. Take the service provided by bank employees to bank customers as an example. When communicating with customers, bank employees mainly rely on experience and intuition to judge customers' emotional state and provide corresponding services. However, the subjective evaluation of customer emotions by bank employees varies from person to person, and is not conducive to comprehensive analysis of bank customers' emotions.

[0003] With the rapid development of sentiment analysis technology, the analysis of customer emotions is realized by automatically identifying and analyzing the emotions of customer status. Existing intelligent sentiment analysis mainly uses text data for analysis and recognition, and its recognition ability for other forms of emotional expressions such as customer voice and facial images is relatively weak. In terms of identifying customer emotions, it has limited semantic understanding ability and low accuracy of sentiment analysis. In addition, the single emotion recognition model cannot meet the personalized needs of customers. Summary of the invention

[0004] The embodiments of the present application provide a multi-dimensional customer emotion recognition method, device and medium, which solve the technical problems of the prior art in that the recognition ability of other forms of emotional expressions such as customer voice and facial images is poor, the accuracy of emotion analysis is low, and the emotion analysis is single.

[0005] In a first aspect, an embodiment of the present application provides a multi-dimensional customer emotion recognition method, characterized in that the method includes: acquiring real-time emotion data, and performing principal component analysis on the real-time emotion data to obtain emotion parameter data to be recognized; wherein the emotion parameter data to be recognized includes: voice data frame, global face image grayscale data, micro-expression image grayscale data; performing emotion feature extraction on the emotion parameter data to be recognized to determine the emotion feature data; performing emotion parameter feature connection on the emotion feature data, and inputting the emotion feature data after the emotion parameter feature connection into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; based on the first emotion category data, determining the second emotion category data through emotion relevance judgment; according to the second emotion category data, determining the emotion change data of the bank customer through real-time emotion change judgment.

[0006] In one implementation of the present application, principal component analysis is performed on real-time emotional data to obtain emotional parameter data to be identified, specifically including: data classification of real-time emotional data to obtain real-time emotional data to be processed; wherein the real-time emotional data to be processed includes: voice data, global facial image data; the voice data collection amount and collection times of the voice data are obtained to obtain a voice data frame; based on the global facial image data, the global facial image grayscale data is determined by a grayscale conversion method; based on the facial image grayscale data, micro-expression image grayscale data is obtained by micro-expression positioning and cropping.

[0007] In one implementation of the present application, emotion feature extraction is performed on emotion parameter data to be identified to determine emotion feature data, specifically including: performing one-dimensional convolution on a speech data frame to obtain speech feature data; performing two-dimensional convolution on global facial image grayscale data to obtain global facial image feature data; performing two-dimensional convolution on micro-expression image grayscale data to obtain micro-expression image feature data; and determining the emotion feature data based on the speech feature data, global facial image feature data and micro-expression image feature data.

[0008] In one implementation of the present application, emotion feature data is connected with emotion parameter features, specifically including: based on the emotion feature data, obtaining the emotion feature vector to be connected through a fully connected layer operation; and integrating and splicing the emotion feature vector to be connected.

[0009] In one implementation of the present application, based on the first emotion category data, the second emotion category data is determined through emotion relevance determination, specifically including: performing a speech relevance mapping determination on the first emotion category data to determine a first relevance of the speech mapping emotion category to the expression mapping emotion category; performing a face relevance mapping determination on the first emotion category data to determine a second relevance of the expression mapping emotion category to the speech mapping emotion category; when the correlation difference between the first correlation and the second correlation is less than a preset correlation threshold, determining the first emotion category data as second emotion category data; when the correlation difference between the first correlation and the second correlation is greater than a preset correlation threshold, selecting the emotion category corresponding to the correlation with a larger correlation value between the first correlation and the second correlation as the second emotion category data.

[0010] In one implementation of the present application, based on the second emotion category data, the emotion change data of the bank customer is determined through real-time emotion change judgment, which specifically includes: coordinate processing of the second emotion category data to obtain a real-time emotion change coordinate graph; based on the real-time emotion change coordinate graph, the emotion change data is determined through emotion conversion node judgment; wherein the emotion change data includes: emotion change time node, emotion change trend.

[0011] In one implementation of the present application, after determining the bank customer's emotion change data through real-time emotion change judgment based on the second emotion category data, the method also includes: determining the bank customer's emotion fluctuation data through emotion parameterization processing based on the emotion change data; inputting the emotion fluctuation data into a preset emotion processing module to determine a service response strategy; and tracking the emotional state of the implementation process of the service response strategy to obtain the accuracy of the service response strategy.

[0012] In one implementation of the present application, after tracking the emotional state of the implementation process of the service response strategy to obtain the accuracy of the service response strategy, the method also includes: based on the accuracy of the service response strategy, iterating the accuracy of the service response strategy to obtain an updated service strategy; and performing emotional service simulation on the updated service strategy to determine the directional service strategy.

[0013] In a second aspect, an embodiment of the present application also provides a multi-dimensional bank customer emotion recognition device, characterized in that the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: obtain real-time emotion data, and perform principal component analysis on the real-time emotion data to obtain emotion parameter data to be identified; wherein the emotion parameter data to be identified includes: voice data frames, global face image grayscale data, micro-expression image grayscale data; perform emotion feature extraction on the emotion parameter data to be identified to determine the emotion feature data; perform emotion parameter feature connection on the emotion feature data, and input the emotion feature data after the emotion parameter feature connection into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; based on the first emotion category data, determine the second emotion category data through emotion relevance judgment; according to the second emotion category data, determine the emotion change data of the bank customer through real-time emotion change judgment.

[0014] In a third aspect, an embodiment of the present application further provides a non-volatile computer storage medium for multi-dimensional bank customer emotion recognition, storing computer executable instructions, characterized in that the computer executable instructions are configured to: obtain real-time emotion data, and perform principal component analysis on the real-time emotion data to obtain emotion parameter data to be recognized; wherein the emotion parameter data to be recognized includes: voice data frames, global face image grayscale data, micro-expression image grayscale data; perform emotion feature extraction on the emotion parameter data to be recognized to determine the emotion feature data; perform emotion parameter feature connection on the emotion feature data, and input the emotion feature data after the emotion parameter feature connection into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; based on the first emotion category data, determine the second emotion category data through emotion relevance judgment; according to the second emotion category data, determine the emotion change data of the bank customer through real-time emotion change judgment.

[0015] The embodiments of the present application provide a multi-dimensional customer emotion recognition method, device and medium. By acquiring the customer's facial and voice parameters in real time and performing a comprehensive analysis of the voice and facial parameters, the technical problems of the prior art in terms of poor recognition ability of other forms of emotional expressions such as customer voice and facial images, low accuracy of emotion analysis and single-minded emotion analysis are solved. The corresponding relationship between the analysis of customer voice and facial images and emotions is refined, the recognition ability and accuracy of emotional expressions are improved, and the single-minded emotion analysis is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flow chart of a multi-dimensional customer emotion recognition method provided in an embodiment of the present application;

[0018] Figure 2 A multi-dimensional customer emotion recognition preprocessing flow chart provided in an embodiment of the present application;

[0019] Figure 3 A multi-dimensional customer emotion classification flow chart provided for an embodiment of the present application;

[0020] Figure 4 A schematic diagram of the internal structure of a multi-dimensional customer emotion recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] The embodiments of the present application provide a multi-dimensional customer emotion recognition method, device and medium. By acquiring the customer's facial and voice parameters in real time and performing a comprehensive analysis of the voice and facial parameters, the technical problems of the prior art in terms of poor recognition ability of other forms of emotional expressions such as customer voice and facial images, low accuracy of emotion analysis and single-minded emotion analysis are solved. The corresponding relationship between the analysis of customer voice and facial images and emotions is refined, the recognition ability and accuracy of emotional expressions are improved, and the single-minded emotion analysis is avoided.

[0023] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A flowchart of a multi-dimensional customer emotion recognition method provided in an embodiment of the present application. Figure 1 As shown, a multi-dimensional customer emotion recognition method provided by an embodiment of the present application specifically includes the following steps:

[0025] Step 101: Acquire real-time emotion data, and perform principal component analysis on the real-time emotion data to obtain emotion parameter data to be identified.

[0026] Among them, the emotion parameter data to be recognized include: voice data frame, global face image grayscale data, and micro-expression image grayscale data.

[0027] Specifically, the method includes: classifying the real-time emotion data to obtain the real-time emotion data to be processed; wherein the real-time emotion data to be processed includes: voice data and global face image data; obtaining the voice data collection amount and collection times of the voice data to obtain the voice data frame; based on the global face image data, determining the global face image grayscale data through the grayscale conversion method; according to the face image grayscale data, obtaining the micro-expression image grayscale data through micro-expression positioning and cropping.

[0028] This application obtains the customer's real-time emotional data and performs principal component analysis on the real-time emotional data to obtain the emotional parameter data to be identified, thereby achieving accurate processing of the customer's real-time emotional data, providing a data basis for the customer's real-time emotional analysis, and making different real-time emotions more conducive to data splicing.

[0029] Figure 2A multi-dimensional customer emotion recognition preprocessing flowchart is provided in an embodiment of the present application. In the embodiment of the present application, it is explained in detail through the following Example 1.

[0030] Example 1: First, the customer's voice data, global face image data and local micro-expression image data are collected through the bank's emotion monitoring device or the voice microphone, smart camera and high-speed camera encapsulated inside the service robot.

[0031] Matrix Represents the voice data frame of a bank customer, in the form of a matrix Represents the customer's global face image data, matrix Represents the customer's local micro-expression image data.

[0032] Among them, N1 is the amount of voice data collected from each customer;

[0033] N2 is the grayscale value of the global facial image data of each customer;

[0034] N3 is the grayscale value of each customer’s partial face image data;

[0035] C is the number of data collection times for each customer.

[0036] N1 is set to 128 frames, N2 and N3 are both set to 48, and the collected speech signal data and face image data are subjected to data denoising and preprocessing using the principal component analysis (PCA) algorithm.

[0037] Step 102: extract emotion features from the emotion parameter data to be identified to determine emotion feature data.

[0038] Specifically, the method includes: performing one-dimensional convolution on the speech data frame to obtain speech feature data; performing two-dimensional convolution on the global face image grayscale data to obtain global face image feature data; performing two-dimensional convolution on the micro-expression image grayscale data to obtain micro-expression image feature data; and determining the emotion feature data based on the speech feature data, the global face image feature data and the micro-expression image feature data.

[0039] The present application extracts emotion features from emotion parameter data to be identified to determine emotion feature data, thereby realizing feature extraction of emotion data described by the customer, determining the convolution dimensions of speech data frames, global facial image grayscale data, and micro-expression image grayscale data, and improving the splicing efficiency of feature vectors.

[0040] Step 103, connect the emotion feature data with the emotion parameter features, and input the emotion feature data after the emotion parameter features are connected into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category.

[0041] Specifically, it includes: based on the emotion feature data, obtaining the emotion feature vector to be connected through the full connection layer operation; and integrating and splicing the emotion feature vector to be connected.

[0042] Figure 3 A multi-dimensional customer emotion classification flowchart is provided for an embodiment of the present application. In the embodiment of the present application, it is explained in detail through the following Example 2.

[0043] Example 2: The feature data after emotional feature extraction are: voice feature data 32*32, global face image feature data 128*8 and micro-expression image feature data 128*8.

[0044] First, the obtained speech features, global face image features and local micro-expression image features are aligned using a fully connected function to ensure that the obtained feature vectors have the same size, which is 1024.

[0045] Then, the Connect operation is used to integrate and concatenate the features to construct the customer sentiment feature with a size of 3072.

[0046] Finally, a linear mapping function is used to map the customer's emotional features into a binary emotional space, and the softmax classifier is used to perform emotional classification to obtain speech mapping emotional categories and expression mapping emotional categories.

[0047] The speech mapping emotion category represents the emotion trend corresponding to the bank customer speech data. Similarly, the expression mapping emotion category represents the emotion trend corresponding to the global facial image feature data and the micro-expression image feature data. Among them, the trend refers to the degree of emotional tendency represented by the feature data. The higher the correlation, the stronger the trend.

[0048] It should be noted that the size of the feature vector can be adjusted accordingly according to the actual usage, but the sizes of the feature vectors need to be the same.

[0049] Step 104: Determine second emotion category data based on the first emotion category data by determining emotion relevance.

[0050] Specifically, it includes: performing a speech relevance mapping judgment on the first emotion category data to determine the first relevance of the speech mapping emotion category to the expression mapping emotion category; performing a face relevance mapping judgment on the first emotion category data to determine the second relevance of the expression mapping emotion category to the speech mapping emotion category; when the correlation difference between the first correlation and the second correlation is less than a preset correlation threshold, the first emotion category data is judged as the second emotion category data; when the correlation difference between the first correlation and the second correlation is greater than the preset correlation threshold, the emotion category corresponding to the correlation with the larger correlation value between the first correlation and the second correlation is selected as the second emotion category data.

[0051] This application determines the second emotion category data based on the first emotion category data through emotion relevance judgment, realizes the relevance detection of bank customer emotion analysis, improves the mapping accuracy of emotion category data and the collected real-time emotion data, and improves the accuracy of customer emotion recognition.

[0052] In the examples of the present application, this is explained in detail through the following Example 3.

[0053] Example 3: Because people's facial expressions are very complex and there are so many customers in a bank, it is inevitable that facial expressions or voices will change very little with emotions.

[0054] First, the speech relevance mapping is determined for the first emotion category data, and the correlation between the source global facial image data and the source micro-expression image data to be analyzed and the speech data is judged respectively to determine the correlation between the emotional trend corresponding to the source global facial image data and the emotional trend corresponding to the speech data.

[0055] For the source micro-expression image data, it is necessary to determine the correlation between the emotional trend of the source micro-expression image data and the emotional trend corresponding to the voice data, that is, the first correlation.

[0056] Similarly, the face relevance mapping judgment is performed on the first emotion category data, and the emotional trend corresponding to the source voice data is judged separately with the emotional trend corresponding to the global face image data and the emotional trend corresponding to the source micro-expression image data, so as to determine the second relevance of the expression mapping emotion category to the voice mapping emotion category.

[0057] When the correlation difference between the first correlation and the second correlation is less than a preset correlation threshold, determining the first emotion category data as the second emotion category data;

[0058] When the correlation difference between the first correlation and the second correlation is greater than a preset correlation threshold, the emotion category corresponding to the correlation with a larger correlation value between the first correlation and the second correlation is selected as the second emotion category data.

[0059] Step 105: Determine the bank customer's emotion change data based on the second emotion category data by judging the emotion change in real time.

[0060] Specifically, it includes: coordinate processing of the second emotion category data to obtain a real-time emotion change coordinate graph; based on the real-time emotion change coordinate graph, determining the emotion change data through emotion conversion node judgment; wherein the emotion change data includes: emotion change time node, emotion change trend.

[0061] After determining the bank customer's emotion change data through real-time emotion change judgment based on the second emotion category data, the method also includes: determining the bank customer's emotion fluctuation data through emotion parameterization processing based on the emotion change data; inputting the emotion fluctuation data into a preset emotion processing module to determine a service response strategy; and tracking the emotion state of the implementation process of the service response strategy to obtain the accuracy of the service response strategy.

[0062] This application realizes real-time analysis of the customer's emotional state by analyzing the emotional changes of the customer's real-time emotional state, thereby improving the efficiency of formulating service response strategies.

[0063] After tracking the emotional state of the implementation process of the service response strategy to obtain the correctness of the service response strategy, the method also includes: based on the correctness of the service response strategy, iterating the correctness of the service response strategy to obtain an updated service strategy; and performing emotional service simulation on the updated service strategy to determine the directional service strategy.

[0064] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides a multi-dimensional customer emotion recognition device, whose structure is as follows: Figure 4 shown.

[0065] Figure 4 A schematic diagram of the internal structure of a multi-dimensional customer emotion recognition device provided in an embodiment of the present application. Figure 4 As shown, the device includes:

[0066] at least one processor 401;

[0067] and, a memory 402 communicatively coupled to the at least one processor;

[0068] The memory 402 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 401 to enable at least one processor 401 to:

[0069] Real-time emotion data is acquired, and principal component analysis is performed on the real-time emotion data to obtain emotion parameter data to be identified; wherein the emotion parameter data to be identified includes: speech data frame, global face image grayscale data, micro-expression image grayscale data; emotion feature extraction is performed on the emotion parameter data to be identified to determine emotion feature data; emotion feature data is connected with emotion parameter features, and the emotion feature data after the emotion parameter feature connection is input into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; based on the first emotion category data, second emotion category data is determined by emotion relevance judgment; according to the second emotion category data, emotion change data of bank customers is determined by real-time emotion change judgment.

[0070] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium for multi-dimensional customer emotion recognition stores computer executable instructions, wherein the computer executable instructions are configured to: obtain real-time emotion data, and perform principal component analysis on the real-time emotion data to obtain emotion parameter data to be recognized; wherein the emotion parameter data to be recognized includes: voice data frame, global face image grayscale data, micro-expression image grayscale data; perform emotion feature extraction on the emotion parameter data to be recognized to determine emotion feature data; perform emotion parameter feature connection on the emotion feature data, and input the emotion feature data after the emotion parameter feature connection into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; based on the first emotion category data, determine the second emotion category data through emotion relevance determination; according to the second emotion category data, determine the emotion change data of the bank customer through real-time emotion change determination.

[0071] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0072] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0073] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0074] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0078] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0079] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0080] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0081] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A multi-dimensional customer emotion recognition method, characterized in that: The method comprises: Acquire real-time emotion data, and perform principal component analysis on the real-time emotion data to obtain emotion parameter data to be identified; wherein the emotion parameter data to be identified includes: voice data frame, global face image grayscale data, micro-expression image grayscale data; Extracting emotion features from the emotion parameter data to be identified to determine emotion feature data; The emotion feature data is connected with emotion parameter features, and the emotion feature data after the emotion parameter features are connected is input into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; Based on the first emotion category data, determining second emotion category data by emotion relevance determination; According to the second emotion category data, the emotion change data of the bank customer is determined through real-time emotion change judgment.

2. A multi-dimensional customer emotion recognition method according to claim 1, characterized in that: The real-time emotion data is subjected to principal component analysis to obtain emotion parameter data to be identified, specifically including: Classifying the real-time emotion data to obtain real-time emotion data to be processed; wherein the real-time emotion data to be processed includes: voice data and global face image data; Acquire the voice data collection amount and collection times of the voice data to obtain the voice data frame; Based on the global face image data, determining the global face image grayscale data by a grayscale conversion method; According to the facial image grayscale data, the micro-expression image grayscale data is obtained by micro-expression positioning and cropping.

3. A multi-dimensional customer emotion recognition method according to claim 1, characterized in that: Extracting emotion features from the emotion parameter data to be identified to determine emotion feature data specifically includes: Performing one-dimensional convolution on the speech data frame to obtain speech feature data; Performing two-dimensional convolution on the global face image grayscale data to obtain global face image feature data; Performing two-dimensional convolution on the micro-expression image grayscale data to obtain micro-expression image feature data; The emotion feature data is determined based on the voice feature data, the global facial image feature data and the micro-expression image feature data.

4. The multi-dimensional customer emotion recognition method according to claim 1, characterized in that: The emotion characteristic data is connected with emotion parameter characteristics, specifically including: Based on the emotion feature data, a to-be-connected emotion feature vector is obtained through a fully connected layer operation; The emotion feature vectors to be connected are integrated and spliced.

5. The multi-dimensional customer emotion recognition method according to claim 1, characterized in that: Based on the first emotion category data, determining the second emotion category data through emotion relevance determination specifically includes: Performing a speech relevance mapping determination on the first emotion category data to determine a first relevance of the speech-mapped emotion category to the expression-mapped emotion category; Performing face relevance mapping determination on the first emotion category data to determine a second relevance of the expression mapping emotion category to the voice mapping emotion category; When the correlation difference between the first correlation and the second correlation is less than a preset correlation threshold, determining the first emotion category data as the second emotion category data; When the correlation difference between the first correlation and the second correlation is greater than a preset correlation threshold, the emotion category corresponding to the correlation with a larger correlation value between the first correlation and the second correlation is selected as the second emotion category data.

6. A multi-dimensional customer emotion recognition method according to claim 1, characterized in that: According to the second emotion category data, the emotion change data of the bank customer is determined by real-time emotion change determination, specifically including: Performing coordinate processing on the second emotion category data to obtain a real-time emotion change coordinate graph; Based on the real-time emotion change coordinate diagram, the emotion change data is determined through emotion conversion node judgment; wherein the emotion change data includes: emotion change time node and emotion change trend.

7. The multi-dimensional customer emotion recognition method according to claim 1, characterized in that: After determining the bank customer's emotion change data through real-time emotion change determination according to the second emotion category data, the method further includes: Based on the emotion change data, determining the emotion fluctuation data of the bank customer through emotion parameterization processing; Inputting the emotion fluctuation data into a preset emotion processing module to determine a service response strategy; The emotional state of the implementation process of the service response strategy is tracked to obtain the accuracy of the service response strategy.

8. A multi-dimensional customer emotion recognition method according to claim 7, characterized in that: After tracking the emotional state of the implementation process of the service response strategy to obtain the accuracy of the service response strategy, the method further includes: Based on the correctness of the service response strategy, iterating the correctness of the service response strategy to obtain an updated service strategy; The updated service strategy is subjected to emotional service simulation to determine a directional service strategy.

9. A multi-dimensional customer emotion recognition device, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: Acquire real-time emotion data, and perform principal component analysis on the real-time emotion data to obtain emotion parameter data to be identified; wherein the emotion parameter data to be identified includes: voice data frame, global face image grayscale data, micro-expression image grayscale data; Extracting emotion features from the emotion parameter data to be identified to determine emotion feature data; The emotion feature data is connected with emotion parameter features, and the emotion feature data after the emotion parameter features are connected is input into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; Based on the first emotion category data, determining second emotion category data by emotion relevance determination; According to the second emotion category data, the emotion change data of the bank customer is determined through real-time emotion change judgment.

10. A non-volatile computer storage medium for multi-dimensional customer emotion recognition, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Acquire real-time emotion data, and perform principal component analysis on the real-time emotion data to obtain emotion parameter data to be identified; wherein the emotion parameter data to be identified includes: voice data frame, global face image grayscale data, micro-expression image grayscale data; Extracting emotion features from the emotion parameter data to be identified to determine emotion feature data; The emotion feature data is connected with emotion parameter features, and the emotion feature data after the emotion parameter features are connected is input into a preset emotion classification module to obtain first emotion category data; wherein the first emotion category data includes: speech mapping emotion category, expression mapping emotion category; Based on the first emotion category data, determining second emotion category data by emotion relevance determination; According to the second emotion category data, the emotion change data of the bank customer is determined through real-time emotion change judgment.

Images