Seat person emotion recognition method and device, terminal equipment and storage medium

Abstract

The invention discloses a seat personnel emotion recognition method and device, terminal equipment and a storage medium, and belongs to the technical field of face recognition, and the method comprises the steps: obtaining original face video stream data and a real identity type of a to-be-recognized seat personnel, carrying out the convolution operation, and extracting a time domain attention feature and a space domain attention feature; inputting the original face video stream data and the real identity category into a preset identity discriminator, minimizing the difference between the distribution of the identity category under the expression features and the distribution of the real identity category, and separating the expression features; and finally, splicing the three obtained features and inputting the spliced features into a preset emotion recognition model to recognize the emotion of the to-be-recognized seat personnel. By implementing the method and the device, the problem that in the prior art, a face recognition model excessively pays attention to differences caused by identity features and cannot extract accurate expression features, so that the accuracy of a recognition result is reduced can be solved.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a method, device, terminal equipment and storage medium for identifying emotions of seat personnel. Background Art

[0002] Agents are employees working in call centers or customer service departments. Their primary responsibilities include answering and making outbound calls to handle services, sales, data collection, information research, and business follow-up. To improve employee efficiency and ensure service quality, companies often need to monitor agent status to assess their work attitude.

[0003] Most current facial expression recognition models rely on facial expression datasets constructed by a limited number of experimental participants making specific facial expressions. The differences in expression features caused by the influence of people's identity characteristics are greater than the differences between different expression features. In other words, the feature differences between different expressions of the same person are smaller than the differences between the same expression of different people. Therefore, there is a problem that the model may pay too much attention to these differences caused by identity characteristics during the recognition process and fail to extract accurate expression features, which in turn leads to reduced accuracy of emotion recognition results. Summary of the Invention

[0004] The present invention provides a method, apparatus, terminal device and storage medium for agent emotion recognition, which can solve the problem in the prior art that the model may pay too much attention to the differences caused by identity characteristics during the emotion recognition process and fail to extract accurate expression features, thereby reducing the accuracy of the emotion recognition results.

[0005] An embodiment of the present invention provides a method for identifying emotions of an agent, including:

[0006] Obtaining the original facial video stream data of the seat to be identified and the real identity category of the seat to be identified;

[0007] After performing convolution operations on the above original face video stream data, temporal domain attention features and spatial domain attention features are extracted;

[0008] Inputting the original face video stream data and the true identity category into a preset identity discriminator, so that the identity discriminator separates the expression features in the original face video stream data by minimizing the difference between the distribution of identity categories under the expression features and the distribution of true identity categories;

[0009] The above-mentioned time domain attention features, spatial domain attention features and expression features are spliced ​​together to obtain the emotional features of the above-mentioned seat to be identified, and the above-mentioned emotional features are input into the preset emotion recognition model so that the above-mentioned preset emotion recognition model can identify the emotions of the above-mentioned seat to be identified based on the above-mentioned emotional features.

[0010] Furthermore, after performing the convolution operation on the original face video stream data, the temporal domain attention features and the spatial domain attention features are extracted, including:

[0011] Calculate the pixel change value between two adjacent video frames in the original face video stream data;

[0012] For every two adjacent video frames, if the pixel change value is greater than a preset threshold, increase the weight of the second video frame; otherwise, decrease the weight of the second video frame;

[0013] Obtaining first face video stream data according to the weight-adjusted video frame;

[0014] Slicing the first facial video stream data according to a preset depth, preset width, and preset height to obtain a facial video frame sequence;

[0015] After performing three-dimensional convolution on the above-mentioned face video frame sequence, the time domain attention features of the above-mentioned original face video stream data are obtained;

[0016] After performing two-dimensional convolution on the above-mentioned face video frame sequence, the spatial domain attention features of the above-mentioned original face video stream data are obtained.

[0017] Furthermore, before slicing the first face video stream data according to a preset depth, a preset width, and a preset height, the method further includes:

[0018] Grayscale the first face video stream data.

[0019] Furthermore, after performing three-dimensional convolution on the above-mentioned facial video frame sequence, the time domain attention features of the above-mentioned original facial video stream data are obtained, including:

[0020] Performing three-dimensional convolution on all sliced ​​video frames in the face video frame sequence to obtain the first feature of each sliced ​​video frame;

[0021] Based on the above first feature, mean pooling is performed on each slice video frame to obtain a compressed second feature;

[0022] Fully connect the second features of all sliced ​​video frames to obtain a one-dimensional temporal feature matrix;

[0023] The above one-dimensional time feature matrix is ​​normalized to obtain the above time domain attention feature.

[0024] Furthermore, after performing two-dimensional convolution on the above-mentioned facial video frame sequence, the spatial domain attention features of the above-mentioned original facial video stream data are obtained, including:

[0025] Splitting the face video frame sequence at preset intervals to obtain a plurality of video frame sequences;

[0026] Pooling and convolution are performed on each video frame sequence in sequence to obtain the third feature;

[0027] Normalize all video frame sequences according to the third feature to obtain a one-dimensional spatial feature matrix;

[0028] Connect the above one-dimensional spatial feature matrices to obtain the above spatial domain attention features.

[0029] Furthermore, the training of the above-mentioned preset identity discriminator includes:

[0030] Obtaining a number of face video stream sample data with real labels; wherein the real labels are used to represent the real identity categories of the face video stream sample data;

[0031] Inputting the above-mentioned facial video stream sample data into the identity discriminator to be trained for iterative training until the difference between the current first distribution probability and the current second distribution probability is less than a preset probability threshold, thereby generating a trained preset identity discriminator; wherein the above-mentioned first distribution probability is: the identity category distribution probability under the current predicted expression feature; the above-mentioned second distribution probability is: the distribution probability of the above-mentioned true identity category;

[0032] Among them, during each iterative training, the predicted expression features and the predicted identity category features are predicted based on the current face video stream sample data; the current first distribution probability and the current second distribution probability are calculated based on the current predicted expression features, predicted identity category features and the above-mentioned true labels; it is judged whether the difference between the current first distribution probability and the current second distribution probability is less than the preset probability threshold; if it is less, the current identity discriminator is used as the trained preset identity discriminator; otherwise, after adjusting the parameters of the current identity discriminator, the current identity discriminator continues to be trained.

[0033] Furthermore, after identifying the emotions of the agent to be identified, the following steps are further included:

[0034] The emotion recognition result is drawn on the above-mentioned original face video stream data and sent to the supervisor, so that the supervisor can analyze the emotional state of the seat to be identified based on the emotion recognition result and the above-mentioned original face video stream data.

[0035] Based on the above method embodiment, the present invention provides a corresponding device embodiment;

[0036] The present invention provides a device for identifying emotions of seat personnel, comprising:

[0037] Data acquisition module, feature extraction module, expression feature separation module and emotion recognition module;

[0038] The data acquisition module is used to obtain the original face video stream data of the seat to be identified and the real identity category of the seat to be identified;

[0039] The feature extraction module is used to extract temporal and spatial attention features after performing a convolution operation on the original face video stream data.

[0040] The expression feature separation module is configured to input the original facial video stream data and the true identity category into a preset identity discriminator, so that the identity discriminator separates the expression features in the original facial video stream data by minimizing the difference between the distribution of identity categories under the expression features and the distribution of true identity categories;

[0041] The above-mentioned emotion recognition module is used to splice the above-mentioned time domain attention features, spatial domain attention features and expression features to obtain the emotional features of the above-mentioned seat to be identified, and input the above-mentioned emotional features into the preset emotion recognition model so that the above-mentioned preset emotion recognition model can identify the emotions of the above-mentioned seat to be identified based on the above-mentioned emotional features.

[0042] Based on the above method embodiment, the present invention provides a corresponding terminal device embodiment;

[0043] The present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-mentioned method for recognizing emotions of an agent according to any embodiment of the present invention.

[0044] Based on the above method embodiment, the present invention provides a storage medium embodiment;

[0045] The present invention provides a storage medium comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for recognizing emotions of an agent according to any embodiment of the present invention is implemented.

[0046] The embodiments of the present invention have the following beneficial effects:

[0047] The present invention provides a method, apparatus, terminal device and storage medium for identifying emotions of an agent. The method comprises: first obtaining original facial video stream data of the agent to be identified and the real identity category of the agent to be identified; then performing a convolution operation on the original facial video stream data to extract temporal domain attention features and spatial domain attention features; then inputting the original facial video stream data and the real identity category into a preset identity discriminator, so that the identity discriminator separates the expression features in the original facial video stream data by minimizing the difference between the distribution of identity categories under expression features and the distribution of real identity categories; finally, splicing the temporal domain attention features, spatial domain attention features and expression features to obtain the emotion features of the agent to be identified, and inputting the emotion features into a preset emotion recognition model, so that the preset emotion recognition model recognizes the emotions of the identified agent based on the emotion features. Therefore, the present invention uses an identity discriminator to minimize the difference between the distribution of real identity categories under expression features and the distribution of real identity categories, separates the expression features and identity features of the agent, and realizes the extraction of expression features. The expression features are then spliced ​​with the time domain attention and spatial domain attention obtained by convolution to obtain emotional features for identifying the emotions of the agent, and emotion recognition is completed based on the emotional features. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The figure is a flow chart of a method for identifying emotions of an agent provided by one embodiment of the present invention.

[0049] Figure 2 This is a structural diagram of a seat emotion recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0052] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0053] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0055] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0056] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0057] See also Figure 1To address the problem in existing technology where models may overemphasize differences caused by identity features during emotion recognition and fail to extract accurate facial features, thereby reducing the accuracy of emotion recognition results, an embodiment of the present invention provides a method for agent emotion recognition, comprising:

[0058] Step S101: obtaining original facial video stream data of the seat to be identified and the real identity category of the seat to be identified;

[0059] Specifically, the original facial video stream data of the corresponding person is obtained through the monitoring device.

[0060] Step S102: After performing a convolution operation on the original face video stream data, extracting the temporal domain attention features and the spatial domain attention features;

[0061] In a preferred embodiment, after performing the convolution operation on the original face video stream data, the temporal domain attention features and the spatial domain attention features are extracted, including:

[0062] Calculate the pixel change value between two adjacent video frames in the original face video stream data;

[0063] For every two adjacent video frames, if the pixel change value is greater than a preset threshold, increase the weight of the second video frame; otherwise, decrease the weight of the second video frame;

[0064] Obtaining first face video stream data according to the weight-adjusted video frame;

[0065] Slicing the first facial video stream data according to a preset depth, preset width, and preset height to obtain a facial video frame sequence;

[0066] Specifically, if the original depth of the first facial video stream data is D, the width is W, and the height is H, then in the sliced ​​facial video frame sequence, the above-mentioned preset depth of each sliced ​​video frame is D, the preset width is W / H, and the preset height is H / N, where N represents the number of sliced ​​video frames.

[0067] After performing three-dimensional convolution on the above-mentioned face video frame sequence, the time domain attention features of the above-mentioned original face video stream data are obtained;

[0068] After performing two-dimensional convolution on the above-mentioned face video frame sequence, the spatial domain attention features of the above-mentioned original face video stream data are obtained.

[0069] Specifically, the original face video stream data can be regarded as a video frame sequence. The video frames in this sequence are divided into video frames with strong discriminability and video frames with weak discriminability through the inter-frame difference algorithm. That is, the change amplitude between two adjacent video frames is calculated through the inter-frame difference algorithm, that is, the above-mentioned pixel change value is calculated. If this pixel change value is greater than the preset threshold, it means that the change amplitude of the current two video frames is large, and the second video frame is a frame with strong discriminability, and its weight is increased; otherwise, it means that the change amplitude of the current two video frames is small, and the second video frame is a frame with weak discriminability, and its weight is lowered.

[0070] Preferably, by adjusting the weight of each video frame through an inter-frame difference algorithm, a feature weight distribution can be established, which helps to highlight key information based on this feature weight distribution in subsequent convolution operations and feature extraction processes.

[0071] Preferably, after slicing the first face video stream data, the same parts in the image can be merged together to facilitate the subsequent extraction of time domain attention features.

[0072] In this preferred embodiment, different weights are assigned to each video frame through the inter-frame difference algorithm, and the weighted facial video frame sequence is convolved to obtain the time domain attention features and spatial domain attention features of the original facial video stream data.

[0073] In another preferred embodiment, before slicing the first face video stream data according to a preset depth, a preset width, and a preset height, the method further includes:

[0074] Grayscale the first face video stream data.

[0075] Specifically, each pixel of each video frame in the first face video stream data is grayed out by using the mean method, so as to limit the dimension of feature extraction in subsequent solutions.

[0076] In this preferred embodiment, the first face video stream data is grayscaled using the mean method.

[0077] In another preferred embodiment, after performing three-dimensional convolution on the facial video frame sequence, the temporal domain attention features of the original facial video stream data are obtained, including:

[0078] Performing three-dimensional convolution on all sliced ​​video frames in the face video frame sequence to obtain the first feature of each sliced ​​video frame;

[0079] Specifically, the first feature is obtained by performing three-dimensional convolution using the following formula:

[0080] slice i =3DConv(X),i∈[1,N]

[0081] Where, slice i represents the first feature of the i-th slice video frame, 3DConv(·) represents a three-dimensional convolution operation, and 3DConv(X) represents a three-dimensional convolution operation on the face video frame sequence X, where X represents the face video frame sequence.

[0082] Based on the above first feature, mean pooling is performed on each slice video frame to obtain a compressed second feature;

[0083] Fully connect the second features of all sliced ​​video frames to obtain a one-dimensional temporal feature matrix;

[0084] Specifically, we use mean pooling to compress the first feature of each sliced ​​video frame, and use a fully connected layer to learn the distribution of attention to obtain a one-dimensional temporal feature matrix:

[0085]

[0086] Where, output T represents a one-dimensional time feature matrix, FC(·) represents a fully connected operation, AvgPool(·) represents an average pooling operation, Indicates that the second feature after mean pooling is fully connected, AvgPool(slice i ) means performing mean pooling on each slice of video frame.

[0087] The above one-dimensional time feature matrix is ​​normalized to obtain the above time domain attention feature.

[0088] Specifically, the time domain attention feature is obtained by the following formula:

[0089] A T (X)=Softmax(output T )

[0090] Where A T (X) represents the time domain attention feature, Softmax(·) represents the normalization operation, Softmax(output T ) represents the normalization of the one-dimensional time feature matrix.

[0091] In this preferred embodiment, the above-mentioned facial video frame sequence is subjected to three-dimensional convolution, mean pooling, full connection and normalization to obtain the time domain attention features of the original facial video stream data.

[0092] In another preferred embodiment, after performing two-dimensional convolution on the facial video frame sequence, the spatial domain attention features of the original facial video stream data are obtained, including:

[0093] Splitting the face video frame sequence at preset intervals to obtain a plurality of video frame sequences;

[0094] Specifically, the entire face video frame sequence is divided into M segments, and the depth of each segmented video frame sequence is D / M, the width is W, and the height is H.

[0095] Pooling and convolution are performed on each video frame sequence in sequence to obtain the third feature;

[0096] Normalize all video frame sequences according to the third feature to obtain a one-dimensional spatial feature matrix;

[0097] Connect the above one-dimensional spatial feature matrices to obtain the above spatial domain attention features.

[0098] Specifically, according to the feature weight distribution obtained by weight adjustment, a two-dimensional convolution is performed on each video frame sequence to obtain the spatial domain attention feature.

[0099] Specifically, the spatial domain attention feature is calculated by the following formula:

[0100]

[0101] Where, seg i represents the sequence features of the i-th video frame sequence, Segment(·) represents the segmentation function, Segment(X) represents the segmentation of the face video frame sequence X, and output s Represents a one-dimensional spatial feature matrix, Softmax{Conv[AvgPool(seg i );MaxPool(seg i )]} means normalizing the third feature of all video frame sequences, Conv[AvgPool(seg i );MaxPool(seg i )] represents the two-dimensional convolution of the pooled video frame sequence, AvgPool(seg i ) represents the mean pooling of the sequence features of the i-th video frame sequence, MaxPool(seg i ) represents the maximum pooling of the sequence features of the i-th video frame sequence, MaxPool(·) represents the maximum pooling operation, A S (X) represents the above-mentioned spatial domain attention features, Indicates connecting the one-dimensional spatial feature matrix.

[0102] In this preferred embodiment, spatial domain attention features are obtained by performing two-dimensional convolution on each video frame sequence.

[0103] Step S103: Inputting the original face video stream data and the true identity category into a preset identity discriminator, so that the identity discriminator separates the expression features in the original face video stream data by minimizing the difference between the distribution of identity categories under the expression features and the distribution of true identity categories;

[0104] Specifically, the identity discriminator is used to determine the correlation between expression features and an individual's identity category. Under the premise that the extracted expression features are independent of the identity features corresponding to the identity category, the identity discriminator aims to minimize the difference between the distribution of identity categories under the expression features and the distribution of true identity categories, thereby establishing an objective function to maximize the independence between the extracted expression features and the individual's identity features. The objective function is:

[0105] P(s|F(x))=P(s)

[0106] Where s represents the identity category, x represents the original face video stream data, F(x) represents the extracted expression features, P(s) represents the marginal distribution corresponding to the true identity category, and P(s|F(x)) represents the distribution of identity categories under the expression features.

[0107] Preferably, the existence of the identity discriminator can solve the problem of how to learn and extract expression features that are not related to the individual, thereby ensuring that the results of emotion recognition can have a high accuracy rate in practical applications.

[0108] In a preferred embodiment, the training of the preset identity discriminator includes:

[0109] Obtaining a number of face video stream sample data with real labels; wherein the real labels are used to represent the real identity categories of the face video stream sample data;

[0110] Inputting the above-mentioned facial video stream sample data into the identity discriminator to be trained for iterative training until the difference between the current first distribution probability and the current second distribution probability is less than a preset probability threshold, thereby generating a trained preset identity discriminator; wherein the above-mentioned first distribution probability is: the identity category distribution probability under the current predicted expression feature; the above-mentioned second distribution probability is: the distribution probability of the above-mentioned true identity category;

[0111] Among them, during each iterative training, the predicted expression features and the predicted identity category features are predicted based on the current face video stream sample data; the current first distribution probability and the current second distribution probability are calculated based on the current predicted expression features, predicted identity category features and the above-mentioned true labels; it is judged whether the difference between the current first distribution probability and the current second distribution probability is less than the preset probability threshold; if it is less, the current identity discriminator is used as the trained preset identity discriminator; otherwise, after adjusting the parameters of the current identity discriminator, the current identity discriminator continues to be trained.

[0112] Specifically, during training, the general rule relative algorithm calculates the current first and second distribution probabilities to determine the correlation between the predicted expression features and the predicted identity category features. Simultaneously, the identity discriminator reduces the distance between facial video stream samples with the same expression in feature space based on the expression features.

[0113] In this preferred embodiment, the identity discriminator is trained using face video stream sample data to obtain a trained preset identity discriminator.

[0114] Step S104: Concatenate the above-mentioned time domain attention features, spatial domain attention features and expression features to obtain the emotional features of the above-mentioned seat to be identified, and input the above-mentioned emotional features into the preset emotion recognition model so that the above-mentioned preset emotion recognition model can identify the emotions of the above-mentioned seat to be identified based on the above-mentioned emotional features.

[0115] Specifically, the temporal domain attention features and the spatial domain attention features are first merged to obtain the spatiotemporal attention features, and then the spatiotemporal attention features are fused and spliced ​​with the expression features to obtain the emotion features.

[0116] Preferably, splicing the time domain attention features, the space domain attention features and the expression features can reduce the information loss and noise introduction during emotion recognition, thereby improving the accuracy of emotion recognition.

[0117] In a preferred embodiment, after identifying the emotion of the agent to be identified, the process further includes:

[0118] The emotion recognition result is drawn on the above-mentioned original face video stream data and sent to the supervisor, so that the supervisor can analyze the emotional state of the seat to be identified based on the emotion recognition result and the above-mentioned original face video stream data.

[0119] Preferably, the obtained emotion recognition results can also be plotted on the real-time facial monitoring video stream and displayed to the front end, providing real-time feedback to the supervisor on the current agent's real-time emotional state, making it easier for the supervisor to globally control the status of the agent's customer service staff. In addition, the emotion recognition results and the URL address of the corresponding original facial video stream data can also be saved to facilitate subsequent video playback and the statistics of the agent's emotion recognition data. Therefore, you can try to check the emotional state of the agent so that the manager can quickly check the work status of a specific individual agent.

[0120] In this preferred embodiment, by drawing the emotion recognition results on the original face video stream data and sending it to the supervisor, the supervisor can analyze and evaluate the emotional state of the seat staff based on this data.

[0121] Based on the above method embodiments, the present invention provides corresponding device embodiments.

[0122] like Figure 2 As shown, an embodiment of the present invention provides an agent emotion recognition device, comprising:

[0123] Data acquisition module, feature extraction module, expression feature separation module and emotion recognition module;

[0124] The data acquisition module is used to obtain the original face video stream data of the seat to be identified and the real identity category of the seat to be identified;

[0125] The feature extraction module is used to extract temporal and spatial attention features after performing a convolution operation on the original face video stream data.

[0126] The expression feature separation module is configured to input the original facial video stream data and the true identity category into a preset identity discriminator, so that the identity discriminator separates the expression features in the original facial video stream data by minimizing the difference between the distribution of identity categories under the expression features and the distribution of true identity categories;

[0127] The above-mentioned emotion recognition module is used to splice the above-mentioned time domain attention features, spatial domain attention features and expression features to obtain the emotional features of the above-mentioned seat to be identified, and input the above-mentioned emotional features into the preset emotion recognition model so that the above-mentioned preset emotion recognition model can identify the emotions of the above-mentioned seat to be identified based on the above-mentioned emotional features.

[0128] It should be noted that the device embodiment described above is merely illustrative, wherein the modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement it without paying any creative work. The above schematic diagram is only an example of an agent emotion recognition device, and does not constitute a limitation on an agent emotion recognition device, and may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0129] Based on the above method embodiment, the present invention provides a corresponding terminal device embodiment.

[0130] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-mentioned method for identifying emotions of an agent in any embodiment of the present invention.

[0131] For example, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the device.

[0132] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, or a cloud server. The device may include, but is not limited to, a processor and a memory;

[0133] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of the device, connecting the various parts of the device using various interfaces and lines.

[0134] The above-mentioned memory can be used to store the above-mentioned computer programs and / or modules. The above-mentioned processor realizes various functions of the above-mentioned device by running or executing the computer programs and / or modules stored in the above-mentioned memory, and calling the data stored in the memory. The above-mentioned memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; in addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0135] Based on the above method embodiment, the present invention provides a corresponding storage medium embodiment.

[0136] Another embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the above-mentioned method for identifying emotions of an agent in any embodiment of the present invention.

[0137] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, an executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0138] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for identifying emotions of seat personnel, characterized in that: include: Obtaining original facial video stream data of the seat person to be identified and the real identity category of the seat person to be identified; After performing a convolution operation on the original face video stream data, temporal domain attention features and spatial domain attention features are extracted; Inputting the original face video stream data and the real identity category into a preset identity discriminator, so that the identity discriminator separates the expression features in the original face video stream data by minimizing the difference between the distribution of identity categories under the expression features and the distribution of the real identity categories; The time domain attention features, spatial domain attention features and expression features are spliced ​​together to obtain the emotional features of the seat to be identified, and the emotional features are input into a preset emotion recognition model so that the preset emotion recognition model can identify the emotions of the seat to be identified based on the emotional features.

2. The method for identifying emotions of seat personnel according to claim 1, characterized in that: After performing the convolution operation on the original face video stream data, the temporal domain attention features and the spatial domain attention features are extracted, including: Calculating a pixel change value between two adjacent video frames in the original face video stream data; For every two adjacent video frames, if the pixel change value is greater than a preset threshold, increase the weight of the second video frame; otherwise, decrease the weight of the second video frame; Obtaining first face video stream data according to the weight-adjusted video frame; Slicing the first facial video stream data according to a preset depth, a preset width, and a preset height to obtain a facial video frame sequence; After performing three-dimensional convolution on the facial video frame sequence, a time domain attention feature of the original facial video stream data is obtained; After performing two-dimensional convolution on the facial video frame sequence, the spatial domain attention features of the original facial video stream data are obtained.

3. The method for identifying emotions of seat personnel according to claim 2, characterized in that: Before slicing the first face video stream data according to a preset depth, a preset width, and a preset height, the method further includes: Grayscale the first face video stream data.

4. The method for identifying emotions of seat personnel according to claim 3, characterized in that: After performing three-dimensional convolution on the facial video frame sequence, the temporal domain attention features of the original facial video stream data are obtained, including: Performing three-dimensional convolution on all sliced ​​video frames in the face video frame sequence to obtain a first feature of each sliced ​​video frame; Perform mean pooling on each sliced ​​video frame according to the first feature to obtain a compressed second feature; Fully connect the second features of all sliced ​​video frames to obtain a one-dimensional temporal feature matrix; The one-dimensional time feature matrix is ​​normalized to obtain the time domain attention feature.

5. The method for identifying emotions of seat personnel according to claim 4, characterized in that: After performing two-dimensional convolution on the face video frame sequence, the spatial domain attention features of the original face video stream data are obtained, including: Splitting the face video frame sequence at preset intervals to obtain a plurality of video frame sequences; Pooling and convolution are performed on each video frame sequence in sequence to obtain the third feature; Normalizing all video frame sequences according to the third feature to obtain a one-dimensional spatial feature matrix; The one-dimensional spatial feature matrix is ​​connected to obtain the spatial domain attention feature.

6. The method for identifying emotions of seat personnel according to claim 5, characterized in that: The training of the preset identity discriminator includes: Acquire a number of face video stream sample data with real labels; wherein the real labels are used to represent the real identity categories of the face video stream sample data; Inputting the facial video stream sample data into the identity discriminator to be trained for iterative training until the difference between the current first distribution probability and the current second distribution probability is less than a preset probability threshold, thereby generating a trained preset identity discriminator; wherein the first distribution probability is: the identity category distribution probability under the current predicted expression feature; the second distribution probability is: the distribution probability of the actual identity category; Among them, during each iterative training, the predicted expression features and the predicted identity category features are predicted based on the current face video stream sample data; the current first distribution probability and the current second distribution probability are calculated based on the current predicted expression features, the predicted identity category features and the true label; it is judged whether the difference between the current first distribution probability and the current second distribution probability is less than the preset probability threshold; if it is less, the current identity discriminator is used as the trained preset identity discriminator; otherwise, after adjusting the parameters of the current identity discriminator, the current identity discriminator continues to be trained.

7. The method for identifying emotions of seat personnel according to claim 6, characterized in that: After identifying the emotion of the agent to be identified, the method further includes: The emotion recognition result is drawn on the original face video stream data and sent to the supervisor, so that the supervisor can analyze the emotional state of the seat to be identified based on the emotion recognition result and the original face video stream data.

8. A seat staff emotion recognition device, characterized in that: include: Data acquisition module, feature extraction module, expression feature separation module and emotion recognition module; The data acquisition module is used to obtain the original face video stream data of the seat person to be identified and the real identity category of the seat person to be identified; The feature extraction module is used to extract the time domain attention features and the spatial domain attention features after performing a convolution operation on the original face video stream data; The expression feature separation module is used to input the original face video stream data and the real identity category into a preset identity discriminator, so that the identity discriminator separates the expression features in the original face video stream data by minimizing the difference between the distribution of identity categories under the expression features and the distribution of real identity categories; The emotion recognition module is used to splice the time domain attention features, spatial domain attention features and expression features to obtain the emotion features of the seat to be identified, and input the emotion features into a preset emotion recognition model so that the preset emotion recognition model can identify the emotions of the seat to be identified based on the emotion features.

9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for recognizing emotions of an agent according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the agent emotion recognition method according to any one of claims 1 to 7.

Images