Emotion recognition method and device, equipment, storage medium and program product

Abstract

The invention provides an emotion recognition method and device, equipment, a storage medium and a program product, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring a target image containing a target object; performing feature extraction on the target image through an R-FCN algorithm to obtain an individual model corresponding to the target object; obtaining target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object through an R-FCN algorithm based on the individual model; based on the reference emotion data set, the target expression recognition data, the target posture recognition data and the target action recognition data, determining an expression score corresponding to the target expression recognition data, a posture score corresponding to the target posture recognition data and an action score corresponding to the target action recognition data; and determining a target emotional state corresponding to the target object based on the expression score, the posture score and the action score. According to the invention, the target emotional state corresponding to the target object can be identified more accurately.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an emotion recognition method, electronic device, storage medium and program product. Background Art

[0002] Emotion recognition refers to the technology of identifying and classifying human emotions by analyzing various data sources (such as speech, facial expressions, text or physiological signals, etc.). Emotion recognition is a research hotspot in the field of artificial intelligence and has been widely used in many fields.

[0003] At present, emotion recognition is usually performed based on deep learning-based target detection algorithms, such as regional convolutional neural networks (R-CNN). However, in some scenarios, such as using bank intelligent robots to provide business services to customers at bank branches instead of humans, bank intelligent robots cannot accurately identify the emotional state of customers, resulting in the inability to provide better services to customers. Summary of the invention

[0004] The present application provides an emotion recognition method, apparatus, device, storage medium and program product to solve technical problems in some scenarios, such as when a bank intelligent robot replaces humans in providing business services to customers at a bank branch, but the bank intelligent robot is unable to accurately identify the emotional state of the customer, resulting in an inability to better serve the customer.

[0005] In a first aspect, the present application provides an emotion recognition method, comprising:

[0006] Acquire a target image containing a target object;

[0007] The target image is feature extracted through the regional fully convolutional network R-FCN algorithm to obtain the individual model corresponding to the target object;

[0008] Based on the individual model, the target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object are obtained through the R-FCN algorithm;

[0009] Based on a reference emotion data set, target expression recognition data, target body posture recognition data, and target action recognition data, determining an expression score corresponding to the target expression recognition data, a body posture score corresponding to the target body posture recognition data, and an action score corresponding to the target action recognition data, wherein the reference emotion data set is obtained through big data training;

[0010] Based on the expression score, posture score and action score, the target emotional state corresponding to the target object is determined.

[0011] In a second aspect, the present application provides an emotion recognition device, comprising:

[0012] A first acquisition module, used for acquiring a target image containing a target object;

[0013] The extraction module is used to extract features of the target image through the regional fully convolutional network R-FCN algorithm to obtain the individual model corresponding to the target object;

[0014] The second acquisition module is used to acquire the target expression recognition data, the target posture recognition data and the target action recognition data corresponding to the target object through the R-FCN algorithm based on the individual model;

[0015] A first determination module is used to determine an expression score corresponding to the target expression recognition data, a posture score corresponding to the target posture recognition data, and an action score corresponding to the target action recognition data based on a reference emotion data set, target expression recognition data, target posture recognition data, and target action recognition data, wherein the reference emotion data set is obtained through big data training;

[0016] The second determination module is used to determine the target emotional state corresponding to the target object based on the expression score, the posture score and the action score.

[0017] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0018] Memory stores computer-executable instructions;

[0019] The processor executes the computer-executable instructions stored in the memory to implement the above first aspect and / or various possible implementations of the first aspect.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed, the above first aspect and / or various possible implementation methods of the first aspect are implemented.

[0021] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above first aspect and / or various possible implementation methods of the first aspect when the computer program is executed.

[0022] The emotion recognition method, apparatus, device, storage medium and program product provided by the present application can extract features of a target image containing a target object through the R-FCN algorithm, so as to more accurately obtain an individual model corresponding to the target object; based on the individual model, the target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object are obtained through the R-FCN algorithm, that is, the expression, posture and action of the target object in the target image are accurately separated and recognized; based on the reference emotion data set, the target expression recognition data, the target posture recognition data and the target action recognition data, the expression score corresponding to the target expression recognition data, the posture score corresponding to the target posture recognition data and the action score corresponding to the target action recognition data are determined; based on the expression score, the posture score and the action score, the target emotional state corresponding to the target object is determined, so as to more accurately recognize the expression, posture and action of the target object from multiple angles, so as to more accurately identify the target emotional state corresponding to the target object, provide corresponding services and assistance to the target object according to the target emotional state, and improve the interaction effect with the target object. The emotion recognition method provided in the embodiment of the present application is more suitable for bank intelligent robots to identify the emotional state of customers in bank branches, improve the human-computer interaction effect, thereby optimizing the business services of the intelligent robots, helping to improve the intelligence and intimacy of the intelligent robots, and thus improving the service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0024] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0025] Figure 2 Schematic diagram of the process of the emotion recognition method provided in the embodiment of the present application Figure 1 ;

[0026] Figure 3 Schematic diagram of the process of the emotion recognition method provided in the embodiment of the present application Figure 2 ;

[0027] Figure 4 A schematic diagram of a residual structure of a residual network provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of obtaining a candidate area provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of a candidate area provided in an embodiment of the present application;

[0030] Figure 7Schematic diagram of the process of the emotion recognition method provided in the embodiment of the present application Figure 3 ;

[0031] Figure 8 A schematic diagram of the structure of an emotion recognition device provided in one embodiment of the present application;

[0032] Fig. 9 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0033] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] In addition, this application involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and the use of artificial intelligence technology for automated decision-making, and a technical solution for making decisions that have a significant impact on personal rights and interests based on the results of automated decision-making. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered.

[0037] It should be noted that the emotion recognition method, device, equipment, storage medium and product provided in this application can be used in the field of artificial intelligence, and can also be used in any field other than artificial intelligence. The application field of the emotion recognition method, device, equipment, storage medium and program product in this application is not limited.

[0038] First, some technical terms involved in this application are explained:

[0039] Motion recognition technology is a technology that understands human behavioral intentions by analyzing and identifying human movements and postures and combining them with big data model training.

[0040] Emotion recognition refers to the technology of identifying and classifying human emotions by analyzing various data sources (such as voice, facial expressions, text or physiological signals).

[0041] At present, emotion recognition is usually performed based on deep learning object detection algorithms. Among them, deep learning-based object detection algorithms currently mainly include two categories: one is a two-stage object detection algorithm that first generates candidate regions, such as Region-based Convolutional Neural Network (R-CNN), Fast R-CNN and Faster R-CNN; the other is a one-stage object detection algorithm that does not need to generate candidate regions first, such as the YOLO algorithm and the Single Shot MultiBox Detector (SSD) algorithm. Although the YOLO algorithm has fast detection speed and high efficiency, it is not effective for detecting smaller objects in images. However, in some scenarios, such as using bank intelligent robots to provide business services to customers at bank outlets instead of humans, bank intelligent robots cannot accurately judge customers' emotions through expressions, postures and movements like humans, resulting in a failure to provide better services to customers.

[0042] Based on the above problems, considering that the Region-based Fully Convolutional Network (R-FCN) algorithm has the characteristics of stronger applicability, higher accuracy and better detection effect for small objects when performing target detection, the present application provides an emotion recognition method, for the acquired target image containing the target object, the R-FCN algorithm is used to identify the individual model corresponding to the target object, and the R-FCN algorithm is used to separate and identify the expression, posture and action of the target object in the individual model. The R-FCN algorithm has higher accuracy in recognition. Therefore, the present application can more accurately identify the target emotional state corresponding to the target object, provide corresponding services and assistance to the target object according to the target emotional state, and improve the interaction effect with the target object (such as improving interaction efficiency, interacting more friendly, etc.).

[0043] It can be seen that the emotion recognition method provided by this application is more suitable for bank intelligent robots to identify the emotional state of customers in bank branches, improve the human-computer interaction effect, thereby helping to improve the intelligence and intimacy of intelligent robots, and thus improve service quality.

[0044] In the following, the application scenarios of the solution provided in this application are first illustrated.

[0045] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, in this application scenario, the bank intelligent robot obtains a target image containing a target customer through a camera (not shown in the figure). The bank intelligent robot determines the target emotional state corresponding to the target customer based on the target image through the emotion recognition method provided in the embodiment of the present application, thereby providing corresponding services and assistance to the target customer.

[0046] It should be noted that Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present application. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices is limited.

[0047] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] Figure 2 Schematic diagram of the process of the emotion recognition method provided in the embodiment of the present application Figure 1 The emotion recognition method may be executed by software and / or hardware devices. For example, the hardware device may be an emotion recognition device, and the emotion recognition device may be an electronic device or a processing chip in the electronic device. Figure 2 As shown, the method of the embodiment of the present application includes:

[0049] S201: Acquire a target image containing a target object.

[0050] In the embodiment of the present application, for example, the target object is a target customer of a bank branch, and the bank intelligent robot can obtain a target image containing the target customer in real time through a camera (image system).

[0051] S202, extracting features of the target image through the R-FCN algorithm to obtain an individual model corresponding to the target object.

[0052] In this step, after obtaining the target image containing the target object, the target image is preprocessed for object feature recognition, that is, the behavior of different objects in the target image is feature extracted by the R-FCN algorithm, and the different objects in the target image are separated into individual models, so that the individual model corresponding to the target object can be obtained. Exemplarily, first, for example, the residual network (ResNet) in the R-FCN algorithm can be used to extract the features of the target image to obtain the corresponding feature map; secondly, the region proposal network (RPN) in the R-FCN algorithm can be used to generate multiple candidate regions corresponding to the feature map, that is, the region of interest (ROI); then, a special convolution layer is added at the end of the RPN network to construct a set of position-sensitive score maps for each ROI, and position information is added to the ROI; finally, the position-sensitive score map is average-pooled and classified, and the different objects in the target image are separated into individual models, so that the individual model corresponding to the target object can be obtained.

[0053] S203, based on the individual model, obtaining target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object through the R-FCN algorithm.

[0054] In this step, after obtaining the individual model corresponding to the target object, the expression, posture and behavior of the target object can be captured from the individual model through the R-FCN algorithm based on the individual model, so that the target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object can be obtained. Exemplarily, referring to the example of step S202, first, for example, the residual network (ResNet) in the R-FCN algorithm can be used to extract features of the individual model to obtain the corresponding feature map; secondly, multiple candidate regions (ROIs) corresponding to the feature map can be generated through the RPN network; then a special convolution layer is added at the end of the RPN network to construct a set of position-sensitive score maps for each ROI, and position information is added to the ROI; finally, the position-sensitive score map is average-pooled and classified to capture the expression, posture and behavior of the target object in the target image, so that the target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object can be obtained.

[0055] S204, based on the reference emotion data set, the target expression recognition data, the target body posture recognition data and the target action recognition data, determining the expression score corresponding to the target expression recognition data, the body posture score corresponding to the target body posture recognition data and the action score corresponding to the target action recognition data, wherein the reference emotion data set is obtained through big data training.

[0056] In this step, the reference emotion data set is obtained in advance through big data training based on face recognition technology and action recognition technology. The reference emotion data set includes, for example, a reference expression data set, a reference posture data set, and a reference action data set. It can be understood that the action recognition technology is to analyze and identify human actions through technologies such as human posture (posture) estimation and human key point detection. Among them, human posture estimation refers to estimating the position, direction, size and other information of the human body from an image or video; human key point detection refers to detecting the key points of the human body from an image or video, such as the head, shoulders, elbows and knees. Through action recognition technology, supplemented by big data modeling training, the behavior of individual models can be accurately identified and their behavioral intentions can be judged. Exemplarily, based on the reference emotion data set, the characteristics of the individual model can be analyzed to judge the emotions, thereby determining the expression score corresponding to the target expression recognition data, the posture score corresponding to the target posture recognition data, and the action score corresponding to the target action recognition data.

[0057] Further, optionally, the reference emotion data set includes a reference expression data set, a reference posture data set and a reference action data set; based on the reference emotion data set, the target expression recognition data, the target posture recognition data and the target action recognition data, determining the expression score corresponding to the target expression recognition data, the posture score corresponding to the target posture recognition data and the action score corresponding to the target action recognition data, may include: determining the expression score corresponding to the target expression recognition data based on the similarity between the target expression recognition data and the data in the reference expression data set; determining the posture score corresponding to the target posture recognition data based on the similarity between the target posture recognition data and the data in the reference posture data set; determining the action score corresponding to the target action recognition data based on the similarity between the target action recognition data and the data in the reference action data set.

[0058] Exemplarily, emotion judgment can be performed based on a reference emotion data set. Specifically, the target expression recognition data can be compared with the data in the reference expression data set, that is, the similarity between the target expression recognition data and the data in the reference expression data set is obtained, and the situation after expression recognition is obtained, thereby determining the expression score corresponding to the target expression recognition data. The target posture recognition data can be compared with the data in the reference posture data set, that is, the similarity between the target posture recognition data and the data in the reference posture data set is obtained, and the situation after posture recognition is obtained, thereby determining the posture score corresponding to the target posture recognition data. The target action recognition data can be compared with the data in the reference action data set, that is, the similarity between the target action recognition data and the data in the reference action data set is obtained, and the situation after action recognition is obtained, thereby determining the action score corresponding to the target action recognition data.

[0059] S205: Determine the target emotional state corresponding to the target object based on the expression score, posture score and action score.

[0060] In this step, after the expression score, posture score and action score are obtained, the target emotional state corresponding to the target object can be determined based on the expression score, posture score and action score.

[0061] Further, optionally, determining the target emotional state corresponding to the target object based on the expression score, posture score and action score may include: taking a weighted sum of the expression score, posture score and action score to obtain the target emotional score; and determining the target emotional state corresponding to the target object based on the target emotional score.

[0062] Exemplarily, a first weight corresponding to the expression score, a second weight corresponding to the posture score, and a third weight corresponding to the action score may be pre-set, so that the expression score, posture score, and action score may be weighted and summed according to the expression score, the first weight, the posture score, the second weight, the action score, and the third weight to obtain a target emotion score. After obtaining the target emotion score, the target emotion state corresponding to the target object may be determined based on the correspondence between the emotion score and the emotion state and the target emotion score. For example, the current emotion state of a customer in a bank branch may be determined, and personalized services may be provided to the customer in combination with the behavior identifier and the rule model to ensure customer satisfaction.

[0063] The emotion recognition method provided in the embodiment of the present application performs feature extraction on a target image containing a target object through the R-FCN algorithm, so as to more accurately obtain an individual model corresponding to the target object; based on the individual model, the target expression recognition data, target body posture recognition data and target action recognition data corresponding to the target object are obtained through the R-FCN algorithm, that is, the expression, body posture and action of the target object in the target image are accurately separated and recognized; based on the reference emotion data set, the target expression recognition data, the target body posture recognition data and the target action recognition data, the expression score corresponding to the target expression recognition data, the body posture score corresponding to the target body posture recognition data and the action score corresponding to the target action recognition data are determined; based on the expression score, the body posture score and the action score, the target emotional state corresponding to the target object is determined, so as to more accurately recognize the expression, body posture and action of the target object from multiple angles, so as to more accurately determine the target emotional state corresponding to the target object, provide corresponding services and help to the target object according to the target emotional state, and improve the interaction effect with the target object (such as improving interaction efficiency, interacting in a more friendly manner, etc.). The emotion recognition method provided in the embodiment of the present application is more suitable for bank intelligent robots to identify the emotional state of customers in bank branches, improve the human-computer interaction effect, thereby optimizing the business services of the intelligent robots, helping to improve the intelligence and intimacy of the intelligent robots, and thus improving the service quality.

[0064] Figure 3 Schematic diagram of the process of the emotion recognition method provided in the embodiment of the present application Figure 2 Based on the above embodiments, the present invention is Figure 2 The step S202 in the embodiment shown is further described. Figure 3 As shown, the method of the embodiment of the present application may include:

[0065] S301 , extracting features of a target image using an R-FCN algorithm to obtain a first feature map.

[0066] It is understandable that, in general, the depth of the deep learning network has a greater impact on the final classification and recognition effects. The greater the network depth, the better the classification and recognition effects. However, when only a simple network stacking method is used to increase the network depth, the gradient disappearance phenomenon will become more and more obvious, resulting in the network finally experiencing degradation problems. This degradation means that when the number of layers of the network model increases, the error rate increases. The cause of this degradation problem can be attributed to the optimization problem, that is, when the network model becomes more complex, the optimization of the stochastic gradient descent method (SGD) becomes more difficult, resulting in the final trained model not achieving a good learning effect. The residual network (RestNet) can better solve the degradation problem that occurs when the network depth increases. Figure 4 A schematic diagram of a residual structure of a residual network provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, in the residual structure of the residual network, the residual network adds an identity mapping to convert the original function H(x) to be learned into F(x)+x, where x represents the input variable. The above two expressions have the same effect, but the difficulty of optimization is not the same. The optimization of F(x) is simpler than that of H(x). This is because the residual vector encoding in image processing can decompose a problem into residual problems of multiple sizes through a conversion, which can optimize the training effect. In addition, before the second activation function of the residual network is output, the original input is superimposed on the main output of the residual network, that is, a shortcut connection is added. The shortcut connection allows the data flow to perform nonlinear superposition operations in a cross-layer manner without adding additional parameters and calculations to the network. At the same time, it can improve the speed of network training and the training effect. Among them, the first activation function and the second activation function are both, for example, rectified linear unit (Rectified LinearUnit, ReLU) functions. When the number of layers of the model is deepened, the residual structure of the residual network can solve the degradation problem well.

[0067] In this step, illustratively, for example, the target image may be subjected to feature extraction through a residual network in an R-FCN algorithm to obtain a first feature map.

[0068] S302: Generate multiple first candidate regions corresponding to the first feature map according to the first feature map through the R-FCN algorithm.

[0069] Exemplarily, a plurality of first candidate regions (ROIs) corresponding to the first feature map may be generated according to the first feature map through an RPN network in the R-FCN algorithm.

[0070] Further, optionally, generating multiple first candidate regions corresponding to the first feature map according to the first feature map through the R-FCN algorithm may include: generating multiple first candidate regions corresponding to the first feature map through the RPN network in the R-FCN algorithm according to the first feature map, the first candidate regions corresponding to a preset number of candidate boxes of different sizes, and the preset number of candidate boxes of different sizes are obtained by learning the geometric size of the target box on the training data set based on the K-means clustering algorithm with intersection-over-union ratio.

[0071] For example, Figure 5 A schematic diagram of obtaining a candidate region provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the RPN network generates N candidate boxes (corresponding to candidate areas) of different sizes on the feature map by sliding windows. It can be understood that due to the differences in the size and aspect ratio of the target to be detected, it is often necessary to set candidate boxes of multiple geometric sizes. In the original RPN network, the candidate box size is composed of three default sizes (128x 128, 256x 256, and 512x 512) and three aspect ratios (1:1, 1:2, and 2:1), with a total of 9 sizes of candidate boxes, which are used to detect targets of different sizes.

[0072] Considering that the target to be detected in the embodiment of the present application is a person in the image, the overall difference in the size of the target to be detected is not large, and the candidate box of the original size is not suitable for the application of the embodiment of the present application. Therefore, in the embodiment of the present application, 6 sizes (i.e., a preset number of sizes) of candidate boxes are selected to avoid generating candidate boxes of redundant and useless sizes, and at the same time, the method of generating candidate boxes in the RPN network is improved. For example, the K-means clustering algorithm based on the intersection-over-union ratio can be used to learn the prior knowledge of the geometric size of the target box for a training data set containing a large amount of training data, and obtain 6 sizes of candidate boxes suitable for the prediction of the two size features in this article. Using this as a reference to generate candidate boxes can reduce the initial error of RPN network training, improve the speed of network training and the accuracy of target positioning.

[0073] Specifically, after all bounding boxes (candidate boxes) in the training data set are clustered together, M are randomly selected from all annotated bounding boxes as initial centroid boxes, and the n annotated boxes and the M initial centroid boxes are clustered on the width and height plane using the K-means clustering algorithm. The distance between the bounding box of the lth centroid box and the bounding box of the ith annotated box is As shown in the following formula 1:

[0074]

[0075] in, and Respectively represent the intersection area and union area of ​​the lth centroid frame and the i-th annotation frame, then Represents the intersection of the area of ​​the lth centroid box and the i-th annotation box.

[0076] Using the distance defined by the above formula 1, M clusters can be generated. The median width and median height of all bounding boxes in each cluster are calculated as the width and height of the new centroid box in the cluster. The above process is repeated continuously. When the update error of the centroid box of each cluster is less than the given error, clustering is stopped.

[0077] In this embodiment, a plurality of first candidate regions corresponding to the first feature map can be generated according to the first feature map through the RPN network in the R-FCN algorithm, and the first candidate regions correspond to candidate boxes of 6 sizes.

[0078] S303: for each first candidate region among the multiple first candidate regions, construct a first position sensitive score map corresponding to each first candidate region according to the first candidate region by using the R-FCN algorithm, and add position information to the first candidate region.

[0079] It can be understood that the position-sensitive score map is the core of the entire R-FCN algorithm. After generating the first feature map, the R-FCN algorithm generates k for each category target. 2 Position sensitive score map, the classification type, for example, there are C categories, object types plus background total C + 1 categories, finally a total of k 2 Output layer with (C+1) channels.

[0080] Figure 6 A schematic diagram of a candidate area provided in an embodiment of the present application, such as Figure 6 As shown, for an ROI with a width of w and a height of h, the size of a bin (also known as a sub-region of the ROI) is k is a predefined hyperparameter. When k is 3, there are 9 position-sensitive score maps, representing the sensitivity scores of a single class of objects, and the numbers they represent are the spatial position information of the (upper-left, upper-middle, upper-right, middle-left, ..., lower-left) regions of the single class of objects.

[0081] In this embodiment, the R-FCN algorithm can be used to construct a first position sensitive score map corresponding to each first candidate region according to the first candidate region, and the position information is added to the first candidate region.

[0082] S304 . Perform average pooling processing on the first position sensitive score map by using the R-FCN algorithm to obtain a first mean value corresponding to the first candidate region.

[0083] It can be understood that the ROI pooling layer is a position-sensitive pooling layer. Only when the position-sensitive score map of the corresponding position is encoded can the corresponding pooling reaction be generated. For any sub-region bin(i,j) in the ROI, the pooling operation is shown in the following formula 2:

[0084]

[0085] Among them, Θ represents the network parameters; x0 and y0 represent the coordinates of the upper left corner of the ROI; bin(i,j) represents the (i,j)th spatial position of the ROI; (x,y) represents the coordinates of the point at this spatial position; and n represents the number of points in this spatial position.

[0086] Then the obtained k 2 The corresponding position-sensitive score maps are averaged and pooled to output the mean. Finally, the confidence that the ROI is category C can be obtained by normalized exponential function (softmax) regression.

[0087] In this embodiment, the first position sensitive score map may be average pooled using the R-FCN algorithm to obtain a first mean value corresponding to the first candidate region.

[0088] S305 . Classify the first candidate region according to the first mean value by using the R-FCN algorithm to obtain an individual model corresponding to the target object.

[0089] In this step, after obtaining the first mean corresponding to the first candidate region, for example, softmax regression can be used to obtain the confidence that the first candidate region is category C, so that the first candidate region can be classified to obtain the individual model corresponding to the target object.

[0090] It can be understood that after the region of interest pooling (ROI-Pooling) maps the feature map of the candidate target to generate a target feature map (feature map), the classification probability and bounding box regression can be jointly trained by detecting the classification probability (softmax Loss) and detecting the bounding box regression (SmoothL1 Loss). In the embodiment of the present application, the two size features output by the feature extraction network are connected to the RPN network respectively, and are trained separately. For the candidate boxes generated by the two RPN networks, for example, the non-maximum suppression method can be used to filter out the same target box. The embodiment of the present application adopts a training method with dynamic alternation of the learning rate. The batch size (batch_size) is, for example, 128. The training samples are constructed and the R-FCN algorithm is allowed to learn from them to obtain an excellent R-FCN algorithm.

[0091] In an embodiment of the present application, a feature of a target image is extracted through an R-FCN algorithm to obtain a first feature map; a plurality of first candidate regions corresponding to the first feature map are generated through the R-FCN algorithm according to the first feature map; for each first candidate region among the plurality of first candidate regions, a first position sensitive score map corresponding to each first candidate region is constructed through the R-FCN algorithm according to the first candidate region, and position information is added to the first candidate region; the first position sensitive score map is average pooled through the R-FCN algorithm to obtain a first mean corresponding to the first candidate region; the first candidate regions are classified through the R-FCN algorithm according to the first mean to obtain an individual model corresponding to the target object, which can more accurately separate and identify the individual model corresponding to the target object in the target image, thereby helping to improve the recognition accuracy of the target emotional state corresponding to the target object.

[0092] Figure 7 Schematic diagram of the process of the emotion recognition method provided in the embodiment of the present application Figure 3 Based on the above embodiments, the present invention is Figure 2 The step S203 in the embodiment shown is further described. Figure 7 As shown, the method of the embodiment of the present application may include:

[0093] S701. Extract features of the individual model using the R-FCN algorithm to obtain a second feature map.

[0094] Exemplarily, referring to the example of step S301, for example, the feature extraction of the individual model can be performed through the residual network in the R-FCN algorithm to obtain the second feature map.

[0095] S702: Generate multiple second candidate regions corresponding to the second feature map according to the second feature map through the R-FCN algorithm.

[0096] Exemplarily, referring to the example of step S302, multiple second candidate regions corresponding to the second feature map can be generated according to the first feature map through the RPN network in the R-FCN algorithm. The second candidate regions correspond to candidate boxes of, for example, 6 sizes.

[0097] S703: for each second candidate region among the multiple second candidate regions, construct a second position sensitive score map corresponding to each second candidate region according to the second candidate region by using the R-FCN algorithm, and add position information to the second candidate region.

[0098] Exemplarily, referring to the example of step S303, a second position-sensitive score map corresponding to each second candidate region may be constructed by the R-FCN algorithm, and position information may be added to the second candidate region.

[0099] S704: Perform average pooling processing on the second position sensitive score map by using the R-FCN algorithm to obtain a second mean value corresponding to the second candidate region.

[0100] Exemplarily, referring to the example of step S304, the second position sensitive score map may be average pooled using the R-FCN algorithm to obtain a second mean corresponding to the second candidate region.

[0101] S705 , classifying the second candidate region according to the second mean value by using the R-FCN algorithm to obtain target expression recognition data, target posture recognition data, and target action recognition data corresponding to the target object.

[0102] Exemplarily, referring to the example of step S305, after obtaining the second mean corresponding to the second candidate area, softmax regression can be used to obtain the confidence that the second candidate area is category C, so that the second candidate area can be classified to obtain the target expression recognition data, target body recognition data and target action recognition data corresponding to the target object.

[0103] In the embodiment of the present application, the R-FCN algorithm is used to extract features of the individual model to obtain a second feature map; the R-FCN algorithm is used to generate multiple second candidate areas corresponding to the second feature map according to the second feature map; for each second candidate area among the multiple second candidate areas, the R-FCN algorithm is used to construct a second position-sensitive score map corresponding to each second candidate area according to the second candidate area, and position information is added to the second candidate area; the second position-sensitive score map is average-pooled by the R-FCN algorithm to obtain a second mean corresponding to the second candidate area; the second candidate areas are classified by the R-FCN algorithm according to the second mean to obtain target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object, which can more accurately separate and identify the target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object, so as to be used to more accurately perform multi-angle recognition of the expression, posture and action of the target object, which helps to improve the recognition accuracy of the target emotional state corresponding to the target object.

[0104] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0105] Figure 8 This is a schematic diagram of the structure of an emotion recognition device provided in one embodiment of the present application. Figure 8 As shown, the emotion recognition device 800 of the embodiment of the present application includes: a first acquisition module 801, an extraction module 802, a second acquisition module 803, a first determination module 804 and a second determination module 805. Among them:

[0106] The first acquisition module 801 is used to acquire a target image containing a target object.

[0107] The extraction module 802 is used to extract features of the target image by using the R-FCN algorithm to obtain an individual model corresponding to the target object.

[0108] The second acquisition module 803 is used to acquire the target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object through the R-FCN algorithm based on the individual model.

[0109] The first determination module 804 is used to determine the expression score corresponding to the target expression recognition data, the posture score corresponding to the target posture recognition data, and the action score corresponding to the target action recognition data based on the reference emotion data set, the target expression recognition data, the target posture recognition data, and the target action recognition data. The reference emotion data set is obtained through big data training.

[0110] The second determination module 805 is used to determine the target emotional state corresponding to the target object based on the expression score, the posture score and the action score.

[0111] In some embodiments, the extraction module 802 can be specifically used to: perform feature extraction on the target image through the R-FCN algorithm to obtain a first feature map; generate multiple first candidate regions corresponding to the first feature map based on the first feature map through the R-FCN algorithm; for each first candidate region among the multiple first candidate regions, construct a first position sensitive score map corresponding to each first candidate region based on the first candidate region through the R-FCN algorithm, and add position information to the first candidate region; perform average pooling processing on the first position sensitive score map through the R-FCN algorithm to obtain a first mean corresponding to the first candidate region; classify the first candidate regions according to the first mean through the R-FCN algorithm to obtain an individual model corresponding to the target object.

[0112] Optionally, when the extraction module 802 is used to generate multiple first candidate regions corresponding to the first feature map according to the first feature map through the R-FCN algorithm, it can be specifically used to: generate multiple first candidate regions corresponding to the first feature map through the region proposal network RPN in the R-FCN algorithm according to the first feature map, the first candidate regions correspond to a preset number of candidate boxes of different sizes, and the preset number of candidate boxes of different sizes are obtained by learning the geometric size of the target box on the training data set based on the K-means clustering algorithm with intersection-over-union ratio.

[0113] In some embodiments, the second acquisition module 803 can be specifically used to: extract features from individual models through the R-FCN algorithm to obtain a second feature map; generate multiple second candidate regions corresponding to the second feature map based on the second feature map through the R-FCN algorithm; for each second candidate region among the multiple second candidate regions, construct a second position sensitive score map corresponding to each second candidate region based on the second candidate region through the R-FCN algorithm, and add position information to the second candidate region; perform average pooling processing on the second position sensitive score map through the R-FCN algorithm to obtain a second mean corresponding to the second candidate region; classify the second candidate regions according to the second mean through the R-FCN algorithm to obtain target expression recognition data, target body recognition data and target action recognition data corresponding to the target object.

[0114] In some embodiments, the reference emotion data set includes a reference expression data set, a reference posture data set and a reference action data set; the first determination module 804 can be specifically used to: determine the expression score corresponding to the target expression recognition data based on the similarity between the target expression recognition data and the data in the reference expression data set; determine the posture score corresponding to the target posture recognition data based on the similarity between the target posture recognition data and the data in the reference posture data set; determine the action score corresponding to the target action recognition data based on the similarity between the target action recognition data and the data in the reference action data set.

[0115] In some embodiments, the second determination module 805 may be specifically used to: perform weighted summation of the expression score, posture score and action score to obtain a target emotion score; and determine a target emotion state corresponding to the target object according to the target emotion score.

[0116] The emotion recognition device of the embodiment of the present application can be used to execute the technical solution of any of the above-mentioned method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0117] Fig. 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig. 9 As shown, the electronic device 900 may include: at least one processor 901 and a memory 902 .

[0118] The memory 902 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer-executable instructions.

[0119] The memory 902 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0120] The processor 901 is used to execute the computer-executable instructions stored in the memory 902 to implement the emotion recognition method described in the aforementioned method embodiment. The processor 901 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. Specifically, when implementing the emotion recognition method described in the aforementioned method embodiment, the electronic device may be, for example, an electronic device with processing functions such as a server or a bank intelligent robot.

[0121] Optionally, the electronic device 900 may further include a communication interface 903. In a specific implementation, if the communication interface 903, the memory 902 and the processor 901 are implemented independently, the communication interface 903, the memory 902 and the processor 901 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0122] Optionally, in a specific implementation, if the communication interface 903, the memory 902 and the processor 901 are integrated on a chip, the communication interface 903, the memory 902 and the processor 901 can communicate through an internal interface.

[0123] The electronic device is used to execute the emotion recognition method in any of the aforementioned method embodiments, and its implementation principle and technical effects are similar and will not be repeated here.

[0124] The present application also provides a computer-readable storage medium, in which computer program instructions are stored. When a processor executes the computer program instructions, the above-mentioned emotion recognition method is implemented.

[0125] The present application also provides a computer program product, including a computer program, which implements the above emotion recognition method when executed by a processor.

[0126] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0127] An exemplary readable storage medium is coupled to the processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist in the emotion recognition device as discrete components.

[0128] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0131] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory (RAM), a disk or an optical disk.

[0132] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0133] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application, are not limited to the precise structures described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An emotion recognition method, characterized in that: include: Acquire a target image containing a target object; Extract features of the target image using a regional fully convolutional network (R-FCN) algorithm to obtain an individual model corresponding to the target object; Based on the individual model, obtaining target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object through the R-FCN algorithm; Determining an expression score corresponding to the target expression recognition data, a posture score corresponding to the target posture recognition data, and an action score corresponding to the target action recognition data based on a reference emotion data set, the target expression recognition data, the target posture recognition data, and the target action recognition data, wherein the reference emotion data set is obtained through big data training; A target emotional state corresponding to the target object is determined based on the expression score, the posture score, and the action score.

2. The emotion recognition method according to claim 1, characterized in that: The feature extraction of the target image by using the regional fully convolutional network R-FCN algorithm to obtain the individual model corresponding to the target object includes: Perform feature extraction on the target image by using the R-FCN algorithm to obtain a first feature map; Generate a plurality of first candidate regions corresponding to the first feature map according to the first feature map by using the R-FCN algorithm; For each first candidate region among the multiple first candidate regions, construct a first position sensitive score map corresponding to each first candidate region according to the first candidate region by using the R-FCN algorithm, and add position information to the first candidate region; Performing average pooling processing on the first position sensitive score map by using the R-FCN algorithm to obtain a first mean value corresponding to the first candidate region; The first candidate region is classified according to the first mean value by using the R-FCN algorithm to obtain an individual model corresponding to the target object.

3. The emotion recognition method according to claim 2, characterized in that: The generating a plurality of first candidate regions corresponding to the first feature map according to the first feature map by using the R-FCN algorithm includes: According to the first feature map, a plurality of first candidate regions corresponding to the first feature map are generated by the region proposal network RPN in the R-FCN algorithm, and the first candidate regions correspond to candidate boxes of a preset number of sizes, and the candidate boxes of the preset number of sizes are obtained by learning the geometric size of the target box of the training data set based on the K-means clustering algorithm of the intersection-over-union ratio.

4. The emotion recognition method according to claim 3, characterized in that: The step of obtaining target expression recognition data, target posture recognition data, and target action recognition data corresponding to the target object through the R-FCN algorithm based on the individual model includes: Perform feature extraction on the individual model by using the R-FCN algorithm to obtain a second feature map; Generate a plurality of second candidate regions corresponding to the second feature map according to the second feature map by using the R-FCN algorithm; For each second candidate region among the multiple second candidate regions, construct a second position sensitive score map corresponding to each second candidate region according to the second candidate region by using the R-FCN algorithm, and add position information to the second candidate region; Performing average pooling processing on the second position sensitive score map by using the R-FCN algorithm to obtain a second mean corresponding to the second candidate region; The second candidate area is classified according to the second mean value by using the R-FCN algorithm to obtain target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object.

5. The emotion recognition method according to any one of claims 1 to 4, characterized in that: The reference emotion data set includes a reference expression data set, a reference posture data set, and a reference action data set; determining the expression score corresponding to the target expression recognition data, the posture score corresponding to the target posture recognition data, and the action score corresponding to the target action recognition data based on the reference emotion data set, the target expression recognition data, the target posture recognition data, and the target action recognition data, includes: Determining an expression score corresponding to the target expression recognition data based on the similarity between the target expression recognition data and the data in the reference expression data set; Determining a posture score corresponding to the target posture recognition data based on the similarity between the target posture recognition data and the data in the reference posture data set; Based on the similarity between the target action recognition data and the data in the reference action data set, an action score corresponding to the target action recognition data is determined.

6. The emotion recognition method according to any one of claims 1 to 4, characterized in that: The determining, based on the expression score, the posture score, and the action score, a target emotional state corresponding to the target object includes: Performing a weighted summation on the expression score, the posture score, and the action score to obtain a target emotion score; A target emotional state corresponding to the target object is determined according to the target emotional score.

7. An emotion recognition device, characterized in that: include: A first acquisition module, used for acquiring a target image containing a target object; An extraction module, used to extract features of the target image through a regional fully convolutional network (R-FCN) algorithm to obtain an individual model corresponding to the target object; A second acquisition module is used to acquire target expression recognition data, target posture recognition data and target action recognition data corresponding to the target object through the R-FCN algorithm based on the individual model; A first determination module is used to determine an expression score corresponding to the target expression recognition data, a posture score corresponding to the target posture recognition data, and an action score corresponding to the target action recognition data based on a reference emotion data set, the target expression recognition data, the target posture recognition data, and the target action recognition data, wherein the reference emotion data set is obtained through big data training; The second determination module is used to determine the target emotional state corresponding to the target object based on the expression score, the posture score and the action score.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the emotion recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the emotion recognition method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the emotion recognition method according to any one of claims 1 to 6 is implemented.

Images