A service evaluation method, device and storage device based on expression recognition

Abstract

The present invention provides a service evaluation method based on expression recognition, an apparatus and a storage device, which obtain a first expression image sequence and a second expression image sequence of a service object; input the first expression image sequence and the second expression image sequence into a trained convolutional neural network to obtain classification results of the first expression image sequence and the second expression image sequence respectively; obtain a corresponding emotion score value based on the classification results of the first expression image sequence and the second expression image sequence; calculate the difference between the score value of the first expression image sequence and the score value of the second expression image sequence, and evaluate the service according to the difference. The present invention can optimize the expression recognition effect for service evaluation in the government field, has a fast expression training speed and a high accuracy rate, and provides a scientific data basis for window service evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic government affairs, and in particular to a service evaluation method, device and storage device based on expression recognition. Background Art

[0002] With the continuous advancement of the "relax, regulate and serve" reform, in order to promote the improvement of the service quality of government service windows, it is necessary to use artificial intelligence technology to provide an evaluation reference for the services provided by staff.

[0003] However, previous facial expression recognition had problems such as insufficient standard data sets and models that needed to be improved, resulting in poor recognition accuracy. At the same time, the traditional seven-category facial expression recognition has obvious biases, which is not conducive to direct use for service evaluation. Therefore, it is necessary to innovate a method with high classification accuracy and suitable for service evaluation. Summary of the invention

[0004] The purpose of the present invention is to provide a service evaluation method, device and storage device based on expression recognition, so as to solve the above-mentioned problems existing in the prior art.

[0005] In order to achieve the above object, the present invention provides a service evaluation method based on expression recognition, comprising the following steps:

[0006] Acquire a first expression image sequence and a second expression image sequence of a service object;

[0007] Inputting the first expression image sequence and the second expression image sequence into a trained convolutional neural network to obtain classification results of the first expression image sequence and the second expression image sequence respectively;

[0008] Based on the classification results of the first expression image sequence and the second expression image sequence, obtaining corresponding emotion score values;

[0009] The difference between the score value of the first expression image sequence and the score value of the second expression image sequence is calculated, and the service is evaluated according to the difference.

[0010] Furthermore, the expression classification result of the convolutional neural network includes several emotion levels, and the emotion levels are used to evaluate the negative and / or positive emotions of the service recipients.

[0011] Furthermore, it also includes:

[0012] Setting a score value of the emotion level according to the emotion level;

[0013] The difference between the scores of the first expression image sequence and the second expression image sequence is calculated according to the score of the emotion level.

[0014] Furthermore, it also includes:

[0015] When the difference is greater than a first threshold, the service evaluation is very satisfactory;

[0016] When the difference is less than or equal to the first threshold and greater than the second threshold, the service evaluation is relatively satisfactory;

[0017] When the difference is less than or equal to the second threshold and greater than the third threshold, the service evaluation is average;

[0018] When the difference is less than or equal to a third threshold and greater than a fourth threshold, the service evaluation is relatively unsatisfactory;

[0019] When the difference is less than or equal to a fourth threshold, the service evaluation is very unsatisfactory.

[0020] Furthermore, the training process of the convolutional neural network includes:

[0021] Use generative adversarial networks to process the images in the dataset, improve the resolution and annotation quality, and expand the expression category labels from seven to eight.

[0022] Inputting a facial expression data set into a convolutional neural network for training, wherein the convolutional neural network selects features of the expression according to the classification labels;

[0023] The extreme learning machine is used to train the deep learning classifier to complete the classification of facial expression data.

[0024] Furthermore, the pooling layer of the convolutional neural network adopts a random pooling algorithm to perform random sampling calculations on feature points.

[0025] On the other hand, the present invention also provides a service evaluation device based on expression recognition, comprising:

[0026] An image acquisition module, used to acquire a first expression image sequence and a second expression image sequence of a service object;

[0027] An image scoring module, for classifying and identifying the first expression image sequence and the second expression image sequence using a trained convolutional neural network to obtain corresponding emotion level score values;

[0028] The image comparison module is used to calculate the difference between the score value of the first expression image sequence and the score value of the second expression image sequence, and evaluate the service according to the difference.

[0029] On the other hand, the present invention further provides a storage device, wherein the storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the steps in the above-mentioned service evaluation method based on expression recognition.

[0030] The present invention provides a service evaluation method based on expression recognition, an apparatus and a storage device, which obtain a first expression image sequence and a second expression image sequence of a service object; input the first expression image sequence and the second expression image sequence into a trained convolutional neural network to obtain classification results of the first expression image sequence and the second expression image sequence respectively; obtain a corresponding emotion score value based on the classification results of the first expression image sequence and the second expression image sequence; calculate the difference between the score value of the first expression image sequence and the score value of the second expression image sequence, and evaluate the service according to the difference. The present invention can optimize the expression recognition effect for service evaluation in the government field, has a fast expression training speed and a high accuracy rate, and provides a scientific data basis for window service evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 The present invention is a flowchart of a method for evaluating a service based on expression recognition according to an embodiment of the present invention.

[0033] Figure 2 This is a network structure diagram of a convolutional neural network according to an embodiment of the present invention.

[0034] Figure 3 The present invention discloses a service evaluation device based on expression recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0039] The following describes a service evaluation method based on expression recognition according to an embodiment of the present invention, an apparatus and a storage device with reference to the accompanying drawings. First, a service evaluation method based on expression recognition according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0040] Figure 1 FIG. 1 is a flow chart of a method for evaluating a service based on expression recognition according to an embodiment of the present invention. Figure 1 As shown, the evaluation method includes the following steps:

[0041] Step S1 obtains a first expression image sequence and a second expression image sequence of a service object.

[0042] Among them, the service object refers to a real person in real life. In some embodiments, the scene where the person is located is a service window, such as a government service window, a bank service window, etc. The person can be a man, a woman, an old man, a child, etc., to receive window service.

[0043] Those skilled in the art should understand that the first expression image sequence and the second expression image sequence can be acquired using any device in the prior art, and as long as the target expression image can be acquired, it should be within the protection scope of the present invention, regardless of the specific acquisition device used, such as a camera, a still camera, a mobile phone, an infrared thermal imaging device, a sensor, etc.

[0044] In some embodiments, the acquisition of the first expression image sequence and the second expression image sequence can be achieved through a camera, and multiple expression images are generated by capturing the facial expressions of the service objects. In order to eliminate the impact of different facial features on the scores of different objects, three expression images can be selected. The expression images can be acquired by video screenshots or directly generated by photos.

[0045] Step S2 inputs the first expression image sequence and the second expression image sequence into a trained convolutional neural network to obtain classification results of the first expression image sequence and the second expression image sequence respectively.

[0046] Those skilled in the art should understand that convolutional neural network, as a type of artificial neural network, is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system to adjust the interconnected relationship between a large number of internal nodes to achieve the purpose of processing information. As long as the connection between the sampling parameter information and the attribute parameter information of the preset model object can be established after model training, it should be within the scope of protection of the present invention.

[0047] Those skilled in the art should understand that the training data set used when training the model is a facial expression data set, which may include test images, public verification images, and private verification images. The facial expression data set has multiple expression categories, and each expression category corresponds to a corresponding expression label.

[0048] In some embodiments, the present invention uses the FER2013 facial expression dataset as a training dataset. In order to improve the expression dataset, a generative adversarial network is used to classify and annotate the FER2013 facial expression dataset, and the expression labels are classified into eight labels: anger, disgust, fear, sadness, surprise, neutral, laughter and smile.

[0049] Specifically, the objective function of the generative adversarial network is:

[0050]

[0051] Among them, x: expression picture, z: noise input to the generator, G(z): picture generated by the generator, D(x): probability that the judger judges whether the expression picture is a classification label, D(x): probability that the judger judges whether the expression picture is a classification label, D(G(x)): probability that the judger judges whether the picture generated by the generator is a classification label.

[0052] In the above objective function, the purpose of the judge is to find D with the maximum objective function. The stronger the ability of D, the larger D(x), the smaller D(G(x)), and the larger V(D,G), so it is D that is the maximum objective function. The purpose of the generator G: The goal is to make the generated image as close to the real image as possible, that is, G when V(D,G) is the smallest.

[0053] According to the above objective function, we can see that the solution of the objective function is divided into two steps:

[0054] (1) First fix the generator G and obtain the discriminator D by maximizing the objective function;

[0055] (2) Fix the discriminator D and find the generator G by minimizing the objective function.

[0056] After such an iterative cycle, we finally obtain the generator G that maximizes the objective function, so that the judgement machine D cannot detect whether the judgment result of the generator G is true or false, thereby classifying and labeling the expression labels.

[0057] In some embodiments, when using a convolutional neural network for model training, the expression features are first extracted according to the classification of the training data set, and then an extreme learning machine is used as a linear classifier of the convolutional neural network to complete the classification of the facial expression data. Among them, the pooling layer of the convolutional neural network uses a random pooling algorithm to perform random sampling calculations on feature points.

[0058] Figure 2 This is a network structure diagram of a convolutional neural network according to an embodiment of the present invention.

[0059] Specifically, Figure 2 As shown in the figure, the convolutional neural network used in the present invention is formed by continuously stacking and accumulating convolutional layers, pooling layers, normalization layers and fully connected layers, and finally using a classifier to achieve classification. In the classifier, C1 and C2 indicate that the first and third layers are convolutional layers, S1 and S2 indicate that the second and fourth layers are pooling layers, and FC indicates that the fifth layer is a fully connected layer.

[0060] Further, in Figure 1 In , C1 and C2 are both convolutional layers. Generally, the first convolutional layer input image is directly connected. Assume that the input is the expression image matrix X of n×n, the convolution kernel size is the matrix K of m×m, and the convolution kernel moving step F is 1. Then the size of the feature map after convolution calculation is (n-m+1)×(n-m+1). The convolution calculation formula can be expressed as:

[0061]

[0062] in, Represents the feature map of the previous layer output, is the bias of the jth feature map of layer l, f(x) is the activation function. Usually, the value calculated by convolution will be further processed by the activation function. The activation function is the ReLU function, and the expression is:

[0063] f(x)=max(0,X)

[0064] In some embodiments of the present invention, the pooling layer adopts a random pooling algorithm. The random pooling algorithm uses the weight of each element in the pooling domain as the probability for probability sampling. The calculation method is:

[0065]

[0066]

[0067] Furthermore, the present invention adopts an extreme learning machine as a classifier of a convolutional neural network, determines the correlation between the output of the hidden layer unit and the classification result by calculating the information gain, and determines the number of hidden layer units by setting a threshold through the Otsu method.

[0068] Assume there are m categories t(t1,t2,…,t m ), hidden units, then the information brought by each hidden unit to the classification is:

[0069]

[0070] Among them, p(t i ) is the classification probability of each category; d represents the number of hidden layer unit output area divisions (because the output is a continuous value from 0 to 1, it is necessary to divide the output into segmented areas); p(h k,l ) is the probability that the output of the kth hidden unit belongs to the lth segment division area; p(t i |h k,l ) indicates that the hidden unit state is h k,l When , the classification result is t i probability.

[0071] Step S3 obtains corresponding emotion score values ​​based on the classification results of the first expression image sequence and the second expression image sequence.

[0072] The expression classification result of the convolutional neural network includes several emotion levels, which are used to evaluate the negative and / or positive emotions of the service object. Specifically, after obtaining the classification results of the first expression image sequence and the second expression image sequence, the classification labels of the first expression image sequence and the second expression image sequence are obtained. The classification labels are divided into 8 emotion levels from low to high according to negative emotions to positive emotions, and each emotion level has a corresponding score. Among them, anger is -25 points, disgust is -20 points, fear is -15 points, sadness is -10 points, surprise is -5 points, neutral is 0 points, laughter is 10 points and smile is 20 points.

[0073] Step S4 calculates the difference between the score of the first expression image sequence and the score of the second expression image sequence, and evaluates the service according to the difference.

[0074] Among them, when the difference is greater than the first threshold, the service evaluation is very satisfactory; when the difference is less than or equal to the first threshold and greater than the second threshold, the service evaluation is relatively satisfactory; when the difference is less than or equal to the second threshold and greater than the third threshold, the service evaluation is average; when the difference is less than or equal to the third threshold and greater than the fourth threshold, the service evaluation is relatively unsatisfactory; when the difference is less than or equal to the fourth threshold, the service evaluation is very unsatisfactory.

[0075] In some embodiments of the present invention, if the difference is positive and the absolute value exceeds 20 points, the service evaluation is very satisfactory; if it is positive and the absolute value exceeds 10 points, the service evaluation is relatively satisfactory; if the difference is positive and the absolute value exceeds 10 points, the service evaluation is relatively satisfactory; if the difference is negative and the absolute value exceeds 20 points, the service evaluation is very unsatisfactory. If the absolute value of the difference is within 10, the service evaluation is average.

[0076] like Figure 3 As shown, the present invention also provides a service evaluation device based on expression recognition, comprising:

[0077] An image acquisition module 101 is used to acquire a first expression image sequence and a second expression image sequence of a service object;

[0078] An image scoring module 102 is used to classify and identify the first expression image sequence and the second expression image sequence using a trained convolutional neural network to obtain corresponding emotion level score values;

[0079] The image comparison module 103 is used to calculate the difference between the score value of the first expression image sequence and the score value of the second expression image sequence, and evaluate the service according to the difference.

[0080] Specifically, an embodiment of the present invention further proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured as the above method.

[0081] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0082] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0083] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0084] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0085] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0086] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A service evaluation method based on expression recognition, characterized in that: include: Acquire a first expression image sequence and a second expression image sequence of a service object; Among them, the FER2013 facial expression dataset is used as the training dataset, and the generative adversarial network is used to classify and annotate the FER2013 facial expression dataset, and the expression labels are classified into eight labels: anger, disgust, fear, sadness, surprise, neutral, laughter and smile. The objective function of the generative adversarial network is: Where x is the expression picture, z is the noise input to the generator, G(z): the picture generated by the generator, D(x): the probability that the judger judges whether the expression picture is a classification label, D(x): the probability that the judger judges whether the expression picture is a classification label, D(G(x)): the probability that the judger judges whether the picture generated by the generator is a classification label; In the above objective function, the purpose of the judge is to find D with the maximum objective function; the stronger the ability of D, the larger D(x), the smaller D(G(x)), and the larger V(D, G), so it is D when the objective function is maximized. The purpose of the generator G is to make the generated image as close to the real image as possible, that is, G when V(D, G) is the smallest. According to the above objective function, the solution of the objective function is divided into two steps: (1) First fix the generator G and obtain the discriminator D by maximizing the objective function; (2) Fix the judgement D and find the generator G by minimizing the objective function; This iterative cycle finally obtains the generator G that maximizes the objective function, so that the judgement D cannot detect whether the judgment result of the generator G is true or false, thereby classifying and labeling the expression labels; Inputting the first expression image sequence and the second expression image sequence into a trained convolutional neural network, respectively obtaining classification results of the first expression image sequence and the second expression image sequence, wherein the expression classification results of the convolutional neural network include a plurality of emotion levels, and the emotion levels are used to evaluate the negative and / or positive emotions of the service object; Among them, the convolutional neural network is formed by continuously stacking and accumulating convolutional layers, pooling layers, normalization layers and fully connected layers, and finally using a classifier to achieve classification. Among them, C1 and C2 indicate that the first and third layers are convolutional layers, S1 and S2 indicate that the second and fourth layers are pooling layers, and FC indicates that the fifth layer is a fully connected layer. Assume that the input is an n×n expression image matrix X, a convolution kernel size is an m×m matrix K, and a convolution kernel moving step F is 1. The size of the feature map after convolution calculation is (n-m+1)×(n-m+1). The convolution calculation formula can be expressed as: in, Represents the feature map of the previous layer output, is the bias of the jth feature map of layer l, f(x) is the activation function. Usually, the value calculated by convolution will be further processed by the activation function. The activation function is the ReLU function, and the expression is: f(x)=max(0,X) The pooling layer uses a random pooling algorithm. The random pooling algorithm uses the weight of each element in the pooling domain as the probability for probability sampling. The calculation method is: The extreme learning machine is used as the classifier of the convolutional neural network. The correlation between the output of the hidden layer unit and the classification result is determined by calculating the information gain method, and the threshold is set by the Otsu method to determine the number of hidden layer units. Assume there are m categories t(t1,t2,…,t m ), hidden units, then the information brought by each hidden unit to the classification is: Among them, p(t i ) is the classification probability of each category; d represents the number of hidden unit output area divisions. Since the output is a continuous value from 0 to 1, it is necessary to divide the output into segmented areas; p(h k,l ) is the probability that the output of the kth hidden unit belongs to the lth segment division area; p(t i |h k,l ) indicates that the hidden unit state is h k,l When , the classification result is t i probability; Based on the classification results of the first expression image sequence and the second expression image sequence, setting a score value of the emotion level according to the emotion level; The difference between the score value of the first expression image sequence and the score value of the second expression image sequence is calculated according to the emotion level, and the service is evaluated according to the difference.

2. A service evaluation method based on expression recognition as claimed in claim 1, characterized in that: Also includes: When the difference is greater than a first threshold, the service evaluation is very satisfactory; When the difference is less than or equal to the first threshold and greater than the second threshold, the service evaluation is relatively satisfactory; When the difference is less than or equal to the second threshold and greater than the third threshold, the service evaluation is average; When the difference is less than or equal to a third threshold and greater than a fourth threshold, the service evaluation is relatively unsatisfactory; When the difference is less than or equal to a fourth threshold, the service evaluation is very unsatisfactory.

3. A service evaluation method based on expression recognition as claimed in claim 1, characterized in that: The training process of the convolutional neural network includes: Use generative adversarial networks to process the images in the dataset, improve the resolution and annotation quality, and expand the expression category labels from seven to eight. Inputting a facial expression data set into a convolutional neural network for training, wherein the convolutional neural network selects features of the expression according to the classification labels; The extreme learning machine is used to train the deep learning classifier to complete the classification of facial expression data.

4. A service evaluation method based on expression recognition as claimed in claim 3, characterized in that: The pooling layer of the convolutional neural network adopts a random pooling algorithm to perform random sampling calculations on feature points.

5. A service evaluation device based on expression recognition, characterized in that: include: An image acquisition module is used to acquire a first expression image sequence and a second expression image sequence of a service object, and acquire a first expression image sequence and a second expression image sequence of a service object; Among them, the FER2013 facial expression dataset is used as the training dataset, and the generative adversarial network is used to classify and annotate the FER2013 facial expression dataset, and the expression labels are classified into eight labels: anger, disgust, fear, sadness, surprise, neutral, laughter and smile. The objective function of the generative adversarial network is: Where x is the expression picture, z is the noise input to the generator, G(z): the picture generated by the generator, D(x): the probability that the judger judges whether the expression picture is a classification label, D(x): the probability that the judger judges whether the expression picture is a classification label, D(G(x)): the probability that the judger judges whether the picture generated by the generator is a classification label; In the above objective function, the purpose of the judge is to find D with the maximum objective function; the stronger the ability of D, the larger D(x), the smaller D(G(x)), and the larger V(D, G), so it is D when the objective function is maximized. The purpose of the generator G is to make the generated image as close to the real image as possible, that is, G when V(D, G) is the smallest. According to the above objective function, the solution of the objective function is divided into two steps: (1) First fix the generator G and obtain the discriminator D by maximizing the objective function; (2) Fix the judgement D and find the generator G by minimizing the objective function; This iterative cycle finally obtains the generator G that maximizes the objective function, so that the judgement D cannot detect whether the judgment result of the generator G is true or false, thereby classifying and labeling the expression labels; Inputting the first expression image sequence and the second expression image sequence into a trained convolutional neural network, respectively obtaining classification results of the first expression image sequence and the second expression image sequence, wherein the expression classification results of the convolutional neural network include a plurality of emotion levels, and the emotion levels are used to evaluate the negative and / or positive emotions of the service object; Among them, the convolutional neural network is formed by continuously stacking and accumulating convolutional layers, pooling layers, normalization layers and fully connected layers, and finally using a classifier to achieve classification. Among them, C1 and C2 indicate that the first and third layers are convolutional layers, S1 and S2 indicate that the second and fourth layers are pooling layers, and FC indicates that the fifth layer is a fully connected layer. Assume that the input is an n×n expression image matrix X, a convolution kernel size is an m×m matrix K, and a convolution kernel moving step F is 1. The size of the feature map after convolution calculation is (n-m+1)×(n-m+1). The convolution calculation formula can be expressed as: in, Represents the feature map of the previous layer output, is the bias of the jth feature map of layer l, f(x) is the activation function. Usually, the value calculated by convolution will be further processed by the activation function. The activation function is the ReLU function, and the expression is: f(x)=max(0,X) The pooling layer uses a random pooling algorithm. The random pooling algorithm uses the weight of each element in the pooling domain as the probability for probability sampling. The calculation method is: The extreme learning machine is used as the classifier of the convolutional neural network. The correlation between the output of the hidden layer unit and the classification result is determined by calculating the information gain method, and the threshold is set by the Otsu method to determine the number of hidden layer units. Assume there are m categories t(t1,t2,…,t m ), hidden units, then the information brought by each hidden unit to the classification is: Among them, p(t i ) is the classification probability of each category; d represents the number of hidden unit output area divisions. Since the output is a continuous value from 0 to 1, it is necessary to divide the output into segmented areas; p(h k,l ) is the probability that the output of the kth hidden unit belongs to the lth segment division area; p(t i |h k,l ) indicates that the hidden unit state is h k,l When , the classification result is t i The probability of An image scoring module, for classifying and identifying the first expression image sequence and the second expression image sequence using a trained convolutional neural network to obtain corresponding emotion level score values; The image comparison module is used to calculate the difference between the score value of the first expression image sequence and the score value of the second expression image sequence, and evaluate the service according to the difference.

6. A storage device, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the service evaluation method based on expression recognition as described in any one of claims 1 to 4.

Images