User expression recognition method and device, computer device, and storage medium

Abstract

Embodiments of the present application disclose a user expression recognition method and device, computer equipment and a storage medium. The method comprises: obtaining image data to be recognized; inputting the image data to be recognized into a recognition model for user expression recognition to obtain a recognition result; and outputting the recognition result; wherein the recognition model is obtained by training a twin network after labeling an image containing user expression data. By implementing the method of the embodiments of the present application, a small amount of accurate labeled data can be used to effectively learn a large amount of unlabeled visual image data, form a recognition model, and accurately recognize user emotions.

Description

Technical Field

[0001] This invention relates to facial expression recognition methods, and more specifically to user facial expression recognition methods, devices, computer equipment, and storage media. Background Technology

[0002] With the continuous development of AI (Artificial Intelligence) technology, intelligent digital humans are emerging in more and more scenarios, various virtual anchors are springing up, and visual interaction is gradually becoming the main way people communicate. In the process of visual communication, how virtual digital humans can accurately identify the user's emotions and make more timely and subtle responses is one of the key challenges in the overall intelligence of intelligent digital humans.

[0003] While there is a vast amount of visual information data, finely labeled user emotion data is extremely scarce. This results in a severe shortage of supervised user emotion recognition data resources, making it impossible to accurately identify users' true emotions in actual production.

[0004] Therefore, it is necessary to design a new method that can effectively learn from massive amounts of unlabeled visual image data using a small amount of precisely labeled data to form a recognition model for accurately identifying user emotions. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a user facial expression recognition method, device, computer equipment, and storage medium.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a user facial expression recognition method, comprising:

[0007] Acquire the image data to be recognized;

[0008] The image data to be recognized is input into the recognition model to perform user facial expression recognition in order to obtain the recognition result;

[0009] Output the recognition result;

[0010] The recognition model is obtained by training a Siamese network after annotating images containing user facial expression data.

[0011] The further technical solution is as follows: the recognition model is obtained by training a Siamese network after annotating images containing user facial expression data, including:

[0012] Collect images containing user facial expression data to obtain facial expression image data;

[0013] A small sample of user facial expressions is annotated on the facial expression image data to obtain an annotated dataset;

[0014] The labeled dataset is input into a Siamese network for image feature extraction to obtain user facial expression features;

[0015] The user's facial expression features are input into a twin network to classify the user's facial expression categories, thereby obtaining the user's facial expressions;

[0016] Calculate the loss value of the user's facial expression and the actual user's facial expression in the facial expression image data;

[0017] Determine whether the loss value tends to stabilize;

[0018] If the loss value does not tend to be stable, the parameters of the Siamese network are adjusted, and the labeled dataset is input into the Siamese network to extract image features to obtain user expression features.

[0019] If the loss value tends to stabilize, then the Siamese network is determined to be the recognition model.

[0020] The further technical solution is as follows: the acquisition of images containing user facial expression data to obtain facial expression image data includes:

[0021] Obtain the raw visual image;

[0022] The original visual image is divided into frames to obtain several frames of images;

[0023] For each frame of the image, an image person detection algorithm is used to detect the front or side view of the face in order to obtain facial expression image data.

[0024] A further technical solution is as follows: The expression image data is annotated with a small sample of user expressions to obtain an annotated dataset, including:

[0025] The facial expression image data is labeled with actual user facial expressions and data augmentation processing is performed to obtain a labeled dataset.

[0026] The further technical solution is as follows: the annotation of the facial expression image data with actual user facial expressions and the subsequent data augmentation processing to obtain an annotated dataset include:

[0027] The facial expression image data is annotated with actual user facial expressions to obtain annotated image data;

[0028] Based on the source information of the labeled image data, the image information of adjacent frames is searched, and the searched images and the labeled image data are converted into feature vectors based on the HOG feature information.

[0029] Cosine similarity is used to calculate the similarity between feature vectors, and images that meet the similarity requirements are assigned the same label information to obtain labeled image data;

[0030] The labeled image data is subjected to image rotation, image noise addition, image subtraction, and image compression to obtain the labeled dataset.

[0031] The further technical solution is as follows: The step of inputting the labeled dataset into a Siamese network for image feature extraction to obtain user facial expression features includes:

[0032] The labeled dataset is then subjected to grayscale and normalization processing to obtain standard images.

[0033] The standard image is used to extract local features by employing a CNN convolutional network within a Siamese network to obtain intermediate features of the local features;

[0034] A recurrent neural network within a Siamese network is used to extract global features from the intermediate features of the local features to obtain global features;

[0035] The attention mechanism within the Siamese network is used to extract high-dimensional features from the global features to obtain user expression features.

[0036] The further technical solution is as follows: The step of inputting the user's facial expression features into a twin network to classify the user's facial expression categories, in order to obtain the user's facial expressions, includes:

[0037] The user facial expression features are completed according to the category of user facial expression to form a training dataset;

[0038] The Siamese network is used to perform feature differentiation extraction on every two features in the training dataset to determine whether each two features belong to the same category, so as to obtain the user's expression; wherein, one of the two features in the training dataset is a supporting sample in the user's expression category.

[0039] The present invention also provides a user facial expression recognition device, comprising:

[0040] The data acquisition unit is used to acquire the image data to be recognized;

[0041] The recognition unit is used to input the image data to be recognized into the recognition model to perform user expression recognition in order to obtain the recognition result;

[0042] An output unit is used to output the recognition result;

[0043] Also includes:

[0044] The recognition model generation unit is used to train a Siamese network by annotating images containing user facial expression data to form a recognition model.

[0045] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0046] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0047] The beneficial effects of this invention compared with the prior art are as follows: This invention acquires image data to be identified and inputs the image data to be identified into a recognition model to perform user expression recognition. The recognition model is obtained by training a Siamese network after annotating images containing user expression data. The training process of the recognition model involves user expression image data acquisition, image small sample annotation set processing, image feature extraction, and user expression small sample classification learning. This enables the effective learning of massive amounts of unlabeled visual image data using a small amount of accurately labeled data to form a recognition model for accurately identifying user emotions.

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram illustrating an application scenario of the user facial expression recognition method provided in this embodiment of the invention;

[0051] Figure 2 This is a flowchart illustrating the user facial expression recognition method provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of a sub-process of the user facial expression recognition method provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of a sub-process of the user facial expression recognition method provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of a sub-process of the user facial expression recognition method provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of a sub-process of the user facial expression recognition method provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of a sub-process of the user facial expression recognition method provided in an embodiment of the present invention;

[0057] Figure 8 This is a schematic block diagram of a user facial expression recognition device provided in an embodiment of the present invention;

[0058] Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0061] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0062] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0063] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the user facial expression recognition method provided in an embodiment of the present invention. Figure 2This is a schematic flowchart illustrating the user expression recognition method provided in an embodiment of the present invention. The user expression recognition method is applied in a server. The server interacts with the terminal, receiving image data to be recognized from the terminal, performing user expression recognition on the input image data, and outputting the recognition result. This recognition model is obtained by training a Siamese network on images containing user expression data after enhancement processing and annotation. This allows for effective learning from massive amounts of unlabeled visual image data using only a small amount of precisely labeled data to form a recognition model for accurately recognizing user emotions.

[0064] Figure 2 This is a flowchart illustrating the user facial expression recognition method provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S130.

[0065] S110. Obtain the image data to be recognized.

[0066] In this embodiment, the image data to be recognized refers to the user visual image information for which user expression recognition is required. It can be an image or an image formed after visual video segmentation.

[0067] S120. Input the image data to be recognized into the recognition model to perform user expression recognition and obtain the recognition result.

[0068] In this embodiment, the recognition result refers to the user's emotion recognition result, that is, the user's facial expression.

[0069] The recognition model is obtained by training a Siamese network after annotating images containing user facial expression data.

[0070] In one embodiment, please refer to Figure 3 The recognition model is obtained by training a Siamese network after annotating images containing user facial expression data, and may include steps S121 to S128.

[0071] S121. Collect images containing user facial expression data to obtain facial expression image data.

[0072] In this embodiment, facial expression image data refers to a set of image data containing the front or side view of a human face.

[0073] In one embodiment, please refer to Figure 4 The above step S121 may include steps S1211 to S1212.

[0074] S1211. Obtain the original visual image.

[0075] In this embodiment, the original visual image refers to an image with rich depiction of human facial expressions.

[0076] For user facial expression recognition, the first step is to collect and organize images containing user facial expression data. In actual production, although there are a large number of visual images or videos, most of the visual and image data can be used for user facial expression recognition. A relatively coarse algorithm is needed to filter this data. Generally, movie or speech videos or images have rich depictions of human facial expressions; while other video data, such as financial data explanations, educational videos, and travel guides, have very few depictions of human facial expressions. Therefore, the raw visual image data used is extracted from movie and speech videos.

[0077] S1212. Perform frame segmentation on the original visual image to obtain several frames of image data.

[0078] In this embodiment, several frames of images refer to the collection of each frame of images formed after the original visual image is processed into frames.

[0079] Specifically, tools such as FFmpeg are used to divide these movie or speech videos into frames according to the screen. Since movie and speech videos have N frames per second (usually 24), and facial expression shots usually last for at least 2 to 3 seconds, in order to avoid a large amount of repeated image data, the video is divided into frames, and M frames (M≤N, usually 4) of the video are used at equal intervals per second.

[0080] S1213. For each frame of the image, use an image person detection algorithm to detect the front or side of the face to obtain expression image data.

[0081] In this embodiment, a coarse and fast image person detection algorithm is used for each frame of the image to detect whether each frame contains a frontal or side view of a person. The HOG (Histogram of Oriented Gradient) and SVM (Support Vector Machine) combination is used to detect faces. HOG mainly extracts gradient features from the image, while SVM performs fast classification based on the extracted features. Currently, HOG+SVM, as a classic algorithm, has been integrated into the OpenCV visual image open tool library, so the relevant algorithm interface module can be directly used for coarse classification and identification.

[0082] After the above processing, a large set of image data containing the front or side views of a human face can be established. This data is the raw data for facial expression recognition.

[0083] S122. Perform small-sample user expression annotation on the expression image data to obtain an annotation dataset.

[0084] In this embodiment, the labeled dataset refers to the labels that annotate the facial expression image data with user facial expression categories.

[0085] Specifically, the facial expression image data is labeled with actual user facial expressions and data augmentation processing is performed to obtain a labeled dataset.

[0086] In one embodiment, please refer to Figure 5 The above step S122 may include steps S1221 to S1224.

[0087] S1221. The facial expression image data is annotated with the actual user's facial expressions to obtain annotated image data.

[0088] In this embodiment, labeled image data refers to images formed after labeling facial expression image data with the actual user's facial expression category.

[0089] S1222. Search for adjacent frame image information based on the source information of the labeled image data, and convert the searched images and the labeled image data into feature vectors based on HOG feature information.

[0090] In this embodiment, the feature vector refers to the feature vector formed by finding adjacent frame image information based on the source information of the labeled image data, and converting the found image with the labeled image data according to the HOG feature information.

[0091] S1223. Calculate the similarity between feature vectors using Cosine similarity, and assign the same label information to images that meet the similarity requirements to obtain labeled image data.

[0092] In this embodiment, labeled image data refers to image data formed by assigning the same label to images whose feature vectors meet the similarity requirements.

[0093] S1224. Perform image rotation, image noise addition, image subtraction, and image compression on the labeled image data to obtain the labeled dataset.

[0094] Specifically, a large number of images containing human facial expressions were acquired. Based on the source information of the original images (movie or video source, time, etc.), representative image information was specifically selected for user expression category labeling, mainly including categories such as laughing, smiling, crying, frowning, and fear. Since the labeled data cannot be too large, small sample augmentation techniques are needed to enhance the data. The augmentation process mainly includes the following methods: First, automatic source tracing to expand the sample label set. Based on the source information of the labeled image data (movie or video source, time, etc.), adjacent frame image information is searched, and the images are converted into feature vectors based on HOG feature information. The Cosine similarity algorithm is used to calculate the similarity between the labeled image data and the adjacent frame image information. If the similarity meets a certain threshold, these images are also labeled with the same information, thereby automatically expanding the labeled image data set. Second, all the labeled image data obtained above undergoes a series of processing operations such as image rotation, image noise addition, image subtraction, and image compression using multiple angle or size parameters, thereby obtaining a larger labeled data set.

[0095] S123. Input the labeled dataset into the Siamese network to extract image features to obtain user expression features.

[0096] In this embodiment, user facial expression features refer to the results obtained after extracting image features from the labeled dataset using feature extraction techniques based on attention and long short-term memory models.

[0097] Having acquired a substantial amount of relevant image data, the next step is to extract features from these labeled datasets for user facial expression recognition. To extract image features effectively and in a timely manner, a feature extraction technique based on attention and long short-term memory mechanisms was employed.

[0098] In one embodiment, please refer to Figure 6 The above step S123 may include steps S1231 to S1234.

[0099] S1231. Perform grayscale and normalization processing on the labeled dataset to obtain a standard image.

[0100] In this embodiment, a standard image refers to an image formed after grayscale and normalization processing of the labeled dataset.

[0101] Specifically, the labeled dataset is binarized into grayscale images, and all images are standardized and scaled to the standard size. Finally, each image is standardized into a standard image of H*W, where H*W represents the height of H pixels and the width of W pixels.

[0102] S1232. The standard image is extracted using a CNN convolutional network within a Siamese network to obtain intermediate features of the local features.

[0103] In this embodiment, the intermediate features of local features refer to the results obtained by the CNN convolutional network extracting local features of the standard image.

[0104] Specifically, CNN (Convolutional Neural Networks) is used for local feature extraction of the image. The binarized standard image is subjected to CNN convolutional feature extraction using K feature convolutional blocks to obtain local features within the image, thus ultimately transforming the H*W standardized image into h*w local feature intermediate features.

[0105] S1233. Use a recurrent neural network within a Siamese network to extract global features from the intermediate features of the local features to obtain global features.

[0106] In this embodiment, global features refer to the result obtained by the recurrent neural network extracting global features from the intermediate features of the local features.

[0107] Specifically, a variant of the Recurrent Neural Network (RNN), LSTM (Long Short-Term Memory), is used to encode the image, improving the representation ability of image features. Specifically, the intermediate image of the aforementioned local features is regarded as a time series vector with a time length of h and w feature points per time series, and is input into the variant LSTM network. After feature extraction and calculation by the LSTM multi-layer network, a total of h+1 time series points are output, with each time series outputting a w-dimensional feature data. LSTM can comprehensively associate the local features of CNN to obtain global features.

[0108] S1234. Use the attention mechanism within the Siamese network to extract high-dimensional features from the global features to obtain user expression features.

[0109] In this embodiment, user facial expression features refer to high-dimensional features extracted from the global features by the attention mechanism.

[0110] Specifically, attention mechanisms are widely used in the image processing domain. Drawing on human attention mechanisms, they play a crucial role in the process of building image feature extraction models, further extracting image information into effective information and learning the importance of different local features. Mathematically, attention mechanisms can be understood as weighted summation, typically using the Softmax form. By employing a multi-attention mechanism, the output of the RNN is ultimately transformed into a one-dimensional vector of length F, thus completing the representation of user facial expression features in the image, i.e., a K-dimensional vector.

[0111] S124. Input the user's facial expression features into the twin network to classify the user's facial expression categories, so as to obtain the user's facial expressions.

[0112] In this embodiment, user facial expressions refer to the user facial expression categories corresponding to the user facial expression features identified by the twin network.

[0113] User facial expression features were extracted from each labeled data set. However, due to the limited overall labeled data, even after small sample data augmentation, the number of labeled data remained small. Directly classifying facial expressions based on the K-dimensional vector features still resulted in insufficient training and low recognition accuracy. Therefore, a supervised training method using a Siamese Network was employed to train the Siamese Network and form a recognition model.

[0114] In one embodiment, please refer to Figure 7 The above step S124 may include steps S1241 to S1242.

[0115] S1241. The user facial expression features are completed according to the category of user facial expression to form a training dataset.

[0116] In this embodiment, the training dataset refers to the feature set formed after the user's facial expression features are completed according to the category of the user's facial expression.

[0117] Specifically, the training set is constructed according to C categories of user expressions, such as laughing, smiling, serious, and panicked, with N sample data for each category, for a total of (C*N) sample data. If the labeled data, i.e., a certain category of the user expression features, is less than N, the data is supplemented to N by repeated sampling.

[0118] S1242. The Siamese network is used to perform feature differential extraction on every two features in the training dataset to determine whether every two features belong to the same category, so as to obtain the user's expression; wherein, one of the two features in the training dataset is a supporting sample in the user's expression category.

[0119] Specifically, during each iteration of the recognition model's training, two categories are randomly selected from C categories, and one image is randomly selected from each of the K images in the two categories as input to the Siamese recognition network. The Siamese network is a dual-path neural network. During training, different pairs of samples are constructed through combination and input into the network. At the top layer, the distance between the sample pairs determines whether the two inputs belong to the same class, generating a corresponding probability distribution, thereby achieving the recognition of differences between different categories.

[0120] Specifically, the input to the Siamese network is the original images, i.e., the labeled dataset. The feature extraction process for the two images in the entire Siamese network is the K-dimensional vector obtained after feature extraction. That is, the feature extraction network structure is the feature extraction module of the entire Siamese network. Next, a module for determining whether the two images have the same features is constructed. Using the K-dimensional vectors of the two images, feature difference extraction is performed, mainly calculating the feature difference and feature variance. Then, these are concatenated into a 2*K difference feature vector. A Softmax classification function is nested on this 2*K one-dimensional vector to train and predict whether they belong to the same category. Finally, the parameters are updated iteratively through backpropagation of the neural network by comparing the difference between the predicted value (whether they belong to the same category) and the actual labeled value (whether they belong to the same category).

[0121] When predicting and recognizing user expressions on a brand-new image (the image data to be recognized), a Siamese network is used to process the image data to be recognized and form a total of C*M image input pairs with M support samples from C categories. The probability value of each input pair being of the same category is calculated, thereby obtaining the weighted support probability of each category. Finally, the category with the highest probability on the support set is selected as the prediction probability, thus completing the user emotion recognition.

[0122] S125. Calculate the loss value of the actual user expression and the expression image data;

[0123] S126. Determine whether the loss value tends to stabilize;

[0124] S127. If the loss value does not tend to stabilize, adjust the parameters of the Siamese network and execute step S123.

[0125] S128. If the loss value tends to stabilize, then the Siamese network is determined to be the recognition model.

[0126] During the training process, if the calculated loss value of the user's facial expression and the actual user's facial expression image data tends to stabilize, it indicates that the current Siamese network has been trained and converged, and can be used as a recognition model; otherwise, it is necessary to adjust the parameters of the Siamese network and perform the training of the Siamese network again until the loss value tends to stabilize.

[0127] In practical applications, the image data to be recognized is used as input and fed into the recognition model for feature extraction. The M supporting samples from the C categories, which are the user expression features of different categories, are also used as input and fed into the recognition model. Based on the features extracted from the image data to be recognized, feature differentiation extraction is performed to determine which of the M supporting samples from the C categories is closest to the features extracted from the image data to be recognized, thereby determining the recognition result.

[0128] S130, Output the recognition result;

[0129] The recognition results are output to the terminal for display.

[0130] The aforementioned user expression recognition method acquires image data to be recognized and inputs it into a recognition model for user expression recognition. The recognition model is obtained by training a Siamese network on images containing user expression data after labeling them. The training process of the recognition model involves user expression image data acquisition, processing of small sample image annotation sets, image feature extraction, and user expression small sample classification learning. This enables the effective learning of massive amounts of unlabeled visual image data using a small amount of accurately labeled data to form a recognition model for accurately recognizing user emotions.

[0131] Figure 8 This is a schematic block diagram of a user facial expression recognition device 300 provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above user expression recognition method, the present invention also provides a user expression recognition device 300. This user expression recognition device 300 includes a unit for performing the above-described user expression recognition method, and the device can be configured in a server. Specifically, please refer to... Figure 8 The user expression recognition device 300 includes a data acquisition unit 301, a recognition unit 302, and an output unit 303.

[0132] The data acquisition unit 301 is used to acquire image data to be recognized; the recognition unit 302 is used to input the image data to be recognized into the recognition model to perform user expression recognition in order to obtain the recognition result; the output unit 303 is used to output the recognition result.

[0133] The user facial expression recognition device 300 also includes a recognition model generation unit, which is used to train a Siamese network by annotating images containing user facial expression data to form a recognition model.

[0134] In one embodiment, the recognition model generation unit includes an image acquisition subunit, an annotation subunit, a feature extraction subunit, a classification subunit, a loss value calculation subunit, a judgment subunit, an adjustment subunit, and a determination subunit.

[0135] The system includes the following subunits: an image acquisition subunit for acquiring images containing user facial expression data to obtain facial expression image data; an annotation subunit for annotating the facial expression image data with small samples of user facial expressions to obtain an annotated dataset; a feature extraction subunit for inputting the annotated dataset into a Siamese network to extract image features to obtain user facial expression features; a classification subunit for inputting the user facial expression features into the Siamese network to classify user facial expression categories to obtain user facial expressions; a loss value calculation subunit for calculating the loss value of the user facial expressions and the actual user facial expressions in the facial expression image data; a judgment subunit for judging whether the loss value tends to be stable; an adjustment subunit for adjusting the parameters of the Siamese network if the loss value does not tend to be stable, and performing the step of inputting the annotated dataset into the Siamese network to extract image features to obtain user facial expression features; and a determination subunit for determining the Siamese network as a recognition model if the loss value tends to be stable.

[0136] In one embodiment, the image acquisition subunit includes an image acquisition module, a frame processing module, and a detection module.

[0137] The image acquisition module is used to acquire the original visual image; the frame processing module is used to perform frame processing on the original visual image to obtain several frames of image data; the detection module is used to perform face detection on each frame of image data by using an image person detection algorithm to detect the front or side of the face to obtain facial expression image data.

[0138] In one embodiment, the annotation subunit is used to annotate the facial expression image data with actual user facial expressions and perform data augmentation processing to obtain an annotated dataset.

[0139] In one embodiment, the annotation subunit includes an expression annotation module, a conversion module, a label setting module, and an image processing module.

[0140] The expression annotation module is used to annotate the expression image data with actual user expressions to obtain annotated image data; the conversion module is used to find adjacent frame image information based on the source information of the annotated image data, and convert the found image and the annotated image data into feature vectors based on HOG feature information; the label setting module is used to calculate the similarity between feature vectors using Cosine similarity, and to assign the same label information to images that meet the similarity requirements to obtain annotated image data; the image processing module is used to perform image rotation, image noise addition, image subtraction, and image compression on the annotated image data to obtain an annotated dataset.

[0141] In one embodiment, the feature extraction subunit includes a standardization processing module, a local feature extraction module, a global feature extraction module, and a high-dimensional feature extraction module.

[0142] The standardization processing module performs grayscale and normalization processing on the labeled dataset to obtain a standard image. The local feature extraction module uses a CNN convolutional network within the Siamese network to extract local features from the standard image to obtain intermediate local features. The global feature extraction module uses a recurrent neural network within the Siamese network to extract global features from the intermediate local features to obtain global features. The high-dimensional feature extraction module uses an attention mechanism within the Siamese network to extract high-dimensional features from the global features to obtain user expression features.

[0143] In one embodiment, the classification subunit includes a completion processing module and a differential extraction module.

[0144] The completion processing module is used to complete the user expression features according to the category of user expression to form a training dataset; the differential extraction module is used to use the Siamese network to perform feature differential extraction on every two features in the training dataset to determine whether every two features belong to the same category to obtain the user expression; wherein, one of the two features in the training dataset is a supporting sample in the user expression category.

[0145] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned user expression recognition device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0146] The aforementioned user facial expression recognition device 300 can be implemented as a computer program, which can, for example... Figure 9 It runs on the computer device shown.

[0147] Please see Figure 9, Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0148] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0149] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a user facial expression recognition method.

[0150] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0151] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a user expression recognition method.

[0152] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0154] Acquire image data to be recognized; input the image data to be recognized into the recognition model to perform user expression recognition, so as to obtain the recognition result; output the recognition result;

[0155] The recognition model is obtained by training a Siamese network after annotating images containing user facial expression data.

[0156] In one embodiment, when the processor 502 implements the step of training a Siamese network by annotating images containing user facial expression data, it specifically implements the following steps:

[0157] Images containing user facial expression data are acquired to obtain facial expression image data; small-sample user facial expression annotations are performed on the facial expression image data to obtain an annotated dataset; the annotated dataset is input into a Siamese network for image feature extraction to obtain user facial expression features; the user facial expression features are input into the Siamese network for user facial expression category classification to obtain user facial expressions; the loss value of the user facial expressions and the actual user facial expressions in the facial expression image data is calculated; it is determined whether the loss value tends to be stable; if the loss value does not tend to be stable, the parameters of the Siamese network are adjusted, and the process of inputting the annotated dataset into the Siamese network for image feature extraction to obtain user facial expression features is performed; if the loss value tends to be stable, the Siamese network is determined to be the recognition model.

[0158] In one embodiment, when the processor 502 implements the step of acquiring an image containing user facial expression data to obtain facial expression image data, it specifically implements the following steps:

[0159] Acquire the original visual image; perform frame segmentation on the original visual image to obtain several frame images; use an image person detection algorithm to detect the front or side of a person's face in each frame image to obtain facial expression image data.

[0160] In one embodiment, when the processor 502 performs the step of annotating the facial expression image data with a small sample of user facial expressions to obtain an annotated dataset, it specifically implements the following steps:

[0161] The facial expression image data is labeled with actual user facial expressions and data augmentation processing is performed to obtain a labeled dataset.

[0162] In one embodiment, when the processor 502 performs the step of annotating the facial expression image data with actual user facial expressions and performing data augmentation processing to obtain an annotated dataset, it specifically implements the following steps:

[0163] The facial expression image data is labeled with actual user facial expressions to obtain labeled image data; adjacent frame image information is searched based on the source information of the labeled image data, and the searched images and the labeled image data are converted into feature vectors based on HOG feature information; the similarity between feature vectors is calculated using Cosine similarity, and images that meet the similarity requirements are labeled with the same information to obtain labeled image data; the labeled image data is then subjected to image rotation, image noise addition, image subtraction, and image compression to obtain a labeled dataset.

[0164] In one embodiment, when the processor 502 implements the step of inputting the labeled dataset into the Siamese network for image feature extraction to obtain user facial expression features, the specific steps are as follows:

[0165] The labeled dataset is grayscaled and normalized to obtain standard images. A CNN convolutional network within a Siamese network is used to extract local features from the standard images to obtain intermediate local features. A recurrent neural network within the Siamese network is used to extract global features from the intermediate local features to obtain global features. An attention mechanism within the Siamese network is then used to extract high-dimensional features from the global features to obtain user expression features.

[0166] In one embodiment, when the processor 502 implements the step of inputting the user's facial expression features into the Siamese network to classify the user's facial expression category in order to obtain the user's facial expression, the processor 502 specifically implements the following steps:

[0167] The user expression features are completed according to the user expression category to form a training dataset; the Siamese network is used to perform feature differentiation extraction on every two features in the training dataset to determine whether every two features belong to the same category, so as to obtain the user expression; wherein, one of the two features in the training dataset is a supporting sample in the user expression category.

[0168] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0169] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0170] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps:

[0171] Acquire image data to be recognized; input the image data to be recognized into the recognition model to perform user expression recognition, so as to obtain the recognition result; output the recognition result;

[0172] The recognition model is obtained by training a Siamese network after annotating images containing user facial expression data.

[0173] In one embodiment, when the processor executes the computer program to implement the step of training a Siamese network by annotating images containing user facial expression data, it specifically implements the following steps:

[0174] Images containing user facial expression data are acquired to obtain facial expression image data; small-sample user facial expression annotations are performed on the facial expression image data to obtain an annotated dataset; the annotated dataset is input into a Siamese network for image feature extraction to obtain user facial expression features; the user facial expression features are input into the Siamese network for user facial expression category classification to obtain user facial expressions; the loss value of the user facial expressions and the actual user facial expressions in the facial expression image data is calculated; it is determined whether the loss value tends to be stable; if the loss value does not tend to be stable, the parameters of the Siamese network are adjusted, and the process of inputting the annotated dataset into the Siamese network for image feature extraction to obtain user facial expression features is performed; if the loss value tends to be stable, the Siamese network is determined to be the recognition model.

[0175] In one embodiment, when the processor executes the computer program to implement the step of acquiring an image containing user facial expression data to obtain facial expression image data, it specifically implements the following steps:

[0176] Acquire the original visual image; perform frame segmentation on the original visual image to obtain several frame images; use an image person detection algorithm to detect the front or side of a person's face in each frame image to obtain facial expression image data.

[0177] In one embodiment, when the processor executes the computer program to perform small-sample user expression annotation on the expression image data to obtain an annotated dataset, it specifically implements the following steps:

[0178] The facial expression image data is labeled with actual user facial expressions and data augmentation processing is performed to obtain a labeled dataset.

[0179] In one embodiment, when the processor executes the computer program to implement the step of annotating the facial expression image data with actual user facial expressions and performing data augmentation processing to obtain an annotated dataset, the specific steps are as follows:

[0180] The facial expression image data is labeled with actual user facial expressions to obtain labeled image data; adjacent frame image information is searched based on the source information of the labeled image data, and the searched images and the labeled image data are converted into feature vectors based on HOG feature information; the similarity between feature vectors is calculated using Cosine similarity, and images that meet the similarity requirements are labeled with the same information to obtain labeled image data; the labeled image data is then subjected to image rotation, image noise addition, image subtraction, and image compression to obtain a labeled dataset.

[0181] In one embodiment, when the processor executes the computer program to implement the step of inputting the labeled dataset into a Siamese network for image feature extraction to obtain user facial expression features, it specifically implements the following steps:

[0182] The labeled dataset is grayscaled and normalized to obtain standard images. A CNN convolutional network within a Siamese network is used to extract local features from the standard images to obtain intermediate local features. A recurrent neural network within the Siamese network is used to extract global features from the intermediate local features to obtain global features. An attention mechanism within the Siamese network is then used to extract high-dimensional features from the global features to obtain user expression features.

[0183] In one embodiment, when the processor executes the computer program to implement the step of inputting the user's facial expression features into the Siamese network for classification of user facial expression categories to obtain user facial expressions, the processor specifically implements the following steps:

[0184] The user expression features are completed according to the user expression category to form a training dataset; the Siamese network is used to perform feature differentiation extraction on every two features in the training dataset to determine whether every two features belong to the same category, so as to obtain the user expression; wherein, one of the two features in the training dataset is a supporting sample in the user expression category.

[0185] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0187] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0188] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0190] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A user expression recognition method, characterized by, The method comprises the following steps: acquiring image data to be recognized; inputting the image data to be recognized into a recognition model to recognize the user's expression, to obtain a recognition result; outputting the recognition result; wherein the recognition model is obtained by training a Siamese network after labeling images containing user expression data; the recognition model is obtained by training a Siamese network after labeling images containing user expression data, comprising: collecting images containing user expression data to obtain expression image data; performing small-sample user expression labeling on the expression image data to obtain a labeled data set; inputting the labeled data set into a Siamese network to extract image features to obtain user expression features; inputting the user expression features into the Siamese network to classify the user expression categories to obtain user expressions; calculating the loss value of the user expression and the actual user expression of the expression image data; determining whether the loss value tends to be stable; if the loss value does not tend to be stable, adjusting the parameters of the Siamese network and performing the inputting of the labeled data set into the Siamese network to extract image features to obtain user expression features; if the loss value tends to be stable, determining that the Siamese network is a recognition model; the small-sample user expression labeling on the expression image data to obtain a labeled data set comprises: labeling the actual user expression of the expression image data and performing data enhancement processing to obtain a labeled data set; the small-sample user expression labeling on the expression image data to obtain a labeled data set comprises: labeling the actual user expression of the expression image data to obtain labeled image data; finding adjacent frame image information according to the source information of the labeled image data, and converting the found image and the labeled image data into feature vectors according to HOG feature information; calculating the similarity between the feature vectors using Cosine similarity, and labeling the images that meet the similarity requirement with the same label information to obtain labeled image data; performing image rotation, image noise addition, image reduction, and image compression on the labeled image data to obtain a labeled data set. 2.The user expression recognition method of claim 1, wherein, the collecting of images containing user expression data to obtain expression image data comprises: acquiring an original visual image; performing frame processing on the original visual image to obtain a plurality of frame image pictures; detecting the front or side face of the person in each frame image picture using an image person detection algorithm to obtain expression image data. 3.The user expression recognition method of claim 1, wherein, the inputting of the labeled data set into the Siamese network to extract image features to obtain user expression features comprises: performing grayscale and normalization processing on the labeled data set to obtain a standard image; extracting local feature intermediate features of the standard image using a CNN convolution network in the Siamese network to obtain local feature intermediate features; extracting global features from the local feature intermediate features using a recurrent neural network in the Siamese network to obtain global features; extracting high-dimensional features in the global features using an attention mechanism in the Siamese network to obtain user expression features. 4.The user expression recognition method of claim 1, wherein, The user expression feature is input into a twin network for classification of a user expression category to obtain a user expression. The user expression features are completed according to the user expression category to form a training data set; The twin network is used to extract feature differences of each two features in the training data set to determine whether each two features belong to the same category to obtain a user expression; wherein one of each two features in the training data set is a support sample in the user expression category.

5. The user expression recognition apparatus, characterized by, The user expression recognition method of any one of claims 1-4, comprising: a data acquisition unit configured to acquire image data to be recognized; a recognition unit configured to input the image data to be recognized into a recognition model for user expression recognition to obtain a recognition result; an output unit configured to output the recognition result; Further comprising: a recognition model generation unit configured to train a twin network after labeling an image containing user expression data to form a recognition model.

6. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method of any one of claims 1-4.

7. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-4.

Images