Image classification method and device, equipment and medium

Abstract

The invention discloses an image classification method and device, equipment and a medium. The method comprises the following steps: acquiring an image to be classified; acquiring a public facial expression data set; wherein the image data in the public facial expression data set comprises various types of facial expressions; the clear degree of the facial expression is greater than or equal to a degree threshold value; performing expression extraction processing on the to-be-classified image based on the public face expression data set, and determining expression data included in the to-be-classified image; and performing expression classification on the to-be-classified image based on the expression data.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image classification method, device, equipment and medium. Background Art

[0002] In the field of large-scale facial expression recognition (FER) technology, accurate classification of facial images and their expression categories is crucial to the FER model training process and the expression recognition ability of the FER model. However, in practical applications, the expression categories in most FER face sample data are not accurate enough to meet the actual FER model training and recognition requirements. Summary of the invention

[0003] Based on the above technical problems, the embodiments of the present application provide an image classification method, device, equipment and medium.

[0004] The technical solution provided by the embodiment of the present application is as follows:

[0005] The present application provides an image classification method, the method comprising:

[0006] Get the image to be classified;

[0007] Acquire a public facial expression data set; wherein the image data in the public facial expression data set contains multiple types of facial expressions; the clarity of the facial expressions is greater than or equal to a degree threshold;

[0008] Performing expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified;

[0009] Perform expression classification on the image to be classified based on the expression data.

[0010] In some embodiments, the performing expression classification on the image to be classified based on the expression data includes:

[0011] If the expression data does not include the facial expression, the image to be classified is classified as a first type.

[0012] In some embodiments, the performing expression classification on the image to be classified based on the expression data includes:

[0013] If the expression data includes the facial expression, the image to be classified is determined to be a reference image for facial expression recognition.

[0014] In some embodiments, the performing expression classification on the image to be classified based on the expression data includes:

[0015] If the blur degree of the facial expression included in the expression data is greater than a preset degree, the image to be classified is classified as the second type.

[0016] In some embodiments, the performing expression classification on the image to be classified based on the expression data includes:

[0017] Obtain a sub-expression set including multiple face sub-regions;

[0018] Based on the sub-expression set and the expression data, expression classification is performed on the image to be classified.

[0019] In some embodiments, the performing expression classification on the image to be classified based on the sub-expression set and the expression data includes:

[0020] cropping the expression data to obtain sub-expression data of a plurality of face sub-regions;

[0021] Determining sub-expression types of the multiple facial sub-regions based on a degree of matching between the sub-expression data of the multiple facial sub-regions and the sub-expressions in the sub-expression set;

[0022] Based on the sub-expression types of the multiple facial sub-regions, expression classification is performed on the image to be classified.

[0023] In some embodiments, the method further comprises:

[0024] Obtaining the initial expression type of the image to be classified;

[0025] Based on the classification result obtained by performing expression classification on the image to be classified, the initial expression type is modified.

[0026] In some embodiments, performing expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified includes:

[0027] Determining facial key points in the image to be classified;

[0028] The expression data is determined by performing expression extraction processing on the image to be classified based on the facial key points through a feature extraction network.

[0029] In some embodiments, determining the key points of the face in the image to be classified includes:

[0030] The image to be classified is corrected and aligned by using a face alignment model to obtain a corrected alignment result, and features are extracted from the corrected alignment result to obtain the face key points.

[0031] In some embodiments, the method further comprises:

[0032] The initial face alignment model is trained based on the public facial expression data set to obtain the face alignment model; wherein the image data in the public facial expression data set contains sample key points; the sample key points include the face key points in the image data.

[0033] In some embodiments, the method further comprises:

[0034] Acquire sample key points of the public facial expression data set; wherein the sample key points include facial key points in the image data of the public facial expression data set;

[0035] Based on the sample key points and the public facial expression data set, an initial feature extraction network is trained to obtain the feature extraction network.

[0036] In some embodiments, obtaining the image to be classified includes:

[0037] Get video data;

[0038] At least part of the image frames in the video data are extracted to obtain the image to be classified.

[0039] The present application also provides an image classification device, the device comprising:

[0040] An acquisition module, used for acquiring an image to be classified and a public facial expression data set; wherein the image data in the public facial expression data set contains multiple types of facial expressions; and the clarity of the facial expression is greater than or equal to a degree threshold;

[0041] A processing module, used for performing expression extraction processing on the image to be classified based on a public facial expression data set, and determining the expression data contained in the image to be classified;

[0042] A classification module is used to perform expression classification on the image to be classified based on the expression data.

[0043] An embodiment of the present application further provides an electronic device, comprising a processor and a memory; a computer program is stored in the memory; when the computer program is executed by the processor, it can implement any of the above-described image classification methods.

[0044] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored; when the computer program is executed by a processor of an electronic device, it can implement any of the above-mentioned image classification methods.

[0045] The image classification method provided in the embodiment of the present application, after obtaining the image to be classified and the public facial expression data set, performs expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified. Since the public facial expression data set contains multiple types of facial expressions and the clarity of the facial expressions is greater than or equal to the degree threshold image data, through the above process, with the help of the high-definition diversified facial expressions carried by the image data in the public facial expression data set, it is possible to achieve accurate and comprehensive extraction of the expression data in the image to be classified, thereby improving the clarity and completeness of the expression data; and by treating the image based on the expression data Classifying images for expression classification can improve the expression classification accuracy of the images to be classified; at the same time, through the above process, not only can accurate expression classification of any image be achieved, but also, in the field of FER technology, accurate screening of image data containing unclear facial expressions and no expressions in sample data can be achieved; on the other hand, through the above process, automatic expression classification of large-scale FER data sets can be achieved, so that any image data in the FER training set can be accurately screened, and then the negative impact of the uncertainty of the expression category of the sample data on the FER training process can be weakened, and the training efficiency of the FER model and the facial expression recognition accuracy of the FER model obtained after training can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the process of an image classification method provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the image classification process provided in the embodiment of the present application;

[0048] Figure 3 A schematic diagram of the structure of the expression correction system provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the structure of an image classification device provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0052] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Facial expressions are one of the most natural, powerful and universal signals for humans to express their emotional states and thoughts. In practical applications, automatic recognition of facial expressions is of great help to computers in understanding and interacting with human behavior. In practical applications, FER has been widely used in the automatic recognition of human expressions.

[0054] In the field of FER technology, in order to speed up the training of FER models, researchers usually obtain image data as sample data through various channels and train FER models based on the sample data. However, the sample data obtained through the above channels, such as from the Internet, contains a large amount of image data with blurred expressions and noisy expression labels. These image data have a significant negative impact on the training and application of FER models, so that the FER model trained in the above way cannot meet the needs of actual facial expression recognition.

[0055] At the same time, in the sample data obtained by the above method, the uncertainty of expression labels exists from high-quality obvious facial expressions to low-quality micro-expressions, resulting in inconsistent labels and noisy labels, which seriously hinders the development of large-scale FER, especially the training and application development of FER models based on data-driven deep learning.

[0056] Based on the above problems, the embodiments of the present application provide an image classification method, device, equipment and medium. It should be noted that the above image classification method can be implemented by a processor of an electronic device; the above processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller (MCU), and a microprocessor.

[0057] Exemplarily, the electronic device may be a physical device or a virtual machine device.

[0058] Exemplarily, the electronic device may be a computer device.

[0059] Figure 1A flow chart of the image classification method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the process may include the following steps:

[0060] Step 101: Obtain an image to be classified.

[0061] In one embodiment, the image to be classified may or may not contain facial expressions.

[0062] In one implementation, the number of images in the images to be classified may be at least one.

[0063] In one implementation, the images to be classified may include images collected in real time, or may include historical images obtained from a network or storage space.

[0064] Step 102: Obtain a public facial expression dataset.

[0065] Among them, the image data in the public facial expression dataset contains multiple types of facial expressions; the clarity of the facial expressions is greater than or equal to the degree threshold.

[0066] In one implementation, facial expressions may include static expressions and / or dynamic expressions.

[0067] In one embodiment, facial expressions may include micro-expressions and obvious expressions whose degree of obviousness is greater than a preset threshold.

[0068] In one embodiment, the facial expression may include an overall facial expression or a local facial expression.

[0069] In one embodiment, when the image data includes an overall expression, the completeness of the facial expression may be greater than or equal to a completeness threshold, and the clarity may be greater than or equal to a degree threshold.

[0070] In one embodiment, the public facial expression dataset may include at least one of a Real-world Affective Faces Database (RAF-DB), a Facial Expression Representations Lesser Used Skill (FERPlus) dataset, and a facial expression data collection (AffectNet).

[0071] In one implementation, facial expressions may exist as expression labels of image data in a public facial expression dataset, that is, there may be a one-to-one correspondence between facial expressions and image data, or facial expressions may exist as attribute data of image data.

[0072] It should be noted that the execution order of step 101 and step 102 can be adjusted successively or executed in parallel, which is not limited in the embodiment of the present application.

[0073] Step 103: performing expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified.

[0074] In one implementation, the expression data may include expression features or expression types carried by at least a portion of an image region in the image to be classified.

[0075] In one implementation, performing expression extraction processing on the image to be classified based on the public facial expression dataset can be achieved by any of the following methods:

[0076] Based on the degree of matching between the image data in the public facial expression dataset and the corresponding area or pixel set in the image to be classified, the expression features carried by at least part of the pixel area are extracted from the image to be classified, and the above expression features are determined as expression data.

[0077] A preliminary recognition is performed on the image to be classified to determine the target pixel area and the initial expression of the target pixel area, and then at least one image data associated with the initial expression is selected from a public facial expression data set, and based on the facial expression corresponding to the at least one image data, the initial expression is corrected or compensated, and the result of the correction or compensation is determined as the expression data.

[0078] Step 104: performing expression classification on the image to be classified based on the expression data.

[0079] In one implementation, expression classification of the image to be classified can be achieved by any of the following methods:

[0080] Based on the completeness of the expression data, the images to be classified are divided into partial expression categories and complete expression categories.

[0081] Based on the expression amplitude contained in the expression data, the images to be classified are divided into obvious expression categories and micro-expression categories.

[0082] From the above, it can be seen that the image classification method provided in the embodiment of the present application, after obtaining the image to be classified and the public facial expression data set, performs expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified. Since the public facial expression data set contains multiple types of facial expressions and the clarity of the facial expressions is greater than or equal to the degree threshold image data, through the above process, with the help of the high-definition diverse facial expressions carried by the image data in the public facial expression data set, it is possible to achieve accurate and comprehensive extraction of the expression data in the image to be classified, thereby improving the clarity and completeness of the expression data; and, by using the expression data set By performing expression classification on the image to be classified, the expression classification accuracy of the image to be classified can be improved; at the same time, through the above process, not only can the accurate expression classification of any image be achieved, but also, in the field of FER technology, the accurate screening of image data containing unclear facial expressions and no expressions in the sample data can be achieved; on the other hand, through the above process, the automatic expression classification of large-scale FER data sets can be achieved, so that any image data in the FER training set can be accurately screened, and then the negative impact of the uncertainty of the expression category of the sample data on the FER training process can be weakened, and the training efficiency of the FER model and the facial expression recognition accuracy of the FER model obtained after training can be improved.

[0083] Based on the above embodiments, in the image classification method provided in the embodiments of the present application, expression classification of the image to be classified based on expression data can be achieved in the following ways:

[0084] If the expression data does not contain facial expressions, the image to be classified is classified as the first type.

[0085] Exemplarily, if the expression data includes facial expressions, the image to be classified may not be classified as the first type.

[0086] In one implementation, the first type may indicate that the image to be classified does not contain any facial expression; illustratively, the first type may indicate that the image to be classified cannot be classified as a FER sample.

[0087] In one embodiment, the expression data not including facial expressions may include: the expression data does not include overall facial expressions or does not include facial expressions in a specified area; wherein the facial expressions in the specified area may include at least one of an eye area, a mouth area, and a nose area.

[0088] From the above, it can be seen that in the image classification method provided by the embodiment of the present application, if the expression data does not contain facial expressions, the image to be classified is classified as the first type. In this way, based on whether the image to be classified contains facial expressions in the public facial expression data set, it is possible to accurately divide the types of images to be classified; and, by adjusting the public facial expression data set, it is possible to flexibly and controllably divide the expression types of images to be classified, thereby meeting diverse expression classification needs and diverse FER training needs; at the same time, through the above operations, it is possible to filter any images to be classified that do not contain facial expressions, thereby reducing the number of noise samples in the FER training process.

[0089] Based on the above embodiments, in the image classification method provided in the embodiments of the present application, expression classification of the image to be classified based on expression data can also be achieved in the following ways:

[0090] If the expression data includes facial expressions, the image to be classified is determined to be a reference image for facial expression recognition.

[0091] Exemplarily, if the expression data does not include facial expressions, it can be determined that the image to be classified is not a reference image.

[0092] In one implementation, if the expression data includes facial expressions, and the clarity of the facial expressions is greater than or equal to a clarity threshold, then the image to be classified may be determined to be a reference image.

[0093] In one embodiment, the reference image may include sample data upon which the FER training process is based.

[0094] In one embodiment, the expression data includes facial expressions, which may include at least one of expression data including overall facial expressions and expression data including local expressions of at least one region of the face.

[0095] From the above, it can be seen that in the image classification method provided in the embodiment of the present application, if the expression data contains facial expressions, the image to be classified is determined to be a benchmark image for facial expression recognition. In this way, it is possible to accurately divide the image to be classified into a benchmark image for facial expression recognition based on whether the image to be classified contains facial expressions in a public facial expression data set; and by adjusting the public facial expression data set, it is possible to achieve flexible and controllable division of the expression types of the images to be classified, thereby meeting diverse expression classification needs and diverse FER training needs.

[0096] Based on the above embodiments, in the image classification method provided in the embodiments of the present application, expression classification of the image to be classified based on expression data can also be achieved in the following ways:

[0097] If the blur degree of the facial expression contained in the expression data is greater than a preset degree, the image to be classified is classified as the second type.

[0098] Exemplarily, if the blur degree of the facial image contained in the expression data is less than or equal to a preset degree, the image to be classified may not be classified as the second type.

[0099] In one implementation, the second type may represent that the blur degree of the facial expression of the image to be classified is greater than a preset degree.

[0100] In one implementation, the blurriness of the facial expression contained in the expression data is greater than a preset degree, which may include the blurriness of the overall facial expression contained in the expression data is greater than a preset degree, or the blurriness of the local facial expression contained in the expression data is greater than a preset degree.

[0101] From the above, it can be seen that in the image classification method provided by the embodiment of the present application, if the degree of blurriness of the facial expression contained in the expression data is greater than the preset degree, the image to be classified is classified as the second type. In this way, the fine-grained and accurate division of the image to be classified based on the degree of blurriness of the facial expression in the public facial expression data set contained in the image to be classified is achieved; and by adjusting the public facial expression data set, the flexible and controllable division of the expression type of the image to be classified can be achieved, so as to meet the diverse expression classification needs and the diverse FER training needs; at the same time, through the above operation, any image to be classified containing fuzzy facial expressions can be filtered, so as to reduce the number of noise samples in the FER training process.

[0102] Based on the above embodiments, in the image classification method provided in the embodiments of the present application, expression classification of the image to be classified based on expression data can also be achieved in the following ways:

[0103] A sub-expression set including a plurality of face sub-regions is obtained; and based on the sub-expression set and the expression data, expression classification is performed on the image to be classified.

[0104] In one implementation, the sub-expressions in the sub-expression set may correspond one-to-one to the sub-regions of the human face.

[0105] In one implementation, the sub-expression set may correspond to at least one of the expressions of a face sub-region contained in an image in a public face image dataset.

[0106] In one implementation, based on the sub-expression set and the expression data, expression classification of the image to be classified can be achieved by any of the following methods:

[0107] If the expression data includes expressions corresponding to respective sub-regions of the face represented by the sub-expression set, the image to be classified can be determined as a reference image.

[0108] If the expression data does not include the expression corresponding to the specified sub-region in the sub-expression set, it can be determined that the image to be classified is of the first type.

[0109] If the expression data includes expressions in the sub-expression set, but the blurriness thereof is greater than or equal to a preset degree, the image to be classified may be classified as the second type.

[0110] From the above, it can be seen that in the image classification method provided by the embodiment of the present application, after obtaining the sub-expression set including multiple face sub-regions, the expression classification of the image to be classified is performed based on the sub-expression set and the expression data. In this way, through the above operation, the expression data carried by the image to be classified can be finely divided in the face sub-region dimension, thereby improving the accuracy and granularity of the expression classification of the image to be classified.

[0111] Based on the foregoing embodiment, in the image classification method provided in the embodiment of the present application, based on the sub-expression set and the expression data, expression classification of the image to be classified can be achieved by the following steps:

[0112] Step A1: crop the expression data to obtain sub-expression data of multiple face sub-regions.

[0113] In one implementation, the sub-expression data may correspond one-to-one to the sub-regions of the face.

[0114] In one implementation, the expression data may be clipped by any of the following methods:

[0115] Based on the preset pixel interval, the expression data is evenly cropped to obtain sub-expression data.

[0116] Based on the distribution characteristics of the expression data in the image to be classified, the expression data is cropped to obtain sub-expression data; for example, the target cropping line can be determined according to the distribution characteristics of the expression data in the image to be processed, and then the expression data is cropped based on the target cropping line.

[0117] Step A2: determining the sub-expression types of the multiple face sub-regions based on the matching degree between the sub-expression data of the multiple face sub-regions and the sub-expressions in the sub-expression set.

[0118] In one embodiment, the sub-expression type may include at least one of an identifier of a facial sub-region corresponding to the sub-expression data, an amplitude of the sub-expression data, and a degree of blurriness of the sub-expression data; exemplarily, the identifier of the facial sub-region may include a name or number of the facial sub-region; exemplarily, the amplitude of the sub-expression data may indicate that the sub-expression data is a micro-expression or an obvious expression.

[0119] In one implementation, the sub-expression types of multiple facial sub-regions may be determined in the following manner:

[0120] If the degree of matching between the mth data in the sub-expression data and the kth sub-expression in the sub-expression set is greater than the matching threshold, it can be determined that the facial sub-region corresponding to the mth data is the facial region associated with the kth sub-expression, and the sub-expression type corresponding to the mth data is determined to be the expression type corresponding to the kth sub-expression; wherein m and k can both be integers greater than 0.

[0121] Step A3: Based on the sub-expression types of multiple face sub-regions, expression classification is performed on the image to be classified.

[0122] In one implementation, expression classification of the image to be classified can be achieved by any of the following methods:

[0123] Based on the correlation between sub-expression data of multiple facial sub-regions, determine whether the image to be classified contains a complete overall facial expression, and based on whether the image to be classified contains an overall facial expression, classify the image to be classified into a complete expression type or a partial expression type; exemplarily, the above-mentioned correlation may include the correlation between facial sub-regions associated with the sub-expression data, and the feature change state between the sub-expression data corresponding to adjacent facial sub-regions.

[0124] Based on the blurriness of the sub-expression data of the plurality of facial sub-regions, it is determined whether to classify the image to be classified into the second type.

[0125] Based on the amount of sub-expression data of the plurality of face sub-regions, it is determined whether to classify the image to be classified as the first type.

[0126] From the above, it can be seen that in the image classification method provided in the embodiment of the present application, the expression data is cropped to obtain sub-expression data of multiple facial sub-regions. In this way, the expression characteristics of each facial sub-region contained in the image to be classified can be accurately displayed through the sub-expression data; and based on the degree of matching between the sub-expression data of multiple facial sub-regions and the sub-expressions in the sub-expression set, the sub-expression types of the multiple facial sub-regions are determined, which can comprehensively reflect the types of expressions of the facial sub-regions contained in the image to be classified; on this basis, the expression classification of the image to be classified is performed based on the sub-expression types of multiple facial sub-regions, which can improve the accuracy and comprehensiveness of expression classification.

[0127] Based on the above embodiments, the image classification method provided in the embodiments of the present application may further include the following steps:

[0128] The initial expression type of the image to be classified is obtained; based on the classification result obtained by performing expression classification on the image to be classified, the initial expression type is corrected.

[0129] In one implementation, the initial expression type may include preset attribute information such as whether the image to be classified contains a facial expression, the area of ​​the facial expression contained, the clarity, and whether it is complete.

[0130] In one implementation, the initial expression type may include a preset expression tag.

[0131] In one implementation, the classification result may include the first type, the second type, and the type of determining the image to be classified as a reference image in the aforementioned embodiment.

[0132] In one implementation, the initial expression type is modified by any of the following methods:

[0133] If the initial expression type is different from the classification result, update the initial expression type to the classification result.

[0134] If the initial expression type is the same as the classification result, the initial classification expression is kept.

[0135] If the classification result is additional description information or supplementary description information of the initial classified expression, the classification result will be added to the initial classified expression; for example, if the initial classified expression is the overall facial expression and the classification result is a smiling expression, the corrected initial classified expression can be: the overall facial expression in a smiling state.

[0136] From the above, it can be seen that the image classification method provided in the embodiment of the present application, after obtaining the initial expression type of the image to be classified, corrects the initial expression type based on the classification result obtained by performing expression classification on the image to be classified. In this way, not only the automatic correction of the initial expression type of the image to be classified is realized, thereby reducing the probability that the expression type of the image to be classified is a noise label; and through the above operation, the probability of overfitting caused by the initial expression type of large-scale uncertain sample data during the FER model training process can be reduced, thereby improving the training efficiency of the FER model.

[0137] Based on the foregoing embodiment, in the image classification method provided in the embodiment of the present application, expression extraction processing is performed on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified, which can be achieved by the following steps:

[0138] Step B1: determine the key points of the face in the image to be classified.

[0139] In one implementation, the facial key points may include pixel coordinate points of at least a portion of a facial region included in the image to be classified.

[0140] In one implementation, key points of the image to be classified may be identified by a neural network, thereby determining key points of the face in the image to be classified.

[0141] Step B2: extracting expression from the image to be classified based on key points of the face through a feature extraction network to determine expression data.

[0142] In one implementation, the feature extraction network may include a convolutional neural network (CNN) and an attention module, referred to as attention CNN.

[0143] In one embodiment, the expression data may be determined in the following manner:

[0144] The facial key points and the image to be classified are input into the feature extraction network to instruct the feature extraction network to extract the features of local expressions or overall expressions with the pixel coordinates indicated by the facial key points as the center, and determine the extraction results as expression data.

[0145] From the above, it can be seen that the image classification method provided in the embodiment of the present application can achieve preliminary locking of the expression area in the image to be classified by determining the facial key points in the image to be classified; and, by performing expression extraction processing on the image to be classified based on the facial key points through the feature extraction network, it is possible to guide the direction and / or area of ​​expression extraction processing on the image to be classified, thereby improving the efficiency of expression extraction processing.

[0146] Based on the above embodiments, in the image classification method provided in the embodiments of the present application, determining the key points of the face in the image to be classified can be achieved in the following ways:

[0147] The image to be classified is corrected and aligned through the face alignment model to obtain a corrected alignment result, and features are extracted from the corrected alignment result to obtain face key points.

[0148] In one implementation, correcting and aligning the image to be classified can reduce the probability of facial expressions in the image to be classified being blurred or missing due to at least one of posture, occlusion, and illumination deviation.

[0149] In one implementation, the face alignment model can perform correction alignment and key point extraction processing on the image to be classified based on the face alignment algorithm (Deep Alignment Network, DAN). In this case, the face alignment model can be called a DAN network.

[0150] Among them, the DAN network is a cascade-like neural network whose input is the entire face image. The DAN network can effectively overcome the problems caused by head posture and achieve better detection effects.

[0151] Exemplarily, the DAN network may be a multi-layer network, the input data of the first layer of which is a vertebral template of facial key points and an image to be processed, wherein the standard template of facial key points may include image data in a public face dataset labeled with facial key points.

[0152] Exemplarily, the first layer of the DAN network sends the input data to the feedforward neural network, which can be composed of ten convolutional layers, five pooling layers and one fully connected layer; wherein, every two convolutional layers are followed by a pooling layer, the activation function can be a ReLU function, and its output data can be the deviation value of each position of the facial key points.

[0153] Exemplarily, the second layer of the DAN network can be a posture estimation network, which mainly calculates the posture direction of the face in the image to be classified, and at the same time performs affine transformation according to the vertebral marker template to extract and straighten the face area in the image to be classified, thereby realizing the above-mentioned correction and alignment operation and obtaining the correction and alignment result.

[0154] Exemplarily, the third layer of the DAN network is a key point heat map generation network; wherein the key point heat map is a probability map of the area near each key point in the correction alignment result, and the farther the pixel is from the key point, the smaller its pixel value is.

[0155] Exemplarily, in the process of determining the key points of the face by the DAN network, the continuous frames of the images to be classified can be grayscaled and gamma corrected to obtain the correction results; then the correction results are subjected to the Histogram of Oriented Gradient (HOG) feature extraction. In this process, the coordinates of the calibrated key points in the previous frame of the image to be classified can be used as the coordinates of the candidate key points in the next frame, thereby establishing the gradient feature vector generation area of ​​the two frames of the images to be classified.

[0156] Exemplarily, the regional gradient feature vector is obtained by calculating the gradient size and gradient direction of the gradient feature vector generating area, and the similarity result α1 of the regional gradient feature vectors of the two frames to be classified is calculated; then, convolution feature comparison areas of the two frames to be classified are established with the key points as the center, the convolution feature comparison areas are convolved multiple times to obtain regional convolution features, and the similarity result α2 of the regional convolution features in the two frames to be classified is calculated; then the similarity result α1 is added to the similarity result α2 according to the proportion to obtain the similarity result β.

[0157] Exemplarily, a 15*15 key point search area can be established with the key point (x1, y1) in the first image to be processed as the center, and a horizontal search is performed first, followed by a vertical search, until a point (x2, y2) above the gradient threshold appears. (x2, y2) is the key point of the second image to be classified.

[0158] For example, if there is still no point higher than the gradient threshold after searching the entire search area, all points in the search area are sorted according to their similarity with (x1, y1), and the point (xn, yn) with the highest similarity is taken as the key point. Where n can be an integer greater than 1.

[0159] Among them, the HOG feature is a feature descriptor used for object detection in computer vision and image processing. The HOG feature constructs the feature by calculating and counting the gradient direction histogram of the local area of ​​the image.

[0160] From the above, it can be seen that the image classification method provided in the embodiment of the present application uses a face alignment model to correct and align the image to be classified to obtain a corrected alignment result, which can weaken the negative impact of blurred or missing facial expressions in the image to be classified due to at least one of posture, occlusion and lighting deviation; and, by performing feature extraction through the corrected alignment result to obtain face key points, not only the accuracy of the face key points can be improved, but also the integrity of the face key points can be improved.

[0161] Based on the above embodiments, the image classification method provided in the embodiments of the present application may further include the following steps:

[0162] The initial face alignment model is trained based on the common facial expression dataset to obtain a face alignment model.

[0163] Among them, the image data in the public facial expression data set contains sample key points; the sample key points include facial key points in the image data.

[0164] In one implementation, the image data in the public facial expression dataset may be pre-set with facial key points.

[0165] In one implementation, the initial alignment model is trained and obtained by:

[0166] The image data in the public facial expression dataset is input into the initial face alignment model. After being processed by the initial face alignment model, the calculated key points are obtained. Then, according to the matching degree between the calculated key points and the above sample key points, the correction parameters of the initial alignment model are adjusted until the matching degree between the above calculated key points and the sample key points is greater than or equal to the matching threshold.

[0167] From the above, it can be seen that the image data classification method provided in the embodiment of the present application trains the initial face alignment model based on the public face expression data set to obtain the face alignment model. Since the image data of the public face expression data set contains sample key points, and the sample key points include the face key points in the image data, the above operation can improve the recognition accuracy of the face alignment model for the face key points.

[0168] Based on the above embodiments, the image classification method provided in the embodiments of the present application may further include the following steps:

[0169] Step C1: Obtain sample key points of a public facial expression dataset.

[0170] The sample key points include facial key points in image data of a public facial expression dataset.

[0171] Step C2: Based on the sample key points and the public facial expression data set, the initial feature extraction network is trained to obtain a feature extraction network.

[0172] In one embodiment, the public facial expression dataset may further include expression labels for each image data; illustratively, the expression labels may include the types of facial expressions contained in the image data, which may include local expressions or overall facial expressions, and may also include obvious expressions or micro-expressions.

[0173] In one implementation, training the initial feature extraction network may be implemented in any of the following ways:

[0174] The image data and sample key points in the public facial expression dataset are input into the initial feature extraction network to obtain the output expression of the initial feature extraction network, and then the parameters of the initial feature extraction network are adjusted based on the matching degree between the output expression and the expression label to obtain the feature extraction network.

[0175] When the initial feature extraction network is an attention CNN in an initial state, loss weights for each local region and global region of the image data are defined, and the image data and sample key points are input into the attention CNN in the initial state to obtain output results, and then the expected losses for each local region and global region are obtained based on the loss weights, and the parameters of the attention CNN in the initial state are adjusted in the form of back propagation to obtain the attention CNN; wherein the expected losses for each local region and global region can be obtained based on the loss weights through a pre-set loss function.

[0176] From the above, it can be seen that in the image classification method provided by the embodiment of the present application, after obtaining the sample key points of the public facial expression data set, the initial feature extraction network is trained based on the sample key points and the public facial expression data set to obtain the feature extraction network. Therefore, since the sample key points include the facial key points in the image data of the public facial expression data set, through the above steps, the initial feature extraction network can be guided to determine the pixel range of the image data in the public facial expression data set that carries the expression data, thereby improving the training efficiency of the initial feature extraction network and shortening the training time; and, due to the clarity of the image data in the public facial expression data set and the diversity of facial expressions, the stability of feature tracking and extraction of facial features can be improved.

[0177] Based on the above embodiments, in the image classification method provided in the embodiments of the present application, obtaining the image to be classified can be achieved in the following manner:

[0178] Acquire video data; extract at least part of the image frames in the video data to obtain images to be classified.

[0179] In one implementation, the video data may be collected in real time or acquired historically.

[0180] In one implementation, there may be multiple video data, and different video data may come from different acquisition devices or storage devices.

[0181] In one implementation, the image to be classified may be obtained in the following manner:

[0182] Acquire an image extraction strategy, extract at least part of the image frames from the video data based on the image extraction strategy, and determine the extracted image frames as images to be classified; wherein the image extraction strategy can be determined according to the acquisition method of the video data and / or the distribution status of the target object in the video data; exemplarily, the above-mentioned acquisition method can include single-device acquisition or multi-device acquisition. For example, if it is multi-device acquisition, it is necessary to determine how to extract the image to be classified based on the correlation between multiple video data collected by multiple devices in the target object dimension; exemplarily, the distribution status can include the occurrence frequency of the target object, the pixel distribution interval of the target object in the video data, etc.; exemplarily, the above-mentioned target object can include a face.

[0183] As can be seen from the above, in the image classification method provided in the embodiment of the present application, after obtaining the video data, the image to be classified is obtained by extracting at least part of the image frames in the video data. In this way, through the above method, the source of the image to be classified is expanded, thereby enriching the categories and scenes of the image to be classified.

[0184] Figure 2 A schematic diagram of the image classification process provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, the process may include the following steps:

[0185] Step 201: Preprocess video data.

[0186] Exemplarily, the video data preprocessing may include the process of extracting the image to be classified from the video data in the aforementioned embodiment, and correcting and aligning the image to be classified by using a face alignment model.

[0187] Exemplarily, the result of the video data preprocessing can be recorded as the preprocessing result. Meanwhile, in the above correction and alignment process, the facial key points of the image to be classified can also be obtained.

[0188] Step 202: Extract facial expression features.

[0189] Exemplarily, the attention CNN provided in the aforementioned embodiment can be used to perform feature extraction on the preprocessing result based on facial key points, thereby obtaining the expression data contained in the image to be classified.

[0190] Step 203: facial expression feature recognition.

[0191] Exemplarily, the expression data can be cropped to obtain sub-expression data of multiple facial sub-regions; then, based on the degree of matching between the sub-expression data of the multiple facial sub-regions and the sub-expressions in the sub-expression set, the sub-expression types of the multiple facial sub-regions are determined; and based on the sub-expression types of the multiple facial sub-regions, the expression classification of the image to be classified is performed, thereby realizing feature recognition of facial expressions.

[0192] Step 204: Image classification.

[0193] Exemplarily, the method provided by the aforementioned embodiment can be used to classify the image to be classified based on whether the feature recognition result contains facial expressions and the degree of blurriness of the facial expressions.

[0194] Exemplarily, the initial expression type of the image to be classified may also be corrected based on the classification result obtained by classifying the image to be classified.

[0195] From the above, it can be seen that the image classification method provided in the embodiment of the present application realizes the classification correction of image data in video data from any source through video data preprocessing, facial expression feature extraction and recognition, and image classification, thereby overcoming the negative impact of noise labels carried by sample data in the FER field in the related art on the FER model training process.

[0196] Based on the aforementioned embodiments, the embodiments of the present application also provide an expression correction system, which can implement the image classification method provided in any of the previous embodiments, as well as the correction operation on the initial expression type of any image. Figure 3 A schematic diagram of the structure of the expression correction system provided in an embodiment of the present application.

[0197] like Figure 3 As shown, the expression correction system 301 may include a data preprocessing module 3011, a feature extraction module 3012, a training module 3013 and a calibration module 3014; wherein, the data preprocessing module 3011 may implement the data preprocessing operation in the aforementioned embodiment, the feature extraction module 3012 may include the attention CNN in the aforementioned embodiment, for realizing expression feature extraction, the training module 3013 may realize the training of the initial expression alignment module and the initial attention CNN, and the calibration module 3014 may calibrate the image data through the expression alignment module during the process of preprocessing the video data in the data preprocessing module, so as to weaken the negative effects of posture, occlusion, expression and lighting on the feature extraction process of the feature extraction module 3012.

[0198] From the above, it can be seen that in the expression correction system provided in the embodiment of the present application, the functions of each module are independent of each other and self-contained, thereby being able to achieve flexible expression extraction and correction of video data or images to be classified from any source.

[0199] Based on the above embodiments, the present application also provides an image classification device. Figure 4 A schematic diagram of the structure of an image classification device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device 4 may include:

[0200] The acquisition module 401 is used to acquire the image to be classified and the public facial expression data set; wherein the image data in the public facial expression data set contains multiple types of facial expressions; the clarity of the facial expression is greater than or equal to the degree threshold;

[0201] Processing module 402, used for performing expression extraction processing on the image to be classified based on the public facial expression data set to obtain expression data in the image to be classified;

[0202] The classification module 403 is used to perform expression classification on the image to be classified based on the expression data.

[0203] In some embodiments, the classification module 403 is used to classify the image to be classified into the first type if the expression data does not include a facial expression.

[0204] In some embodiments, the classification module 403 is used to determine that the image to be classified is a reference image for facial expression recognition if the expression data includes facial expressions.

[0205] In some embodiments, the classification module 403 is used to classify the image to be classified into the second type if the blur degree of the facial expression contained in the expression data is greater than a preset degree.

[0206] In some embodiments, the acquisition module 401 is used to acquire a sub-expression set including a plurality of face sub-regions;

[0207] The classification module 403 is used to classify the expression of the image to be classified based on the sub-expression set and the expression data.

[0208] In some embodiments, the processing module 402 is used to crop the expression data to obtain sub-expression data of multiple face sub-regions; determine the sub-expression types of the multiple face sub-regions based on the matching degree between the sub-expression data of the multiple face sub-regions and the sub-expressions in the sub-expression set;

[0209] The classification module 403 is used to classify the expression of the image to be classified based on the sub-expression types of multiple facial sub-regions.

[0210] In some embodiments, the acquisition module 401 is used to acquire the initial expression type of the image to be classified;

[0211] The processing module 402 is used to modify the initial expression type based on the classification result obtained by performing expression classification on the image to be classified.

[0212] In some embodiments, the processing module 402 is used to determine the key points of the face in the image to be classified; perform expression extraction processing on the image to be classified based on the key points of the face through a feature extraction network to determine the expression data.

[0213] In some embodiments, the processing module 402 is used to perform correction alignment on the image to be classified by using a face alignment model to obtain a correction alignment result, and perform feature extraction on the correction alignment result to obtain facial key points.

[0214] In some embodiments, the image classification device 4 may also include a training module for training an initial face alignment model based on a public facial expression data set to obtain a face alignment model; wherein the image data in the public facial expression data set contains sample key points; the sample key points include face key points in the image data.

[0215] In some embodiments, the acquisition module 401 is used to acquire sample key points of a public facial expression data set; wherein the sample key points include facial key points in image data of the public facial expression data set;

[0216] The training module is used to train the initial feature extraction network based on sample key points and a public facial expression data set to obtain a feature extraction network.

[0217] In some embodiments, the acquisition module 401 is used to acquire video data;

[0218] The processing module 402 is used to extract at least part of the image frames in the video data to obtain images to be classified.

[0219] The image classification device provided by the embodiment of the present application, after acquiring the image to be classified and the public facial expression data set, performs expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified. Since the public facial expression data set contains multiple types of facial expressions and the clarity of the facial expressions is greater than or equal to the degree threshold image data, through the above process, with the help of the high-definition diverse facial expressions carried by the image data in the public facial expression data set, it is possible to achieve accurate and comprehensive extraction of the expression data in the image to be classified, thereby improving the clarity and completeness of the expression data; and by treating the image to be classified based on the expression data Classifying images for expression classification can improve the expression classification accuracy of the images to be classified; at the same time, through the above process, not only can accurate expression classification of any image be achieved, but also, in the field of FER technology, accurate screening of image data containing unclear facial expressions and no expressions in sample data can be achieved; on the other hand, through the above process, automatic expression classification of large-scale FER data sets can be achieved, so that any image data in the FER training set can be accurately screened, and then the negative impact of the uncertainty of the expression category of the sample data on the FER training process can be weakened, and the training efficiency of the FER model and the facial expression recognition accuracy of the FER model obtained after training can be improved.

[0220] Based on the above embodiments, the present application also provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device 5 may include a processor 501 and a memory 502; wherein the memory 502 stores a computer program, and when the computer program is executed by the processor 501, it can implement the image classification method provided in any of the previous embodiments.

[0221] The above-mentioned processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0222] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (Random Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), flash memory, a hard disk drive (Hard Disk Drive, HDD) or a solid state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor.

[0223] Based on the foregoing embodiments, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor of an electronic device, the image classification method provided in any of the previous embodiments can be implemented.

[0224] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0225] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0226] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0227] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0228] It should be noted that the above-mentioned computer-readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0229] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0230] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0231] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus necessary general hardware nodes, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0232] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0233] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0234] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0235] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image classification method, characterized in that: The method comprises: Get the image to be classified; Acquire a public facial expression data set; wherein the image data in the public facial expression data set contains multiple types of facial expressions; the clarity of the facial expressions is greater than or equal to a degree threshold; Performing expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified; Perform expression classification on the image to be classified based on the expression data.

2. The method according to claim 1, characterized in that The performing expression classification on the image to be classified based on the expression data comprises: If the expression data does not include the facial expression, the image to be classified is classified as a first type.

3. The method according to claim 1, characterized in that The performing expression classification on the image to be classified based on the expression data comprises: If the expression data includes the facial expression, the image to be classified is determined to be a reference image for facial expression recognition.

4. The method according to claim 1, characterized in that: The performing expression classification on the image to be classified based on the expression data comprises: If the blur degree of the facial expression included in the expression data is greater than a preset degree, the image to be classified is classified as the second type.

5. The method according to claim 1, characterized in that: The performing expression classification on the image to be classified based on the expression data comprises: Obtain a sub-expression set including multiple face sub-regions; Based on the sub-expression set and the expression data, expression classification is performed on the image to be classified.

6. The method according to claim 5, characterized in that The performing expression classification on the image to be classified based on the sub-expression set and the expression data comprises: cropping the expression data to obtain sub-expression data of a plurality of face sub-regions; Determining sub-expression types of the multiple facial sub-regions based on a degree of matching between the sub-expression data of the multiple facial sub-regions and the sub-expressions in the sub-expression set; Based on the sub-expression types of the multiple facial sub-regions, expression classification is performed on the image to be classified.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining the initial expression type of the image to be classified; Based on the classification result obtained by performing expression classification on the image to be classified, the initial expression type is modified.

8. The method according to claim 1, characterized in that The step of performing expression extraction processing on the image to be classified based on the public facial expression data set to determine the expression data contained in the image to be classified includes: Determining facial key points in the image to be classified; The expression data is determined by performing expression extraction processing on the image to be classified based on the facial key points through a feature extraction network.

9. The method according to claim 8, characterized in that The determining of key points of the face in the image to be classified includes: The image to be classified is corrected and aligned by using a face alignment model to obtain a corrected alignment result, and features are extracted from the corrected alignment result to obtain the face key points.

10. The method according to claim 9, characterized in that The method further comprises: The initial face alignment model is trained based on the public facial expression data set to obtain the face alignment model; wherein the image data in the public facial expression data set contains sample key points; the sample key points include the face key points in the image data.

11. The method according to claim 8, characterized in that The method further comprises: Acquire sample key points of the public facial expression data set; wherein the sample key points include facial key points in the image data of the public facial expression data set; Based on the sample key points and the public facial expression data set, an initial feature extraction network is trained to obtain the feature extraction network.

12. The method according to claim 1, characterized in that The step of obtaining the image to be classified comprises: Get video data; At least part of the image frames in the video data are extracted to obtain the image to be classified.

13. An image classification device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be classified and a public facial expression data set; wherein the image data in the public facial expression data set contains multiple types of facial expressions; and the clarity of the facial expression is greater than or equal to a degree threshold; A processing module, used for performing expression extraction processing on the image to be classified based on a public facial expression data set, and determining the expression data contained in the image to be classified; A classification module is used to perform expression classification on the image to be classified based on the expression data.

14. An electronic device, characterized in that: The electronic device includes a processor and a memory; a computer program is stored in the memory; when the computer program is executed by the processor, it can implement the image classification method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The storage medium stores a computer program; when the computer program is executed by a processor of an electronic device, it can implement the image classification method as described in any one of claims 1 to 12.

Images