Robust facial expression recognition method using facial landmark-based keypoint feature extraction and attention technique, and computing device for performing the same

Abstract

A method for recognizing facial expressions and a computing device for performing the same are disclosed. A method according to one disclosed embodiment comprises the steps of: inputting a face image into a face alignment network to extract a first feature map containing geometric shape information of the face and landmark coordinates of a plurality of faces; inputting the face image into a backbone network to generate a second feature map containing external feature information of the face; fusing the first feature map and the second feature map to generate a global feature map; extracting keypoint features for classifying facial expressions based on the global feature map and landmark coordinates; and classifying facial expressions based on the extracted keypoint features.

Description

Technology Field

[0001] The present invention relates to a robust facial expression recognition method using facial landmark-based keypoint feature extraction and an attention technique, and a computing device for performing the same. Background Technology

[0003] Facial expressions are representative non-verbal information indicating a person's emotional state, intention, and reaction, and are utilized as important information in various fields such as human-computer interaction, emotional interfaces, driver monitoring systems, service robots, and remote consultation systems. In particular, video-based facial expression recognition technology using cameras is being actively researched because it can naturally identify a user's emotions or state in a non-contact manner.

[0004] Generally, in image-based facial expression recognition, the face region is detected from the input image, and the size, rotation, and position of the face are normalized using face alignment techniques. Features for expression classification are then extracted from the normalized face images. This face alignment process detects major facial landmarks such as the eyes, nose, and mouth to correct geometric deformations, thereby enabling more consistent feature representation even in different shooting environments. Accordingly, face alignment technology is recognized as a critical element in the preprocessing stage of facial expression recognition.

[0005] However, conventional facial expression recognition technologies have limitations in that they do not fully utilize facial alignment information. For example, a significant number of conventional technologies use landmark information obtained from facial alignment merely for geometric normalization purposes, such as cropping or rotating facial images, and fail to actively incorporate it during the feature extraction stage for actual expression identification. Furthermore, while some technologies utilize feature maps generated by facial alignment networks, they often remain limited to a method of comprehensively using the entire feature map, rather than precisely selecting and utilizing specific facial regions around the eyes, nose, and mouth where expression changes are prominent.

[0006] Meanwhile, when recognizing the facial expressions of others, humans tend to judge emotions based on areas where changes in expression are concentrated, such as the corners of the eyes, nose, and mouth, rather than interpreting the entire face with equal weight. Nevertheless, conventional technologies have faced problems where unnecessary information is learned along with the core areas, leading to reduced computational efficiency or an increased possibility of overfitting, because they process information with low direct relevance to expression discrimination, such as the facial outline or background, with a weight similar to that of the core areas. In particular, although features of the facial outline region may be relatively low in terms of expression discrimination ability, conventional technologies have not sufficiently reflected these differences in importance by region.

[0007] Furthermore, the accuracy of facial expression recognition in real-world environments can be degraded by various disturbance factors. For instance, errors in the predicted location of facial landmarks may occur due to factors such as significant changes in facial posture, partial occlusion, lighting variations, image quality degradation, and background complexity. Conventional technology has not been able to sufficiently mitigate the impact of such landmark prediction errors on expression recognition performance; it suffers from the problem that even slight deviations in landmark locations lead to inaccurate feature extraction of key facial regions, causing a rapid decline in final expression recognition performance. In other words, conventional technology has the limitation of lacking robustness against landmark detection errors occurring in unstructured environments (in the wild). Prior art literature

[0009] Korean Registered Patent Publication No. 10-2188970 (2020.11.25) The problem to be solved

[0010] An embodiment of the present invention is intended to provide a facial expression recognition method capable of increasing the accuracy of facial expression recognition and a computing device for performing the same.

[0011] The embodiments of the present invention are intended to provide a facial expression recognition method capable of securing excellent robustness and generalization performance even in actual unstructured environments, and a computing device for performing the same. means of solving the problem

[0013] A method according to one disclosed embodiment is performed in a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, comprising: inputting a face image into a face alignment network to extract a first feature map containing geometric shape information of a face and a plurality of landmark coordinates of a face; inputting the face image into a backbone network to generate a second feature map containing external feature information of a face; fusing the first feature map and the second feature map to generate a global feature map; extracting keypoint features for facial expression classification based on the global feature map and the landmark coordinates; and classifying the facial expression of the face image based on the extracted keypoint features.

[0014] The step of generating the global feature map may include: a step of matching spatial sizes by performing upsampling or downsampling on at least one of the first feature map and the second feature map; and a step of combining the first feature map and the second feature map with matched spatial sizes in a channel direction, and passing the combined feature map through at least one convolutional layer to generate a global feature map in which geometric shape information of the face and external feature information of the face are reflected together.

[0015] The step of extracting the keypoint features above may extract keypoint features by selectively sampling feature vectors at locations on the global feature map corresponding to each of the multiple landmark coordinates.

[0016] The step of extracting the keypoint features may include: a step of obtaining feature values ​​at a plurality of adjacent grid points surrounding the landmark coordinates when the landmark coordinates are real value coordinates that do not correspond to integer grid points of the global feature map; and a step of calculating a feature vector of the landmark coordinate location by interpolating the feature values ​​according to the distance relationship between the landmark coordinates and each adjacent grid point.

[0017] The step of extracting keypoint features may include, in the learning step, generating perturbed landmark coordinates by adding a random offset of a preset range to each of the plurality of landmark coordinates, and extracting keypoint features at a location on the global feature map corresponding to the perturbed landmark coordinates.

[0018] The above offset is randomly selected from a uniform distribution or a probability distribution, and in the inference step, may further include the step of extracting keypoint features at locations on the global feature map corresponding to each of the original landmark coordinates to which the random offset was not added.

[0019] The step of classifying the facial expression may include: a step of setting a keypoint feature located near the geometric center of the face among a plurality of keypoint features as a representative feature; a step of replicating the representative feature to correspond to the number of landmarks; a step of calculating an attention score representing the importance of each keypoint feature by combining the replicated representative features and the plurality of keypoint features; and a step of re-weighting the plurality of keypoint features based on the attention score.

[0020] The above representative feature may be a keypoint feature extracted at a landmark location corresponding to the nasal base of the face.

[0021] The step of reweighting the keypoint features includes the step of generating attention weights by normalizing the attention scores with a softmax function, and the step of classifying facial expressions may further include the step of generating a final feature vector by multiplying the attention weights by each of the plurality of keypoint features and adding the original keypoint features using a residual linkage method; and the step of inputting the final feature vector into one or more fully connected layers to calculate classification scores for a plurality of expression classes and classifying facial expressions based on the calculated classification scores.

[0022] The step of extracting keypoint features includes generating perturbed landmark coordinates by adding a random offset of a preset range to each of the plurality of landmark coordinates, and extracting keypoint features at a location on the global feature map corresponding to the perturbed landmark coordinates; and the step of classifying facial expressions may include setting a keypoint feature located near the geometric center of the face among the plurality of keypoint features as a representative feature; replicating the representative feature to correspond to the number of the plurality of landmarks; calculating an attention score representing the importance of each keypoint feature by combining the replicated representative feature and the plurality of keypoint features; and reweighting the plurality of keypoint features based on the attention score.

[0023] A computing device according to one disclosed embodiment is a computing device comprising: a processor; and a memory for storing one or more programs executed by the processor, wherein the processor performs the operation of inputting a face image into a face alignment network to extract a first feature map containing geometric shape information of the face and a plurality of landmark coordinates of the face; inputting the face image into a backbone network to generate a second feature map containing external feature information of the face; fusing the first feature map and the second feature map to generate a global feature map; extracting keypoint features for facial expression classification based on the global feature map and the landmark coordinates; and classifying the facial expression of the face image based on the extracted keypoint features. Effects of the invention

[0025] According to the disclosed embodiment, by utilizing face landmark coordinates as a guide for directly extracting features of key areas on a global feature map, information around the eyes, nose, and mouth, where facial expression changes occur intensively, can be reflected more effectively. Accordingly, while reducing the influence of facial outline or background information that is not directly related to facial expression determination, local features that are significant for facial expression recognition can be utilized intensively, thereby improving the accuracy of facial expression recognition.

[0026] Furthermore, by fusing external feature information extracted from the backbone network with structural shape information extracted from the face alignment network to generate a global feature map, and by extracting keypoint features at locations corresponding to each of multiple landmark coordinates, a precise feature representation that reflects both the texture and geometric information of the face can be secured. In particular, by applying bilinear interpolation to landmark coordinates in the form of real values, quantization error is reduced, and feature vectors corresponding to landmark locations can be obtained more accurately, thereby further improving the extraction precision of local facial expression features.

[0027] In addition, by applying a normalization technique that adds random perturbations to landmark coordinates during the learning phase, facial expression features can be stably captured even in environments where some error occurs in the landmark detection location due to changes in lighting, occlusion, or orientation. Accordingly, even when the landmark location is slightly off, surrounding contextual information can be learned together, thereby ensuring excellent robustness and generalization performance in real-world unstructured environments.

[0028] Furthermore, by readjusting the importance of each keypoint feature based on a representative feature corresponding to the geometric center of the face, adaptive feature enhancement reflecting the relationship between the central part of the face and areas of facial expression change is possible. Accordingly, features in areas important for distinguishing expressions, such as the corners of the mouth and the eyes, are further emphasized, and features with relatively low discriminative power can also be significantly corrected, enabling more precise recognition of even subtle changes in facial expressions.

[0029] Furthermore, compared to complex methods that process the entire face area in a batch, high facial expression recognition performance can be achieved while effectively reducing unnecessary background noise through landmark-based selective feature extraction and attention-based feature correction. Therefore, since the present invention ensures both accuracy and efficiency while maintaining a relatively simple structure, it can be effectively applied to various fields such as intelligent driver monitoring systems, emotional interaction robots, non-face-to-face consultation systems, and learner immersion analysis systems. Brief explanation of the drawing

[0031] FIG. 1 is a block diagram showing the configuration of a face expression recognition device according to one embodiment of the present invention. FIG. 2 is a diagram showing the overall framework of a face expression recognition model in one embodiment of the present invention. FIG. 3 is a diagram illustrating the process of extracting keypoint features by landmark perturbation in one embodiment of the present invention. Figure 4 is a diagram showing the change in the distribution of keypoint features before and after being adjusted by attention weights in one embodiment of the present invention. FIG. 5 is a flowchart illustrating a facial expression recognition method according to an embodiment of the present invention. FIG. 6 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. Specific details for implementing the invention

[0032] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0033] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0034] Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0036] FIG. 1 is a block diagram showing the configuration of a face expression recognition device according to one embodiment of the present invention, and FIG. 2 is a diagram showing the overall framework of a face expression recognition model in one embodiment of the present invention.

[0037] Referring to FIGS. 1 and 2, the face expression recognition device (100) may include a global feature generation module (102) and a keypoint feature processing module (104). The face expression recognition device (100) is a device for artificial neural network-based face expression recognition. The face expression recognition device (100) has a face expression recognition model, and the face expression recognition model may be implemented as a global feature generation module (102) and a keypoint feature processing module (104).

[0038] The global feature generation module (102) can generate a global feature map by receiving a face image that is the subject of facial expression recognition. Here, the face image may be a video captured in real-time from a camera, or it may be received from a storage medium or an external server. The face image may be a static image containing a single face, but is not limited thereto, and may be each frame of a video consisting of multiple frames.

[0039] The global feature generation module (102) may include a face alignment network (111), a backbone network (113), and a feature fusion unit (115).

[0040] The face alignment network (111) may include a neural network that receives a face image as input and extracts landmark coordinates of the face. The face alignment network (111) may be configured to extract geometric shape information of the face from the input face image and to predict multiple face landmarks.

[0041] The face alignment network (111) can generate a first feature map containing geometric shape information from a face image. The face alignment network (111) can extract landmark coordinates corresponding to the locations of major parts of the face, such as eyes, eyebrows, nose, mouth, and face contours, from the first feature map. In one embodiment, the face alignment network (111) can extract 68 landmark coordinates corresponding to the eyes, eyebrows, nose, mouth, face contours, etc. of the face, but the number of landmarks is not limited thereto. Landmark coordinates can be expressed as real-valued coordinates. Landmark coordinates can be used as guide information for selectively extracting feature vectors of corresponding locations on a global feature map.

[0042] The backbone network (113) can receive a face image as input and generate a second feature map containing visual information of the face (i.e., external feature information). The backbone network (113) can generate a second feature map corresponding to the skin texture, fine wrinkles, changes in brightness and contrast, and local texture changes according to facial expressions from the face image. In one embodiment, the backbone network (113) may be composed of a deep neural network including a plurality of convolutional layers and residual blocks. At this time, a plurality of feature maps can be output at different layer depths. The backbone network (113) may be arranged in parallel with the face alignment network (111) to process the face images respectively.

[0043] The feature fusion unit (115) can generate a global feature map by fusing the first feature map of the face alignment network (111) and the second feature map of the backbone network (113). That is, since the first feature map and the second feature map have different resolutions and number of channels, the spatial size is matched through upsampling or downsampling, and then combined in the channel direction.

[0044] The feature fusion unit (115) can generate a global feature map in which geometric shape information of the face and external feature information of the face are reflected together by performing upsampling or downsampling on the first feature map and the second feature map, respectively, to align spatial sizes and combine them in the channel direction, and then passing through at least one convolutional layer. The global feature map includes high-density feature representations for the entire face area and can be used for local feature extraction centered on landmark locations in the keypoint feature processing module (104) described later.

[0045] In one embodiment, the feature fusion unit (115) can generate a global feature map by fusing the first feature map and the second feature map according to the following mathematical formula 1.

[0046] (Mathematical Formula 1)

[0047]

[0048] : Global Features Map

[0049] : i-th 3×3 convolutional layer

[0050] : Feature map of the j-th block hierarchy

[0051] BBN: Index representing the backbone network

[0052] FAN: Index representing the face alignment network

[0053] D: Downsampling

[0054] U: Upsampling

[0055] : An operation that concatenates feature maps along the channel dimension

[0056] To explain Mathematical Formula 1, The dimension of is is, and is downsampled It becomes. The dimension of is is, and without change in size It passes through. The dimension of is is, and is upsampled After becoming As it passes through, the dimension is reduced, and finally It becomes. The dimension of is It is. If you connect each of these A feature map of the size is generated, and then A global feature map of the same dimension passing through is generated.

[0057] The keypoint feature processing module (104) can extract keypoint features corresponding to each landmark location based on a global feature map and landmark coordinates, and classify facial expressions based on the keypoint features. The keypoint feature processing module (104) may include a keypoint extraction unit (121), a landmark perturbation unit (123), an attention unit (125), and an expression classification unit (127).

[0058] The keypoint extraction unit (121) can extract keypoint features corresponding to each landmark location based on a global feature map and landmark coordinates. Instead of using the entire global feature map collectively, the keypoint extraction unit (121) can extract keypoint features by selectively sampling feature vectors from locations on the global feature map corresponding to each of the multiple landmark coordinates. Through this, information on local areas sensitive to changes in facial expression, such as the eyes, nose, and mouth, can be utilized intensively. Here, the keypoint features can be defined by the following mathematical formula 2.

[0059] (Mathematical Formula 2)

[0060]

[0061] Key Features

[0062] : Landmark coordinates

[0063] : x-coordinates of the landmark

[0064] : y-coordinates of the landmark

[0065] Here, if the landmark coordinates are real value coordinates that do not correspond to integer grid points, the key point extraction unit (121) can precisely extract the feature vector of the corresponding coordinate location using bilinear interpolation.

[0066] Specifically, when the landmark coordinate does not match an integer grid point of the global feature map, the keypoint extraction unit (121) can obtain feature values ​​from a plurality of adjacent grid points surrounding the landmark coordinate and calculate a feature vector of the landmark coordinate location by interpolating the feature values ​​according to the distance relationship between the landmark coordinate and each adjacent grid point. For example, when the keypoint extraction unit (121) obtains feature values ​​from four adjacent grid points surrounding the landmark coordinate, it can calculate a weight for the feature value of each grid point based on the relative distance between the landmark coordinate (i.e., real value coordinate) and the four adjacent grid points, and generate a feature vector corresponding to the landmark coordinate by weighting the feature values ​​at the four grid points using the calculated weight.

[0067] Accordingly, it is possible to stably obtain keypoint features that are meaningful for facial expression discrimination while reducing quantization error. The keypoint extraction unit (121) can extract keypoint features using double linear interpolation as shown in Equation 3 below.

[0068] (Mathematical Formula 3)

[0069]

[0070] : bilinear interpolation function

[0071] The landmark perturbation unit (123) can add a random offset (i.e., noise) to the landmark coordinates to ensure robustness against landmark prediction errors during the learning process. That is, since keypoint features are extracted based on the landmark coordinates of the face, they essentially depend on the accuracy of the landmark coordinates. However, in real environments, landmark prediction may become inaccurate due to factors such as large changes in posture or occlusion. Accordingly, by introducing a random perturbation to the landmark coordinates, robustness against landmark prediction errors is ensured.

[0072] The landmark perturbation unit (123) can generate perturbed landmark coordinates by adding noise that follows a probability distribution or a uniform distribution within a pre-set range for each landmark coordinate. At this time, the keypoint extraction unit (121) can extract keypoint features using the perturbed landmark coordinates during the learning process. This allows each neural network to learn not only the accurate landmark point but also the surrounding contextual information together. Extracting keypoint features using perturbed landmark coordinates can be represented by Equation 4.

[0073] (Mathematical Formula 4)

[0074]

[0075]

[0076] : Predicted landmark coordinates

[0077] : [ Uniform distribution within the range

[0078] : An indicator function that is 1 during learning and 0 otherwise

[0079] According to mathematical formula 4, in the learning process Noise (random offset) sampled from is added to the landmark coordinates predicted by the face alignment network (111), after which keypoint extraction is performed. On the other hand, in the inference process, the predicted landmark coordinates are used as is without perturbation. Hyperparameters defines the maximum magnitude of the perturbation and can be adjusted to optimize robustness. Here, a uniform distribution was used, but it is not limited to this, and other probability distributions such as a Gaussian distribution may also be used.

[0080] FIG. 3 is a diagram illustrating the process of extracting keypoint features by landmark perturbation in one embodiment of the present invention. Referring to FIG. 3, a random offset is added to the landmark coordinates during the learning process, thereby introducing intentional noise into the keypoint features. This method provides a normalization effect by generating somewhat inaccurate keypoint features and improves the generalization performance of the neural network on data containing noise.

[0081] The attention unit (125) may set one of the multiple keypoint features as a representative feature. Among the multiple keypoint features, some keypoint features located along the outer edge of the face tend to have relatively low expression classification performance. However, since these keypoint features also still contain unique and useful information, completely excluding these keypoint features does not yield optimal expression classification results. In the embodiment disclosed herein, in order to improve the expression classification performance of all keypoint features while preserving the contribution of each keypoint feature, one of the multiple keypoint features may be set as a representative feature, and an attention score for each keypoint feature may be calculated based on the representative feature.

[0082] Specifically, the attention unit (125) may set a key point feature at a landmark located near the geometric center of the face as a representative feature among multiple key point features. For example, the attention unit (125) may set a key point feature at a nasal base landmark as a representative feature (P). Since the nasal base moves the least in position even when facial expressions change, it can serve as the geometric center of the face, i.e., a reference point. However, it is not limited to this, and a key point feature at the tip of the nose may also be set as a representative feature.

[0083] The attention unit (125) can generate combined features by replicating representative features as many times as the number of landmarks and combining them with each keypoint feature. In one embodiment, the attention unit (125) can combine each representative feature with each keypoint feature in the channel direction. The attention unit (125) can input the combined features into a fully connected layer to calculate an attention score representing the importance of each keypoint feature.

[0084] The attention unit (125) can convert the calculated attention score into an attention weight by normalizing it through a softmax function. The attention weight can be applied to refine each keypoint feature on a channel-wise basis. The attention unit (125) can generate a final feature vector by multiplying the attention weight by each keypoint feature to readjust the importance of each keypoint feature. At this time, the final feature vector can be generated while suppressing information loss by adding the original keypoint features using a residual linkage method. The generation of the final feature vector can be represented by the following mathematical equation 5.

[0085] (Mathematical Formula 5)

[0086]

[0087] : Final feature vector

[0088] Key Features

[0089] : Fully connected layer

[0090] N: Number of landmarks

[0091] : Operator that repeats N times

[0092] Figure 4 is a diagram showing the change in the distribution of keypoint features before and after adjustment by attention weights in one embodiment of the present invention. Referring to Figure 4, the non-coarse keypoint features show high class separation even before application, and the face outline keypoint features show that the distribution between classes is more clearly separated after the application of RKFA (Representative Keypoint Feature Attention), thereby improving discriminability.

[0093] The facial expression classification unit (127) can classify facial expressions based on the final feature vector. The facial expression classification unit (127) may include one or more fully connected layers. For example, the facial expression classification unit (127) may include a first fully connected layer (127a) and a second fully connected layer (127b). After transforming the feature dimension in the first fully connected layer (127a), an activation function and a normalization operation are performed, and a classification score corresponding to one of a plurality of facial expression classes can be output in the second fully connected layer (127b).

[0094] Here, the expression classes may include, for example, happiness, sadness, surprise, fear, anger, disgust, and neutrality, but are not limited thereto. The expression classification unit (127) may output the expression class having the highest classification score as the recognition result, or output probability values ​​for each expression class together.

[0095] The face expression recognition device (100) can control each component to be learned in an end-to-end manner. The face expression recognition device (100) can calculate a loss value based on the error between the correct label and the expression classification result, and update the parameters of each neural network using a backpropagation algorithm.

[0096] In one embodiment, the face expression recognition device (100) may use a cross-entropy function as a loss function. At this time, the loss function ( ) can be expressed by the following mathematical formula 6. The face expression recognition device (100) has a loss function ( The parameters of the face expression recognition model can be trained so that ) is minimized.

[0097] (Mathematical Formula 6)

[0098]

[0099] : Total number of expression classes

[0100] : Probability of predicting the input face image as the c-th expression class

[0101] : Correct answer value of the expression class for the input face image

[0102] The face expression recognition device (100) converts the output value of the expression classification unit (127) into a probability value, and at this time, a softmax function may be applied so that the sum of the probability values ​​of all expression classes becomes 1. Probability value ( ) can be expressed by the following mathematical formula 7.

[0103] (Mathematical Formula 7)

[0104]

[0105] : Raw values ​​(logit) for each expression class output from the last layer of the face expression recognition model

[0106] In this specification, the term "module" may refer to a functional and structural combination of hardware for carrying out the technical concept of the present invention and software for driving said hardware. For example, the "module" may refer to a logical unit of a specific code and a hardware resource for executing said code, and does not necessarily refer to physically connected code or a single type of hardware.

[0108] FIG. 5 is a flowchart illustrating a facial expression recognition method according to an embodiment of the present invention. In the illustrated flowchart, the method is described by dividing it into a plurality of steps, but at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed with one or more steps not illustrated added.

[0109] Referring to FIG. 5, a face expression recognition device (100) can input a face image into a face alignment network to extract a first feature map containing geometric shape information of the face and landmark coordinates of a plurality of faces (S 101).

[0110] Additionally, the face expression recognition device (100) can input a face image into a backbone network to generate a second feature map containing external feature information of the face (S 103).

[0111] Next, the face expression recognition device (100) can generate a global feature map by fusing the first feature map and the second feature map (S 105).

[0112] Next, the face expression recognition device (100) can extract keypoint features for face expression classification based on a global feature map and multiple landmark coordinates (S 107).

[0113] Next, the face expression recognition device (100) can classify the face expression of a face image based on extracted keypoint features (S 109).

[0115] FIG. 6 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0116] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a device for receiving a face image and recognizing a facial expression. The computing device (12) may be a facial expression recognition device (100).

[0117] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0118] The processor (14) can perform technical features according to the disclosed embodiments by executing at least one instruction stored in a computer-readable storage medium (16). The processor (14) can train a neural network or model designed in a machine learning or deep learning manner. To this end, the processor (14) can perform calculations for training the neural network, such as processing input data for training, extracting features from input data, calculating loss, and updating the weights of the neural network using backpropagation. Additionally, the processor (14) can perform inference for a specific purpose using a model implemented in an artificial neural network manner.

[0119] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0120] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0121] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and the network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0123] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0125] 10: Computing Environment 12: Computing device 14 : Processor 16: Computer-readable storage media 18: Communication bus 20 : Program 22 : Input / Output Interface 24 : Input / Output Devices 26: Network communication interface 100: Facial expression recognition device 102 : Global Feature Generation Module 104: Keypoint Feature Processing Module 111: Face Alignment Network 113 : Backbone Network 115 : Feature Fusion Part 121 : Keypoint Extraction Unit 123 : Landmark East Perturbation 125 : Attention section 127 : Facial Expression Classification Unit

Claims

Claim 1 A method performed on a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, comprising: inputting a face image into a face alignment network to extract a first feature map containing geometric shape information of the face and a plurality of face landmark coordinates; inputting the face image into a backbone network to generate a second feature map containing external feature information of the face; aligning the spatial sizes of the first feature map and the second feature map, and combining the spatially aligned first feature map and the second feature map in a channel direction to generate a global feature map in which the geometric shape information and the external feature information are reflected together; and extracting a plurality of keypoint features based on the global feature map and the plurality of face landmark coordinates, wherein, in a learning step, a random offset is added to at least some of the plurality of face landmark coordinates to generate perturbed landmark coordinates and the keypoint features are extracted at a location on the global feature map corresponding to the perturbed landmark coordinates, and in an inference step, the keypoint features are extracted at a location on the global feature map corresponding to a face landmark coordinate to which the random offset is not added. A step of setting a keypoint feature extracted from face landmark coordinates located near the geometric center of the face among the plurality of keypoint features as a representative feature; a step of replicating the representative feature to correspond to the number of the plurality of keypoint features, and combining each of the replicated representative feature with each of the plurality of keypoint features to calculate an attention score representing the importance of each of the plurality of keypoint features;A method comprising the step of reweighting the plurality of keypoint features based on the attention score, and classifying the facial expression of the face image based on a final feature vector generated by adding the original keypoint features to the reweighted keypoint features using a residual linkage method. Claim 2 delete Claim 3 The method of claim 1, wherein the step of extracting keypoint features comprises extracting keypoint features by selectively sampling feature vectors at locations on the global feature map corresponding to each of a plurality of landmark coordinates. Claim 4 The method of claim 3, wherein the step of extracting keypoint features comprises: a step of obtaining feature values ​​at a plurality of adjacent grid points surrounding the landmark coordinates when the landmark coordinates are real value coordinates that do not correspond to integer grid points of the global feature map; and a step of calculating a feature vector of the landmark coordinate location by interpolating the feature values ​​according to the distance relationship between the landmark coordinates and each adjacent grid point. Claim 5 delete Claim 6 The method of claim 1, wherein the offset is randomly selected from a uniform distribution or a probability distribution. Claim 7 delete Claim 8 The method of claim 1, wherein the representative feature is a keypoint feature extracted at a landmark location corresponding to the nasal base of the face. Claim 9 The method according to claim 1, wherein the step of reweighting the keypoint features includes the step of generating attention weights by normalizing the attention scores with a softmax function, and the step of classifying the facial expressions includes the step of generating a final feature vector by multiplying the attention weights by each of the plurality of keypoint features and adding the original keypoint features in a residual linkage manner; and further including the step of inputting the final feature vector into one or more fully connected layers to calculate classification scores for a plurality of expression classes and classifying the facial expressions based on the calculated classification scores. Claim 10 delete Claim 11 A computing device comprising: a processor; and a memory for storing one or more programs executed by the processor, wherein the processor comprises: an operation of inputting a face image into a face alignment network to extract a first feature map containing geometric shape information of the face and a plurality of face landmark coordinates; an operation of inputting the face image into a backbone network to generate a second feature map containing external feature information of the face; an operation of aligning the spatial sizes of the first feature map and the second feature map, and combining the spatially aligned first feature map and the second feature map in a channel direction to generate a global feature map in which the geometric shape information and the external feature information are reflected together; and an operation of extracting a plurality of keypoint features based on the global feature map and the plurality of face landmark coordinates, wherein in a learning step, a random offset is added to at least some of the plurality of face landmark coordinates to generate perturbed landmark coordinates and the keypoint features are extracted at a location on the global feature map corresponding to the perturbed landmark coordinates, and in an inference step, the keypoint features are extracted at a location on the global feature map corresponding to a face landmark coordinate to which the random offset is not added. A computing device that performs the operation of setting a keypoint feature extracted from face landmark coordinates located near the geometric center of the face among the plurality of keypoint features as a representative feature; the operation of replicating the representative feature to correspond to the number of the plurality of keypoint features, and combining each of the replicated representative feature with each of the plurality of keypoint features to calculate an attention score representing the importance of each of the plurality of keypoint features; and the operation of reweighting the plurality of keypoint features based on the attention score, and classifying the facial expression of the face image based on a final feature vector generated by adding the original keypoint feature to the reweighted keypoint feature using a residual linkage method. Claim 12 delete Claim 13 delete Claim 14 A computing device according to claim 11, wherein the offset is randomly selected from a uniform distribution or a probability distribution. Claim 15 delete Claim 16 A computing device according to claim 11, wherein the operation of reweighting the keypoint features includes the operation of normalizing the attention score with a softmax function to generate attention weights, and the processor further performs the operation of multiplying the attention weights by each of the plurality of keypoint features and adding the original keypoint features in a residual linkage manner to generate a final feature vector; and inputting the final feature vector into one or more fully connected layers to calculate classification scores for a plurality of expression classes and classifying facial expressions based on the calculated classification scores. Claim 17 delete Claim 18 A computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises one or more instructions, and when the instructions are executed by a computing device having one or more processors, the computing device comprises the steps of: inputting a face image into a face alignment network to extract a first feature map containing geometric shape information of the face and a plurality of face landmark coordinates; inputting the face image into a backbone network to generate a second feature map containing external feature information of the face; aligning the spatial sizes of the first feature map and the second feature map, and combining the spatially aligned first feature map and the second feature map in a channel direction to generate a global feature map in which the geometric shape information and the external feature information are reflected together; extracting a plurality of keypoint features based on the global feature map and the plurality of face landmark coordinates, wherein in the learning step, a random offset is added to at least some of the plurality of face landmark coordinates to generate perturbed landmark coordinates, and the keypoint features are extracted at a location on the global feature map corresponding to the perturbed landmark coordinates, and inference In the step, the method comprises: extracting the keypoint feature at a location on the global feature map corresponding to the face landmark coordinates to which the random offset is not applied; setting the keypoint feature extracted at the face landmark coordinates located near the geometric center of the face among the plurality of keypoint features as the representative feature; replicating the representative feature to correspond to the number of the plurality of keypoint features, and combining each of the replicated representative feature with each of the plurality of keypoint features to calculate an attention score representing the importance of each of the plurality of keypoint features.A computer program stored on a non-transient computer-readable storage medium, which performs the step of classifying facial expressions of a face image based on a final feature vector generated by reweighting the plurality of keypoint features based on the attention score and adding the original keypoint features to the reweighted keypoint features using a residual linkage method.

Images