Expression recognition model training method and system based on head posture and facial key points

Abstract

The application provides an expression recognition model training method and system based on head posture and facial key points, and the training method comprises the following steps: first, acquiring multiple frames of training images and corresponding head postures of the same target object as training data; then, pre-processing the data, and constructing a double-frame training network model based on a multilayer perception mechanism; subsequently, importing the pre-processed data, selecting any two frames of images for mixed training, and acquiring common and different characteristics between different data; in the training process or after the training is completed, selecting two frames of untrained images to verify the training effect; if the verification result is the same as the imported image, the model training is completed. The technical problem solved by the application is that: the existing training set focuses on the front view scene, and the facial feature change under the non-front view posture is easy to cause judgment deviation; and the individual facial feature difference is large, the normal feature of part of people is close to the abnormal expression of others, the traditional model is not optimized for individuals in the multi-person mixed training, relies on the general threshold, and is easy to misjudge the physiological difference as the expression change, resulting in recognition error.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of expression recognition, in particular to an expression recognition model training method and system based on head posture and facial key points. BACKGROUND

[0002] With the rapid development of computer vision technology, facial expression recognition is increasingly widely used in intelligent interaction, security monitoring, emotional computing and other fields. Its core requirement is to accurately determine the expression state (such as normal, raised eyebrows, smiling, etc.) of a person through image or video data.

[0003] However, there are at least one of the following problems in the related art: the training is focused on the front view scene, when the identified person is in a non-front view posture such as looking down or looking up, the relative position of the facial feature points, the contour and the visual presentation of the muscle movement change significantly, resulting in deviation of the model in judging the expression; and the facial features of different individuals are quite different, the facial features of some people in a normal state may be close to the non-normal expression features of others, and the traditional model uses a multi-person mixed training mode, is not optimized for individual features, and only relies on general feature thresholds, which is easy to misjudge the individual physiological differences as expression changes, resulting in expression recognition errors. SUMMARY

[0004] The technical problem solved by the present application is that the training is focused on the front view scene, when the identified person is in a non-front view posture such as looking down or looking up, the relative position of the facial feature points, the contour and the visual presentation of the muscle movement change significantly, resulting in deviation of the model in judging the expression; and the facial features of different individuals are quite different, the facial features of some people in a normal state may be close to the non-normal expression features of others, and the traditional model uses a multi-person mixed training mode, is not optimized for individual features, and only relies on general feature thresholds, which is easy to misjudge the individual physiological differences as expression changes, resulting in expression recognition errors.

[0005] To solve the above problems, the present application provides an expression recognition model training method based on head posture and facial key points, comprising: obtaining multiple frames of training images of the same target object and the head posture corresponding to each frame of training image, denoted as training data; preprocessing the training data, and building a double-frame training network model based on a multi-layer perception mechanism; importing the preprocessed training data into the double-frame training network model, and selecting any two frames of training images for mixed training to obtain common characteristics and difference characteristics between different training data; in the mixed training process, and / or after the mixed training is completed, any two frames of untrained training images are selected for training effect verification; if the verification result is the same as the imported training image, the double-frame training network model training is completed.

[0006] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: compared with the related art in which the facial expression recognition training set is focused on the front view, the present application first focuses on individual characteristics, uses only the data of the same target object for training, makes the model focus on learning the facial feature rules of the object, avoids the interference of physiological differences between different individuals on expression judgment, then combines the head posture data and image features, learns through double-frame comparison, makes the imported double-frame training network model understand the feature deformation rules of the same expression in different postures, so that the model can still be stably recognized in a non-front view scene, and based on a multi-layer perception mechanism, compared with a convolution network, is better at processing structured data, that is, the preprocessed key points and posture data are imported into the double-frame training network model, the inference speed is improved by tens of times, and the real-time interaction demand is met, finally, through the verification process during and after training, the model overfitting is avoided, and accurate recognition is ensured in untrained postures and / or expression combinations.

[0007] In an example of the present application, a plurality of training images of the same target object and the head posture corresponding to each training image are obtained, denoted as training data, including: performing face model recognition on any single training image, and framing the image training area; the image training area contains at least one pair of training elements; a plurality of contour points are set around the training elements; a head posture coordinate system is established in any single training image, and the head posture coordinate corresponding to the training image is obtained; a single-frame image data association is established based on the plurality of contour points in the single training image and the head posture coordinate corresponding to the single training image, and single-frame training data corresponding to the training image is generated; wherein the training elements include any facial feature.

[0008] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: by framing the image training area containing the training elements of facial features such as eyebrows and eyes, redundant information is excluded, the model only focuses on the key facial areas related to expressions, such as the relative movement of eyebrows and eyes in the action of raising eyebrows, and the feature extraction efficiency is improved; a plurality of contour points are set around the training elements, the expression change is converted into quantifiable point coordinate change, and specific data is provided for subsequent normalization and comparison learning; finally, the contour points are bound with the posture data through the head posture coordinate system, so that the model can distinguish whether the contour point change is caused by posture or expression, and lay a foundation for subsequent common and difference characteristic learning.

[0009] In an example of the present application, the training data is preprocessed, including: normalizing the plurality of contour point coordinates to obtain first data; normalizing the head posture coordinates to obtain second data; establishing a single-frame image data association based on the plurality of contour points in the single training image and the head posture coordinate corresponding to the single training image, including: splicing the first data and the second data to form the single-frame training data of the single training image.

[0010] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: dividing the contour point coordinates by the width and height of the image training area and dividing the head posture angle by 90° enable all data to be mapped to a unified range, so that the model can focus on the relative relationship rather than the absolute coordinates; finally, the normalized contour point data and the head posture data are spliced to form single-frame training data containing facial features and posture information, so that the model can simultaneously perceive the correlation between the two, and provide complete features for double-frame comparison.

[0011] In an example of the present application, the preprocessed training data is imported into a double-frame training network model, and any two frames of training images are selected for mixed training to obtain common characteristics and difference characteristics between different training data, including: selecting any two single-frame training data for team training to obtain double-frame training data; wherein the team training includes splicing processing of the selected two single-frame training data; the double-frame training data is imported into the double-frame training network model; the double-frame training network model is controlled to perform comparative learning on the training elements in the double-frame training data to obtain the common characteristics and difference characteristics of the selected two single-frame spliced data; the double-frame training network model is controlled to output the recognition result of the double-frame training data; wherein the common characteristics at least include the relative distance between adjacent contour points under the same expression at different head postures; the difference characteristics at least include the relative distance between contour points under different expressions at the same head posture.

[0012] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: first, the common characteristics are determined, i.e., the relative distance between adjacent contour points under the same expression at different head postures is learned, so that the model understands that the core feature ratio of the expression does not change with the change of the posture, and avoids misjudgment of the expression due to the change of the posture; then, the difference characteristics are captured, i.e., the relative distance between contour points under different expressions at the same head posture is learned, so that the model can accurately identify the expression change under the same posture; finally, through comparison of the two frames of data, the model focuses on the change trend of the expression rather than the single-frame static feature, and the accuracy of dynamic expression recognition is improved.

[0013] In an example of the present application, the preprocessed training data is imported into a double-frame training network model, and any two frames of training images are selected for mixed training to obtain common characteristics and difference characteristics between different training data, and further including: at least three groups of expression combination labels are preset according to the double-frame training data, wherein the image expression combination labels correspond to the expression state changes of the two frames of images respectively; the double-frame training network model is controlled to output the recognition result of the double-frame training data, including: when the double-frame training data is mixed trained, the probability result distribution of the at least three groups of expression combination labels corresponding to the double-frame training data is output; the expression combination label with the maximum probability is selected as the output result.

[0014] Compared with the prior art, the technical effects reached by adopting the technical scheme are as follows: the preset combination label enables the model to not only identify a single-frame expression but also clearly identify the expression change trend between two frames, adapt to a dynamic scene, and output a probability distribution of at least three types of labels to provide a credibility reference for the identification result, so that the probability value can be adjusted for judgment in a fuzzy scene; finally, the maximum probability label is selected as the output to avoid the singleness of a single label, so that the model can still stably output the most possible result in a complex scene.

[0015] In an example of the present application, during the mixed training process and / or after the mixed training is completed, any two frames of untrained training images are selected for training effect verification, including: selecting two single-frame training data that have not been trained in combination to perform splicing processing to obtain verification data; importing the verification data into the double-frame training network model to obtain the output result of the verification data; if the output result is the same as the imported two frames of training images, the double-frame training network model passes the verification.

[0016] Compared with the prior art, the technical effects reached by adopting the technical scheme are as follows: the verification data comes from single-frame data that has not participated in the combination training, ensuring that the model is tested under unseen poses and / or expression combinations to prevent the model from memorizing the training data; the consistency between the verification result and the actual image can be used to quantize the accuracy of the model in different scenes, ensuring that the model is stable and effective in actual application.

[0017] In an example of the present application, after the double-frame training network model is trained, the expression recognition model training method further includes expression recognition prediction based on the double-frame training network model, specifically including: obtaining real-time multi-frame expression images of a target object, and importing the preprocessed real-time multi-frame expression images into the double-frame training network model; selecting two real-time single-frame training data at any frame interval to perform splicing processing to form real-time double-frame training data; importing the real-time double-frame training data into the double-frame training network model, and controlling the double-frame training network model to output the probability distribution of the expression combination label corresponding to the real-time double-frame training data; selecting the expression combination label with the maximum probability as a prediction result, and determining the expression state change of the target object according to the prediction result.

[0018] Compared with the prior art, the technical effects reached by adopting the technical scheme are as follows: the preprocessing of the real-time image is completely consistent with the processing method of the training data, ensuring that the feature distribution input into the model matches that during the training, and avoiding a decrease in recognition accuracy caused by a difference in data distribution; the double-frame data at any frame interval can be extracted to flexibly adapt to expressions of different dynamic speeds, ensuring that the complete expression change process is captured; the expression combination label with the maximum probability can be used to clearly identify the expression change trend of the target object.

[0019] In an example of the present application, before acquiring the plurality of training images of the same target object and the plurality of training head poses corresponding to the training images, the expression recognition model training method further comprises: recording a plurality of expression images of the same target object under different head poses; according to the types of the expression images, classifying and labeling the plurality of expression images into first images and second images; and taking the classified and labeled plurality of expression images as training images.

[0020] Compared with the prior art, the technical effects achieved by adopting the technical solution are: the expression images of the same target object under different head poses are clearly recorded, ensuring that the training data contains all the pose scenes required by the model, and avoiding the situation that the model has no rules to learn under a specific pose due to data missing; the images are classified and labeled as first images and second images, so that the model learns the real feature difference between normal and abnormal expressions; and the standardized classification and labeling provides a basis for subsequent expression combination labels, ensuring the consistency of the label logic during double-frame training.

[0021] In an example of the present application, an expression recognition system based on head pose and facial key points is also provided, which can implement the expression recognition model training method in any of the above examples. The expression recognition system comprises: an acquisition module, which is used to acquire a plurality of training images of the same target object and a training head pose corresponding to each training image, denoted as training data; a preprocessing module, which is used to preprocess the training data; a model construction module, which constructs a double-frame training network model based on a multilayer perception mechanism; a training module, which is used to import the preprocessed training data into the double-frame training network model, and select any two training images for mixed training to obtain common characteristics and difference characteristics between different training data; and a verification module, which is used to select any two untrained training images for training effect verification during the mixed training process and / or after the mixed training is completed.

[0022] After adopting the technical solution of the present application, the following technical effects can be achieved:

[0023] (1) The present application first focuses on individual characteristics, and only uses the data of the same target object for training, so that the model focuses on learning the facial feature rules of the object, avoiding the interference of physiological differences between different individuals on expression judgment; then the head posture data and image features are combined, and through double-frame comparison learning, the imported double-frame training network model understands the feature deformation rules of the same expression under different postures, so that it can still be recognized stably in a non-frontal scene; and a multi-layer perception mechanism is built into the network, which is better at processing structured data than convolutional networks, that is, the preprocessed key points and posture data are imported into the double-frame training network model, the inference speed is improved by tens of times, and the real-time interaction demand is met; finally, through the verification process during and after training, overfitting of the model is avoided, and accurate recognition is ensured under untrained postures and / or expression combinations;

[0024] (2) The present application trains the image training area containing facial features such as eyebrows and eyes, excludes redundant information, and makes the model only pay attention to the key facial area related to expression, such as the relative movement of eyebrows and eyes in the action of raising eyebrows, to improve the feature extraction efficiency; a plurality of contour points are set around the training elements, the expression change is converted into quantifiable point coordinate change, and specific data is provided for subsequent normalization and comparison learning; finally, the contour points are bound with the posture data through the head posture coordinate system, so that the model can distinguish whether the contour point change is caused by posture or expression, laying a foundation for subsequent common and difference characteristic learning;

[0025] (3) The present application first learns the relative distance of adjacent contour points under the same expression in different head postures, so that the model understands that the core feature ratio of the expression does not change with the change of the posture, and avoids misjudgment of the expression due to the change of the posture; then the relative distance of contour points under the same head posture is learned, so that the model can accurately identify the expression change under the same posture; finally, through the comparison of two frames of data, the model pays attention to the change trend of the expression rather than the single-frame static feature, improving the accuracy of dynamic expression recognition. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating laborious work;

[0027] Figure 1 A flowchart of an expression recognition model training method based on head posture and facial key points is provided for the embodiments of the present application;

[0028] Figure 2 For Figure 1A specific flowchart of the middle S6;

[0029] Figure 3 An expression image schematic diagram provided by the present application;

[0030] Figure 4 A profile point and head posture splicing processing schematic diagram provided by the present application;

[0031] Figure 5 A structure schematic diagram of double-frame training data provided by the present application;

[0032] Figure 6 A module schematic diagram of an expression recognition system based on head posture and facial key points provided by the present application.

[0033] Explanation of reference signs:

[0034] 100, expression recognition system; 10, acquisition module; 20, preprocessing module; 30, model construction module; 40, training module; 50, verification module; 60, image training area. DETAILED DESCRIPTION

[0035] The embodiments of the present application will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts under the premise that the present application falls within the scope of protection of the present application.

[0036] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, it can be fixedly connected, or connected, or integrally connected; it can be mechanically connected, or electrically connected; it can be directly connected, or indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0037] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0038] The image learning method of the expression recognition model in the related art mostly adopts the mode of "single-frame static feature" and "multi-person mixed training"; it usually collects a large number of facial expression images of different individuals, focuses on the expression features in the front view posture; after extracting the high-bit features of the single-frame image through the convolutional neural network, the mapping learning from "expression category" to "label annotation" is directly performed; for example, when training the "eyebrow raising" expression recognition, the model learns the average distance threshold of the eyebrows and the eyes / cheeks of all individuals when raising eyebrows, and uses it as the judgment standard; the core logic of this method is to learn the universal law of "human expression" through massive data, but there are limitations: first, the adaptability to non-front view posture is poor, when the identified person is looking down or is in profile, the geometric deformation of the facial features will break the average threshold, leading to model misjudgment; second, individual differences are ignored, for example, different people have different eyebrow bone heights and eyebrow thicknesses, and the unified average threshold is difficult to adapt to the actual features of each person, and the recognition robustness is low.

[0039] In view of the above limitations, the present application uses double-frame dynamic contrast combined with individual exclusive training, only uses the image data of the same target object for training, avoiding the interference of physiological characteristics of different individuals; instead of learning "the commonality of eyebrow raising of all people", the model focuses on "the eyebrow raising feature change law of a certain person in different postures", and this individual exclusive "shortened ratio" will be remembered by the model and become the basis for recognition; on this basis, the head posture data and the facial contour point features are normalized and fused to form the joint representation of "spatial features combined with posture features"; when learning, the model will pay attention to "the relative position change of the contour points" and "the associated influence of the posture angle", for example, when detecting that the yaw angle increases, the model will automatically call the individual feature library in this posture, instead of relying on the judgment standard in the front view, so as to maintain the recognition stability in the non-front view scene.

[0040] Referring to Figure 1 , Figure 1 A flowchart of an expression recognition model training method based on head posture and facial key points is provided for the embodiments of the present application; combined Figures 2 to 5 The expression recognition model training method based on head posture and facial key points comprises:

[0041] S1: recording a plurality of expression images of the same target object in different head postures;

[0042] S2: classifying and marking the plurality of expression images according to the types of the expression images, and dividing them into first images and second images;

[0043] S3: taking the plurality of expression images classified and marked as training images;

[0044] S4: obtaining a plurality of training images of the same target object and the head posture corresponding to each training image, denoted as training data;

[0045] S5: Preprocess the training data and build a double-frame training network model based on a multilayer perception mechanism;

[0046] S6: Import the preprocessed training data into the double-frame training network model and select any two frames of training images for mixed training to obtain common characteristics and difference characteristics between different training data;

[0047] S7: During the mixed training process, and / or after the mixed training is completed, select any two frames of untrained training images for training effect verification;

[0048] S71: If the verification result is the same as the imported training image, the double-frame training network model training is completed;

[0049] S72: If the verification result is different from the imported training image, re-mix the training of the two frames of training images.

[0050] In combination with specific use cases, before image training, first record multiple expression images of the same target object under different head poses; and according to the type of the recorded expression images, classify all the expression images and mark them as first images and second images; for example, the recorded expression images are divided into two categories, including: normal expression images and eyebrow-raising images; the first image is a normal expression image, and the second image is an eyebrow-raising image; after each training image is marked, it is used as a training image.

[0051] In the image processing stage, first, multiple frames of training images of the same target object and the head pose corresponding to each training image are obtained, so that each frame of training image can obtain its corresponding training data, and the training data of each frame of image in the training set is made; then the training data of each frame of image in the training set is preprocessed, and a double-frame training network model is built based on a multilayer perception mechanism; the preprocessed training data is imported into the double-frame training network model, and the training data of any two frames of training images are randomly selected for mixed training to obtain the common characteristics and difference characteristics between the two frames of training images; wherein each training image has a specific image number.

[0052] Further, in the mixed training process, and after the mixed training is completed, any two frames of training images that are not combined for training can be selected for training effect verification; for example, during the training process, every 10 iterations, 20 training images are extracted from the reserved images and input into the double-frame training network model for verification; similarly, after the training data for the same target object is trained, the remaining training images that are not combined for training are used for effect verification again; during the verification, the double-frame training network model compares the output results of the several groups of training images with the actual expression labels of the images, if the matching rate exceeds the preset threshold, the verification is passed, the double-frame training network model is trained and the parameters are saved; if the output result of a certain group of verification data is wrong, the data is re-added to the training set and re-trained until the verification is passed.

[0053] Further, please refer to Figure 3 , obtain multiple frames of training images of the same target object and the head posture corresponding to each frame of training image, denoted as training data, including:

[0054] Perform face model recognition on any single frame of training image, and frame the image training area; the image training area contains at least a pair of training elements;

[0055] A plurality of contour points are set around the training elements;

[0056] A head posture coordinate system is established in any single frame of training image, and the head posture coordinate corresponding to the training image is obtained;

[0057] Single frame image data association is established based on the plurality of contour points in the single frame of training image and the head posture coordinate corresponding to the single frame of training image, and single frame training data corresponding to the training image is generated;

[0058] Wherein, the training elements include any facial feature.

[0059] Preferably, the training elements at least include: eyes, eyebrows, lips and nose, and the like facial features.

[0060] In order to facilitate the understanding of the present scheme, the present application selects eyebrows and eyes as the training elements in the embodiments of the present application.

[0061] In combination with a specific use case, after a plurality of training images of the same target are acquired, each training image is processed. First, a face model is recognized in each training image, and then an image training area 60 corresponding to the face is framed. The image training area 60 contains at least the training elements of eyes and eyebrows. A plurality of contour points are set around the outer contour of the eyes and eyebrows. Then, a head posture coordinate system is established in the training image, and the head posture coordinate corresponding to the training image is acquired. Finally, the plurality of contour points and the head posture coordinate in the image are associated to generate single-frame training data corresponding to the image.

[0062] For example, there are 1000 recorded images, including 500 normal expression images and 500 raised eyebrow images. Then, 1000 single-frame training data are generated in the training image processing stage. Each single-frame training data includes a plurality of contour points and a head posture coordinate in the training image.

[0063] Further, please refer to Figure 4 The training data is preprocessed, including:

[0064] The plurality of contour point coordinates are normalized to obtain first data.

[0065] The head posture coordinate is normalized to obtain second data.

[0066] The single-frame image data association is established based on the plurality of contour points in the single-frame training image and the head posture coordinate corresponding to the single-frame training image, including:

[0067] The first data and the second data are spliced to form single-frame training data of the single-frame training image.

[0068] For example, the size of the framed image training area 60 is 120 pixels in length and 84 pixels in width. The coordinate value of each contour point is (Xu, Yu). X represents the horizontal position of the contour point in the image training area 60, Y represents the height position of the contour point in the image training area 60, and u represents the corresponding image number. The normalization of the contour point coordinates includes dividing the coordinates of the contour points on the outer contour of the eyebrows by the length and height dimensions of the image training area 60 to normalize the contour point coordinate information to the range of [0, 1] and obtain the normalized coordinates corresponding to the contour point, i.e. the first data. The normalization of the head posture coordinate includes dividing the three angles of the head posture coordinate by 90° to normalize the angle information of the head posture to the range of [0, 1]. Finally, any one contour point is selected as the basis, and the remaining contour points are arranged horizontally and flattened, and the normalized head posture coordinate is added to form the first data of the training image.

[0069] Referring to Figure 4 In one specific example, the size of the framed image training area 60 is 120 pixels in length W and 84 pixels in width H, respectively; the contour points of the selected eye and eyebrow are a total of 39; the coordinates of the leftmost Q1 contour point are (14, 50), and the coordinates of the adjacent Q2 contour point are (28, 41), and the coordinates of the Q3 and Q4 contour points are (45, 28) and (63, 15), respectively; the coordinates of the remaining contour points are not described in detail here; the contour points are normalized to satisfy the following formulas 1 and 2:

[0070] Formula 1: Norm_x1=Qz.X / W;

[0071] Formula 2: Norm_y1=Qz.Y / H;

[0072] Wherein Q represents the contour point, and z represents the arrangement number of the contour point.

[0073] Further, the normalized coordinates of all contour points are calculated, and 78 values containing X and Y dimensions are obtained, which together form the first data to eliminate the influence of size differences of different image training areas 60.

[0074] Further, the three angles in the coordinate system corresponding to the head pose are Pitch (pitch angle), Yaw (yaw angle) and Roll (roll angle); the values of the head pose in the current training image are Pitch: 4.52, Yaw: 35.2, and Roll: 2.84; and the normalized processing satisfies the following formulas 3, 4 and 5:

[0075] Formula 3: Norm_pitch=Pitch / 90;

[0076] Formula 4: Norm_yaw=Yaw / 90;

[0077] Formula 5: Norm_roll=Roll / 90;

[0078] Further, the normalized head pose angle values are calculated and form the second data.

[0079] Further, as shown in Figure 4 The 78 normalized contour point coordinates in the first data and the three normalized pose angles in the second data are formed into a single-frame training data by horizontal splicing, that is, a 1x81 vector data is finally formed; this splicing method integrates the facial feature contour points and the head pose angles into the same structure, which not only retains the "shift rule of eyebrow peak position relative to the training area when lowering the head", but also integrates the "quantitative information of the lowering angle", providing complete input for the double-frame training network model to learn the internal correlation between the two.

[0080] After preprocessing, single-frame images with different head poses and different training area sizes are all converted into structured data with unified dimensions and consistent scales, ensuring that the model can focus on the relative relationship of features during training, rather than absolute coordinates or angle values, thereby laying a standardized data foundation for subsequent double-frame mixed training.

[0081] Among them, the pitch angle refers to the rotation angle of the head around the horizontal axis (left-right direction), reflecting the "looking up" or "looking down" pose; the yaw angle refers to the rotation angle of the head around the vertical axis (up-down direction), reflecting the "turning head left" or "turning head right" pose; the roll angle refers to the rotation angle of the head around the front-back axis (perpendicular to the face direction), reflecting the "tilting head" pose.

[0082] Further, please refer to Figure 5 The preprocessed training data is imported into the double-frame training network model, and any two training images are selected for mixed training to obtain common and different characteristics between different training data, including:

[0083] S62: Select any two single-frame training data for team training to obtain double-frame training data; wherein the team training includes splicing processing of the selected two single-frame training data;

[0084] S63: Import the double-frame training data into the double-frame training network model;

[0085] S64: Control the double-frame training network model to perform comparative learning on the training elements in the double-frame training data to obtain the common and different characteristics of the selected two single-frame splicing data;

[0086] S65: Control the double-frame training network model to output the recognition result of the double-frame training data;

[0087] Among them, the common characteristics at least include the relative distance between two adjacent contour points of the same expression under different head poses; the different characteristics at least include the relative distance between the contour points of different expressions under the same head pose.

[0088] On the basis of normalizing the single-frame training image, any two frames of the normalized training image are selected for team training; that is, two 1x81 vectors are horizontally spliced in order to form a 1x162 double-frame training data, that is, the double-frame training data, wherein the first 81 values correspond to the first frame of the training image, and the last 81 training data correspond to the second frame of the training image; this splicing method completely retains the facial contour features and head posture information of the two frames of data, providing complete input for subsequent contrast learning; on this basis, the double-frame training data is imported into a double-frame training network model composed of multiple layers of perceptrons, after the 1x162 double-frame training data is input into the model, the double-frame training network model will perform contrast learning on the data in the double-frame training data, the model focuses on the common characteristics, that is, the relative distance between two adjacent contour points in the same expression under different head postures, and the model pays attention to the difference characteristics, that is, the relative distance between contour points in different expressions under the same head posture; finally, the double-frame training network model outputs the recognition result of the double-frame training data.

[0089] It should be noted that the present application uses a multi-layer perceptron instead of a convolutional network, which retains only the core features required for expression recognition by inputting contour points, and is more direct in data processing; in contrast, the convolutional network involves complex convolution kernel sliding, feature mapping and other operations, and the amount of calculation is usually larger; for example, for a small image data set, if a convolutional network is used, the convolution calculation of the convolution kernel and each region of the image requires a lot of time; and the multi-layer perceptron directly performs matrix operations on the coordinate points, which may complete the processing faster under moderate data volume; in addition to the above advantages, other advantages are not described here.

[0090] Further, the preprocessed training data is imported into the double-frame training network model, and any two frames of training images are selected for mixed training to obtain the common characteristics and difference characteristics between different training data, which further includes:

[0091] At least three groups of expression combination labels are preset according to the double-frame training data, wherein the image expression combination labels correspond to the expression state changes of the two frames of images respectively;

[0092] The double-frame training network model outputs the recognition result of the double-frame training data, which includes:

[0093] When the double-frame training data is mixed trained, the probability result distribution of the at least three groups of expression combination labels corresponding to the double-frame training data is output;

[0094] The expression combination label with the maximum probability is selected as the output result.

[0095] The at least three groups of expression combination labels include: "normal-normal", "normal-raising eyebrows" or "raising eyebrows-normal", and "raising eyebrows-raising eyebrows".

[0096] Preferably, the expression combination label is four groups, respectively including: "normal-normal", "normal-raising eyebrows", "raising eyebrows-normal" and "raising eyebrows-raising eyebrows"; wherein "normal-normal" indicates that the previous frame and the next frame are normal expressions; "normal-raising eyebrows" indicates that the previous frame is a normal expression and the next frame is a raising eyebrows expression; "raising eyebrows-normal" indicates that the previous frame is a raising eyebrows expression and the next frame is a normal expression; "raising eyebrows-raising eyebrows" indicates that the previous frame and the next frame are both raising eyebrows expressions.

[0097] Preferably, the preprocessed training data is imported into the double-frame training network model, and any two frames of training images are selected for mixed training to obtain common characteristics and difference characteristics between different training data, and the mixed training further includes:

[0098] S61: According to the double-frame training data, four combination labels of "normal-normal", "normal-raising eyebrows", "raising eyebrows-normal" and "raising eyebrows-raising eyebrows" are preset;

[0099] S65: The double-frame training network model outputs the recognition result of the double-frame training data, including:

[0100] S651: When the double-frame training data is mixed trained, the probability result distribution of the four combination labels of "normal-normal", "normal-raising eyebrows", "raising eyebrows-normal" and "raising eyebrows-raising eyebrows" corresponding to the double-frame training data is outputted;

[0101] S652: The expression combination label with the maximum probability is selected as the output result.

[0102] Specifically, at least the following four cases are included in the random selection of two frames of training images: "normal-normal", "normal-raising eyebrows", "raising eyebrows-normal" and "raising eyebrows-raising eyebrows"; then, due to the contour points and head posture angles, the final recognition result of the double-frame training network model will fall into the four cases, and the double-frame training network model will output the probability result distribution of the four expression combination labels respectively; assuming that the recognition result probability distribution of the model for the double-frame data is: "normal-normal" is 0.10, "normal-raising eyebrows" is 0.85, "raising eyebrows-normal" is 0.02, and "raising eyebrows-raising eyebrows" is 0.03; for the probability result distribution of the above double-frame training data, the probability of "normal-raising eyebrows" is 0.85, which is the maximum probability in the four labels, so "normal-raising eyebrows" is selected as the output result of the double-frame training data.

[0103] Further, due to the significant differences in the relative positions of the facial contour points and the head posture angles under different expression combinations, the double-frame training network model can accurately classify the recognition results into the corresponding expression combination labels by learning these difference features. For example, in the expression combination of "eyebrow-raising-eyebrow-raising", the positions of the eyebrow contour points relative to the eyelids in the two frames are both in the state of upward movement, and the overall distribution of the contour points determined by the head posture angle is consistent. The model can recognize it as "eyebrow-raising-eyebrow-raising" by capturing these features. In the combination of "normal-eyebrow-raising", the eyebrow contour points of the normal expression in the first frame are relatively low, and the eyebrow contour points of the eyebrow-raising expression in the second frame are upward. Combined with the changes in the distribution of the contour points caused by the head posture angle, the model can accurately recognize this expression combination.

[0104] Through the above process, the double-frame training network model can output the corresponding probability result distribution according to the preset expression combination label in the mixed training, and select the label with the maximum probability as the output result, thereby realizing accurate recognition of different expression combinations and providing a reliable basis for further optimization of the model and final expression recognition prediction.

[0105] Further, in the mixed training process, and / or after the mixed training is completed, any two frames of untrained training images are selected for training effect verification, including:

[0106] Two single-frame training data that have not been trained in teams are selected for splicing processing to obtain verification data;

[0107] The verification data is imported into the double-frame training network model to obtain the output result of the verification data;

[0108] If the output result is the same as the imported two frames of training images, the double-frame training network model passes the verification.

[0109] In combination with the actual situation, when verifying the training effect, two single-frame training data that have not been trained in teams are preferentially selected for verification in the above transverse splicing manner. On the one hand, training effect verification during the mixed training process can avoid continuous training of the model in the wrong direction. For example, when performing mixed training, verification is performed every 10 iterations. If the two groups of untrained training data selected are the combination of "normal + eyebrow-raising", and the output result of the double-frame training network model is: the probability of "normal-eyebrow-raising" is 0.4, the probability of "normal-normal" is 0.3, the probability of "eyebrow-raising-normal" is 0.2, and the probability of "eyebrow-raising-eyebrow-raising" is 0.1, the label corresponding to the maximum probability is "normal-normal", which does not match the actual "normal-eyebrow-raising", indicating that the model has deviation in learning the features of "normal→eyebrow-raising" under this posture, and there is a risk of underfitting. At this time, the training strategy needs to be adjusted: increase the training sample size of the "normal-eyebrow-raising" combination under this posture, and compensate for the shortcomings of the model through targeted reinforcement training.

[0110] On the other hand, the final verification carried out after the completion of the mixed training is also determined only according to the maximum probability label. For example, after training for 50 rounds, 10 groups of verification data covering all poses and all expression combinations are used for testing. If the matching rate of the maximum probability label and the actual expression combination reaches a preset matching rate threshold, it is proved that the model has fully mastered the expression change rule of the target object and can be put into practical application. If the maximum probability label of a group of "eyebrow-raising-normal" verification data is "eyebrow-raising-eyebrow-raising", which does not match the actual situation, it needs to be re-added to the training set for additional training until the final verification is passed.

[0111] Further, after the training of the double-frame training network model is completed, the expression recognition model training method further comprises expression recognition prediction based on the double-frame training network model, specifically comprising:

[0112] Obtaining real-time multi-frame expression images of the target object, and importing the pre-processed real-time multi-frame expression images into the double-frame training network model;

[0113] Selecting two real-time single-frame training data at any frame interval for splicing processing to form real-time double-frame training data;

[0114] Importing the real-time double-frame training data into the double-frame training network model, and controlling the double-frame training network model to output the probability distribution of the expression combination label corresponding to the real-time double-frame training data;

[0115] Selecting the expression combination label with the maximum probability as the prediction result, and determining the expression state change of the target object according to the prediction result.

[0116] In combination with specific use cases, in a vehicle-mounted scene, the driver's facial expressions are collected in real time by an in-vehicle camera. The collected real-time images are first pre-processed in the same way as in the training stage: a face detection algorithm is called to frame the image training area 60 containing the eyebrows and eyes; 39 contour points are extracted and the coordinates are recorded, and the contour point coordinates are normalized according to the width and height of the training area to obtain the first data; at the same time, the real-time head pose angle is obtained through a pose sensor; then, the frame interval is flexibly set according to the actual expression change speed, for example, the 10th frame and the 12th frame are extracted from the real-time single-frame training data at a frame interval of 2, the 10th frame corresponds to a normal expression, and the 12th frame may be in the process of eyebrow-raising. The two real-time single-frame training data of 1x81 are horizontally spliced into real-time double-frame training data of 1x162, ensuring that the model can accurately capture the time trend of the expression.

[0117] Further, if the probability of "normal-eyebrow-raising" is the highest in the probability distribution of the expression combination label corresponding to the real-time double-frame training data, it is therefore selected as the prediction result; combined with the label, it can be determined that the expression state change of the driver is from the normal expression of the 10th frame to the eyebrow-raising expression of the 12th frame.

[0118] Further, the present application also provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor; the processor implements the expression recognition model training method in any of the above examples when executing the program.

[0119] Further, please refer to Figure 6 , the present application also provides an expression recognition system 100 based on head pose and facial key points, which can implement the expression recognition model training method in any of the above examples, and the expression recognition system 100 comprises: an acquisition module 10, a preprocessing module 20, a model construction module 30, a training module 40 and a verification module 50; the acquisition module 10 is used for acquiring a plurality of frames of training images of the same target object and a training head pose corresponding to each frame of training image, denoted as training data; the processing module is used for preprocessing the training data; the model construction module 30 constructs a double-frame training network model based on a multilayer perception mechanism; the training module 40 is used for importing the preprocessed training data into the double-frame training network model, and selecting any two frames of training images for mixed training to obtain common characteristics and difference characteristics between different training data; the verification module 50 is used for selecting any two frames of untrained training images for training effect verification in the mixed training process, and / or after the mixed training is completed.

[0120] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application should be subject to the scope defined by the claims.

Claims

1. A method for training an expression recognition model based on head pose and facial key points, characterized in that, include: Acquire multiple training images of the same target object and the head pose corresponding to each training image, and record them as training data; The training data is preprocessed, and a two-frame training network model is constructed based on a multilayer perceptron. The preprocessed training data is imported into the dual-frame training network model, and any two frames of the training images are selected for mixed training to obtain the common and different characteristics between different training data. During the hybrid training process, and / or after the hybrid training is completed, any two untrained training images are selected to verify the training effect; If the verification result is the same as the imported training image, then the training of the dual-frame training network model is complete. The acquisition of multiple training images of the same target object and the head pose corresponding to each training image, denoted as training data, includes: For any single frame of the training image, face model recognition is performed, and the image training area is selected; the image training area contains at least one pair of training elements; Multiple contour points are set around the training element; Establish a head pose coordinate system in any single frame of the training image, and obtain the head pose coordinates corresponding to the training image; A single-frame image data association is established based on multiple contour points in a single frame of the training image and the head pose coordinates corresponding to the single frame of the training image, and single-frame training data corresponding to the training image is generated. The training elements include any facial feature; The preprocessing of the training data includes: The coordinates of the multiple contour points are normalized to obtain the first data; The head pose coordinates are normalized to obtain the second data; The step of establishing a single-frame image data association based on multiple contour points in a single frame of the training image and the head pose coordinates corresponding to that single frame of the training image includes: The first data and the second data are concatenated to form the single-frame training data of the single-frame training image; The step of importing the preprocessed training data into the dual-frame training network model and selecting any two training images for mixed training to obtain the common and difference characteristics between different training data includes: Select any two of the single-frame training data for team training to obtain two-frame training data; wherein, the team training includes splicing the two selected single-frame training data. Import the dual-frame training data into the dual-frame training network model; The dual-frame training network model is controlled to perform comparative learning on the training elements in the dual-frame training data to obtain the common and difference characteristics of the two selected single-frame training data. The dual-frame training network model is controlled to output the recognition result of the dual-frame training data; The common characteristics include at least the relative distance between two adjacent contour points of the same expression under different head postures; the difference characteristics include at least the relative distance between contour points of different expressions under the same head posture.

2. The facial expression recognition model training method according to claim 1, characterized in that, The step of importing the preprocessed training data into the dual-frame training network model and selecting any two training images for mixed training to obtain the common and difference characteristics between different training data also includes: At least three sets of facial expression combination labels are preset based on the dual-frame training data, wherein the facial expression combination labels correspond to the changes in facial expression state in the two frames of images respectively; The control of the dual-frame training network model to output the recognition result of the dual-frame training data includes: When performing mixed training on the two-frame training data, the probability distribution of the two-frame training data corresponding to the at least three sets of expression combination labels is output; The tag combination of the emoticons with the highest probability is selected as the output result.

3. The facial expression recognition model training method according to claim 2, characterized in that, During the hybrid training process, and / or after the hybrid training is completed, selecting any two untrained training images for training effect verification includes: Two single-frame training data that were not subjected to the aforementioned team training are selected and spliced ​​together to obtain validation data; The verification data is imported into the dual-frame training network model to obtain the output result of the verification data; If the output result is the same as the two imported training images, then the dual-frame training network model is verified.

4. The facial expression recognition model training method according to claim 1 or 2, characterized in that, After the dual-frame training network model is trained, the expression recognition model training method further includes: Facial expression recognition and prediction based on the dual-frame trained network model specifically includes: The real-time multi-frame facial expression images of the target object are acquired, and after the preprocessing, they are imported into the dual-frame training network model. Two real-time single-frame training data are extracted at any frame interval and then spliced ​​together to form real-time dual-frame training data. The real-time two-frame training data is imported into the two-frame training network model, and the two-frame training network model is controlled to output the probability distribution of the expression combination labels corresponding to the real-time two-frame training data. The most probable combination of facial expressions is selected as the prediction result, and the change in the facial expression state of the target object is determined based on the prediction result.

5. The facial expression recognition model training method according to any one of claims 1 to 3, characterized in that, Before acquiring multiple training images of the same target object and the head pose corresponding to each training image, the expression recognition model training method further includes: Record multiple facial expression images of the same target object under different head poses; Based on the type of the facial expression images, multiple facial expression images are classified and labeled into a first image and a second image; Multiple labeled facial expression images are used as training images.

6. An expression recognition system based on head pose and facial key points, characterized in that, The facial expression recognition system is capable of implementing the facial expression recognition model training method as described in any one of claims 1 to 5, and includes: The acquisition module is used to acquire multiple training images of the same target object and the head pose corresponding to each training image, which are recorded as training data. A preprocessing module is used to preprocess the training data; A model building module, which builds the dual-frame training network model based on a multilayer perceptron; The training module is used to import the preprocessed training data into the dual-frame training network model, and select any two frames of the training images for mixed training to obtain the common and difference characteristics between different training data. A verification module is used to select any two untrained training images for training effect verification during the hybrid training process and / or after the hybrid training is completed.

Images