Expression recognition model training method and system based on head posture and facial key points

Abstract

The invention provides an expression recognition model training method and system based on head postures and facial key points. The training method comprises the following steps: firstly, acquiring multiple frames of training images of the same target object and corresponding head postures as training data; preprocessing the data, and building a double-frame training network model based on a multi-layer perception mechanism; importing the preprocessed data, selecting any two frames of images for mixed training, and obtaining common and difference characteristics of different data; in the training process or after the training process is completed, selecting two frames of untrained images to verify the training effect; and if the verification result is the same as the imported image, completing model training. The technical problems that existing training is concentrated on a front view scene, and judgment deviation is easily caused by facial feature changes under a non-front view posture are solved; the difference of five sense organs of an individual is large, normal features of some people are close to abnormal expressions of other people, multi-person mixed training of a traditional model is not optimized for the individual and depends on a general threshold value, physiological differences are easily misjudged as expression changes, and recognition errors are caused.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of expression recognition, in particular to an expression recognition model training method and system based on head posture and facial key points. BACKGROUND

[0002] With the rapid development of computer vision technology, facial expression recognition is increasingly widely used in intelligent interaction, security monitoring, emotional computing and other fields. Its core requirement is to accurately determine the expression state (such as normal, raised eyebrows, smiling, etc.) of a person through image or video data.

[0003] However, there are at least one of the following problems in the related art: the training is focused on the front view scene, when the identified person is in a non-front view posture such as looking down or looking up, the relative position of the facial feature points, the contour and the visual presentation of the muscle movement change significantly, resulting in deviation of the model in judging the expression; and the facial features of different individuals are quite different, the facial features of some people in a normal state may be close to the non-normal expression features of others, and the traditional model uses a multi-person mixed training mode, is not optimized for individual features, and only relies on general feature thresholds, which is easy to misjudge the individual physiological differences as expression changes, resulting in expression recognition errors. SUMMARY

[0004] The technical problem solved by the present application is that the training is focused on the front view scene, when the identified person is in a non-front view posture such as looking down or looking up, the relative position of the facial feature points, the contour and the visual presentation of the muscle movement change significantly, resulting in deviation of the model in judging the expression; and the facial features of different individuals are quite different, the facial features of some people in a normal state may be close to the non-normal expression features of others, and the traditional model uses a multi-person mixed training mode, is not optimized for individual features, and only relies on general feature thresholds, which is easy to misjudge the individual physiological differences as expression changes, resulting in expression recognition errors.

[0005] To solve the above problems, the present application provides an expression recognition model training method based on head posture and facial key points, comprising: obtaining multiple frames of training images of the same target object and the head posture corresponding to each frame of training image, denoted as training data; preprocessing the training data, and building a double-frame training network model based on a multi-layer perception mechanism; importing the preprocessed training data into the double-frame training network model, and selecting any two frames of training images for mixed training to obtain common characteristics and difference characteristics between different training data; in the mixed training process, and / or after the mixed training is completed, any two frames of untrained training images are selected for training effect verification; if the verification result is the same as the imported training image, the double-frame training network model training is completed.

[0006] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: compared with the related art in which the facial expression recognition training set is focused on the front view, the present application first focuses on individual characteristics, uses only the data of the same target object for training, makes the model focus on learning the facial feature rules of the object, avoids the interference of physiological differences between different individuals on expression judgment, then combines the head posture data and image features, learns through double-frame comparison, makes the imported double-frame training network model understand the feature deformation rules of the same expression in different postures, so that the model can still be stably recognized in a non-front view scene, and based on a multi-layer perception mechanism, compared with a convolution network, is better at processing structured data, that is, the preprocessed key points and posture data are imported into the double-frame training network model, the inference speed is improved by tens of times, and the real-time interaction demand is met, finally, through the verification process during and after training, the model overfitting is avoided, and accurate recognition is ensured in untrained postures and / or expression combinations.

[0007] In an example of the present application, a plurality of training images of the same target object and the head posture corresponding to each training image are obtained, denoted as training data, including: performing face model recognition on any single training image, and framing the image training area; the image training area contains at least one pair of training elements; a plurality of contour points are set around the training elements; a head posture coordinate system is established in any single training image, and the head posture coordinate corresponding to the training image is obtained; a single-frame image data association is established based on the plurality of contour points in the single training image and the head posture coordinate corresponding to the single training image, and single-frame training data corresponding to the training image is generated; wherein the training elements include any facial feature.

[0008] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: by framing the image training area containing the training elements of facial features such as eyebrows and eyes, redundant information is excluded, the model only focuses on the key facial areas related to expressions, such as the relative movement of eyebrows and eyes in the action of raising eyebrows, and the feature extraction efficiency is improved; a plurality of contour points are set around the training elements, the expression change is converted into quantifiable point coordinate change, and specific data is provided for subsequent normalization and comparison learning; finally, the contour points are bound with the posture data through the head posture coordinate system, so that the model can distinguish whether the contour point change is caused by posture or expression, and lay a foundation for subsequent common and difference characteristic learning.

[0009] In an example of the present application, the training data is preprocessed, including: normalizing the plurality of contour point coordinates to obtain first data; normalizing the head posture coordinates to obtain second data; establishing a single-frame image data association based on the plurality of contour points in the single training image and the head posture coordinate corresponding to the single training image, including: splicing the first data and the second data to form the single-frame training data of the single training image.

[0010] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: dividing the contour point coordinates by the width and height of the image training area and dividing the head posture angle by 90° enable all data to be mapped to a unified range, so that the model can focus on the relative relationship rather than the absolute coordinates; finally, the normalized contour point data and the head posture data are spliced to form single-frame training data containing facial features and posture information, so that the model can simultaneously perceive the correlation between the two, and provide complete features for double-frame comparison.

[0011] In an example of the present application, the preprocessed training data is imported into a double-frame training network model, and any two frames of training images are selected for mixed training to obtain common characteristics and difference characteristics between different training data, including: selecting any two single-frame training data for team training to obtain double-frame training data; wherein the team training includes splicing processing of the selected two single-frame training data; the double-frame training data is imported into the double-frame training network model; the double-frame training network model is controlled to perform comparative learning on the training elements in the double-frame training data to obtain the common characteristics and difference characteristics of the selected two single-frame spliced data; the double-frame training network model is controlled to output the recognition result of the double-frame training data; wherein the common characteristics at least include the relative distance between adjacent contour points under the same expression at different head postures; the difference characteristics at least include the relative distance between contour points under different expressions at the same head posture.

[0012] Compared with the prior art, the technical effects achieved by adopting the technical scheme are as follows: first, the common characteristics are determined, that is, the relative distance between adjacent contour points under the same expression at different head postures is learned, so that the model understands that the core feature ratio of the expression does not change with the change of the posture, and avoids misjudgment of the expression due to the change of the posture; then, the difference characteristics are captured, that is, the relative distance between contour points under different expressions at the same head posture is learned, so that the model can accurately identify the expression change under the same posture; finally, through comparison of the two frames of data, the model focuses on the change trend of the expression rather than the single-frame static feature, and the accuracy of dynamic expression recognition is improved.

[0013] In an example of the present application, the preprocessed training data is imported into a double-frame training network model, and any two frames of training images are selected for mixed training to obtain common characteristics and difference characteristics between different training data, and further including: at least three groups of expression combination labels are preset according to the double-frame training data, wherein the image expression combination labels correspond to the expression state changes of the two frames of images respectively; the double-frame training network model is controlled to output the recognition result of the double-frame training data, including: when the double-frame training data is mixed trained, the probability result distribution of the at least three groups of expression combination labels corresponding to the double-frame training data is output; the expression combination label with the maximum probability is selected as the output result.

[0014] Compared with the prior art, the technical effects reached by adopting the technical scheme are as follows: the preset combination label enables the model to not only identify a single-frame expression but also clearly identify the expression change trend between two frames, adapt to a dynamic scene, and output a probability distribution of at least three types of labels to provide a credibility reference for the identification result, so that the probability value can be adjusted for judgment in a fuzzy scene; finally, the maximum probability label is selected as the output to avoid the singleness of a single label, so that the model can still stably output the most possible result in a complex scene.

[0015] In an example of the present application, during the mixed training process and / or after the mixed training is completed, any two frames of untrained training images are selected for training effect verification, including: selecting two single-frame training data that have not been trained in combination to perform splicing processing to obtain verification data; importing the verification data into the double-frame training network model to obtain the output result of the verification data; if the output result is the same as the imported two frames of training images, the double-frame training network model passes the verification.

[0016] Compared with the prior art, the technical effects reached by adopting the technical scheme are as follows: the verification data comes from single-frame data that has not participated in the combination training, ensuring that the model is tested under unseen poses and / or expression combinations to prevent the model from memorizing the training data; the consistency between the verification result and the actual image can be used to quantize the accuracy of the model in different scenes, ensuring that the model is stable and effective in actual application.

[0017] In an example of the present application, after the double-frame training network model is trained, the expression recognition model training method further includes expression recognition prediction based on the double-frame training network model, specifically including: obtaining real-time multi-frame expression images of a target object, and importing the preprocessed real-time multi-frame expression images into the double-frame training network model; selecting two real-time single-frame training data at any frame interval to perform splicing processing to form real-time double-frame training data; importing the real-time double-frame training data into the double-frame training network model, and controlling the double-frame training network model to output the probability distribution of the expression combination label corresponding to the real-time double-frame training data; selecting the expression combination label with the maximum probability as a prediction result, and determining the expression state change of the target object according to the prediction result.

[0018] Compared with the prior art, the technical effects reached by adopting the technical scheme are as follows: the preprocessing of the real-time image is completely consistent with the processing method of the training data, ensuring that the feature distribution input into the model matches that during the training, and avoiding a decrease in recognition accuracy caused by a difference in data distribution; the double-frame data at any frame interval can be extracted to flexibly adapt to expressions with different dynamic speeds, ensuring that the complete expression change process is captured; the expression combination label with the maximum probability can be used to clearly identify the expression change trend of the target object.

[0019] In an example of the present application, before acquiring the plurality of training images of the same target object and the plurality of training head poses corresponding to the training images, the expression recognition model training method further comprises: recording a plurality of expression images of the same target object under different head poses; according to the types of the expression images, classifying and labeling the plurality of expression images into first images and second images; and taking the classified and labeled plurality of expression images as the training images.

[0020] Compared with the prior art, the technical effects achieved by adopting the technical solution are: the expression images of the same target object under different head poses are clearly recorded, ensuring that the training data contains all the pose scenes required by the model, and avoiding the situation that the model has no rules to learn under a specific pose due to data missing; the images are classified and labeled as first images and second images, so that the model learns the real feature difference between normal and abnormal expressions; and the standardized classification and labeling provides a basis for subsequent expression combination labels, ensuring the consistency of the label logic during double-frame training.

[0021] In an example of the present application, an expression recognition system based on head pose and facial key points is also provided, which can implement the expression recognition model training method in any of the above examples. The expression recognition system comprises: an acquisition module, which is used to acquire a plurality of training images of the same target object and a training head pose corresponding to each training image, denoted as training data; a preprocessing module, which is used to preprocess the training data; a model construction module, which constructs a double-frame training network model based on a multilayer perception mechanism; a training module, which is used to import the preprocessed training data into the double-frame training network model, and select any two training images for mixed training to obtain common characteristics and difference characteristics between different training data; and a verification module, which is used to select any two untrained training images for training effect verification during the mixed training process and / or after the mixed training is completed.

[0022] After adopting the technical solution of the present application, the following technical effects can be achieved: (1) The present application firstly focuses on individual characteristics, and only uses the data of the same target object for training, so that the model focuses on learning the facial feature rules of the object, avoiding the physiological differences between different individuals to interfere with the expression judgment; then the head posture data and image features are combined, and through double-frame comparison learning, the imported double-frame training network model understands the feature deformation rules of the same expression under different postures, so that it can still be recognized stably in a non-frontal scene; and a multi-layer perception mechanism is built into the network, which is better at processing structured data than convolutional networks, that is, the preprocessed key points and posture data are imported into the double-frame training network model, the inference speed is improved by tens of times, and the real-time interaction demand is met; finally, through the verification process during and after training, overfitting of the model is avoided, and accurate recognition is ensured under untrained postures and / or expression combinations; (2) The present application trains the image training area containing facial features such as eyebrows and eyes by framing, excludes redundant information, and makes the model only pay attention to the key facial areas related to expressions, such as the relative movement of eyebrows and eyes in the action of raising eyebrows, to improve the feature extraction efficiency; multiple contour points are set around the training elements, the expression changes are converted into quantifiable point coordinate changes, and specific data are provided for subsequent normalization and comparison learning; finally, the contour points are bound with the posture data through the head posture coordinate system, so that the model can distinguish whether the contour point change is caused by posture or expression, laying a foundation for subsequent common and difference characteristic learning; (3) The present application firstly learns the relative distance of adjacent contour points under the same expression in different head postures, so that the model understands that the core feature ratio of the expression does not change with the posture change, and avoids misjudgment of the expression due to the posture deformation; then it captures the difference characteristics, that is, it learns the relative distance of contour points under the same head posture in different expressions, so that the model can accurately identify the expression change under the same posture; finally, through the comparison of two frames of data, the model pays attention to the change trend of the expression rather than the single-frame static feature, improving the accuracy of dynamic expression recognition. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating laborious work; Figure 1 A flowchart of an expression recognition model training method based on head posture and facial key points is provided for the embodiments of the present application; Figure 2 For Figure 1 A specific flowchart of S6 in the embodiment; Figure 3 An expression image schematic diagram is provided for the present application; Figure 4 A schematic diagram of the profile point and head pose splicing processing provided by the present application after processing; Figure 5 A structure schematic diagram of the double-frame training data provided by the present application; Figure 6 A module schematic diagram of an expression recognition system based on head pose and facial key points provided by the present application.

[0024] Explanation of reference signs: 100, expression recognition system; 10, acquisition module; 20, preprocessing module; 30, model construction module; 40, training module; 50, verification module; 60, image training area. DETAILED DESCRIPTION

[0025] Embodiments of the present application will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0026] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, it can be fixedly connected, or connected, or integrally connected; it can be mechanically connected, or electrically connected; it can be directly connected, or indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0027] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0028] The image learning method of the expression recognition model in the related art mostly adopts the mode of "single-frame static feature" and "multi-person mixed training"; it usually collects a large number of facial expression images of different individuals, focuses on the expression features in the front view posture; after extracting the high-bit features of the single-frame image through the convolutional neural network, the mapping learning from "expression category" to "label annotation" is directly performed; for example, when training the "eyebrow raising" expression recognition, the model learns the average distance threshold of the eyebrows and eyes / cheeks of all individuals when raising eyebrows, and uses it as the judgment standard; the core logic of this method is to learn the universal law of "human expression" through massive data, but there are limitations: on the one hand, it is not suitable for non-front view posture, when the identified person is looking down or sideways, the geometric deformation of the facial features will break the average threshold, leading to model misjudgment; on the other hand, it ignores individual differences, for example, different people have different eyebrow bone heights and eyebrow thicknesses, and the unified average threshold is difficult to adapt to the actual features of each person, and the recognition robustness is low.

[0029] In view of the above limitations, the present application uses double-frame dynamic contrast combined with individual exclusive training, only uses the image data of the same target object for training, avoiding the interference of physiological characteristics of different individuals; instead of learning "the commonality of all people's eyebrow raising", the model focuses on "the eyebrow raising feature change law of a certain person in different postures", and this individual exclusive "shortened ratio" will be remembered by the model and become the basis for recognition; on this basis, the head posture data and facial contour point features are normalized and fused to form a joint representation of "spatial features combined with posture features"; when learning, the model will pay attention to "the relative position change of the contour points" and "the associated influence of the posture angle", for example, when detecting that the yaw angle increases, the model will automatically call the individual feature library under this posture, instead of relying on the judgment standard in the front view, so as to maintain the recognition stability in the non-front view scene.

[0030] Referring to Figure 1 , Figure 1 A flowchart of an expression recognition model training method based on head posture and facial key points is provided for the embodiments of the present application; combined Figures 2 to 5 The expression recognition model training method based on head posture and facial key points comprises: S1: recording a plurality of expression images of the same target object under different head postures; S2: classifying and marking the plurality of expression images according to the types of the expression images, and dividing them into first images and second images; S3: taking the plurality of expression images classified and marked as training images; S4: obtaining a plurality of training images of the same target object and the head posture corresponding to each training image, denoted as training data; S5: preprocessing the training data, and building a double-frame training network model based on a multi-layer perception mechanism; S6: import the preprocessed training data into the double-frame training network model, and select any two frames of training images for mixed training to obtain common characteristics and difference characteristics between different training data; S7: during the mixed training process, and / or after the mixed training is completed, select any two frames of untrained training images for training effect verification; S71: if the verification result is the same as the imported training image, the double-frame training network model training is completed; S72: if the verification result is different from the imported training image, the mixed training of the two frames of training images is re-performed.

[0031] In combination with specific use cases, before image training, first, multiple expression images of the same target object under different head poses are recorded; and according to the types of the recorded expression images, all the expression images are classified and marked as first images and second images; for example, the recorded expression images are divided into two categories, including: normal expression images and raised eyebrow images; the first image is a normal expression image, and the second image is a raised eyebrow image; after each training image is marked, it is used as a training image.

[0032] In the image processing stage, first, multiple frames of training images of the same target object and the head pose corresponding to each training image are obtained, so that each frame of training image can obtain its corresponding training data, and the training data of each frame of image in the training set is made; then the training data of each frame of image in the training set is preprocessed, and a double-frame training network model is built based on a multi-layer perception mechanism; the preprocessed training data is imported into the double-frame training network model, and the training data of any two frames of training images are randomly selected for mixed training to obtain the common characteristics and difference characteristics between the two frames of training images; wherein each training image has a specific image number.

[0033] Further, during the mixed training process and after the mixed training is completed, any two frames of training images that have not been combined for training can be selected for training effect verification; for example, during the training process, every 10 iterations, 20 training images are extracted from the reserved images and input into the double-frame training network model for verification; similarly, after the training of the training data of the same target object is completed, the remaining training images that have not been combined for training are used for effect verification again; during verification, the double-frame training network model compares the output results of the several groups of training images with the actual expression labels of the images, if the matching rate exceeds a predetermined threshold, the verification is passed, the double-frame training network model is trained and the parameters are saved; if the output result of a certain group of verification data is wrong, the data is re-added to the training set and re-trained until the verification is passed.

[0034] For further information, please refer to [link / reference]. Figure 3 Multiple training images of the same target object and the head pose corresponding to each training image are obtained and recorded as training data, including: For any single frame of training image, face model recognition is performed, and the image training area is selected; the image training area contains at least one pair of training elements; Set multiple contour points around the training element; Establish a head pose coordinate system in any single frame of the training image and obtain the head pose coordinates corresponding to the training image. Establish a data association between a single frame of training image and the head pose coordinates corresponding to the single frame of training image, and generate the single frame training data corresponding to the training image. The training elements include any facial feature.

[0035] Preferably, the training elements include at least facial features such as eyes, eyebrows, lips, and nose.

[0036] To facilitate understanding of this solution, the present invention selects eyebrows and eyes as training elements in the embodiments of the present invention.

[0037] Depending on the specific application, after acquiring multiple training images of the same target, each training image is processed. First, face model recognition is performed on each training image, and then the image training area 60 corresponding to the face is selected. This image training area 60 contains at least one pair of training elements: eyes and eyebrows. Multiple contour points are set around the outer contours of the eyes and eyebrows. Then, a corresponding head pose coordinate system is established in the training image, and the head pose coordinates corresponding to the training image are obtained. Finally, the multiple contour points and head pose coordinates in the image are associated with single-frame image data, and single-frame training data corresponding to the image is generated.

[0038] For example, if there are 1,000 recorded images, including 500 images with normal facial expressions and 500 images with raised eyebrows, then 1,000 single-frame training data will be generated during the training image processing stage. Each single-frame training data includes multiple contour points and head pose coordinates in the training image.

[0039] For further information, please refer to [link / reference]. Figure 4 The training data is preprocessed, including: The coordinates of multiple contour points are normalized to obtain the first data; The head pose coordinates are normalized to obtain the second data; A single-frame image data association is established based on multiple contour points in a single-frame training image and the corresponding head pose coordinates of that single-frame training image, including: The first data and the second data are spliced to form single-frame training data of the single-frame training image.

[0040] For example, the size of the framed image training area 60 is 120 pixels in length and 84 pixels in width, and the coordinate value of each contour point is (Xu, Yu), where X represents the horizontal position of the contour point in the image training area 60, Y represents the height position of the contour point in the image training area 60, and u represents the corresponding figure number. The normalization of the contour point coordinates includes dividing the coordinates of the contour points on the eyebrow contour by the length and height of the image training area 60 to normalize the contour point coordinate information to the range of [0, 1] and obtain the normalized coordinates corresponding to the contour point, i.e., the first data. The normalization of the head pose coordinates includes dividing the three angles of the head pose coordinates by 90° to normalize the angle information of the head pose to the range of [0, 1]. Finally, any one contour point is selected as the basis, the remaining contour points are arranged horizontally and flattened, and the normalized head pose coordinates are added to form the first data of the training image.

[0041] For example, the size of the framed image training area 60 is 120 pixels in length and 84 pixels in width, and the coordinate value of each contour point is (Xu, Yu), where X represents the horizontal position of the contour point in the image training area 60, Y represents the height position of the contour point in the image training area 60, and u represents the corresponding figure number. The normalization of the contour point coordinates includes dividing the coordinates of the contour points on the eyebrow contour by the length and height of the image training area 60 to normalize the contour point coordinate information to the range of [0, 1] and obtain the normalized coordinates corresponding to the contour point, i.e., the first data. The normalization of the head pose coordinates includes dividing the three angles of the head pose coordinates by 90° to normalize the angle information of the head pose to the range of [0, 1]. Finally, any one contour point is selected as the basis, the remaining contour points are arranged horizontally and flattened, and the normalized head pose coordinates are added to form the first data of the training image. Figure 4 In one specific example, the size of the framed image training area 60 is 120 pixels in length (W) and 84 pixels in width (H), and the contour points of the selected eye and eyebrow are a total of 39. The coordinates of the leftmost Q1 contour point are (14, 50), and the coordinates of the adjacent Q2 contour point are (28, 41). Similarly, the coordinates of the Q3 and Q4 contour points are (45, 28) and (63, 15), respectively. The coordinates of the remaining contour points are not described in detail here. The normalization of the contour points satisfies the following formulas 1 and 2: Formula 1: Norm_x1=Qz.X / W; Formula 2: Norm_y1=Qz.Y / H; where Q represents the contour point, and z represents the arrangement number of the contour point.

[0042] Further, the normalized coordinates of all contour points are calculated to obtain 78 values containing X and Y dimensions, which together form the first data to eliminate the influence of the size difference of different image training areas 60.

[0043] Further, the three angles in the coordinate system corresponding to the head pose are Pitch (pitch angle), Yaw (yaw angle), and Roll (roll angle). The values of the head pose in the current training image are Pitch: 4.52, Yaw: 35.2, and Roll: 2.84. The normalization satisfies the following formulas 3, 4, and 5: Formula 3: Norm_pitch = Pitch / 90; Formula 4: Norm_yaw = Yaw / 90; Formula 5: Norm_roll = Roll / 90; Further, the angle values of the normalized head posture are calculated and constituted as second data.

[0044] Further, as shown in Figure 4 The 78 normalized contour point coordinates in the first data and the three normalized posture angles in the second data are formed into a single-frame training data in the form of transverse splicing, that is, a 1x81 vector data is finally formed; this splicing method integrates the facial feature contour points and the head posture angles into the same structure, which not only retains the "shift rule of the eyebrow peak position relative to the training area when lowering the head", but also incorporates the "quantitative information of the lowering angle", providing complete input for the dual-frame training network model to learn the internal correlation between the two.

[0045] After preprocessing, the single-frame images of different head postures and different training area sizes are all converted into structured data with unified dimensions and consistent scales, ensuring that the model can focus on the relative relationship of the features during training, rather than absolute coordinates or angle values, laying a standardized data foundation for subsequent dual-frame mixed training.

[0046] The pitch angle refers to the rotation angle of the head around the horizontal axis (left-right direction), reflecting the "head up" or "head down" posture; the yaw angle refers to the rotation angle of the head around the vertical axis (up-down direction), reflecting the "head left" or "head right" posture; the roll angle refers to the rotation angle of the head around the front-back axis (perpendicular to the face direction), reflecting the "head tilt" posture.

[0047] Further, referring to Figure 5 The preprocessed training data is imported into the dual-frame training network model, and any two frames of training images are selected for mixed training to obtain common and different characteristics between different training data, including: S62: selecting any two single-frame training data for team training to obtain dual-frame training data; wherein the team training includes splicing processing of the selected two single-frame training data; S63: importing the dual-frame training data into the dual-frame training network model; S64: controlling the dual-frame training network model to perform comparative learning on the training elements in the dual-frame training data to obtain the common and different characteristics of the selected two single-frame splicing data; S65: controlling the dual-frame training network model to output the recognition result of the dual-frame training data; Among them, common characteristics include at least the relative distance between two adjacent contour points of the same expression under different head postures; and differential characteristics include at least the relative distance between contour points of different expressions under the same head posture.

[0048] Based on the normalization of single-frame training images, two normalized training images are selected for pair training. This involves horizontally concatenating two 1×81 vectors sequentially to form 1×162 two-frame training data, where the first 81 values ​​correspond to the first training image and the last 81 correspond to the second. This concatenation method fully preserves the facial contour features and head pose information of both frames, providing complete input for subsequent comparative learning. The two-frame training data is then imported into a two-frame training network model composed of a multilayer perceptron. After inputting the 1×162 two-frame training data, the model performs comparative learning on the data. The model focuses on common features, namely the relative distance between two adjacent contour points of the same expression under different head poses, and on different features, namely the relative distance between contour points of different expressions under the same head pose. Finally, the two-frame training network model outputs the recognition results of the two-frame training data.

[0049] It should be noted that this invention uses a multilayer perceptron instead of a convolutional network. By inputting contour points, it retains only the core features required for expression recognition, making data processing more direct. In contrast, convolutional networks involve complex operations such as convolutional kernel sliding and feature mapping, which typically require more computation. For example, for a small image dataset, if a convolutional network is used, the convolution calculation between the convolutional kernel and various regions of the image would take a considerable amount of time. However, the multilayer perceptron directly performs matrix operations on the coordinate points, which may complete the processing faster when the amount of data is moderate. In addition to the advantages mentioned above, there are other advantages, which will not be elaborated here.

[0050] Furthermore, the preprocessed training data is imported into a two-frame training network model, and any two training frames are selected for mixed training to obtain the common and differential characteristics between different training data. This also includes: Based on the two-frame training data, at least three sets of expression combination labels are preset, wherein the image expression combination labels correspond to the changes in the expression state of the two frames of images respectively; Controlling the output of the dual-frame training network model to recognize the dual-frame training data includes: When performing mixed training on two-frame training data, output the probability distribution of at least three sets of expression combination labels corresponding to the two-frame training data; Select the emoji combination label with the highest probability as the output result.

[0051] The at least three groups of expression combination labels include "normal-normal", "normal-raised eyebrow", or "raised eyebrow-normal", and "raised eyebrow-raised eyebrow".

[0052] Preferably, the expression combination labels are four groups, including "normal-normal", "normal-raised eyebrow", "raised eyebrow-normal", and "raised eyebrow-raised eyebrow"; wherein "normal-normal" indicates that the previous frame and the next frame are normal expressions; "normal-raised eyebrow" indicates that the previous frame is a normal expression and the next frame is a raised eyebrow expression; "raised eyebrow-normal" indicates that the previous frame is a raised eyebrow expression and the next frame is a normal expression; and "raised eyebrow-raised eyebrow" indicates that the previous frame and the next frame are both raised eyebrow expressions.

[0053] Preferably, the preprocessed training data is imported into the double-frame training network model, and any two frames of training images are selected for mixed training to obtain common characteristics and difference characteristics between different training data, and the method further comprises: S61: According to the double-frame training data, four combination labels of "normal-normal", "normal-raised eyebrow", "raised eyebrow-normal", and "raised eyebrow-raised eyebrow" are preset; S65: The double-frame training network model outputs the recognition result of the double-frame training data, including: S651: When the double-frame training data is mixed trained, the probability result distribution of the four combination labels of "normal-normal", "normal-raised eyebrow", "raised eyebrow-normal", and "raised eyebrow-raised eyebrow" corresponding to the double-frame training data is outputted; S652: The expression combination label with the maximum probability is selected as the output result.

[0054] Specifically, the two frames of training images are randomly selected to include at least the following four cases: "normal-normal", "normal-raised eyebrow", "raised eyebrow-normal", and "raised eyebrow-raised eyebrow"; then, due to the contour points and head posture angles, the final recognition result of the double-frame training network model will fall into the four cases, and the double-frame training network model will output the probability result distribution of the four groups of expression combination labels respectively; assuming that the recognition result probability distribution of the model for the double-frame data is: "normal-normal" is 0.10, "normal-raised eyebrow" is 0.85, "raised eyebrow-normal" is 0.02, and "raised eyebrow-raised eyebrow" is 0.03; for the probability result distribution of the above double-frame training data, the probability of "normal-raised eyebrow" is 0.85, which is the maximum probability among the four groups of labels, and therefore "normal-raised eyebrow" is selected as the output result of the double-frame training data.

[0055] Further, due to the significant differences in the relative positions of the facial contour points and the head posture angles under different expression combinations, the double-frame training network model can accurately classify the recognition results into the corresponding expression combination labels by learning these difference features. For example, in the expression combination of "eyebrow-raising-eyebrow-raising", the positions of the eyebrow contour points relative to the eyelids in the two frames are both in the state of upward movement, and the overall distribution of the contour points determined by the head posture angle is consistent. The model can recognize it as "eyebrow-raising-eyebrow-raising" by capturing these features. In the combination of "normal-eyebrow-raising", the eyebrow contour points of the normal expression in the first frame are relatively low, and the eyebrow contour points of the eyebrow-raising expression in the second frame are upward. Combined with the changes in the distribution of the contour points caused by the head posture angle, the model can accurately recognize this expression combination.

[0056] Through the above process, in the mixed training, the double-frame training network model can output the corresponding probability result distribution according to the preset expression combination label, and select the label with the maximum probability as the output result, thereby realizing accurate recognition of different expression combinations and providing a reliable basis for further optimization of the model and final expression recognition prediction.

[0057] Further, in the mixed training process, and / or after the mixed training is completed, any two frames of untrained training images are selected for training effect verification, including: Two single-frame training data that have not been trained in teams are selected for splicing processing to obtain verification data; The verification data is imported into the double-frame training network model to obtain the output result of the verification data; If the output result is the same as the imported two frames of training images, the double-frame training network model passes the verification.

[0058] In combination with the actual situation, when verifying the training effect, two single-frame training data that have not been trained in teams are preferentially selected for verification in the above transverse splicing manner. On the one hand, training effect verification during the mixed training process can avoid continuous training of the model in the wrong direction. For example, when performing mixed training, verification is performed every 10 iterations. If the two groups of training data selected for verification are the combination of "normal + eyebrow-raising", and the output result of the double-frame training network model is: the probability of "normal-eyebrow-raising" is 0.4, the probability of "normal-normal" is 0.3, the probability of "eyebrow-raising-normal" is 0.2, and the probability of "eyebrow-raising-eyebrow-raising" is 0.1, the label corresponding to the maximum probability is "normal-normal", which does not match the actual "normal-eyebrow-raising", indicating that the model has deviation in learning the features of "normal→eyebrow-raising" under this posture, and there is a risk of underfitting. At this time, the training strategy needs to be adjusted: the number of training samples of the "normal-eyebrow-raising" combination under this posture is increased, and the short board of the model is compensated through targeted reinforcement training.

[0059] On the other hand, the final verification carried out after the completion of the mixed training is also determined only according to the maximum probability label. For example, after training for 50 rounds, 10 groups of verification data covering all posture and expression combinations are used for testing. If the matching rate of the maximum probability label and the actual expression combination reaches a preset matching rate threshold, it is proved that the model has fully mastered the expression change rule of the target object and can be put into practical application. If the maximum probability label of a group of "eyebrow-raising-normal" verification data is "eyebrow-raising-eyebrow-raising", which does not match the actual situation, it needs to be re-added to the training set for additional training until the final verification is passed.

[0060] Further, after the training of the double-frame training network model is completed, the expression recognition model training method further includes performing expression recognition prediction based on the double-frame training network model, specifically including: Obtaining real-time multi-frame expression images of a target object, and importing the real-time multi-frame expression images into the double-frame training network model after preprocessing; Selecting two real-time single-frame training data at any frame interval for splicing processing to form real-time double-frame training data; Importing the real-time double-frame training data into the double-frame training network model, and controlling the double-frame training network model to output the probability distribution of the expression combination label corresponding to the real-time double-frame training data; Selecting the expression combination label with the maximum probability as the prediction result, and determining the expression state change of the target object according to the prediction result.

[0061] In combination with specific use cases, in a vehicle-mounted scene, the driver's facial expressions are collected in real time by an in-vehicle camera. The collected real-time images are first preprocessed in the same way as in the training stage: a face detection algorithm is called to frame the image training area 60 containing the eyebrows and eyes; 39 contour points are extracted and the coordinates are recorded, and the contour point coordinates are normalized according to the width and height of the training area to obtain first data; at the same time, the real-time head posture angle is obtained through a posture sensor; then, the frame interval is flexibly set according to the actual expression change speed, and taking a frame interval of 2 as an example, the 10th frame and the 12th frame of data are extracted from the real-time single-frame training data, the 10th frame corresponds to a normal expression, and the 12th frame may be in the process of eyebrow-raising. The two real-time single-frame training data of 1x81 are horizontally spliced into real-time double-frame training data of 1x162, ensuring that the model can accurately capture the time trend of the expression.

[0062] Further, if the probability distribution of the expression combination label corresponding to the real-time double-frame training data has the highest "normal-eyebrow-raising" probability, the label is selected as the prediction result; and the expression state change of the driver is determined as: from the normal expression of the 10th frame to the eyebrow-raising expression of the 12th frame.

[0063] Further, the present application also provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor; the processor implements the expression recognition model training method in any of the above examples when executing the program.

[0064] Further, please refer to Figure 6 Further, the present application also provides an expression recognition system 100 based on head pose and facial key points, which can implement the expression recognition model training method in any of the above examples, and the expression recognition system 100 comprises: an acquisition module 10, a preprocessing module 20, a model construction module 30, a training module 40 and a verification module 50; the acquisition module 10 is used for acquiring multiple frames of training images of the same target object and the training head pose corresponding to each frame of training image, denoted as training data; the preprocessing module 20 is used for preprocessing the training data; the model construction module 30 constructs a double-frame training network model based on a multilayer perception mechanism; the training module 40 is used for importing the preprocessed training data into the double-frame training network model, and selecting any two frames of training images for mixed training to obtain common characteristics and difference characteristics between different training data; the verification module 50 is used for selecting any two frames of untrained training images for training effect verification in the mixed training process, and / or after the mixed training is completed.

[0065] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application should be subject to the scope defined by the claims.

Claims

1. A method for training an expression recognition model based on head pose and facial landmarks, the method comprising: include: Acquire multiple training images of the same target object and the head pose corresponding to each training image, and record them as training data; The training data is preprocessed, and a two-frame training network model is constructed based on a multilayer perceptron. The preprocessed training data is imported into the dual-frame training network model, and any two frames of the training images are selected for mixed training to obtain the common and different characteristics between different training data. During the hybrid training process, and / or after the hybrid training is completed, any two untrained training images are selected to verify the training effect; If the verification result is the same as the imported training image, then the training of the dual-frame training network model is complete. 2.The expression recognition model training method of claim 1, wherein, The acquisition of multiple training images of the same target object and the head pose corresponding to each training image, denoted as training data, includes: For any single frame of the training image, face model recognition is performed, and the image training area is selected; the image training area contains at least one pair of training elements; Multiple contour points are set around the training element; Establish a head pose coordinate system in any single frame of the training image, and obtain the head pose coordinates corresponding to the training image; A single-frame image data association is established based on multiple contour points in a single frame of the training image and the head pose coordinates corresponding to the single frame of the training image, and single-frame training data corresponding to the training image is generated. The training elements include any facial feature. 3.The expression recognition model training method of claim 2, wherein, The preprocessing of the training data includes: The coordinates of the multiple contour points are normalized to obtain the first data; The head pose coordinates are normalized to obtain the second data; The step of establishing a single-frame image data association based on multiple contour points in a single frame of the training image and the head pose coordinates corresponding to that single frame of the training image includes: The first data and the second data are concatenated to form the single-frame training data of the single-frame training image. 4.The expression recognition model training method of claim 3, wherein, The step of importing the preprocessed training data into the dual-frame training network model and selecting any two training images for mixed training to obtain the common and difference characteristics between different training data includes: Select any two of the single-frame training data for team training to obtain two-frame training data; wherein, the team training includes splicing the two selected single-frame training data. Import the dual-frame training data into the dual-frame training network model; The dual-frame training network model is controlled to perform comparative learning on the training elements in the dual-frame training data to obtain the common and difference characteristics of the two selected single-frame spliced ​​data. The dual-frame training network model is controlled to output the recognition result of the dual-frame training data; The common characteristics include at least the relative distance between two adjacent contour points of the same expression under different head postures; the difference characteristics include at least the relative distance between contour points of different expressions under the same head posture. 5.The expression recognition model training method of claim 4, wherein, The preprocessed training data is imported into the double-frame training network model, and any two frames of training images are selected for mixed training to obtain common characteristics and difference characteristics between different training data, and the method further comprises: At least three groups of expression combination labels are preset according to the double-frame training data, wherein the image expression combination labels correspond to the expression state changes of two frames of images respectively; The control of the double-frame training network model outputs the recognition result of the double-frame training data, comprising: When the double-frame training data is mixed and trained, the probability result distribution of the at least three groups of expression combination labels corresponding to the double-frame training data is outputted; The expression combination label with the maximum probability is selected as the output result. 6.The expression recognition model training method of claim 4 or 5, wherein, In the mixed training process, and / or after the mixed training is completed, any two frames of untrained training images are selected for training effect verification, comprising: Two single-frame training data that have not undergone the team training are selected for splicing processing to obtain verification data; The verification data is imported into the double-frame training network model to obtain the output result of the verification data; If the output result is the same as the imported two frames of training images, the double-frame training network model is verified to pass. 7.The expression recognition model training method of claim 4 or 5, wherein, After the double-frame training network model is trained, the expression recognition model training method further comprises: Based on the double-frame training network model, expression recognition prediction is performed, specifically comprising: Real-time multi-frame expression images of a target object are obtained, and after the preprocessing, the real-time multi-frame expression images are imported into the double-frame training network model; Two real-time single-frame training data are selected at any frame interval for splicing processing to form real-time double-frame training data; The real-time double-frame training data is imported into the double-frame training network model, and the double-frame training network model is controlled to output the probability distribution of the expression combination label corresponding to the real-time double-frame training data; The expression combination label with the maximum probability is selected as the prediction result, and the expression state change of the target object is determined according to the prediction result. 8.The expression recognition model training method of any one of claims 1-5, wherein, Before the multiple training images of the same target object and the multiple training head poses corresponding to the training images are obtained, the expression recognition model training method further comprises: Multiple expression images of the same target object under different head poses are recorded; According to the types of the expression images, the multiple expression images are classified and labeled into first images and second images; The multiple expression images classified and labeled are used as the training images.

9. A head pose and facial landmark based expression recognition system, characterized in that, The expression recognition system can implement the expression recognition model training method as claimed in any one of claims 1 to 8, and the expression recognition system comprises: An acquisition module is configured to acquire multiple frames of training images of a same target object and a training head pose corresponding to each frame of the training images, denoted as training data; A preprocessing module is configured to preprocess the training data; A model construction module is configured to construct the double-frame training network model based on a multi-layer perception mechanism; The training module is configured to import the preprocessed training data into the double-frame training network model, and to select any two frames of the training images for mixed training to obtain common characteristics and difference characteristics between different training data. The verification module is configured to select any two frames of the untrained training images for training effect verification during the mixed training process and / or after the mixed training is completed.

Images