Facial expression analysis method based on deep learning

Abstract

The invention relates to the technical field of facial expression analysis, in particular to a facial expression analysis method based on deep learning, which comprises the following steps: acquiring facial feature point coordinates, analyzing displacement and density distribution, obtaining a facial activity intensity index, calling muscle action data, analyzing synchronization rate and direction consistency, and constructing muscle group interaction data. The method comprises the following steps: tracking facial actions, identifying action time, analyzing an action change rate, calculating an action change contribution ratio, analyzing a ratio variable amplitude, adjusting a weight, obtaining an action contribution adjustment result, and comparing a facial region action collaboration degree to obtain a facial coordination output index. According to the method, through fine comparison of facial feature point displacement and density distribution, the measurement precision of facial activity intensity is improved, emotion information extraction is more accurate, muscle group interaction analysis particularly strengthens measurement of synchronous activity rate and direction consistency, and the recognition capability of micro expressions and fast change expressions is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of facial expression analysis, in particular to a facial expression analysis method based on deep learning. BACKGROUND

[0002] Facial expression analysis technology is an important research direction in the field of computer vision and artificial intelligence, involving the identification and interpretation of individual emotions and psychological states from facial images or video streams through image processing and pattern recognition techniques. This technology relies on facial feature extraction and classification algorithms, mainly including facial recognition, facial key point detection, emotion analysis, etc. In recent years, with the continuous development of deep learning technology, the accuracy and efficiency of facial expression analysis have been significantly improved. This technology has been widely applied in security monitoring, human-computer interaction, mental health monitoring, intelligent customer service, and other fields.

[0003] Among them, the facial expression analysis method based on deep learning uses deep neural networks to automatically analyze facial expressions, which can extract emotional information from complex facial images. By training a deep learning model, it can identify different facial expressions and judge the emotional state of individuals according to the changes in expressions. Its main uses include but are not limited to human-computer interaction, emotion monitoring, and psychological state assessment in medical health, etc.

[0004] The existing technology mainly relies on static facial feature extraction, which is difficult to effectively capture the dynamic changes of facial expressions and the subtle interactions between muscle groups, limiting the accuracy of expression recognition. In addition, existing methods perform poorly in processing time series information, making it difficult to track rapidly changing expressions in real time, affecting the timeliness and accuracy of emotion analysis. Traditional expression classification methods rely too much on a single evaluation dimension, ignoring the synergistic effect between facial regions, resulting in a more one-sided emotion analysis result that cannot fully reflect complex emotional changes, affecting the widespread application of facial expression analysis systems. SUMMARY

[0005] To solve the technical problems of existing technologies that rely on static facial feature extraction, which is difficult to effectively capture the dynamic changes of facial expressions and the subtle interactions between muscle groups, limiting the accuracy of expression recognition, in addition, existing methods perform poorly in processing time series information, making it difficult to track rapidly changing expressions in real time, affecting the timeliness and accuracy of emotion analysis, traditional expression classification methods rely too much on a single evaluation dimension, ignoring the synergistic effect between facial regions, resulting in a more one-sided emotion analysis result that cannot fully reflect complex emotional changes, affecting the widespread application of facial expression analysis systems, the embodiments of the present application provide a facial expression analysis method based on deep learning. The technical solution is as follows: On the one hand, a facial expression analysis method based on deep learning is provided, including the following steps: S1: Obtain facial image and skeleton calibration point data, extract feature point coordinates of eye, mouth corner, eyebrow and forehead region, analyze motion trajectory parameters of feature points, calculate displacement of each region, compare with density distribution, and obtain facial activity intensity evaluation index; S2: Based on the facial activity intensity evaluation index, call action data of orbicularis oculi, zygomaticus minor, frontalis and procerus, analyze synchronous activity rate and direction consistency of data, screen muscle group pairs with high synchronization and direction consistency, construct interaction data between muscle groups, and obtain muscle group interaction coordination metric; S3: Use the muscle group interaction coordination metric to track facial action in each frame of image, identify the start and end time points of the action, analyze the displacement increment and frame change frequency in the time period, and obtain facial action change analysis result; S4: Call the facial action change analysis result, combine the initial intensity and time rate change of facial action, calculate the change contribution ratio of the action, compare with the reference baseline, analyze the change amplitude of the ratio, and adjust the change proportion to obtain the action contribution adjustment result.

[0006] On the other hand, the facial activity intensity evaluation index includes eye activity intensity, mouth corner activity intensity, eyebrow activity intensity, and forehead activity intensity. The muscle group interaction coordination metric includes muscle group interaction frequency, action coordination, coordinated change amplitude, and interaction relevance. The facial action change analysis result includes time sequence change amplitude, expression turning point, and action stability. The action contribution adjustment result includes contribution ratio adjustment value, change amplitude adjustment value, re-allocated contribution weight, and ratio update range.

[0007] On the other hand, the steps of S1 are as follows: S101: Obtain facial image and skeleton calibration point data, extract feature point coordinates of eye, mouth corner, eyebrow and forehead region, analyze horizontal and vertical displacement changes of feature points in image frame sequence, calculate average displacement of feature points in the region, and obtain facial region displacement sequence; S102: Call the facial region displacement sequence, compare the displacement changes in the sequence frame by frame, determine the displacement difference of each region, and compare with the average displacement difference value of all frames in the region, screen the frame segment with key deviation as the characteristic activity frame, and count the displacement difference density to obtain the characteristic activity frame displacement density; S103: Based on the characteristic activity frame displacement density, calculate the displacement gradient growth rate of each region, normalize the data through proportional relationship, obtain the motion intensity of each region, and obtain the facial activity intensity evaluation index.

[0008] On the other hand, the steps of S2 are as follows: S201: Based on the facial activity intensity evaluation index, the action data of orbicularis oculi, zygomaticus minor, frontalis and procerus are called, the activity synchronization rate of muscle groups at the target time point is analyzed, whether similar activities are performed at the same time is judged, and time synchronization rate data is obtained; S202: Based on the time synchronization rate data, the action direction consistency of muscle groups is analyzed, whether the movement direction of muscles in expression action is the same is checked, and highly consistent muscle group pairs are screened, and direction consistency score is obtained. S203: The direction consistency score is called, the degree of cooperation between muscle group pairs with high synchronization rate and direction consistency is analyzed, interaction data between muscle groups is constructed, and the coordination of facial expressions is evaluated, and muscle group interaction cooperation degree is obtained.

[0009] On the other hand, the steps of S3 are specifically: S301: Using the muscle group interaction cooperation degree, traverse the image frame sequence, monitor the action of facial muscle groups in each frame, record the frame points of action start and end, determine the duration of action, and mark the frame range of time interval, obtain action time range index; S302: Based on the action time range index, the coordinate increment between related frames is extracted, the displacement increment from each frame to the next frame is calculated, the change rate of displacement increment is analyzed combined with time interval, and rate change parameter is obtained. S303: Based on the rate change parameter, compare with the anatomical region of muscle group, analyze the matching degree of rate and muscle group, evaluate the consistency of matching relationship, and determine the fluctuation direction and amplitude relationship, obtain facial action change analysis result.

[0010] On the other hand, the rate change parameter is compared with the anatomical region of muscle group, and the formula is: ; Analysis of the matching degree of rate and muscle group , wherein, represents the rate change parameter of the first muscle group, represents the anatomical region reference value of the first muscle group, represents the activity intensity of the first muscle group, represents the distance between the rate change and the first muscle group, represents the total number of muscle groups involved in comparison, and e represents the base number of natural logarithm.

[0011] On the other hand, the steps of S4 are specifically: S401: Call the facial movement change analysis results, extract the initial intensity and rate change data of each movement, calculate the data change contribution ratio, and obtain the movement contribution ratio by comparing the contribution ratio of each movement with the average contribution ratio of the movement cycle. S402: Based on the action contribution ratio, set a reference baseline, calculate the deviation rate of each action contribution ratio from the baseline, analyze the variation range of the deviation rate, determine the average fluctuation range of the deviation rate, and mark the action segments that exceed the range to obtain deviation fluctuation analysis data. S403: Based on the deviation fluctuation analysis data, the weights are redistributed according to the magnitude of the difference value, and the contribution of the current action is matched to obtain the action contribution adjustment result.

[0012] On the other hand, the deviation rate between the contribution ratio of each action and the baseline is calculated using the formula: ; Obtain the deviation rate The analysis examines the magnitude of deviation rate changes, determines the average fluctuation range of the deviation rate, and marks the action segments that exceed the range, thus obtaining deviation fluctuation analysis data. Representing the The contribution ratio of each action, This represents a set reference baseline used for comparison with the contribution ratio of each action. Represents the total number of actions.

[0013] On the other hand, the method also includes: S5: Based on the adjustment results of the action contribution, call the action and weight data of the eye, mouth and eyebrow regions, compare the degree of action coordination between each facial region, analyze the matching degree and coordination between region pairs, determine the classification output index range, and obtain the facial coordination output index. The facial coordination output indicators include regional coordination, movement matching, and inter-regional coordination differences.

[0014] On the other hand, step S5 specifically includes: S501: Based on the adjustment results of the action contribution, collect action data and weights of the eye, mouth and eyebrow regions. By analyzing the intensity of the regional actions and the adjusted weights, compare the degree of coordination between each region to obtain coordination intensity data. S502: Based on the aforementioned coordination strength data, analyze the matching degree and coordination degree of actions between different facial regions, calculate the consistency and coordination of actions between regions, quantify the matching degree of region pairs, and obtain a region coordination metric. S503: Using the regional coordination metric, according to the coordination score of facial expression, set the classification output interval, divided into low, medium and high interval level, get the facial coordination output index.

[0015] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects: By fine comparison of facial feature point displacement and density distribution, the measurement accuracy of facial activity intensity is improved, the extraction of emotional information is more accurate, the muscle group interaction analysis particularly strengthens the measurement of synchronous activity rate and direction consistency, the recognition ability of micro-expression and rapidly changing expression is improved, the use of time sequence characteristics makes the tracking of facial movements more real-time, and the dynamic response ability of facial recognition is optimized. Through multi-dimensional analysis of the coordination degree and coordination degree between facial regions, not only the complexity of facial expression analysis is improved, but also the application stability and accuracy of facial recognition in variable environment are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is the main step flowchart of the present application; Figure 2 is the step flowchart of S1 of the present application; Figure 3 is the step flowchart of S2 of the present application; Figure 4 is the step flowchart of S3 of the present application; Figure 5 is the step flowchart of S4 of the present application; Figure 6 is the step flowchart of S5 of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the present application will be described below in combination with the drawings.

[0019] In the embodiments of the present application, the words such as "example", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0020] In the embodiments of the present application, "image" and "picture" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "relevant" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0021] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0022] In order to make the technical problems, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the drawings.

[0023] The embodiments of the present application provide a facial expression analysis method based on deep learning, as shown in Figure 1 The method comprises the following steps: S1: Obtain facial image and skeleton calibration point data, extract feature point coordinates of eye, mouth corner, eyebrow and forehead region, analyze motion trajectory parameters of feature points from image frame sequence, calculate displacement of each region, and compare with density distribution to analyze displacement gradient of each region, and obtain facial activity intensity evaluation index; S2: Based on the facial activity intensity evaluation index, call action data of orbicularis oculi, zygomaticus minor, frontalis and procerus, analyze synchronous activity rate and direction consistency of the data, cross compare according to the activity rate and direction consistency, screen muscle group pairs with high synchronization and direction consistency, construct interaction data between muscle groups, and obtain muscle group interaction coordination metric; S3: Use the muscle group interaction coordination metric to track facial action in each frame of image, identify the start and end time points of the action, analyze the displacement increment and inter-frame change frequency in the time period, compare the change rate with the matching relationship of the muscle group belonging to the region, and obtain facial action change analysis result; S4: Call the facial action change analysis result, combine the starting intensity and time rate change of the facial action, calculate the change contribution ratio of the action, compare with the reference baseline, analyze the change amplitude of the ratio, adjust the change proportion, re-distribute the weight, and obtain the action contribution adjustment result; S5: According to the action contribution adjustment result, call the action and weight data of the eye, mouth and eyebrow regions, compare the action coordination degree between each facial region, analyze the matching degree and coordination degree between the regions, determine the classification output index interval, and obtain the facial coordination output index.

[0024] The facial activity intensity assessment indicators include eye activity intensity, corner of mouth activity intensity, eyebrow activity intensity, and forehead activity intensity. The muscle group interaction coordination measurement includes muscle group interaction frequency, movement coordination, coordination change amplitude, and interaction correlation. The facial movement change analysis results include temporal change amplitude, expression turning point, and movement stability. The movement contribution adjustment results include contribution ratio adjustment value, change amplitude adjustment value, redistributed contribution weight, and ratio update range. The facial coordination output indicators include regional coordination degree, movement matching degree, and inter-regional coordination difference.

[0025] like Figure 2 As shown, the specific steps of S1 are as follows: S101: Obtain facial images and skeleton calibration point data, extract feature point coordinates of the eye, mouth, eyebrow and forehead regions, analyze the horizontal and vertical displacement changes of feature points in the image frame sequence, calculate the average displacement of feature points in the region, and obtain the facial region displacement sequence. Facial images can be acquired using image acquisition devices. For the facial region within the image, edges are detected based on the RGB value distribution and grayscale gradient changes, identifying key areas such as eyebrows, eyes, corners of the mouth, and forehead. Then, by combining the distribution relationship of corresponding regions of the skeleton calibration points, the coordinate information of specific numbered feature points is extracted. For example, the midpoint of the left eyebrow can be obtained by averaging the coordinates of calibration points 17 and 19; similarly, the left corner of the mouth can be determined by the coordinates of calibration point 48. When analyzing the lateral and longitudinal displacement changes of feature points in an image frame sequence, the coordinates of corresponding feature points in the image are read frame by frame. The difference between the x and y coordinates of the current frame and the previous frame is calculated. For example, the lateral displacement of a point in frame t is Δx = x(t) - x(t-1). The vertical displacement Δy = y(t) - y(t-1) is calculated. Then, within each frame, the arithmetic mean of the horizontal and vertical displacements of all feature points in the region is calculated. If a region contains n feature points, its average horizontal displacement is Σ(Δx_i) / n, and its average vertical displacement is Σ(Δy_i) / n. This is used to construct the displacement sequence of the facial region on the time axis. For example, if the forehead region consists of 5 feature points, their horizontal displacements in the 10th frame are [1.2, 0.9, 1.1, 1.0, 1.3] pixels, then the average horizontal displacement of the forehead region in that frame is (1.2 + 0.9 + 1.1 + 1.0 + 1.3) / 5 = 1.1 pixels. The vertical displacement is processed in the same way. All frames are processed in sequence to obtain the complete facial region displacement sequence.

[0026] S102: Call the facial region displacement sequence, compare the displacement changes in the sequence frame by frame, determine the displacement difference of each region, and compare it with the average displacement difference of all frames in the region. Select the frame segments with critical deviations as feature active frames, and calculate their displacement difference density to obtain the displacement density of feature active frames. The displacement sequence of the facial region is called, and the displacement value of each frame in the sequence is compared with the adjacent frame. Assuming that the displacement value of the region in the t frame is V(t), the displacement difference AV(t)=|V(t)-V(t-1)| is calculated. In this way, the displacement difference value set of all frame pairs is calculated, and then in each region, the average value of the displacement difference value of all frame pairs is calculated μ=∑ΔV(t) / N, where N is the total number of frames minus one. The displacement difference of each frame is compared with the average value. If ΔV(t)>μ, the frame is determined to be a deviation frame. Further filtering all frames that meet the condition forms a preliminary characteristic active frame set. The determination of the μ value in the judgment standard depends on the average value obtained by statistical frame difference data. The value can be obtained by program processing. For example, the region displacement difference value set is [0.4, 0.5, 0.6, 0.7, 1.2, 0.8, 0.3] pixels. The average difference μ=(0.4+0.5+0.6+0.7+1.2+0.8+0.3) / 7≈0.643 pixels. Only the frame segment greater than 0.643 is reserved as the key frame of deviation. The total frame proportion occupied by all characteristic active frames in the time axis is calculated. The displacement density ρ=K / T is calculated, where K is the number of characteristic active frames, and T is the total number of image frames. If K=30 frames and T=120 frames, the displacement density ρ of the characteristic active frame of the region is 30 / 120=0.25, i.e. 25%.

[0027] S103: Based on the displacement density of the characteristic active frame, the displacement gradient growth rate of each region is calculated. The data is normalized by proportional relationship to obtain the motion intensity of each region, and the facial activity intensity evaluation index is obtained. The displacement gradient growth rate of each region needs to be calculated in the characteristic active frame to analyze the trend of the displacement value changing with time. Assuming that the average displacement of the region in the t frame is V(t), the displacement gradient growth rate in the t1 to t2 frame segment can be represented as g=(V(t2)-V(t1)) / (t2-t1). For example, in the eye region, the 20th frame V(20)=1.2 pixels, and the 30th frame V(30)=2.1 pixels. Then g=(2.1-1.2) / 10=0.09 pixels / frame. The g value of each region is normalized by proportion. The maximum g value is set to 1, and the g values of other regions are scaled to the [0, 1] interval in proportion. If the maximum gradient value is 0.15 and the g value of a region is 0.09, the normalized intensity I=0.09 / 0.15=0.6. The normalized motion intensity of each region is the final facial activity intensity evaluation index, and the value range is defined as the [0, 1] interval. The low intensity is defined as [0, 0.3), the medium intensity is defined as [0.3, 0.7), and the high intensity is defined as [0.7, 1]. For example, when the forehead region index is 0.75, it is determined to be a high-intensity region. The facial activity intensity evaluation index set is formed after all regions are determined.

[0028] As Figure 3As shown, the steps of S2 are specifically: S201: Based on the facial activity intensity evaluation index, the action data of orbicularis oculi, zygomaticus minor, frontalis and depressor supercilii are called, the activity synchronization rate of the muscle group at the target time point is analyzed, whether similar activities are performed at the same time is judged, and time synchronization rate data is obtained; The action data of orbicularis oculi, zygomaticus minor, frontalis and depressor supercilii are called, the contraction strength of each muscle on the time axis is positioned, the displacement value of the key feature point in the control area of each muscle is extracted frame by frame, the motion amplitude in its time sequence is calculated, and each muscle action time is marked. Then, whether the actions of multiple muscles at the same time point are synchronized is judged. The specific steps are as follows. In each frame of image, the average displacement intensity of the control area of four muscles is recorded respectively. When the average displacement value of a certain muscle area is greater than a preset threshold value, it is determined that the muscle is in an active state at the current frame. The threshold value is set with reference to the reference intensity value μ of the region and an empirical coefficient λ to form a threshold value T, that is, T=μ+λ. For example, if the frontalis muscle obtains μ of 0.6 pixels in the average resting state during the analysis process, and λ is 0.4, then T=1.0 pixel. If the average displacement of the frontalis muscle area in the 45th frame is 1.3 pixels, the frontalis muscle in this frame is determined to be active. At the same time, whether the other three muscles also meet the active state standard in this frame is recorded. If the number of muscles that meet the standard is n, and the total number of muscles is N, then the time synchronization rate R in this frame is n / N. R is calculated frame by frame to form a time sequence, and the proportion of frames with synchronization rate greater than a set threshold value R0 in the total number of frames is further counted to form the final time synchronization rate data. R0 can be set to 0.75, which means that if 3 or more muscles are activated at the same time in a frame, it is considered that the frame has synchronization characteristics. If the total number of frames is 100, and the number of frames that meet R≥0.75 is 30, then the final time synchronization rate is 0.3, that is, 30%.

[0029] S202: Based on the time synchronization rate data, the action direction consistency of the muscle group is analyzed, whether the movement direction of the muscle in the expression action is the same is checked, and a highly consistent muscle group pair is selected to obtain a direction consistency score; The direction of displacement of each muscle in a specific synchronization frame is analyzed to determine whether the directions are consistent. The basis for the determination is the motion direction vector of the feature points in the muscle region in the synchronization frame. The direction of the main motion of the muscle in the current frame is obtained by weighted averaging the displacement direction vectors of all feature points in each muscle region in the adjacent frame. Let D represent a two-dimensional vector (dx, dy), where dx is the horizontal displacement increment and dy is the vertical displacement increment. If the main motion direction of the orbicularis oculi muscle between the 50th frame and the 51st frame is D1 = (1.2, -0.5) and the main motion direction of the zygomaticus minor muscle is D2 = (1.1, -0.6), the angle θ between the two vectors is calculated. According to the inner product formula cosθ = (D1 · D2) / (|D1||D2|), it is obtained that θ ≈ 5.6°. It is determined that the directions are consistent if θ is less than a set threshold θ0. The value of θ0 is set to 15°, that is, the directions are considered consistent if the angle between the directions is not more than 15°. The proportion of the number of muscle pairs that satisfy the consistent directions to the total number of combinations in all synchronization frames is counted. Let the total number of combinations be M and the number of combinations that satisfy the condition be m. The direction consistency score S is m / M. For example, if there are 3 muscle combinations in each frame in 30 synchronization frames, and 2 pairs of muscles satisfy the consistent directions in each frame on average, then the total m = 2 x 30 = 60, M = 3 x 30 = 90, and S = 60 / 90 = 0.667, that is, 66.7%. The score reflects the strength of the coordination between the muscle groups in the direction of motion. The direction consistency score is obtained.

[0030] S203: The direction consistency score is called to analyze the coordination degree between the muscle groups that have high synchronization rates and consistent directions, to construct the interaction data between the muscle groups, and to evaluate the coordination of the facial expressions, to obtain the muscle group interaction coordination metric. The contraction amplitude variation trends of the screened muscle pairs in the synchronization frames are compared to determine whether they have a positive coordination relationship. The specific operation is as follows. The contraction displacement values of each muscle pair in each frame are constructed into sequences. The product S_i = ΔA_i x ΔB_i of the inter-frame difference values ΔA_i and ΔB_i of the two sequences is calculated. If S_i > 0, the two muscles change in the same direction in the frame. If S_i < 0, the two muscles change in opposite directions. The proportion P of the frames in which S_i > 0 is calculated. If P ≥ a coordination threshold P0, it is considered that the muscle pair has a coordination relationship. P0 can be 0.8, representing that the two muscles change in the same direction in at least 80% of the frames. In an example, if 25 frames are analyzed and the number of frames in which S_i > 0 is 21, then P = 21 / 25 = 0.84 > 0.8, and it is considered that the muscle pair is coordinated. The above processing is repeated to traverse all the direction-consistent muscle pairs to construct a coordination matrix, record the number of coordination relationships between the muscle pairs, and comprehensively evaluate the coordination degree of the entire facial muscle system. The interaction coordination degree C can be defined as the number of coordinated pairs / the number of direction-consistent pairs. If the number of coordinated pairs is 12 and the number of direction-consistent pairs is 15, then C = 0.8. The data is finally used as an evaluation index of the coordination of facial expressions to obtain the muscle group interaction coordination metric.

[0031] As Figure 4 shown, the steps of S3 are specifically: S301: Using muscle group interactive coordination measurement, traversing the image frame sequence, monitoring the action of the facial muscle group in each frame, recording the frame points of the action start and end, determining the duration of the action, and marking the frame range of the time interval to obtain the action time range index; Using muscle group interactive coordination measurement, it is necessary to traverse the image sequence frame by frame, and perform feature point recognition and action state analysis on the control area of all muscle groups in each frame. By extracting the average displacement intensity of the key feature points in each muscle corresponding area, it is determined whether the muscle in the current frame is in the action state. The judgment standard is whether the average displacement value of the muscle in the current frame is higher than the activation threshold T constructed by the static state displacement mean value plus the empirical offset coefficient. If the static mean value μ of a muscle is 0.5 pixels and the empirical coefficient λ is set to 0.3, then T = 0.5 + 0.3 = 0.8 pixels. When the actual value is greater than 0.8 pixels, the muscle is marked as "activated", otherwise as "static". The frame where the state changes from "static" to "activated" in the continuous frame is regarded as the action start frame, and the frame where the state changes from "activated" to "static" is regarded as the action end frame. The start and end frames of each complete action in the entire sequence are recorded, the duration of each action is calculated Δt = end frame number - start frame number + 1, for example, muscle group A starts at frame 30 and ends at frame 45, then Δt = 45-30+1 = 16 frames. At the same time, the frame interval occupied by each action is indexed and marked to construct the start and end frame segment record table, and the complete action time range index is formed.

[0032] S302: Based on the action time range index, the coordinate increment between the related frames is extracted, the displacement increment from each frame to the next frame is calculated, and the change rate of the displacement increment is analyzed combined with the time interval to obtain the rate change parameter; The feature point displacement coordinates of all muscle control areas in the specified frame interval are extracted, and the coordinate difference is calculated point by point between the continuous frames. Let the coordinates of feature point i in frame t be (x i (t), y i (t)), and the coordinates of feature point i in frame t+1 be (x i (t+1), y i (t+1)). Then the displacement increment Δx i = x i (t+1)-x i (t), Δy i =y i (t+1)-y i(t), the average of the increments of all feature points in the frame is used to obtain the overall inter-frame displacement increments Δx_avg and Δy_avg. Then, combined with the inter-frame time interval Δt, the rate of change of this displacement increment per unit time is calculated as v_x = Δx_avg / Δt and v_y = Δy_avg / Δt. For example, between frames 35 and 36, Δt = 1 frame. If Δx_avg = 1.2 pixels and Δy_avg = -0.8 pixels, then the horizontal rate v_x = 1.2 and the vertical rate v_y = -0.8. This operation is repeated to traverse all adjacent frames within the action frame segment, forming a rate change sequence. The difference between the rate sequences is calculated to obtain the rate change between adjacent frames Δv = v(t+1) - v(t). Then, the average change and standard deviation of Δv are statistically analyzed to form a set of rate change parameters. For example, if the continuous rate change is [0.2, 0.3, -0.1, 0.4, 0.2] pixels / frame. 2 The mean is 0.2 and the standard deviation is 0.167. The statistic is used as the rate change parameter for subsequent analysis.

[0033] S303: Based on the rate change parameters, compare them with the anatomical regions of muscle groups, analyze the matching degree between rate and muscle groups, evaluate the consistency of the matching relationship, determine the relationship between the direction and amplitude of fluctuations, and obtain the facial movement change analysis results. Based on the rate change parameter, a comparison was made with the anatomical regions of the muscle group, using the following formula: ; Analysis of the matching degree between speed and muscle groups ,in, Representing the The rate change parameters of each muscle group are used to quantify the dynamic rate of change of the corresponding muscle group in each anatomical region. Representing the The anatomical reference values ​​for each muscle group, i.e., the theoretical standard movement response rate of that muscle group. Representing the The activity intensity of a muscle group reflects the degree of activity of that muscle group during facial movements. Representative rate change and the first The distance between individual muscle groups quantifies the relative distance between the anatomical location and rate of change of muscle groups. The number of muscle groups involved in the comparison is represented by , and e represents the base of the natural logarithm. set up Rate change parameters of the first muscle group Reference value Activity intensity Distance index ; Rate change parameters of the second muscle group Reference value , activity intensity , distance index , the calculation process is as follows: For each muscle group: ; ; Calculate : ; The results show that the average matching degree is 0.561, indicating that the average matching degree of the rate of change and the muscle group in the analysis range is medium and low, reflecting the consistency level between the considered muscle group and the rate of change parameter in the facial action response. Lower matching degree suggests that the motion capture algorithm needs to be adjusted or the reference rate of the muscle group needs to be reevaluated to improve the response accuracy and effectiveness of facial recognition.

[0034] As Figure 5 shown, the steps of S4 are as follows: S401: Call the facial action change analysis result, extract the starting intensity and rate change data of each action within the time, calculate the data change contribution ratio, and obtain the action contribution ratio by comparing the contribution ratio of each action with the average contribution ratio of the action period; First, extract the muscle activity intensity value S0 corresponding to the starting frame position of each action, and obtain the rate change sequence V={v1, v2,..., v n} in the time period covered by the action. Associate each frame rate value in the sequence with the initial intensity S0, calculate the amplitude change Δv i =|v i -S0| of each frame rate relative to the initial intensity, and accumulate the Δv i of all frames to obtain the total change value ΣΔv of the action. Then calculate the change contribution ratio C of the action according to the formula C=ΣΔv / (S0×T), where T is the number of action frames, and ΣΔv represents the total rate change of the action in the entire period. S0×T is used as a normalized reference quantity to represent the standard total change quantity of the action under static intensity. For example, if the starting intensity S0 of an action is 1.0 pixel / frame, the duration T is 20 frames, and the rate sequence is [1.2, 1.5, 1.1, 1.3, …] with a total of 20 items, then ΣΔv=6.8 pixel / frame, and C=6.8 / (1.0×20)=0.34. Then calculate the contribution ratio C_i of each action, and then average the contribution ratios of all actions C̄=ΣC_i / N, where N is the number of actions. Finally, compare the contribution ratio of each action with the average value C̄, record the difference δC=C_i-C̄, and mark the contribution ratio δC of each action.

[0035] S402: Based on the action contribution ratio, set a reference baseline, calculate the deviation rate of each action contribution ratio from the baseline, analyze the change range of the deviation rate, determine the average fluctuation interval of the deviation rate, and mark the action segment that exceeds the interval to obtain deviation fluctuation analysis data; The deviation rate of each action contribution ratio from the baseline is calculated using the formula: ; Obtain the deviation rate , analyze the change range of the deviation rate, determine the average fluctuation interval of the deviation rate, and mark the action segment that exceeds the interval to obtain deviation fluctuation analysis data, wherein, represents the contribution ratio of the i-th action, represents the set reference baseline for comparison with the contribution ratio of each action, represents the total number of actions; There are 5 actions, and their contribution ratios are monitored to be Therefore, ; ; Reference baseline The average value obtained by analyzing past data is set to .

[0036] Using data to calculate : ; The obtained indicates that the overall deviation rate is 0.1414, indicating that the average deviation degree of the action relative to the baseline is low, and the average fluctuation interval of the deviation rate is small, which can be used to determine whether the deviation of any action exceeds the normal fluctuation range, helping to identify the action segment that needs to be improved or adjusted, and then maintaining the consistency and quality standards of operation.

[0037] S403: Based on the deviation fluctuation analysis data, reassign weights according to the size of the difference value, match the contribution of the current action, and obtain the action contribution adjustment result; For each action segment, call its corresponding contribution ratio and deviation rate, compare its relative reference deviation amplitude δR_i=|C_i-B| / B, calculate the difference value δR_i for all actions, and adjust the weight according to its value. The adjustment method is to redistribute the original action weight W_i in the overall evaluation. If δR_i> deviation weight threshold θ, the weight W_i of the action needs to be reduced. If δR_i< θ, maintain or moderately increase W_i. The threshold θ is set as the reference baseline deviation tolerance, for example θ=0.20, which corresponds to a deviation of ±20% within the normal range, and the part exceeding the threshold is abnormal. For each δR_i, according to its size, use the inverse proportion to distribute the weight adjustment factor α_i=1 / (1+δR_i), so that the greater the difference, the smaller the weight. For example, δR_i=0.5, α_i=1 / (1+0.5)=0.666, if the initial weight W0=1.0, then the adjusted weight W_i=W0×α_i=0.666. All action segments perform this adjustment operation to form the action contribution adjustment result, which is a set of weight correction values that match the actual contribution of the action.

[0038] As shown in Figure 6 , the steps of S5 are specifically: S501: According to the action contribution adjustment result, collect the action data and weight of the eye, mouth and eyebrow regions, analyze the intensity of the regional action and the adjusted weight, compare the coordination degree between each region, and obtain the coordination intensity data; First, extract the feature point set of the facial eye, mouth and eyebrow regions and the corresponding action segment. In each action segment, the average displacement amplitude of the feature points in the region is obtained as the action intensity of the region. Let the feature points of the t-th frame of the region R be n, and the displacement between the t-th frame and the t-1-th frame of each feature point be Δd1, Δd2,..., Δd n , then the action intensity A_R(t)=(ΣΔd i) / n, while calling the action weight W(t) obtained in the previous step and assigning it to each frame, and then performing a normalization calculation on the weighted action intensity of the three regions in each frame, and performing a coordination degree analysis on the action data between regions, using the cosine similarity of the three-region weighted intensity vector for comparison. If the vector A=[A_eye, A_mouth, A_brow] and B=[W_eye, W_mouth, W_brow], then the similarity S=(A·B) / (|A||B|) is calculated. This calculation is performed for each frame, and the average of the S values obtained by calculating all frames is the coordination intensity value of the three regions in the action segment. For example, if the eye action intensity is 1.2 pixels, the mouth is 2.3, and the eyebrow is 1.5, and the corresponding weights are 0.8, 1.1, and 0.9, respectively, then the calculated coordination intensity is S≈0.997. By traversing all action segments, the coordination intensity sequence of the corresponding frame segment is generated, and finally the coordination intensity data is formed.

[0039] S502: Based on the coordination intensity data, analyze the matching degree and coordination of the action between the different facial regions, calculate the consistency and coordination of the action between the regions, and quantify the matching degree of the region pair to obtain the region coordination metric; The action pairs of different facial regions are analyzed in pairs, and the region pairs are set as (eye-mouth), (eye-brow), and (mouth-brow). The weighted action intensity sequence of each pair of regions in each action segment is extracted, respectively A(t) and B(t). The difference between the frame vectors of the sequences is compared, the difference D(t)=|A(t)-B(t)| of each frame is calculated, and the action consistency value U of the region pair is obtained by taking the average of all frames, where T is the number of frames in the action segment. The consistency value U ranges from 0 to 1, and the closer U is to 1, the more consistent it is. Then further analyze the coordination by observing whether the trend direction of the action between the regions is consistent, and the judgment standard is whether the sign of the action intensity change direction of the two regions in the adjacent frames is the same. Set ΔA(t)=A(t+1)-A(t), ΔB(t)=B(t+1)-B(t), if ΔA(t)·ΔB(t)>0 represents the same direction, count the number of frames M that satisfy this condition, and the coordination C=M / (T-1). For example, if a segment is 10 frames and 7 frames satisfy the same direction, then C=7 / 9≈0.778. Finally, the arithmetic mean of the consistency U and the coordination C is defined as the matching degree M=(U+C) / 2. The M value of the three region pairs is calculated to form the region coordination metric set.

[0040] S503: Using the region coordination metric, according to the coordination score of the facial expression, set the classification output interval, divide it into low, medium and high interval levels, and obtain the facial coordination output index. The matching degree M of each region pair is scored and output, first, the classification output interval of the coordination score is set, according to the coordination score range [0, 1], three level intervals of low, medium and high are divided, the interval is set as: low coordination [0, 0.4), medium coordination [0.4, 0.7), high coordination [0.7, 1], for each region pair coordination M_i, judge which interval it falls into, and correspondingly assign a classification label L_i, for example, M=0.35 is judged as low coordination, M=0.66 is medium coordination, and M=0.85 is high coordination, finally, the coordination levels of all region pairs are counted and summarized to construct the overall face coordination output index, if two of the three region pairs are high coordination and one is medium coordination, the overall level of expression coordination can be defined as 'high', if two are medium and one is low, the output is'medium', and if most are low, the output is 'low', the output is a coordination output index composed of a clear level label and the coordination of each region pair.

[0041] It should be understood that the term "and / or" herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, wherein A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects, but can also represent an "and / or" relationship, which can be understood in combination with the context.

[0042] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, and c can be single or multiple.

[0043] It should be understood that in various embodiments of the present application, the size of the sequence number of the above processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0044] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0045] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0046] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of units or components can be combined or integrated into another device, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0047] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0048] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0049] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0050] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A facial expression analysis method based on deep learning, characterized by, The method comprises: S1: obtaining facial image and skeleton calibration point data, extracting feature point coordinates of eye, mouth corner, eyebrow and forehead region, analyzing motion trajectory parameters of feature points, calculating displacement of each region, comparing with density distribution to obtain facial activity intensity evaluation index; S2: based on the facial activity intensity evaluation index, calling action data of orbicularis oculi, zygomaticus minor, frontalis and procerus, analyzing synchronous activity rate and direction consistency of data, screening muscle group pairs with high synchronization and direction consistency, constructing interaction data between muscle groups, obtaining muscle group interaction coordination measurement; S3: using the muscle group interaction coordination measurement, tracking facial action in each frame of image, identifying starting and ending time points of action, analyzing displacement increment and interframe change frequency in time period to obtain facial action change analysis result; S4: calling the facial action change analysis result, combining starting intensity and time rate change of facial action, calculating change contribution ratio of action, comparing with reference baseline, analyzing change amplitude of ratio, and adjusting change proportion to obtain action contribution adjustment result. 2.The deep learning-based facial expression analysis method of claim 1, wherein, The facial activity intensity evaluation index includes eye activity intensity, mouth corner activity intensity, eyebrow activity intensity and forehead activity intensity, the muscle group interaction coordination measurement includes muscle group interaction frequency, action coordination, coordinated change amplitude and interaction relevance, the facial action change analysis result includes time sequence change amplitude, expression turning point and action stability, and the action contribution adjustment result includes contribution ratio adjustment value, change amplitude adjustment value, re-allocated contribution weight and ratio update range. 3.The deep learning-based facial expression analysis method of claim 1, wherein, The steps of S1 are specifically: S101: obtaining facial image and skeleton calibration point data, extracting feature point coordinates of eye, mouth corner, eyebrow and forehead region, analyzing horizontal and vertical displacement change of feature points in image frame sequence, calculating average displacement of feature points in region to obtain facial region displacement sequence; S102: calling the facial region displacement sequence, comparing displacement change in sequence frame by frame to determine displacement difference of each region, comparing with average displacement difference value of all frames of region, screening frame segment with key deviation as feature activity frame, and counting displacement difference density thereof to obtain feature activity frame displacement density; S103: based on the feature activity frame displacement density, calculating displacement gradient growth rate of each region, normalizing data through proportional relationship to obtain motion intensity of each region, and obtaining facial activity intensity evaluation index. 4.The deep learning-based facial expression analysis method of claim 1, wherein, The steps of S2 are specifically: S201: based on the facial activity intensity evaluation index, calling action data of orbicularis oculi, zygomaticus minor, frontalis and procerus, analyzing activity synchronous rate of muscle group at target time point to determine whether similar activity is performed at the same time to obtain time synchronization rate data; S202: based on the time synchronization rate data, analyzing action direction consistency of muscle group, checking whether the movement direction of muscle in expression action is the same, screening muscle group pairs with high consistency to obtain direction consistency score; S203: Call the direction consistency score, analyze the degree of coordination between muscle groups with high synchronization rate and consistent direction, construct the interaction data between muscle groups, and evaluate the coordination of facial expressions to obtain the muscle group interaction coordination metric. 5.The deep learning-based facial expression analysis method of claim 1, wherein, The steps of S3 are specifically: S301: Use the muscle group interaction coordination metric to traverse the image frame sequence, monitor the action of the facial muscle group in each frame, record the frame points of the action start and end, determine the duration of the action, and mark the frame range of the time interval to obtain the action time range index; S302: Based on the action time range index, extract the coordinate increment between the associated frames, calculate the displacement increment from each frame to the next frame, combine the time interval, analyze the change rate of the displacement increment, and obtain the rate change parameter; S303: Based on the rate change parameter, compare it with the anatomical region of the muscle group, analyze the matching degree of the rate and the muscle group, evaluate the consistency of the matching relationship, and determine the fluctuation direction and amplitude relationship to obtain the facial action change analysis result. 6.The deep learning-based facial expression analysis method of claim 5, wherein, The rate change parameter is compared with the anatomical region of the muscle group using the formula: ； Matching of analysis rate to muscle group wherein, a rate change parameter representative of the muscle group, an anatomical region reference value representative of the muscle group, an activity intensity representative of the muscle group, a distance representative of the rate change from the muscle group, a total number of muscle groups involved in the comparison, and e represents the base of the natural logarithm. 7.The deep learning-based facial expression analysis method of claim 1, wherein, The steps of S4 are specifically: S401: Call the facial action change analysis result, extract the starting intensity and rate change data of each action within the time, calculate the data change contribution ratio, compare the contribution ratio of each action with the average contribution ratio of the action period, and obtain the action contribution ratio; S402: Based on the action contribution ratio, set a reference baseline, calculate the deviation rate of each action contribution ratio from the baseline, analyze the change amplitude of the deviation rate, determine the average fluctuation interval of the deviation rate, and mark the action segment that exceeds the interval to obtain the deviation fluctuation analysis data; S403: Based on the deviation fluctuation analysis data, reassign weights according to the size of the difference value, match the contribution of the current action, and obtain the action contribution adjustment result. 8.The deep learning-based facial expression analysis method of claim 7, wherein, The formula for calculating the deviation rate of each action contribution ratio from the baseline is: ； Obtaining deviation rate , analyzing the change range of the deviation rate, determining the average fluctuation interval of the deviation rate, and marking the action segment exceeding the interval to obtain deviation fluctuation analysis data, wherein, representing the contribution ratio of the first action, representing the set reference baseline, used for comparison with the contribution ratio of each action, representing the total number of actions. 9.The deep learning-based facial expression analysis method of claim 1, wherein, The method further comprises: S5: According to the action contribution adjustment result, call the action and weight data of the eye, mouth, and eyebrow regions, compare the action coordination degree between each facial region, analyze the matching degree and coordination degree between the regions, determine the classification output indicator interval, and obtain the facial coordination output indicator; The facial coordination output indicator includes region coordination degree, action matching degree, and inter-regional coordination difference. 10.The deep learning-based facial expression analysis method of claim 9, wherein, The steps of S5 are specifically: S501: According to the action contribution adjustment result, collect the action data and weights of the eye, mouth, and eyebrow regions, compare the coordination degree between each region by analyzing the intensity of the regional action and the adjusted weight, and obtain the coordination intensity data; S502: Based on the coordination intensity data, analyze the matching degree and coordination degree of the action between the different facial regions, calculate the consistency and coordination of the action between the regions, and quantify the matching degree of the region pairs to obtain the region coordination metric; S503: Use the region coordination metric to set the classification output interval based on the coordination score of the facial expression, divide it into low, medium, and high interval levels, and obtain the facial coordination output indicator.

Images