Composite emotion recognition method and device based on time sequence feature fusion and LLM driving and storage medium

Abstract

The invention discloses a composite emotion recognition method and device based on time sequence feature fusion and LLM driving and a storage medium. The method comprises the following steps: acquiring multiple frames of to-be-recognized face images of a target user, a calm frame face image and synchronous user voice, wherein the to-be-recognized face image comprises a first facial action unit (AU); calculating a node pair variation feature and a key point movement speed feature of a first facial action unit (AU) corresponding to each frame of to-be-recognized face image; determining an intensity value for each first facial action unit (AU); inputting each intensity value into a space-time convolutional network to obtain an AU intensity-time curve and an AU combined time sequence matrix; screening the AU intensity-time curve and the AU combination time sequence matrix by using an activation threshold value and a screening condition; and processing user voice, the screened AU intensity-time curve and the screened AU combination time sequence matrix through an LLM semantic alignment module to obtain multi-modal fusion features, and inputting the multi-modal fusion features into an LLM-driven AU-emotion dynamic causal reasoning library to determine a composite target expression of a target user.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a composite emotion recognition method, device and storage medium based on temporal feature fusion and LLM-driven approach. Background Technology

[0002] Facial expressions are one of the core nonverbal cues for conveying emotional states and intentions, and their automatic recognition technology has broad application prospects in fields such as human-computer interaction, mental health assessment, safe driving, and intelligent monitoring. Currently, the analysis method based on facial action units (AUs) is the mainstream technical approach in the field of facial expression recognition. This method objectively and quantitatively describes facial expression details by detecting the activity of specific facial muscle groups (such as AU4 corresponding to frowning and AU12 corresponding to raising the corners of the mouth), thereby establishing a correlation between AUs and inner emotions, which is the core technical foundation for achieving accurate emotion inference.

[0003] However, existing AU recognition and expression inference systems still have multiple technical shortcomings, making it difficult to meet the needs of complex real-world scenarios. Specific problems are as follows:

[0004] 1. Weak anti-interference ability and insufficient robustness

[0005] Traditional visual models are susceptible to interference from environmental and dynamic factors. Issues such as changes in lighting and image noise can blur AU features, making it easy to miss some low-intensity, short-duration micro-expressions (such as inner eyebrow lifting AU1). When faced with dynamic scenes such as hand occlusion, hair occlusion, and head turning, the system lacks a mechanism to distinguish non-emotional actions, often misclassifying actions such as blinking (AU43) and head turning as valid AUs. Although some studies have optimized the system through adversarial training, these studies are mostly based on ideal laboratory data and do not cover continuous dynamic occlusion scenarios. Therefore, in non-cooperative environments such as real-time interaction, the system's performance will significantly degrade.

[0006] 2.AU - Staticization of emotional associations makes it difficult to analyze complex emotions.

[0007] Existing technologies primarily rely on fixed AU combination rules (such as Ekman's six basic emotion models) to infer emotions, without considering the dynamic evolutionary characteristics of AUs. Specifically, they ignore the impact of AU activation order (e.g., the transitional state of AU15 (drooping corners of the mouth) followed by AU4 (frowning), intensity changes (e.g., the increased intensity of AU12 (upturned corners of the mouth)), and duration on emotions, and can only output basic emotion categories. Furthermore, they lack effective processing capabilities for the ambiguity of AU combinations in ambiguous expressions (e.g., "AU1+AU2+AU25" may correspond to the initial stage of surprise or fear), struggle to distinguish complex emotions in transitional states (e.g., fear-surprise, sadness-anger), and fail to integrate contextual semantics and individual user differences, resulting in insufficient accuracy in recognizing complex emotions.

[0008] 3. The reasoning mechanism is rigid, lacking interpretability and dynamic adaptability.

[0009] The AU (Authorized Aspect) emotion association relies on a pre-defined rule base, which has certain limitations. For example, the reasoning process lacks interpretability, only outputting emotion categories without demonstrating the "AU feature → emotion" association logic, failing to meet the needs of decision traceability in fields such as medicine and education. Furthermore, it lacks dynamic reasoning capabilities; the AU combinations and weights in the rule base are fixed (e.g., the weight of AU12 in the pleasant emotion remains constant), unable to adjust the reasoning logic based on the temporal relationship of AUs, nor adapt to different scenarios (e.g., driving scenarios require prioritizing AUs related to surprise) and population differences. In addition, the system does not incorporate the dynamic causal reasoning capabilities of a Large Language Model (LLM), making it prone to misjudging AU temporal associations and generating reasoning illusions, further reducing the reliability of complex emotion recognition.

[0010] Therefore, a novel composite emotion recognition method is urgently needed to solve the problems of existing facial expression recognition technology. Summary of the Invention

[0011] To address the technical problems of existing facial expression recognition technologies, such as "weak anti-interference ability, difficulty in parsing complex emotions, and lack of interpretability in reasoning," this application provides a method, device, and storage medium for complex emotion recognition based on temporal feature fusion and LLM-driven approach. Through dynamic AU feature modeling, LLM intelligent reasoning, and cross-modal fusion, it achieves accurate recognition of complex emotions.

[0012] The first aspect of this application provides a composite emotion recognition method based on temporal feature fusion and LLM-driven approach, comprising:

[0013] The system acquires multiple frames of the target user's face image to be identified, a calm frame face image, and synchronized user voice. The face image to be identified includes a first facial action unit (AU).

[0014] Based on the calm frame face image, calculate the node pair change features and key point movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified;

[0015] The intensity value of each first facial action unit (AU) is determined based on the node change characteristics and the key point movement speed characteristics;

[0016] The intensity value of each first facial action unit (AU) is input into the spatiotemporal convolutional network to obtain the AU intensity-time curve and the AU combination temporal matrix;

[0017] The AU intensity-time curves and AU combination time series matrices are filtered using activation thresholds and filtering conditions to obtain the filtered AU intensity-time curves and AU combination time series matrices.

[0018] The user's speech and the filtered AU intensity-time curve and AU combination temporal matrix are processed by the LLM semantic alignment module to obtain multimodal fusion features;

[0019] The multimodal fusion features are input into the LLM-driven AU-emotion dynamic causal reasoning library, and the composite target expression of the target user is determined through causal reasoning. The composite target expression includes basic emotion and / or transitional composite emotion and intensity.

[0020] Furthermore, the calculation of the node pair change features and keypoint movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified includes:

[0021] The first key node pair of the face image to be identified is determined based on the first facial action unit (AU);

[0022] Obtain key point information of the face image to be identified, and calculate the node distance of the first key node pair;

[0023] Obtain the node distance of the first key node pair in the calm frame face image;

[0024] Calculate the difference between the node distances of the face image to be identified and the calm frame face image to obtain the node pair change feature;

[0025] Based on the key point information of two adjacent frames of the face image to be identified, the movement speed of the key points is determined, and the key point movement speed feature is obtained.

[0026] Furthermore, determining the intensity value of each first facial action unit (AU) based on the node pair change feature and the keypoint movement speed feature includes:

[0027] The node pair change features and the key point movement speed features are input into the support vector regression model to obtain the intensity value of the first facial action unit (AU) in each frame of the face image to be identified.

[0028] Furthermore, the AU intensity-time curve is the intensity value of multiple first facial action units (AUs) arranged in frame order, and the AU combination time sequence matrix is ​​a matrix of total number of frames × number of first facial action units (AUs), where each element in the matrix represents the intensity value of a certain first facial action unit (AU) in the corresponding frame.

[0029] Furthermore, the construction of the activation threshold includes:

[0030] Acquire multiple test images of the target user, the test images including a face test image in a calm state, the face test image including a second facial motion unit;

[0031] Determine the intensity value for each of the second facial motion units;

[0032] Calculate the average intensity value and standard deviation of all second facial motion units based on the intensity values;

[0033] The activation threshold is constructed based on the average intensity value, the standard deviation of the intensity value, and the preset noise tolerance coefficient;

[0034] The construction of the filtering criteria includes:

[0035] The intensity value of the first facial motion unit (AU) for a consecutive preset number of frames is not lower than the activation threshold, and the fluctuation range of the first facial motion unit (AU) for a consecutive preset number of frames is less than the preset value.

[0036] Furthermore, the LLM-driven AU-emotion dynamic causal inference library is constructed, including:

[0037] Based on the labeled dataset of the first facial action unit (AU), emotion labels, and causal reasoning samples containing the correspondence between AU activation time sequence and emotion evolution, an initial reasoning knowledge base is constructed.

[0038] Using the accuracy of emotion recognition and the reasonableness of the interpretable text generated by the initial inference base as joint reward signals, the initial inference knowledge base is trained through a policy gradient optimization algorithm. During the training process, first facial action unit (AU) and emotion association data from multiple scenarios are incorporated.

[0039] Furthermore, the method also includes:

[0040] Obtain the current application scenario and the demographic type of the target users;

[0041] The AU-emotion weights in the LLM-driven AU-emotion dynamic causal inference library are adjusted based on the current application scenario and the population type. The AU-emotion weights include the consideration priority and / or correlation strength of facial action units (AUs).

[0042] Furthermore, the process of processing the user speech and the filtered AU intensity-time curve and AU combination temporal matrix through the LLM semantic alignment module to obtain multimodal fusion features includes:

[0043] Extract the speech features of the user's speech, and perform semantic understanding on the speech features to obtain speech semantic features;

[0044] The speech semantic features, the filtered AU intensity-time curves, and the AU combined temporal matrix are input into the LLM semantic alignment module, and the association weights between the speech semantic features and the visual temporal features are calculated based on the cross-modal attention mechanism.

[0045] The speech semantic features are dynamically weighted and fused with the filtered AU intensity-time curve and AU combination time-series matrix according to the association weights to obtain the multimodal fusion features.

[0046] The second aspect of this application proposes a composite emotion recognition device based on temporal feature fusion and LLM-driven approach, comprising:

[0047] The acquisition module is used to acquire multiple frames of face images to be identified, calm frame face images, and synchronized user voice of the target user. The face images to be identified include a first facial action unit (AU).

[0048] The calculation module is used to calculate the node pair change features and key point movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified based on the calm frame face image;

[0049] A determination module is used to determine the intensity value of each first facial action unit (AU) based on the node pair change feature and the key point movement speed feature;

[0050] The input module is used to input the intensity value of each of the first facial action units (AU) into the spatiotemporal convolutional network to obtain the AU intensity-time curve and the AU combination temporal matrix;

[0051] The preprocessing module is used to filter the AU intensity-time curve and AU combination time series matrix using activation thresholds and filtering conditions to obtain the filtered AU intensity-time curve and AU combination time series matrix.

[0052] The multimodal fusion module is used to process the user's speech and the filtered AU intensity-time curve and AU combination temporal matrix through the LLM semantic alignment module to obtain multimodal fusion features;

[0053] The composite target expression generation module is used to input the multimodal fusion features into the LLM-driven AU-emotion dynamic causal reasoning library, and determine the composite target expression of the target user through causal reasoning. The composite target expression includes basic emotion and / or transitional composite emotion and intensity.

[0054] A third aspect of this application proposes a storage medium storing computer instructions, which include execution logic for core modules such as multimodal data processing, dynamic AU feature extraction, LLM inference library invocation, and cross-modal semantic alignment, ensuring that each step runs in a timely and coordinated manner, and enabling the computer to execute the methods described above.

[0055] The beneficial effects of this application are as follows:

[0056] The composite emotion recognition method based on temporal feature fusion and LLM-driven approach provided in this application determines the intensity value of each facial action unit (AU) by calculating the change features of node pairs and the movement speed features of key points. This enables more accurate capture of the dynamic changes in facial movements, thereby effectively reducing the interference of factors such as illumination changes, image noise, and dynamic occlusion on AU recognition. By generating AU intensity-time curves and AU combination temporal matrices through a spatiotemporal convolutional network and combining them with an LLM semantic alignment module to process user speech, deep fusion of multimodal data is achieved. This method not only considers the dynamic evolution features of AUs, such as activation order, intensity changes, and duration, but also integrates scene semantics and individual user differences. It can accurately identify basic emotions and transitional composite emotions and their intensity, thereby solving the problem of difficult composite emotion parsing in the prior art. This allows for more accurate identification and differentiation of complex emotional states (such as fear-surprise, sadness-anger, and other composite emotions).

[0057] Furthermore, this invention is based on an LLM-driven AU-emotion dynamic causal inference library. Through causal inference, it can output the correlation logic of "AU feature-emotion" and determine the AU weight ratio threshold for different scenarios through "scenario-based customization + dynamic adaptation": For target scenarios and groups, a scenario-specific dataset containing facial AU data and emotion labels is collected, and core AUs that have a significant impact on emotions are selected; by verifying the recognition accuracy of different candidate thresholds, the inflection point of accuracy is taken as the initial threshold; the AU weight is dynamically updated online using the analytic hierarchy process, and the threshold is fine-tuned based on individual expression habits and environmental interference learned by the sliding window, solving the problem of poor adaptability of traditional fixed thresholds and improving the accuracy of scenario-based emotion recognition. Attached Figure Description

[0058] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram illustrating the application scenario of the composite emotion recognition method based on temporal feature fusion and LLM-driven approach proposed in this application.

[0061] Figure 2 This is a flowchart of the composite emotion recognition method based on temporal feature fusion and LLM-driven approach proposed in this application;

[0062] Figure 3 This is a schematic diagram of the composite emotion recognition device based on temporal feature fusion and LLM-driven technology proposed in this application. Detailed Implementation

[0063] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0064] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0065] In related technologies, a facial action unit (AU) is a unit composed of the movement of one or more facial muscles. It can be used to represent subtle changes in facial movements, and different changes in movements can constitute different facial expressions. Analysis methods based on facial action units are the mainstream technical approach in the field of expression recognition. This method objectively and quantitatively describes expressions by detecting the activity of specific facial muscle groups, and then infers underlying emotions.

[0066] For example, when a face image shows eyebrows pulled down (AU4), upper eyelids raised (AU5), eyelids tightened (AU7), and lips pursed (AU23), these four facial motion units can be combined to form an angry expression. When a face image shows inner eyebrows raised (AU1), corners of the mouth turned up (AU12), and jaw drooping (AU26), these three facial motion units can be combined to form a "surprised" expression.

[0067] Current facial motion unit (FMU) recognition systems generally lack robustness and struggle to handle complex real-world scenarios. Furthermore, their mapping between FMUs and emotions is too rigid and static, failing to effectively address ambiguous and complex emotions. Existing technologies largely rely on predefined, fixed FMU-emotion combination rules (such as Ekman's six basic emotion models), neglecting crucial dynamic information such as FMU intensity and duration, resulting in only coarse emotion categories that cannot meet the demands of refined analysis.

[0068] This application provides specific implementation examples of composite emotion recognition based on temporal feature fusion and LLM-driven methods, applied to, for example... Figure 1 In the application environment shown, terminal 10 and server 20 communicate via a network or line. Terminal 10 acquires an image of a face to be recognized and transmits it to server 20. Server 20 analyzes the image to obtain a composite target expression and stores it on server 20 or transmits it back to terminal 10. Terminal 10 can be a smartphone, tablet, laptop, desktop computer, smart home appliance, vehicle terminal, aircraft, or wearable device terminal with a camera. Server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0069] The first aspect of this application provides a composite emotion recognition method based on temporal feature fusion and LLM-driven approach, the process of which is as follows: Figure 2 As shown, the steps include:

[0070] S101. Acquire multiple frames of the target user's face image to be identified, a calm frame face image, and synchronized user voice. The face image to be identified includes a first facial action unit (AU).

[0071] S102. Based on the calm frame face image, calculate the node pair change features and key point movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified;

[0072] S103. Determine the intensity value of each first facial motion unit (AU) based on the node pair change feature and the key point movement speed feature;

[0073] S104. Input the intensity value of each of the first facial action units (AU) into the spatiotemporal convolutional network to obtain the AU intensity-time curve and the AU combination temporal matrix;

[0074] S105. The AU intensity-time curve and AU combination time series matrix are filtered using activation threshold and filtering conditions to obtain the filtered AU intensity-time curve and AU combination time series matrix.

[0075] S106. The user's speech and the filtered AU intensity-time curve and AU combination temporal matrix are processed by the LLM semantic alignment module to obtain multimodal fusion features;

[0076] S107. Input the multimodal fusion features into the LLM-driven AU-emotional dynamic causal inference library to determine the composite target expression of the target user. The composite target expression includes basic emotion and / or transitional composite emotion and intensity.

[0077] The following will provide a detailed description of the composite emotion recognition based on temporal feature fusion and LLM-driven approach disclosed in the first aspect of this application. In this application, the first facial action unit refers to a first set containing at least one facial action unit, and the second facial action unit refers to a second set containing at least one facial action unit.

[0078] In step S101, multiple frames of the target user's face image to be identified, a calm frame face image, and synchronized user voice are acquired. The face image to be identified includes a first facial action unit (AU).

[0079] Acquiring multiple frames of the target user's face image to be identified can be done by capturing the target user's face images in real time over a continuous period of time, or by acquiring a video containing the target user's face images over a continuous period of time and dividing it into multiple consecutive frames of face images. Each face image to be identified includes at least one facial motion unit (FMU), thus providing a basis for recognition. This step involves acquiring multiple consecutive frames of facial motion units, which provide analytical data for subsequent steps to analyze aspects such as the intensity and duration of the facial motion units.

[0080] A calm frame face image refers to a baseline face image captured when the target user is relaxed and in a state without any emotion or muscle activity. Ideally, no facial action units (AUs) should appear in a calm frame face image. However, considering that the target user's face will have static features in a relaxed state, a weak intensity of facial action units may still appear in the calm frame face image. For such weak AUs, this embodiment first calculates the average intensity of the AU within the calm frames (taking the average of 5-8 consecutive calm frames) and uses it as the "individual baseline value" of the user's AU. The AU intensity of subsequent face images to be identified needs to be subtracted from this baseline value before participating in the calculation, to avoid individual static features interfering with the identification of emotion-related AUs.

[0081] Calm frame face images serve as a baseline for the target user's facial expressions. Obtaining calm frame face images can be used to analyze the differences between the face image to be identified and the calm frame face images, and to measure the intensity of facial action units contained in the face image to be identified.

[0082] To further enhance the accuracy of emotion recognition, this implementation can also synchronize user voice, which contains semantic cues related to emotions. Specifically, AU4 (frowning) can correspond to both anger and sadness; if "sadness" appears in the voice, the weight of sadness can be increased. If "surprise followed by happiness" appears in the voice, "surprise" or similar complex emotions can be prioritized for recognition.

[0083] In step S102, the node pair change features and key point movement speed features of the first facial action unit corresponding to each frame of the face image to be identified are calculated based on the calm frame face image.

[0084] Specifically, to accurately capture and quantify changes in the target user's facial expressions, calm frame face images are used as a reference benchmark. Calm frame face images record the user's facial state when there is no emotional expression, providing a stable reference point for subsequent analysis. Based on calm frames, detailed feature calculations can be performed on each frame of the face image to be identified.

[0085] Specifically, two key features of the first facial action unit (AU) in each frame of the face image to be identified are calculated: node pair change feature and keypoint movement velocity feature. The node pair change feature is the difference in node distance between key node pairs in the current frame and a calm frame, used to reflect the specific change in the facial action unit. The keypoint movement velocity feature is the rate of change of keypoint displacement in adjacent frames, used to reflect the movement rate of the facial action unit, the relative displacement change between specific nodes (such as the inner corner of the eyebrow, the corner of the mouth, etc.). These changes reflect the degree of facial muscle activity and are the basis for recognizing facial action units. For example, when a user frowns, the distance between the nodes at the inner corner of the eyebrow decreases, and this change can be accurately identified.

[0086] Simultaneously, keypoint velocity features are calculated, which are the speed at which these facial keypoints move between consecutive frames. Velocity features can capture dynamic changes in facial movements, helping to distinguish between rapid facial movements (such as blinking) and slow emotional expressions (such as a gradually unfolding smile). For example, the speed at which the corners of the mouth turn up can reflect the naturalness of the smile; a rapid upward turn may indicate a sudden joy, while a slow upward turn may be a gentle emotional expression.

[0087] Optionally, in some embodiments, calculating the node pair change feature and keypoint movement speed feature of the first facial action unit (AU) corresponding to each frame of the face image to be identified includes: determining the first key node pair of the face image to be identified based on the first facial action unit (AU); obtaining keypoint information of the face image to be identified and calculating the node distance of the first key node pair; obtaining the node distance of the first key node pair in the calm frame face image; calculating the difference between the node distances of the face image to be identified and the calm frame face image to obtain the node pair change feature; and determining the movement speed of the keypoints based on the keypoint information of two adjacent frames of the face image to be identified to obtain the keypoint movement speed feature.

[0088] Specifically, a key node pair refers to one or more pairs of nodes used to measure the intensity of a specific facial motion unit. For example, the key node pair corresponding to the facial motion unit "upturned corner of the mouth" (AU12) is "left corner of the mouth node (node ​​61 based on MediaPipe 478 facial keypoints) and left cheekbone node (node ​​13)". The intensity of the facial motion unit "upturned corner of the mouth" (AU12) is reflected by the node distance between the left corner of the mouth node and the left cheekbone node. For a given facial motion unit, its key node pair is preset.

[0089] For a face image to be identified, determining its facial action units (FAUs) is crucial. It's understood that one FAU corresponds to one key node pair. Therefore, for a face image to be identified, the first key node pair is determined through its facial action units. The first key node pair refers to a set that includes at least one key node pair.

[0090] Key point information of multiple frames of a face image to be recognized is acquired to determine key node pairs in the face image. Key node pairs are key locations when facial action units are activated. Specifically, the key point information may include, for example, 478 key points of the face image to be recognized. Key points corresponding to nodes in the first key node pair are found from these key points. The node distance of the first key node pair in each frame is calculated by using the distances between the corresponding key points in each frame.

[0091] Obtain key point information from the calm frame face image, find the key points corresponding to the nodes of the first key node pair, and calculate the node distance of the first key node pair in the calm frame face image.

[0092] The difference between the node distance of the first key node pair in multiple frames of the face image to be identified and the node distance of the first key node pair in the calm frame face image is calculated to obtain the node pair change feature. The movement velocity of key points in two adjacent frames of the multiple frames of the face image to be identified is calculated to obtain the key point movement velocity feature.

[0093] In step S103, the intensity value of each first facial action unit (AU) is determined based on the node pair change feature and the key point movement speed feature.

[0094] Specifically, to quantify the activity level of each first facial action unit (AU), the intensity value can be determined based on the node pair change characteristics and key point movement velocity characteristics obtained in step S103. Specifically, the node pair change characteristics reflect the relative displacement between key facial nodes, and the change in this relative displacement can intuitively demonstrate the amplitude of facial muscle activity. For example, when a person changes from a calm state to a smiling state, the distance between the corner of the mouth and the key cheek nodes increases significantly; this change in distance can serve as an important indicator for measuring the intensity of the smile.

[0095] Meanwhile, keypoint velocity features capture the dynamic changes of facial keypoints between consecutive frames, i.e., the speed at which keypoints move. This speed reflects the rate and duration of facial movements. For example, in rapid blinking (AU43), the speed of eye keypoint movement is much higher than in a slow smile. By combining the changes in node pairs with the speed of keypoint movement, the activity intensity of each AU can be assessed more comprehensively. The intensity value can be calculated using specific algorithms or models to quantify the activation level of the AU, thus providing a quantitative indicator for subsequent emotion recognition and analysis.

[0096] Optionally, in some embodiments, determining the intensity value of each first facial action unit (AU) based on the node pair change features and the keypoint movement speed features includes: inputting the node pair change features and the keypoint movement speed features into a support vector regression model to obtain the intensity value of the first facial action unit (AU) in each frame of the face image to be identified.

[0097] Specifically, to accurately determine the intensity value of each first facial action unit (AU), a support vector regression model from machine learning can be used as the machine learning model. Specifically, node pair change features and keypoint movement speed features are used as input data and fed into the support vector regression model. The support vector regression model can, for example, use a radial basis function (RBF) as the kernel function, and optimize the penalty coefficient, kernel function parameters, and error tolerance through 5-fold cross-validation.

[0098] For example, based on the first facial action unit in the image of the face to be identified, the support vector regression model corresponding to the first facial action unit is invoked, and the change feature and key point movement speed feature are input into the support vector regression model. The support vector regression model outputs the single-frame intensity values ​​of these two features, which are represented by continuous values ​​from 0 to 1, where 0 represents no action and 1 represents maximum intensity, corresponding to levels A to E in the FACS standard. The mean square error of the single-frame intensity value estimation does not exceed 0.071.

[0099] In step S104, the intensity value of each first facial action unit is input into the spatiotemporal convolutional network to obtain the AU intensity-time curve and the AU combined temporal matrix.

[0100] Specifically, the intensity value of each first facial action unit (AU) is input into a spatiotemporal convolutional network. Spatiotemporal convolutional networks are deep learning models used to process data with spatiotemporal dependencies, capable of simultaneously capturing feature changes in both spatial and temporal dimensions. Specifically, this spatiotemporal convolutional network extracts local features of each AU in the spatial dimension through convolution operations, and captures the changes of these features over time through convolution operations in the temporal dimension. This spatiotemporal combined processing approach can effectively identify and analyze the dynamic behavior of AUs, such as the activation order, duration, and intensity changes of AUs.

[0101] Through spatiotemporal convolutional network processing, the system can generate two important outputs: AU intensity-time curves and AU combination temporal matrices. The AU intensity-time curve is a continuous graph that shows the intensity changes of each AU at different time points. This curve can intuitively reflect the activation process of AUs; for example, an AU may gradually increase in the early stages of emotional expression, reach a peak, and then gradually decrease.

[0102] In one scenario, a Spatiotemporal Convolutional Network (STCN) can be configured with, for example, three spatiotemporal convolutional layers and a kernel size of 3×3. The single-frame intensity values ​​output from the Support Vector Regression model are input into the STCN, which outputs an AU intensity-time curve and an AU combined temporal matrix. The AU intensity-time curve records the intensity values ​​of each facial motion unit in frame order and can be smoothed for noise using a 5-frame sliding window averaging method.

[0103] Optionally, in some embodiments, the AU intensity-time curve is the intensity value of multiple first facial action units (AUs) arranged in frame order, and the AU combination time sequence matrix is ​​a matrix of total number of frames × number of first facial action units (AUs), where each element in the matrix represents the intensity value of a certain first facial action unit (AU) in the corresponding frame.

[0104] Specifically, the AU intensity-time curve is a graph formed by arranging the intensity values ​​of multiple first facial action units (AUs) in frame order. Specifically, the intensity value of each first facial action unit (AU) in each frame of the face image to be recognized is formed into a time series. This time series reflects the dynamic changes of each AU in consecutive frames, visually demonstrating how the activation intensity of each AU evolves over time. For example, when a person gradually transitions from a calm state to a smiling state, the intensity value of the AU with the corners of the mouth turned up (such as AU12) will gradually increase over time; this change can be clearly seen on the intensity-time curve.

[0105] The AU combination temporal matrix is ​​a two-dimensional matrix, with dimensions equal to the total number of frames multiplied by the number of first facial action units (AUs). Each row in the matrix corresponds to a frame of the face to be identified, and each column corresponds to a specific first facial action unit (AU). Each element in the matrix represents the intensity value of a particular first facial action unit (AU) in the corresponding frame. Using this two-dimensional matrix, the intensity changes of multiple AUs across all frames can be observed simultaneously, as well as the relationships between them. For example, if a row in the matrix shows that the intensity values ​​of AU1 (inner brow lift) and AU4 (frowning) increase simultaneously, this may indicate that the person in that frame is in a state of anxiety or worry.

[0106] In step S105, the AU intensity-time curve and AU combined time series matrix are filtered using activation threshold and filtering conditions to obtain the filtered AU intensity-time curve and AU combined time series matrix.

[0107] Specifically, in order to extract valuable information for emotion recognition from the first facial action unit (AU) data, filtering can be performed. Specifically, an activation threshold can be set to filter AU signals that are too weak and may not have a substantial impact on emotion judgment. Only when the intensity of an AU exceeds the activation threshold will it be considered a valid signal.

[0108] Furthermore, the AU intensity-time curve and AU combination time series matrix can be further filtered based on preset filtering criteria. Filtering criteria can include various factors such as the duration of an AU, its frequency of occurrence, and its combination patterns with other AUs. For example, some AUs may only have a clear indicative effect on emotions under specific time series combinations, or only when a certain AU appears continuously for a certain period of time does it indicate that the user is in a specific emotional state.

[0109] Understandably, if the user corresponding to the face image to be identified is an existing user in the database, then the threshold and filtering conditions corresponding to that user are retrieved from the database in this step; if the user corresponding to the face image to be identified is a new user not included in the database, then the activation threshold and filtering conditions corresponding to that new user are constructed in this step.

[0110] Optionally, in one embodiment, the construction of the activation threshold includes: acquiring multiple test images of the target user, the test images including a face test image in a calm state, the face test image including a second facial motion unit;

[0111] Determine the intensity value of each second facial motion unit; calculate the average intensity value and standard deviation of all second facial motion units based on the intensity value; construct the activation threshold based on the average intensity value, standard deviation of intensity value, and a preset noise tolerance coefficient.

[0112] Specifically, in the process of constructing the activation threshold, multiple test images of the target user can be acquired, including a face test image of the user in a calm state. The face test image includes a second facial action unit (AU), which is typically inactive or in a low-activity state in the calm state.

[0113] Each second facial motion unit (AU) in each frame of the test image is analyzed to determine its intensity value. The intensity value reflects the activity level of each AU in a resting state. By calculating the intensity values ​​of all second facial motion units (AUs), an average intensity value can be obtained, which represents the typical activity level of the AUs in a resting state. Simultaneously, the standard deviation of these intensity values ​​is calculated to reflect the distribution range and fluctuation of the intensity values, helping the system understand the natural variability of AU intensity in a resting state.

[0114] The activation threshold is constructed based on the calculated average intensity value, the standard deviation of the intensity value, and a preset noise tolerance coefficient. The noise tolerance coefficient is an important parameter that allows for a certain degree of tolerance for minor fluctuations in a calm state, avoiding misinterpretation of normal fluctuations as emotional expressions. The activation threshold is obtained by combining the average intensity value and the standard deviation and multiplying it by the noise tolerance coefficient. The activation threshold can distinguish between minor AU activity in a calm state and significant AU activity during emotional expressions.

[0115] Optionally, in some embodiments, the construction of the filtering conditions includes: the intensity value of the first facial action unit (AU) for a consecutive preset number of frames is not lower than the activation threshold and the fluctuation range of the first facial action unit (AU) for a consecutive preset number of frames is less than a preset value.

[0116] Specifically, the selection criterion can be that the intensity value of a certain first facial action unit (AU) is not lower than the previously mentioned activation threshold within a consecutive preset number of frames. That is, only when an AU maintains a high activation intensity continuously within a certain time range will it be considered a valid signal. For example, if the preset number of frames is 5 frames, the intensity value of the AU must exceed the activation threshold within these 5 frames, indicating that the activation of the AU has a certain degree of persistence and stability.

[0117] Furthermore, the filtering criteria can also be to apply "activation threshold + stability threshold" to filter temporal features, remove noise sequences where the intensity of a single frame exceeds the activation threshold but consecutive frames do not meet the stability threshold, and retain stable sequences of 3 to 5 consecutive frames that meet the dual threshold constraints.

[0118] In step S106, the user speech and the filtered AU intensity-time curve and AU combination temporal matrix are processed by the LLM semantic alignment module to obtain multimodal fusion features.

[0119] In this step, the semantic information in the user's speech is fused and aligned with the filtered facial action unit (AU) intensity-time curves and AU combination temporal matrices to obtain multimodal fusion features. Specifically, by performing speech recognition on the user's speech and converting it into text data, the semantic understanding capabilities of LLM (Large Language Model) can accurately extract the emotional tendency, semantic content, and subtle emotion-related features from the text. Simultaneously, the filtered AU intensity-time curves and AU combination temporal matrices can represent the dynamic trajectory of facial expressions. The LLM semantic alignment module can accurately align the semantic features in the speech with the temporal features of the AUs in the time dimension, ensuring consistency in emotional expression. For example, when the speech contains a certain emotion, the corresponding AU change can be matched, thereby achieving collaborative analysis of speech and facial expressions.

[0120] Optionally, in one embodiment, the step of processing the user's speech and the filtered AU intensity-time curve and AU combination time-series matrix through the LLM semantic alignment module to obtain multimodal fusion features includes: extracting speech features from the user's speech and performing semantic understanding on the speech features to obtain speech semantic features; inputting the speech semantic features and the filtered AU intensity-time curve and AU combination time-series matrix into the LLM semantic alignment module, and calculating the association weight between the speech semantic features and the visual time-series features based on a cross-modal attention mechanism; and dynamically weighting and fusing the speech semantic features with the filtered AU intensity-time curve and AU combination time-series matrix according to the association weight to obtain the multimodal fusion features.

[0121] Specifically, user speech can be analyzed to extract its speech features. These speech features can include information such as pitch, speech rate, and rhythm, which can reflect the speaker's emotional state. Then, semantic understanding is performed on these speech features to transform them into speech semantic features that can express specific meanings.

[0122] The obtained speech semantic features, along with the filtered AU intensity-time curve and AU combined temporal matrix, are input into the LLM semantic alignment module. In the LLM semantic alignment module, a cross-modal attention mechanism can be used to calculate the association weights between the speech semantic features and the AU intensity-time curve and AU combined temporal matrix (i.e., visual temporal features). The cross-modal attention mechanism can automatically evaluate the importance of different modal features in the current context. For example, when speech and facial expressions jointly express strong emotions, higher weights are given; while when emotional expression is more subtle, the weight allocation can be adjusted according to the specific situation.

[0123] Based on the calculated association weights, the speech semantic features are dynamically weighted and fused with the filtered AU intensity-time curves and AU combined time-series matrix. This will be explained in detail below with specific examples.

[0124] In step S107, the multimodal fusion features are input into the LLM-driven AU-emotion dynamic causal reasoning library, and the composite target expression of the target user is determined through causal reasoning. The composite target expression includes basic emotion and / or transitional composite emotion and intensity.

[0125] In step S107, the multimodal fusion features are input into the AU-emotion dynamic causal inference library driven by the large language model (LLM). This inference library is based on the fused speech and facial action unit (AU) features and accurately determines the composite target expression of the target user through a complex dynamic causal inference process.

[0126] Complex target expressions can include basic emotions (such as happiness, sadness, anger, and other basic emotion categories), as well as transitional complex emotions (such as the transition from sadness to anger, or a mixture of fear and surprise). Furthermore, the inference library can further quantify the intensity of emotions, that is, assess the strength of a user's emotions, thus providing more refined analytical results for emotion recognition.

[0127] Optionally, in one embodiment, constructing the LLM-driven AU-emotion dynamic causal reasoning library includes: constructing an initial reasoning knowledge base based on the labeled dataset of the first facial action unit (AU), emotion labels, and causal reasoning samples containing the correspondence between AU activation time sequence and emotion evolution; using the emotion recognition accuracy and the reasonableness of the interpretable text output by the initial reasoning library as joint reward signals, training the initial reasoning knowledge base through a policy gradient optimization algorithm, and incorporating first facial action unit (AU) and emotion association data from multiple scenarios during the training process.

[0128] Specifically, the initial inference knowledge base can be based on a labeled dataset of first facial action units (AUs), which records the performance of various AUs under different emotional states. Simultaneously, combined with emotion labels, it indicates the emotion category corresponding to each AU combination. The initial inference knowledge base also includes causal inference samples, which record the temporal relationships of AU activation and the evolution of emotions, helping to establish a dynamic causal link between AUs and emotions.

[0129] After the initial inference knowledge base is built, its performance can be optimized through training. During training, the accuracy of emotion recognition and the interpretability of the inference results can be used as joint evaluation criteria. The constructed LLM-driven AU - Emotion Dynamic Causal Inference Library should not only have high accuracy in emotion recognition, but also ensure that the inference process can output reasonable and clear explanatory text, which is easy to understand and apply. For example, a policy gradient optimization algorithm can be used to train the initial inference knowledge base, and by continuously adjusting the parameters, its performance in complex emotion recognition tasks can be improved.

[0130] Furthermore, to enhance the generalization and adaptability of the LLM-driven AU-emotion dynamic causal inference library, training can incorporate AU-emotion correlation data from various scenarios. This data could include, for example, human-computer interaction, mental health assessments, and safe driving. Through training with diverse data, the LLM-driven AU-emotion dynamic causal inference library can learn the complex relationships between AUs and emotions in different scenarios, thus achieving higher accuracy and reliability in practical applications.

[0131] Specifically, based on the analytical logic of AU combinations, core combinations that can accurately reflect specific emotions can be selected.

[0132] For example, the core AU combination for the emotion of "anger" is AU4 (frowning), AU5 (brows down), AU17 (chin tucked in), AU23 (lips pulled tight), and AU24 (teeth clenched). The synergistic activation of these AUs can clearly indicate the presence of anger.

[0133] In practical applications, the emotion of "anger" can be represented by the core facial movement unit (AU) using the following formula:

[0134] Anger = AU4+AU5+AU17+AU23+AU24;

[0135] The above formula can be used to represent the core AU combination corresponding to the emotion of "anger".

[0136] As for the emotion of "surprise", its core AU is AU25 (eyes wide open), and AU25 can be used as the key marker of the emotion of "surprise".

[0137] For example, for the emotion of "anger," the core AU combination is AU4 (frowning), AU5 (brows down), AU17 (chin tucked in), AU23 (lips pulled tight), and AU24 (teeth clenched) (which conforms to the typical muscle activation pattern of anger in the FACS standard). The synergistic activation of these AUs can clearly indicate the presence of anger. In practical applications, the emotion of "anger" can be represented by the core facial movement units (AUs) using the following formula:

[0138] Anger = AU4+AU5+AU17+AU23+AU24;

[0139] The above formula clearly defines the core AU combination corresponding to the emotion of "anger." Its core dynamic characteristics are: first, AU4 is activated (its intensity gradually increases to a high level) → AU5 is activated simultaneously (its intensity subsequently increases to a moderate to high level) → then AU23 (tightening of the lips) and AU24 (clenching of the teeth) are activated in sequence (both reaching moderate or higher intensity), while AU17 (tightening of the jaw) maintains a stable intensity. The LLM inference library can identify through causal reasoning: "The initial activation of AU4 + AU5 establishes the baseline muscle tension state of anger, and the subsequent synergistic activation of AU23 + AU24 further enhances the intensity of the emotion."

[0140] For the emotion of "surprise," the core AU combination is AU2 (outer eyebrow lift), AU5 (upper eyelid lift), and AU25 (lip separation) (which conforms to the typical muscle activation pattern of surprise in the FACS standard). The sequential activation of these AUs is a key indicator of the emotion of "surprise." In practical application, the core AU combination of the emotion of "surprise" can be represented as:

[0141] Surprise = AU2 + AU5 + AU25;

[0142] The above formula clearly defines the core AU components of the emotion "surprise," whose core dynamic characteristics are: AU2 is activated first (intensity gradually increases) → AU5 is activated simultaneously (intensity rapidly rises to a high level) → AU25 is activated with a delay (lips separate, intensity reaches a medium level). The LLM inference library can output through causal reasoning: "The initial activation of AU2 (outer eyebrow lift) + AU5 (upper eyelid lift) corresponds to the visual expansion response of surprise, and the subsequent activation of AU25 (lips separate) further strengthens the emotional expression. This sequential activation perfectly matches the emotional evolution logic of 'surprise.'"

[0143] Optionally, in one embodiment, the method further includes: obtaining the current application scenario and the population type to which the target user belongs; adjusting the AU-emotion weights in the LLM-driven AU-emotion dynamic causal inference library based on the current application scenario and the population type, wherein the AU-emotion weights include the consideration priority and / or association strength of facial action units (AUs).

[0144] Specifically, information about the current application scenario is obtained, such as whether it is used for human-computer interaction, mental health assessment, or safe driving. Simultaneously, the demographic type of the target user can be obtained, such as age, gender, and occupation, as different groups may exhibit differences in emotional expression. Based on the current application scenario and the demographic type of the target user, the AU-emotion weights in the LLM-driven AU-emotion dynamic causal inference library are adjusted.

[0145] Specifically, after determining the core combination as described above, a ranking-aware strategy can be further introduced to dynamically adjust the AU-emotion weights. The core of this strategy lies in ranking and adjusting the contribution of AUs to emotion expression based on different application scenarios and the complexity of emotion expression.

[0146] For example, in online education scenarios, to identify complex emotions like "sadness-anger," the first step is to rank the contribution of each Active Character (AU) to the emotion. Analysis reveals that AU15 (drooping corners of the mouth) contributes significantly to "sadness," while AU4 (frowning) contributes significantly to "anger." Based on this ranking, the weights can be dynamically adjusted. For instance, the weight of AU15 for "sadness" can be increased from 0.3 to 0.6 to enhance sensitivity to sadness; simultaneously, the weight of AU4 for "anger" can be decreased from 0.5 to 0.3 to avoid misjudging the intensity of anger due to overactivation of AU4.

[0147] For example, in driving scenarios, priority can be given to the emotion of "surprise", and the weight of the core facial action unit AU25 (mouth opening) for the emotion of "surprise" can be increased; in social scenarios, priority can be given to the emotion of "anxiety", and the weight of the core facial action unit AU1 (inner eyebrow raised) for the emotion of "anxiety" can be increased.

[0148] For example, for complex emotions, the weights of facial motion units can be dynamically adjusted based on their activation sequence. When facial motion units representing different emotions appear sequentially, the weight of the first facial motion unit can be increased to improve the accuracy of emotion recognition. For example, "activating AU15 (sadness) first → activating AU4 (anger) later" increases the weight of AU15 for sadness and decreases the weight of AU4 for anger.

[0149] The priority of facial action units (AUs) can be set according to the current application scenario and user group. For example, in a safe driving scenario, AUs related to surprise (such as AU25) can be given a higher priority; while in a social scenario, AUs related to anxiety (such as AU1, raised inner eyebrows) can be given a higher priority, so as to adjust the attention given to different AUs during the reasoning process according to the priority.

[0150] The strength of the association between AU and emotion can vary among different populations when expressing the same emotion. For example, the intensity of AU12 (upturned corners of the mouth) may differ between adults and children when expressing happiness. The strength of the association between AU and emotion can be adjusted according to population type to more accurately reflect the emotional expression characteristics of different groups.

[0151] Furthermore, to determine the weighting of AUs in different scenarios, the following methods can be used to dynamically adjust the AU weights to adapt to different scenarios and individual differences. For example, in a question-and-answer scenario, the focus is on the impact of the user's emotional state on answering questions. For instance, users may exhibit emotions such as nervousness, confidence, or confusion when answering questions. In this scenario, voice and facial expressions are key features because they directly reflect the user's emotional state. Voice characteristics such as speech rate, tone, and pauses can reflect the user's level of nervousness. For example, a faster speech rate may indicate nervousness or anxiety, while a slower speech rate may indicate confidence or deep thought. Facial action units (AUs), such as AU4 (frowning) and AU12 (upturned corners of the mouth), can reflect the user's confusion or confidence. For example, a frown may indicate confusion, while an upturned corner of the mouth may indicate confidence.

[0152] In this scenario, a differentiable decision tree model can be used to dynamically adjust the weights: the initial weight allocation can be 0.6 for the speech modality and 0.4 for the facial expression modality. The differentiable decision tree model analyzes the emotional features in the current question-and-answer scenario. If speech features (such as increased speech rate) are found to contribute more to emotion recognition, the weight of the speech modality is increased to 0.7, and the weight of the facial expression modality is decreased to 0.3. Conversely, if facial expression features (such as frequent frowning) contribute more to emotion recognition, the weight of the facial expression modality is increased to 0.5, and the weight of the speech modality is decreased to 0.5.

[0153] It is understood that the above description of the AU-emotion weight adjustment is merely exemplary, and those skilled in the art can select and set the AU-emotion weight according to actual needs, as long as the technical principles of this application can be achieved.

[0154] In summary, the composite emotion recognition method 100 based on temporal feature fusion and LLM-driven approach provided in this application determines the intensity value of each facial action unit (AU) by calculating the change features of node pairs and the movement speed features of key points. This enables more accurate capture of the dynamic changes in facial movements, thereby effectively reducing the interference of factors such as illumination changes, image noise, and dynamic occlusion on AU recognition. By generating AU intensity-time curves and AU combination temporal matrices through a spatiotemporal convolutional network and combining them with an LLM semantic alignment module to process user speech, deep fusion of multimodal data is achieved. This method not only considers the dynamic evolution features of AUs, such as activation order, intensity changes, and duration, but also integrates scene semantics and individual user differences. It can accurately identify basic emotions and transitional composite emotions and their intensity, thereby solving the problem of difficult composite emotion parsing in the prior art. This allows for more accurate identification and differentiation of complex emotional states (such as fear-surprise, sadness-anger, and other composite emotions).

[0155] Furthermore, this invention is based on an LLM-driven AU-emotion dynamic causal inference library. Through causal inference, it can output the correlation logic of "AU feature-emotion" and determine the AU weight ratio threshold for different scenarios through "scenario-based customization + dynamic adaptation": For target scenarios and groups, a scenario-specific dataset containing facial AU data and emotion labels is collected, and core AUs that have a significant impact on emotions are selected; by verifying the recognition accuracy of different candidate thresholds, the inflection point of accuracy is taken as the initial threshold; the AU weight is dynamically updated online using the analytic hierarchy process, and the threshold is fine-tuned based on individual expression habits and environmental interference learned by the sliding window, solving the problem of poor adaptability of traditional fixed thresholds and improving the accuracy of scenario-based emotion recognition.

[0156] The second aspect of this application provides a composite emotion recognition device based on temporal feature fusion and LLM-driven architecture. Figure 3 This is a schematic diagram of the composite emotion recognition device based on temporal feature fusion and LLM-driven architecture proposed in this application. Figure 3 As shown, the device divides each function into functional modules, including:

[0157] The acquisition module 201 is used to acquire multiple frames of face images to be identified, calm frame face images, and synchronized user voice of the target user. The face images to be identified include a first facial action unit (AU).

[0158] The calculation module 202 is used to calculate the node pair change features and key point movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified based on the calm frame face image;

[0159] The determination module 203 is used to determine the intensity value of each first facial action unit (AU) based on the node pair change feature and the key point movement speed feature;

[0160] The input module 204 is used to input the intensity value of each of the first facial action units (AU) into the spatiotemporal convolutional network to obtain the AU intensity-time curve and the AU combination temporal matrix;

[0161] Preprocessing module 205 is used to filter the AU intensity-time curve and AU combination time series matrix using activation threshold and filtering conditions to obtain the filtered AU intensity-time curve and AU combination time series matrix;

[0162] The multimodal fusion module 206 is used to process the user speech and the filtered AU intensity-time curve and AU combination temporal matrix through the LLM semantic alignment module to obtain multimodal fusion features;

[0163] The composite target expression generation module 207 is used to input the multimodal fusion features into the LLM-driven AU-emotion dynamic causal reasoning library, and determine the composite target expression of the target user through causal reasoning. The composite target expression includes basic emotion and / or transitional composite emotion and intensity.

[0164] The composite emotion recognition device based on temporal feature fusion and LLM-driven method provided in this application includes modules whose functions correspond one-to-one with the steps of the composite emotion recognition method based on temporal feature fusion and LLM-driven method. For a detailed explanation of the composite emotion recognition device based on temporal feature fusion and LLM-driven method, as well as related refinements and optimizations, please refer to the specific embodiments of the method described above, which will not be repeated here.

[0165] In some embodiments, this application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. The computer program includes the execution logic of core modules such as multimodal data processing, dynamic AU feature extraction, LLM inference library calling, and cross-modal semantic alignment, ensuring that each step runs in a timely and coordinated manner. When the computer program is executed by a processor, it can implement the above method steps. For specific implementation processes, please refer to the above embodiments, which will not be repeated here.

[0166] The preferred embodiments disclosed above are merely illustrative of this disclosure. These preferred embodiments do not exhaustively describe all details, nor do they limit the disclosure to specific implementation methods. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A composite emotion recognition method based on temporal feature fusion and LLM-driven approach, characterized in that, include: The system acquires multiple frames of the target user's face image to be identified, a calm frame face image, and synchronized user voice. The face image to be identified includes a first facial action unit (AU). Based on the calm frame face image, calculate the node pair change features and key point movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified; The intensity value of each first facial action unit (AU) is determined based on the node change characteristics and the key point movement speed characteristics; The intensity value of each first facial action unit (AU) is input into the spatiotemporal convolutional network to obtain the AU intensity-time curve and the AU combination temporal matrix; The AU intensity-time curves and AU combination time series matrices are filtered using activation thresholds and filtering conditions to obtain the filtered AU intensity-time curves and AU combination time series matrices. The user's speech and the filtered AU intensity-time curve and AU combination temporal matrix are processed by the LLM semantic alignment module to obtain multimodal fusion features; The multimodal fusion features are input into the LLM-driven AU-emotion dynamic causal reasoning library, and the composite target expression of the target user is determined through causal reasoning. The composite target expression includes basic emotion and / or transitional composite emotion and intensity.

2. The method according to claim 1, characterized in that, The calculation of node pair change features and keypoint movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified includes: The first key node pair of the face image to be identified is determined based on the first facial action unit (AU); Obtain key point information of the face image to be identified, and calculate the node distance of the first key node pair; Obtain the node distance of the first key node pair in the calm frame face image; Calculate the difference between the node distances of the face image to be identified and the calm frame face image to obtain the node pair change feature; Based on the key point information of two adjacent frames of the face image to be identified, the movement speed of the key points is determined, and the key point movement speed feature is obtained.

3. The method according to claim 1, characterized in that, The determination of the intensity value of each first facial action unit (AU) based on the node pair change feature and the keypoint movement speed feature includes: The node pair change features and the key point movement speed features are input into the support vector regression model to obtain the intensity value of the first facial action unit (AU) in each frame of the face image to be identified.

4. The method according to claim 1, characterized in that, The AU intensity-time curve is the intensity value of multiple first facial action units (AUs) arranged in frame order. The AU combination time sequence matrix is ​​a matrix of total number of frames × number of first facial action units (AUs), and each element in the matrix represents the intensity value of a certain first facial action unit (AU) in the corresponding frame.

5. The method according to claim 1, characterized in that, The construction of the activation threshold includes: Acquire multiple test images of the target user, the test images including a face test image in a calm state, the face test image including a second facial motion unit; Determine the intensity value for each of the second facial motion units; Calculate the average intensity value and standard deviation of all second facial motion units based on the intensity values; The activation threshold is constructed based on the average intensity value, the standard deviation of the intensity value, and the preset noise tolerance coefficient; The construction of the filtering criteria includes: The intensity value of the first facial motion unit (AU) for a consecutive preset number of frames is not lower than the activation threshold, and the fluctuation range of the first facial motion unit (AU) for a consecutive preset number of frames is less than the preset value.

6. The method according to claim 1, characterized in that, The construction of the LLM-driven AU-emotion dynamic causal inference library includes: Based on the labeled dataset of the first facial action unit (AU), emotion labels, and causal reasoning samples containing the correspondence between AU activation time sequence and emotion evolution, an initial reasoning knowledge base is constructed. Using the accuracy of emotion recognition and the reasonableness of the interpretable text generated by the initial inference base as joint reward signals, the initial inference knowledge base is trained through a policy gradient optimization algorithm.

7. The method according to claim 6, characterized in that, Also includes: Obtain the current application scenario and the demographic type of the target users; The AU-emotion weights in the LLM-driven AU-emotion dynamic causal inference library are adjusted based on the current application scenario and the population type. The AU-emotion weights include the consideration priority and / or correlation strength of facial action units (AUs).

8. The method according to claim 1, characterized in that, The process of processing the user's speech and the filtered AU intensity-time curve and AU combination temporal matrix through the LLM semantic alignment module yields multimodal fusion features, including: Extract the speech features of the user's speech, and perform semantic understanding on the speech features to obtain speech semantic features; The speech semantic features, the filtered AU intensity-time curves, and the AU combined temporal matrix are input into the LLM semantic alignment module, and the association weights between the speech semantic features and the visual temporal features are calculated based on the cross-modal attention mechanism. The speech semantic features are dynamically weighted and fused with the filtered AU intensity-time curve and AU combination time-series matrix according to the association weights to obtain the multimodal fusion features.

9. A composite emotion recognition device based on temporal feature fusion and LLM-driven architecture, characterized in that, include: The acquisition module is used to acquire multiple frames of face images to be identified, calm frame face images, and synchronized user voice of the target user. The face images to be identified include a first facial action unit (AU). The calculation module is used to calculate the node pair change features and key point movement speed features of the first facial action unit (AU) corresponding to each frame of the face image to be identified based on the calm frame face image; A determination module is used to determine the intensity value of each first facial action unit (AU) based on the node pair change feature and the key point movement speed feature; The input module is used to input the intensity value of each of the first facial action units (AU) into the spatiotemporal convolutional network to obtain the AU intensity-time curve and the AU combination temporal matrix; The preprocessing module is used to filter the AU intensity-time curve and AU combination time series matrix using activation thresholds and filtering conditions to obtain the filtered AU intensity-time curve and AU combination time series matrix. The multimodal fusion module is used to process the user's speech and the filtered AU intensity-time curve and AU combination temporal matrix through the LLM semantic alignment module to obtain multimodal fusion features; The composite target expression generation module is used to input the multimodal fusion features into the LLM-driven AU-emotion dynamic causal reasoning library, and determine the composite target expression of the target user through causal reasoning. The composite target expression includes basic emotion and / or transitional composite emotion and intensity.

10. A storage medium, characterized in that, The computer contains computer instructions for causing the computer to perform the method as described in any one of claims 1-8.

Images