Method for joint emotion recognition of action and micro-expression based on perceptual cross-modal interaction

Abstract

The present application relates to the field of artificial intelligence and computer vision technology, and discloses a method for joint emotion recognition of action and micro-expression based on perception cross-modal interaction, which comprises: acquiring a continuous video stream sequence, extracting an action input sequence and a micro-expression input difference sequence, encoding the input sequence to generate an action feature unit and a micro-expression feature unit, calculating a reliability score to construct an alignment correlation matrix, performing calculation based on the alignment correlation matrix and the reliability score to generate cross-modal interaction features, combining visual condition features, reliability scores and cross-modal conflict features to generate feature fusion weights, performing weighted summation on the feature units and the cross-modal interaction features based on the weights, and outputting a joint decision result. The present application overcomes the defect of modal misalignment by constructing an alignment correlation matrix that fuses reliability bias and dynamically allocating by combining gate weights, thereby improving the inference stability under data degradation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to a method for joint emotion recognition based on action and micro-expression based on perceptual cross-modal interaction. Background Technology

[0002] Emotion recognition relies on extracting feature information from a single modality, which has limitations. The joint inference of dual modalities, combining human action patterns and facial micro-expression features, has become the focus of current research. Action features reflect macro-level behavioral intentions, while micro-expressions contain subtle clues to emotional changes. At present, most cross-modal emotion recognition systems adopt a direct splicing method to fuse multi-source data and rely on deep neural networks to extract joint features.

[0003] Existing technologies face multiple challenges when processing heterogeneous modal data. Human motion features and facial micro-expression features are inconsistent in terms of time scale and duration. Traditional alignment mechanisms are difficult to solve semantic misalignment caused by time frame misalignment. In real-world scenarios, video data is often affected by occlusion and motion blur, leading to a decline in the quality of some modal data. Existing fusion mechanisms lack dynamic evaluation and adaptive filtering for the quality of single-modal data. Once a modal input contains noise, the overall inference system performance deteriorates.

[0004] In complex emotional expression scenarios, there is a semantic conflict between the emotional tendency conveyed by actions and the true emotional state revealed by micro-expressions. Static fixed-weight feature fusion methods cannot handle information contradictions between modalities, cannot adaptively adjust the influence ratio of different features according to environmental conditions, and models lacking conflict perception mechanisms and dynamic fusion weights exhibit poor robustness when dealing with contradictory samples. A novel cross-modal interaction architecture is urgently needed to solve the problems of misalignment, low-quality interference, and semantic conflict. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a joint emotion recognition method based on action and micro-expression based on perceptual cross-modal interaction. This method solves the technical problems of insufficient inference robustness caused by heterogeneous modal time scale misalignment, noise introduced by data degradation, and semantic conflicts between modalities in existing cross-modal emotion recognition methods.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a joint emotion recognition method based on action and micro-expression in perceptual cross-modal interaction, the method comprising the following steps: Acquire a continuous video stream sequence and assign a tracking identifier to the target person; Combine the tracking identifiers to extract motion data and facial data of the same target from the continuous video stream sequence; Spatial reference alignment and time axis extraction are performed on the motion data to construct the motion input sequence; Rigid motion compensation and facial region segmentation are performed on the facial data to construct a micro-expression input difference sequence; The action input sequence and the micro-expression input difference sequence are feature encoded to generate action feature units and micro-expression feature units; The reliability scores of the motion feature unit and the micro-expression feature unit are calculated based on local quality assessment. A cross-modal alignment correlation matrix is ​​constructed by combining time distance constraints, content relevance, and the aforementioned reliability score; Calculations are performed based on the alignment correlation matrix and the reliability score to generate cross-modal interaction features, updated unimodal features, and cross-modal conflict features; The visual conditional features extracted from the video window, the reliability score, and the cross-modal conflict features are combined to generate feature fusion weights; Based on the feature fusion weights, a weighted summation is performed on the updated single-modal features and the cross-modal interaction features to output a joint decision result.

[0007] The aforementioned joint emotion recognition method introduces a reliability score in the feature encoding and fusion stages. By combining temporal distance constraints and content relevance, asynchronous micro-expression and action features are correlated over time. During feature interaction, the reliability score is used as a bias term to reduce the interference of low-quality features on the interaction results. At the same time, feature fusion weights are generated by combining external visual condition features and cross-modal conflict features. When single-modal data is missing or semantic inconsistencies exist between modalities, the feature weight ratio is adjusted to improve the representation results of joint features.

[0008] Furthermore, the steps of acquiring a continuous video stream sequence and assigning a tracking identifier to the target person, and combining the tracking identifier to extract motion data and facial data of the same target from the continuous video stream sequence, include: The algorithm performs human target detection and multi-target tracking, assigning continuous tracking labels across frames to target human figures in the video frame; Extract key human body points and facial image data within the identified target area; The coordinates of the predicted head center are calculated, and the spatial distance deviation is calculated by combining the spatial geometric relationship between the face detection box and the predicted head center coordinates. By combining the tracking identifier and the spatial distance deviation parameter, the action sequence and the micro-expression sequence are bound to the same target instance to obtain the action data and the face data. By calculating the spatial distance deviation parameter, spatial geometric constraints are constructed, which can distinguish individual targets in multi-target or occluded scenarios, and associate the extracted action sequence and micro-expression sequence with the same object.

[0009] Furthermore, the step of performing spatial reference alignment and time axis extraction on the action data to construct the action input sequence includes: The hip center node or the confidence weighting center is selected as the origin of the three-dimensional coordinate system. The overall displacement vector is subtracted and the normalized key point coordinates are generated using the scale-normalized reference parameters. The difference operation is performed along the time axis to extract multi-order kinematic feature parameters. The normalized coordinates, the multi-order kinematic feature parameters, and the node missing mask are concatenated using an execution matrix to generate the action input sequence. The local center is used as the origin, and the overall displacement vector is subtracted to reduce the rigid motion effects caused by camera movement or character translation. The multi-order kinematic feature parameters extracted through differential operations reflect the dynamic changes of the local human body.

[0010] Furthermore, the steps of performing rigid motion compensation and facial region segmentation on the facial data to construct a micro-expression input difference sequence include: Affine transformation matrix is ​​used to eliminate the interference of rigid head motion and obtain a normalized face image sequence; Within a sliding short window, locate the starting phase frame and the peak phase frame of the motion process, and divide the aligned single-frame face into multiple independent facial local blocks; The forward and backward difference matrices of each local facial block are calculated, and the peak phase feature activation tensor and the fused region location encoding are concatenated along the channel dimension to construct the micro-expression input difference sequence. The difference operation is used to reduce the proportion of static facial features and highlight transient change features. Combined with the region location encoding, the spatial topological relationship of each segment of the face can be preserved.

[0011] Furthermore, the step of performing feature encoding on the action input sequence and the micro-expression input difference sequence to generate action feature units and micro-expression feature units includes: The action feature unit is output by aggregating local spatial morphological features and capturing long-term dependencies using spatial graph convolutional layers and temporal network models. Lightweight convolutional layers and region self-attention encoding modules are used to generate the co-variation association weights between various local blocks of the face, and the micro-expression feature units are output. Spatial graph convolutional layers and temporal network models are used to process the spatiotemporal association attributes of action sequences, and the region self-attention encoding module is used to calculate the co-variation relationship between local facial regions, and the initial feature representations of the two modalities are extracted respectively.

[0012] Furthermore, the step of calculating the reliability scores of the action feature unit and the micro-expression feature unit based on local quality assessment includes: The confidence score of key point detection, trajectory continuity and signal-to-noise ratio of local joint motion are aggregated, and the reliability score corresponding to the motion feature unit is generated by mapping through motion multilayer perceptron. The reliability score corresponding to the micro-expression feature unit is generated by combining the visible proportion of the nine-square facial region, the face alignment residual, and the non-rigid motion signal-to-noise ratio through a micro-expression multilayer perceptron mapping. For feature units that generate invalid input states due to missing corresponding modal data, a reliability value of 0 is assigned and a global masking code is generated simultaneously. The reliability score of the corresponding modality is calculated by combining indicators such as key point confidence, regional visibility ratio and signal-to-noise ratio. Feature units with missing modal data are masked with a value of 0, providing a data quality reference for input data for subsequent interactive calculations.

[0013] Furthermore, the step of constructing a cross-modal alignment correlation matrix by combining temporal distance constraints, content relevance, and the reliability score includes: The time dimension constraint bias term is calculated by substituting the difference between the relative event stage positions within the sliding window and the real physical timestamp into the Gaussian decay function. The input tensor is mapped by a linear transformation network and a dot product is performed to obtain the content correlation matrix between actions and micro-expressions. The reliability score of the read feature is extracted and the natural logarithm is performed to generate a reliability bias term. The content relevance matrix, the time dimension constraint bias term, and the reliability bias term are superimposed to generate a cross-modal soft alignment prior matrix as the alignment correlation matrix. The time difference is processed using a Gaussian decay function, and combined with the content relevance matrix and the reliability bias term, a cross-modal soft alignment prior matrix is ​​constructed to establish the time and content correlation between asynchronous actions and micro-expression features.

[0014] Furthermore, the steps of generating cross-modal interaction features, updated unimodal features, and cross-modal conflict features by performing calculations based on the alignment correlation matrix and the reliability score include: Construct a bidirectional interaction path, and superimpose the reliability score as a bias term on the bidirectional cross-modal attention score to penalize low-quality features, and calculate and generate an interaction feature matrix as the cross-modal interaction feature; The cross-modal interaction features are written back to update the original single-modal feature sequence in residual form, forming updated action features and updated micro-expression features as the updated single-modal features; The updated action features and the updated micro-expression features are mapped to a unified semantic space to calculate and generate the cross-modal conflict features that quantify the degree of conflict. In the cross-modal attention calculation, a reliability score is introduced as a bias term to adjust the weight ratio of low reliability features. At the same time, the updated unimodal features are mapped to a unified space to calculate and represent the degree of difference between action features and micro-expression features.

[0015] Furthermore, the step of generating feature fusion weights by combining the visual conditional features extracted from the video window, the reliability score, and the cross-modal conflict features includes: The visual condition features are constructed by extracting scene brightness, motion blur level, and occlusion ratio parameters from the current video window. The visual condition features, the aggregated reliability value of the two branches, the cross-modal conflict features, and the modal missing markers are input into the gating network. Dynamically updated action feature weights, micro-expression feature weights, and cross-modal interaction feature weights are generated by normalized exponential function mapping and used as feature fusion weights. Combined with visual condition features such as scene brightness and motion blur degree, as well as feature conflict, feature fusion weights are calculated through a gating network. The numerical proportion of corresponding features in the fusion calculation is adjusted according to video environment conditions and modal semantic differences.

[0016] Furthermore, the step of performing a weighted summation on the updated single-modal features and the cross-modal interaction features based on the feature fusion weights, and outputting the joint decision result includes: Using the corresponding feature fusion weights, a weighted summation calculation is performed to generate a joint feature representation vector; The classification decision network outputs the discrete probability distribution vector and continuous intensity score of the corresponding emotion category, and calculates the information entropy value and concatenates the cross-modal conflict feature value in the channel dimension to generate the inference uncertainty assessment score. The joint decision result is generated by combining the discrete emotion classification label, the continuous intensity score, and the inference uncertainty assessment score. The output joint decision result integrates the discrete probability distribution vector, the continuous intensity score, and the inference uncertainty assessment score, providing the classification result of the emotion category and the uncertainty assessment data.

[0017] This invention provides a joint emotion recognition method based on action and micro-expression based on perceptual cross-modal interaction. It has the following beneficial effects: 1. This invention constructs an alignment correlation matrix that includes a time dimension constraint bias term and a content correlation matrix, and superimposes the time and content dimension parameters of the action feature unit and the micro-expression feature unit to overcome the misalignment defect of heterogeneous modalities in the time scale and improve the accuracy of cross-modal interaction feature extraction.

[0018] 2. This invention calculates the reliability scores of action feature units and micro-expression feature units, and incorporates the reliability scores as a bias term into the bidirectional cross-modal attention score. In the cross-modal interaction feature generation stage, it adaptively suppresses low-quality feature interference caused by occlusion and blurring, and enhances the model's adaptability to degraded data.

[0019] 3. This invention combines visual condition features, reliability scores, and cross-modal conflict features, inputs them into a gating network to generate dynamically adjusted action feature weights, micro-expression feature weights, and cross-modal interaction feature weights. The weighted summation parameter ratios are adaptively allocated according to the data quality and semantic contradictions of different modalities, thus maintaining the inference stability of the joint decision results. Attached Figure Description

[0020] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention; Figure 3 This is a time-series evolution diagram of the cross-modal features of the present invention; Figure 4 This is a network topology diagram for reliability estimation and feature alignment in this invention; Figure 5 This is a topology diagram of the cross-modal bidirectional interaction and residual update mechanism of the present invention; Figure 6 This is a flowchart of the adaptive gating fusion inference and network training strategy of the present invention; Figure 7 This is a cross-modal attention weight distribution diagram of the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 This invention provides a joint emotion recognition system based on action and micro-expression expressions using perceptual cross-modal interaction, comprising: The video management module is used for video decoding and timestamp marking, performing person detection and multi-target tracking tasks, and maintaining corresponding person status cache data.

[0023] The association module is used to bind the body movement sequence and facial micro-expression sequence of the same person target by combining target identification, spatial position relationship, head and shoulder key points and historical movement trajectory.

[0024] The action construction module is used to perform centering translation and scale normalization operations on keypoint coordinates, calculate kinematic feature parameters, and generate missing masks to construct action input sequences.

[0025] The micro-expression construction module is used to perform face alignment and head rigidity motion compensation, locate the starting and peak stages of facial motion, segment the face into multiple local regions and calculate feature differences.

[0026] The feature encoding module is used to perform intermediate layer feature encoding on the action input sequence and the micro-expression input sequence respectively, generating action feature units and micro-expression feature units that retain the time dimension.

[0027] The reliability estimation module is used to calculate the local quality assessment score of each feature unit and generate the corresponding effective mask bias based on the quality data.

[0028] The feature alignment module is used to establish a soft alignment relationship between action feature units and micro-expression feature units by combining timestamps, relative stages of events, similarity of feature content, and reliability scores of feature units.

[0029] The interaction module is used to perform bidirectional cross-modal attention calculations before a single modality enters the classification process, and combines reliability information to control the magnitude of information interaction and update residual features.

[0030] The adaptive gating module is used to generate fusion weights for action features, micro-expression features, and interaction features by combining the image visual features of the current window, modal reliability, and cross-modal semantic conflict states.

[0031] The output module is used to integrate features of the intermediate representation based on the fusion weights, and output the final sentiment inference result and uncertainty quantification value through the joint decision network.

[0032] See attached document Figure 2 This invention provides a method for joint emotion recognition based on action and micro-expression in cross-modal interaction of perception, the method comprising the following steps: S1: Obtain a continuous video stream sequence and append a timestamp, execute a person target detection and multi-target tracking algorithm, and assign cross-frame continuous tracking identifiers to the target person in the video frame; S2, extract human body key points, hand key points, facial key points and face image data in the determined target person area, and use the tracking markers allocated in step S1 to bind the extracted action sequence and face sequence to the same target instance in combination with the spatial geometric relationship between the human body detection box and the face detection box and the historical motion trajectory. S3, select human feature points as the origin of coordinates to translate the coordinates of the overall key points and perform normalization according to the target body size, calculate the position offset, velocity and acceleration of key points in adjacent frames, and construct action sequence data covering the time window by combining skeleton connectivity vector, key point detection confidence and node missing mask. S4, combine facial stable feature points to perform similar transformation operation on face data to eliminate the interference of rigid head movement, locate the starting stage and peak stage of facial movement process within the sliding short time window, divide the aligned face into multiple independent facial regions and calculate the forward difference and reverse difference data of each corresponding region. S5, the action sequence constructed in step S3 is sent to the action encoder to perform spatiotemporal graph encoding and output action feature units that retain joint and time information. The data constructed in step S4 is sent to the micro-expression encoder to perform shallow feature extraction and output micro-expression feature units that retain facial region and orientation information. At this time, the system does not generate single-modal independent emotion classification results. S6, quantify the reliability score of the action feature unit based on the action branch convergence detection confidence, trajectory continuity and occlusion duration information, quantify the reliability score of the micro-expression feature unit based on the micro-expression branch convergence facial region visibility, face alignment residual and non-rigid motion signal-to-noise ratio, and generate an attention mask by combining the above scores; S7 limits the time distance threshold for cross-modal retrieval. Within the candidate range, it integrates the real timestamp of the feature, the relative window stage position, the cosine similarity of the feature vector content, and the reliability score calculated in step S6 to construct an alignment correlation matrix between the action feature unit sequence and the micro-expression feature unit sequence. The alignment correlation matrix is ​​a cross-modal soft alignment prior matrix generated by combining time features, content similarity, and reliability score, and establishes a time-scale soft correlation relationship between the action sequence and the micro-expression sequence. S8, when performing bidirectional information interaction, the first path uses micro-expression features to query associated action sequence features, and the second path uses action features to query micro-expression sequence features. During the attention calculation process, a reliability score is introduced as a bias term to penalize low-quality features. After obtaining the interaction features, the residual form is used to write back and update the original single-modal feature sequence. S9 extracts scene brightness, motion blur degree, target size ratio and occlusion ratio parameters from the current video window where the target is located to form visual condition features. The gated network combines the visual condition features, the overall reliability aggregation value of the two branches, the modal missing label and the calculated cross-modal conflict features to generate three adaptive fusion weights that are dynamically updated with the time window. S10, combined with the fusion weights generated in step S9, performs weighted summation calculations on independent action features, independent micro-expression features, and cross-modal interaction features respectively. The aggregated joint feature vector is then input into the emotion prediction output head network for inference and outputs the target person's emotion classification label, emotion response intensity, and model uncertainty value.

[0033] In a specific implementation, step S1 provided by the present invention includes the following steps: S101, acquire a continuous video stream sequence and attach a timestamp to construct an input data stream. The continuous video stream sequence is a data stream containing multiple frames of images, which serves as the basic input source for the system's emotion recognition. The video management module receives the raw continuous video stream sequence data input from the external acquisition device and calls the internal hardware decoder to perform decoding operations to extract frame-by-frame image data containing the red, green, and blue channels. The system adjusts the pixel size of each frame of image according to the preset resolution to construct a fixed-dimensional image matrix. The system reads the hardware clock source to assign incremental global timestamps to each frame of image to record the absolute time physical position of the image in the time sequence. The image matrix containing the timestamps is encapsulated to construct a standardized input data stream and pushed into the data cache queue maintained by the system according to the first-in-first-out principle to provide the basic data source.

[0034] S102, perform person target detection and output a two-dimensional detection box matrix. The video management module reads the single-frame image matrix from the data cache queue in chronological order and inputs it into the deployed person target detection model to perform bounding box prediction calculation. The person target detection model adopts a fully convolutional neural network architecture that includes a residual convolutional backbone network, a multi-scale feature pyramid fusion layer, and a bounding box regression prediction head to extract visual features. The forward propagation outputs a set of candidate targets with confidence exceeding the set screening threshold. Each detected target is represented by a two-dimensional matrix numerical value containing the horizontal and vertical coordinates of the bounding box center point, the width of the bounding box, and the height information, forming an independent set of person detection boxes for the current frame, which is stored in the intermediate feature temporary storage area and passed to the downstream stage.

[0035] S103, using a multi-target tracking algorithm to assign continuous tracking markers across frames to the target person in the video frame, the video management module reads the current frame person detection box set in the intermediate feature storage area together with the target trajectory state data left over from the historical frames to perform cross-frame association, the algorithm runs the Kalman filter prediction module to predict the prior position matrix of the current frame based on the target motion speed and position of the previous frame and calculates the intersection-union distance matrix between the actual detection box and the target. The Hungarian matching algorithm is used to complete the optimal allocation mapping of the target bipartite graph. Each independent human target in the image is assigned a globally unique identity tracking identifier number. For lost trajectories that have not been matched with detection boxes in multiple consecutive frames, a culling operation is performed to release system memory. The two-dimensional matrix of the successfully matched bounding boxes is packaged together with the identity tracking identifier number to generate a continuous spatiotemporal motion trajectory dictionary.

[0036] In a specific implementation, step S2 provided by the present invention includes the following steps: S201: Within the identified target person area, extract key points of the human body, hands, and face, and add face image data. The association module reads the detection box matrix stored in the target state cache data structure, performs image cropping to obtain a local human body image, and inputs it into the human pose estimation model. The human pose estimation model adopts a feedforward neural network architecture consisting of a high-resolution convolutional backbone network and a two-dimensional heatmap regression prediction layer. It receives a 256x192 red, green, and blue three-channel pixel matrix after cropping. After forward propagation, it outputs a heatmap tensor representing the spatial probability distribution of body joints, fingertips, and facial contour feature points. The non-maximum suppression algorithm is used to find the peak response coordinates of the heatmap and extract the coordinates of 133 two-dimensional spatial key points to form a coordinate sequence. Simultaneously, the local human body image is sent to a face detection network based on a residual network architecture to perform facial region recognition. The output two-dimensional bounding box matrix surrounding the face region is calculated, merged, and cropped to generate a 112x112 facial image pixel matrix.

[0037] S202, combining the two-dimensional spatial coordinate point sequence output by the human pose estimation model, infers and predicts the head center coordinates. The association module reads two-dimensional vector values ​​representing the coordinates of the left shoulder joint, the right shoulder joint, and the tip of the nose from the coordinate sequence composed of 133 two-dimensional spatial key point coordinates. Based on the physiological connection constraints of the human skeleton and torso space, the average value of the left and right shoulder coordinates is calculated to obtain the shoulder center point vector. The shoulder center point vector is used to extend and map upward along the coordinate direction of the tip of the nose to obtain the predicted head center coordinates. The system extracts the two-dimensional bounding box of the facial region and calculates the diagonal intersection value to define the face center coordinate parameter. The face center coordinate parameter represents the physical center spatial position of the face actually captured by the face detection network. The predicted head center coordinate parameter represents the physical center spatial position of the human head obtained by inferring the body torso action modality.

[0038] S203, combining the spatial geometric relationship between the face detection box and the predicted head center coordinates, plus the historical motion trajectory, binds the action sequence and micro-expression sequence to the same target instance. The association module retrieves the face center coordinate parameters and the predicted head center coordinate parameters, and calculates the square root of the sum of the squares of the differences between the horizontal and vertical coordinates of the two coordinate points to obtain the straight-line Euclidean distance as the spatial distance deviation parameter. Candidate objects whose spatial distance deviation is lower than a preset pixel distance matching threshold are extracted to construct associated combinations. The pixel distance matching threshold is set to a value consisting of 0.2 times the width of the target 2D detection box plus 0.2 times its height. By combining the tracking identifier numbers issued by the video management module to read the matching results of the previous frame and maintain the continuity of the temporal dimension association mapping, a mapping binding table of action sequences and micro-expression sequences is generated. For each independent person entity that passes the matching verification, an independent time sliding window memory cache space is opened on the memory. The action sequences associated with the same identity identifier and the micro-expression sequences are merged and stored in the internal dedicated data queue structure.

[0039] See attached document Figure 3 In a specific implementation, step S3 provided by the present invention includes the following steps: S301, the motion construction module receives the human body key point coordinate data output by the previous stage and performs spatial reference alignment calculation at the single frame image level. The system selects the center node of the hip of the target person as the origin of the human body three-dimensional coordinate system. When the hip node is in a visually obstructed state, the visible shoulder node and torso node are extracted. The action construction module multiplies the spatial coordinates of the visible node by the corresponding detection confidence and performs a weighted average calculation to generate a confidence weighting center to serve as the candidate coordinate origin. The system measures the Euclidean distance between the two shoulders and the longitudinal length of the torso to form the scale-normalized reference parameters. The system performs a spatial translation operation by subtracting the origin vector from the original coordinates of all joints. The system uses the translation coordinates divided by the scale-normalized reference parameters to generate a normalized coordinate vector. The global displacement vector for background motion estimation is extracted using existing optical flow methods. The system subtracts the global displacement vector from the translation coordinates to generate compensation coordinates.

[0040] S302, the motion construction module performs differential operations along the time axis based on the normalized key point coordinates to extract multi-order kinematic feature parameters. The system calculates the spatial position offset and temporal derivative velocity component of a single node between adjacent video frames and extracts the cross-frame acceleration component. The motion construction module extracts the skeleton connectivity vector formed by adjacent joints, the joint angle of limb movement, and the motion amplitude of local nodes. The system aggregates the coordinates of all nodes to calculate the trajectory of the body's center of gravity change and the symmetry index of the left and right limb movements. The system simultaneously extracts the confidence score of key point detection and node missing markers. The system assigns independent missing masks to nodes that are not detected successfully and accumulates the number of undetected frames along the time window to generate a value for the duration of continuous missing data. The action construction module maintains the original numerical state of the physical missing data. The action construction module performs a matrix concatenation operation on the feature channel dimension with normalized coordinates, multi-order kinematic feature parameters and node missing masks to generate an action input sequence, which is then sent to the feature encoding module. The action input sequence is data composed of normalized key point coordinates, kinematic feature parameters and missing masks, representing the body trunk movement pattern of the target person changing over time.

[0041] See attached document Figure 3 In a specific implementation, step S4 provided by the present invention includes the following steps: S401, the micro-expression construction module extracts the facial key points of the face corresponding to the continuous video frames. The system selects stable key points of the corners of the eyes, the tip of the nose, and the corners of the mouth to calculate the parameters of the affine transformation matrix. The micro-expression construction module uses the affine transformation matrix to perform spatial resampling calculation on the facial image sequence contained in the input video frame, along with image rotation and cropping operations. The system uniformly maps facial pixels of any pose to a normalized image space of 224×224 pixels along with the reference frontal orientation. The micro-expression construction module estimates the 3D head Euler angle pose based on the existing 3D dense face alignment algorithm for sequences containing severe head rotation interference. The system extracts non-rigid facial motion information by subtracting the rigid head motion components associated with the 3D pose transformation from the facial motion field. The micro-expression construction module measures the reconstruction error of the corresponding aligned face images in adjacent frames and outputs the face alignment residual value. The system then feeds a normalized face image sequence carrying the alignment residual information to subsequent stages.

[0042] S402, the micro-expression construction module calculates the facial motion activity of the current window by comprehensively considering the changes in local facial texture gradients, non-rigid displacement of key points, and local optical flow field intensity within the sliding time window. The system locks the time node with motion activity below 10% of the activity peak and before the substantial change in facial shape as the starting stage frame of the motion process. The micro-expression construction module searches for video frames within the short time window where the non-rigid facial motion activity is in the highest peak range and defines them as the peak stage frames of the motion process. The system aligns the single-frame face image and divides it into 9 74×74 pixel facial local blocks covering the independent sub-regions of the eye area, between the eyebrows, the nose wing, the corner of the mouth, the cheek, and the jaw. The micro-expression construction module evaluates the visible pixel ratio of each facial local block along with the illumination clarity to generate the region occlusion determination state. The system then feeds the initial stage local block and the peak stage local block, which have completed the region segmentation operation, into the differential computing network.

[0043] S403, the system is configured with a shallow residual network model containing a squeezing excitation module, which receives 74×74 pixel local facial blocks from the initial stage and the peak stage respectively. The shallow residual network model internally arranges sequentially connected two-dimensional convolutional layers, batch normalization layers, global average pooling layers, and fully connected layers to extract local texture feature matrices. Based on a parameter weight sharing mechanism, the system projects the features from both stages into the same semantic space to avoid pseudo-difference generation. The system extracts the baseline tensor of the initial stage features and the activation tensor of the peak stage features, performs matrix subtraction and nonlinear activation operations, and outputs a directional difference matrix. The difference formula is: ; ; In the formula: A positive difference matrix characterizing the enhanced state of facial texture response relative to the initial stage; The inverse difference matrix characterizing the weakening of facial texture response relative to the initial stage; To correct the nonlinear activation function of the linear unit; This is the activation matrix for the region features corresponding to the peak phase of micro-expression movements; This is the regional feature baseline matrix corresponding to the initial stage of micro-expression movement.

[0044] S404, the micro-expression construction module performs a splicing operation along the channel dimension of the forward difference matrix, the reverse difference matrix, and the peak stage feature activation tensor to construct a joint representation unit. The system introduces regional position encoding, along with difference direction marking, event stage label, and relative time embedding, and fuses them into the joint representation unit. The micro-expression construction module extracts the collaborative change correlation of each local facial block from the input region of the joint representation unit through the attention network layer. The system generates micro-expression feature difference data that retains the spatial position information of the facial region along with the details of local motion direction.

[0045] In a specific implementation, step S5 provided by the present invention includes the following steps: S501, the feature encoding module is configured to receive action input sequence data from the action encoder and internally arrange a spatial graph convolutional layer in conjunction with a temporal network model to extract high-order features. The spatial graph convolutional layer aggregates the local spatial morphological features between adjacent joint nodes to generate spatially encoded joint tensors. The action encoder arranges the spatially encoded joint tensors in a time sequence and inputs them into the temporal network model for feature mapping operations. The temporal network model internally arranges a causal-depth separable one-dimensional convolutional layer in conjunction with a channel attention layer in conjunction with a temporal self-attention component. Causal depth separable one-dimensional convolution operation extracts short-term local action patterns along with velocity components. The temporal self-attention component performs forward operation to capture long-term dependencies between the preparation state, peak state, and recovery state. The action encoder truncates the global average pooling layer at the end of the model and, together with the classification projection layer, directly outputs action feature units that maintain the original video time frame dimension along with the human joint node dimension. The action feature units are high-order feature matrices extracted by the action encoder that retain joint and temporal information, providing a deep semantic expression of action modality for subsequent classification and interaction.

[0046] S502, the feature encoding module is configured to receive the micro-expression encoder, which contains the forward difference matrix, the reverse difference matrix, and the basic peak feature tensor. The micro-expression input difference sequence is facial feature data containing the facial forward and reverse difference matrices and position encoding, representing the direction and intensity of non-rigid subtle facial movements. The micro-expression encoder internally employs lightweight convolutional layers along with a region self-attention encoding module to process local facial block information. The lightweight convolutional layers perform forward propagation operations on the input matrix to extract local texture response components. The system superimposes the region location encoding, differential direction markers, event phase embedding vectors, and relative timestamp embedding vectors into the local texture response components to form a region feature expression. The region self-attention encoding module receives the region feature expression, performs calculations to generate collaborative change association weights between various local blocks of the face, and performs feature aggregation. The micro-expression encoder maintains the spatial isolation dimension of the local blocks of the face and outputs micro-expression feature units carrying directional attributes. The micro-expression feature unit is a local facial feature matrix carrying directional attributes extracted by the micro-expression encoder, which provides a deep semantic expression of micro-expression modality for subsequent classification and interaction.

[0047] S503, the feature encoding module implements an intermediate layer feature output control strategy for the action processing branch and the micro-expression processing branch. In the joint inference stage, the system does not output single-modal independent emotion classification results through the action encoder and the micro-expression encoder. The action feature unit and the micro-expression feature unit are input to the feature alignment module and the interaction module. In the training and pre-training stage, the action encoder and the micro-expression encoder are connected to the auxiliary classification output head to perform parameter pre-training. In the joint training and joint inference stages, the auxiliary classification output head is removed or the inference output path of the auxiliary classification output head is cut off.

[0048] See attached document Figure 4 In a specific implementation, step S6 provided by the present invention includes the following steps: S601, the reliability estimation module processes multi-dimensional quality data of action branches to extract the evaluation score of action feature units. The system extracts the confidence score of human key point detection along with the node visibility ratio, and simultaneously obtains the number of consecutive missing frames along with the size of the human external detection box, combined with the trajectory continuity index and the spatial abnormal jump displacement between two adjacent frames. The system introduces the signal-to-noise ratio (SNR) of local joint motion along with the abrupt change in skeleton length. It obtains the SNR by dividing the variance of local joint displacement by the variance of background pixel fluctuation. The system then concatenates the acquired physical parameters to generate a motion quality feature vector, which is fed into a motion multilayer perceptron for mapping. The motion multilayer perceptron consists of a fully connected input layer, a hidden feature transformation layer, and a single-neuron output layer, all connected in series. The single-neuron output layer is configured with a sigmoid activation function to output a reliability score for the motion feature unit, ranging from 0 to 1. The system performs aggregation operations on the reliability scores at each joint level over time to construct the overall reliability of the motion window. The reliability score, calculated based on local quality data, is an evaluation value between 0 and 1, representing the physical confidence level of the feature unit and used to control the amplitude of information interaction and generate penalty bias.

[0049] S602, the reliability estimation module processes the multidimensional local quality data of the micro-expression branch to generate the evaluation score of the micro-expression feature unit. The system extracts the face detection confidence score along with the visible proportion of the face nine-square grid area and obtains the facial image clarity score along with the effective pixel size value. The pixel coordinate offset Euclidean distance generated after performing affine transformation operation on stable key points is used to quantize the alignment residual value. Combined with the spatial Euler angle generated by the 3D head pose estimation network and the alignment residual value, the ratio of the local facial non-rigid motion amplitude to the background noise intensity is calculated to construct the non-rigid motion signal-to-noise ratio. The system concatenates the aforementioned physical parameters along with the temporal continuity index of the transition from the initial stage to the peak stage of facial movement to construct a micro-expression quality feature vector. The reliability estimation module deploys a micro-expression multilayer perceptron with the same hierarchical structure to receive the feature vector input. The micro-expression feature vector is mapped through a fully connected layer and a sigmoid activation function layer to output a micro-expression feature unit reliability score between 0 and 1. The system performs convergence calculation on the local quality scores at each facial segmentation region level to generate the overall reliability of the micro-expression window.

[0050] S603, the reliability estimation module combines the calculated feature unit evaluation score to generate attention control parameters along with network optimization guidance. The system performs numerical judgment on the reliability score of the feature unit along with state monitoring. When the feature unit is missing the corresponding modal data and generates an invalid input state, the reliability estimation module directly assigns a reliability value of 0 and simultaneously generates a global masking code for the corresponding feature unit. During the end-to-end parameter optimization training phase of the joint network, the system configures the quality ranking loss to constrain the update direction of the weight matrix of the multilayer perceptron. The system constructs a degraded version of the action and a degraded version of the micro-expression for the same paired human image training samples. Local key point occlusion, continuous frame forced missing, and facial region masking operations are applied to reduce the physical annotation quality of the corresponding modality. The quality ranking mechanism uses a ranking function with boundary margin to force the reliability score of the output corresponding to the degraded input to be lower than the reliability score of the output of the complete high-quality input. The classification inference cross-entropy loss and the quality ranking loss together guide the network parameters to perform gradient backpropagation operation. The evaluation value output by the multilayer perceptron is used to characterize the physical confidence level of the corresponding local feature parts.

[0051] See attached document Figure 4 In a specific implementation, step S7 provided by the present invention includes the following steps: S701, the feature alignment module extracts the real physical timestamp values ​​attached to the single-modal feature sequence and constructs a time dimension constraint bias term together with the relative event stage position inside the sliding window. The system sets a maximum time distance threshold to limit the feature matching search range based on the statistical law of the distribution of time difference between actions and micro-expressions in the historical sample library. The time distance constraint parameters are calculated by substituting the time distance difference of different modal features into the Gaussian decay function. The variance of the Gaussian decay function is set to be equal to one-third of the time distance threshold value, which generates a time dimension constraint bias term that decreases exponentially as the time distance difference increases. The feature alignment module establishes the base parameters of the feature soft correlation matrix across time scales within the candidate range. The time dimension constraint bias term is encoded by the system into the corresponding dimension bias tensor matrix and directly embedded into the subsequent multi-head attention operation module as the basic position penalty term for adjusting the feature interaction depth.

[0052] S702, the feature alignment module is arranged with a linear transformation network containing a fully connected layer without bias to perform feature space projection, receive input tensors from each modality feature unit and map them to generate a query tensor with uniform dimension, along with key tensors and value tensors. The weight matrix of the fully connected layer is iteratively updated through end-to-end backpropagation via the joint training error of the emotion classification. The system uses the action query tensor to multiply the micro-expression key tensor and divide by the square root of the feature dimension scalar to obtain the content relevance matrix of the action reading micro-expression. Simultaneously, it uses the micro-expression query tensor to multiply the action key tensor and divide by the square root of the feature dimension scalar to obtain the content relevance matrix of the micro-expression reading action. The system extracts the reliability score of the read feature and performs a natural logarithmic operation to generate a reliability bias term. The system presets a very small normal value of one ten-thousandth to avoid zero-value logarithmic overflow error. The low reliability value drives the logarithmic bias term to generate negative extreme values ​​and truncates the interactive calculation path of low-quality local features that are of interest to the target modality query at the tensor operation level.

[0053] S703, the feature alignment module overlays the content relevance matrix, time dimension constraint bias, and reliability bias. The system generates a cross-modal soft alignment prior matrix as the alignment correlation matrix. The feature alignment module outputs the alignment correlation matrix, candidate feature index, time dimension constraint bias, event stage bias, and reliability bias to the interaction module as input data for bidirectional cross-modal attention calculation.

[0054] See attached document Figure 5 In a specific implementation, step S8 provided by the present invention includes the following steps: S801, the interaction module receives action feature units, micro-expression feature units, alignment correlation matrix, candidate feature index, time dimension constraint bias term, event stage bias term and reliability bias term, and constructs the first interaction path for micro-expression reading action and the second interaction path for action reading micro-expression.

[0055] S802, the first interaction path uses micro-expression feature units as the query end and action feature units as the read end, generating an attention response for the micro-expression region, differential direction, and event stage corresponding to the action time and joint information; the second interaction path uses action feature units as the query end and micro-expression feature units as the read end, generating an attention response for the action time and joint information corresponding to the facial region, differential direction, and event stage information. The interaction module superimposes a time dimension constraint bias, an event stage bias, and a reliability bias generated by the reliability of the read feature units into the attention score. The above bidirectional cross-modal attention calculation formula is: ; ; In the formula: This is a micro-expression query action interaction feature matrix generated using micro-expressions as the query retrieval action sequence; This is a feature matrix of micro-expression interactions generated for retrieving micro-expression sequences based on actions. It is a normalized exponential function; The query calculation matrix corresponding to micro-expression modalities; The state is the transpose of the key matrix corresponding to the action mode; The query calculation matrix is ​​the one corresponding to the action modality; The transpose of the key matrix corresponding to the micro-expression modality is the state. This is the value mapping matrix corresponding to the action mode; This is a value mapping matrix corresponding to micro-expression modalities; Scalar factor to scale the feature channel dimensions of the query matrix and the key matrix; The relative time position offset parameter matrix calculated for the micro-expression query action stage; The relative time position offset parameter matrix calculated for the micro-expression stage of the action query; The reliability assessment score of the motion feature unit output by the front-end module; The quality reliability evaluation score of the micro-expression feature unit output by the front-end module.

[0056] S803, the interaction module generates interaction features for micro-expression reading actions and interaction features for action-reading micro-expressions based on the bidirectional cross-modal attention calculation results; the interaction module generates residual update thresholds based on query features, interaction features, mutual reliability, and local conflict degree, and writes the interaction features back to the action branch and micro-expression branch in a residual manner to form updated action features, updated micro-expression features, and cross-modal interaction features. The cross-modal interaction features are interaction feature matrices generated by bidirectional cross-modal attention calculation, which fuse heterogeneous modal information to make up for the deficiencies of single-modal features.

[0057] See attached document Figure 6 In a specific implementation, step S9 provided by the present invention includes the following steps: The S901 adaptive gating module extracts visual condition features from the current person video window. These visual condition features include average brightness, contrast, motion blur, background motion, camera shake, person density, human body frame size, face frame size, occlusion ratio, effective key point ratio, and tracking stability.

[0058] S902, the adaptive gating module performs reliability-weighted convergence on the updated action features, updated micro-expression features and cross-modal interaction features respectively to generate action representation, micro-expression representation and interaction representation; the adaptive gating module maps the action representation and micro-expression representation to a unified semantic space and calculates modal conflict features.

[0059] S903, the adaptive gating module inputs action representation, micro-expression representation, interaction representation, visual conditional features, action modal reliability, micro-expression modal reliability, modal existence marker, and modal conflict features into the gating network, and outputs action feature weights, micro-expression feature weights, and cross-modal interaction feature weights. The generation formulas for the above action feature weights, micro-expression feature weights, and cross-modal interaction feature weights are as follows: ; In the formula: To extract the feature fusion weight parameters assigned to the action modality channel; To extract the feature fusion weight parameters assigned to the micro-expression modality channels; To extract feature fusion weight parameters assigned to cross-modal interactive feature channels; It is a normalized exponential function; A multi-layer nonlinear mapping function configured within the gated network; Extract the current time window pooled feature tensor of the output for the action branch; Extract the current time window pooled feature tensor for the micro-expression branch; Extract the current time window pooled feature tensor of the output for the interactive branch; This is a visual conditional feature vector of scene brightness and motion blur degree extracted from the current video frame of the target. Aggregate the reliability score parameter for the overall action modality within the current time window; Aggregate the reliability score parameter for the overall micro-expression modality within the current time window; A parameter for measuring cross-modal semantic conflict states between hidden features of the action branch and hidden features of the micro-expression branch; Use a mask to mark variables indicating the missing key points in the underlying structure of the action modality under the current time window; This is a variable used to label the local occlusion of the underlying structure of the micro-expression modality within the current time window.

[0060] See attached document Figure 6 The steps S10 and model training process provided by this invention include the following steps in specific implementation: S1001, the output module extracts the action feature weights, micro-expression feature weights, and cross-modal interaction feature weights generated by the adaptive gating module. The system maps the independent action feature representation, independent micro-expression feature representation, and cross-modal interaction feature representation to the same feature dimension, and performs weighted summation calculation using the corresponding weights to generate a joint feature representation vector.

[0061] S1002, the output module is configured with a multi-layer classification decision network to receive the joint feature representation vector input and perform nonlinear mapping calculation. The multi-layer classification decision network is arranged with a fully connected dimensionality reduction layer connected in series with a batch normalization layer and a modified linear unit activation layer to extract high-order nonlinear semantic patterns. The network endpoint accesses the emotion prediction output head network to perform the final complex numerical mapping inference. The emotion prediction output head network is equipped with seven discrete classification neurons that combine with a normalized exponential function to output discrete probability distribution vectors corresponding to the seven basic emotion categories. The emotion prediction output head network is simultaneously equipped with two continuous numerical regression neurons to map and output the continuous intensity score scalar of emotion valence along with arousal and emotional space coordinates. The system integrates discrete emotion classification labels with emotion valence continuous intensity scores to construct a multi-dimensional emotional response state profile and outputs the inference results from the underlying hardware application interface.

[0062] S1003, the output module combines the cross-modal conflict features obtained by the adaptive gating module with the probability distribution information generated by the output head network to calculate the uncertainty evaluation parameters. The cross-modal conflict features are the measurement parameters calculated after mapping the action representation and micro-expression representation to a unified semantic space, which quantifies the degree of conflict or inconsistency between the two modalities at the semantic level. The system extracts the classification probability distribution vector and calculates the information entropy value to represent the probability dispersion of the current inference result; the output module concatenates the information entropy value together with the cross-modal conflict feature value in the channel dimension and inputs it into the auxiliary single-layer linear mapping network to calculate and output the uncertainty evaluation score. The auxiliary single-layer linear mapping network is internally arranged with fully connected neurons and activation layers to map input features to uncertainty assessment scores in the range of zero to one. The uncertainty assessment scores are used to characterize the inference uncertainty corresponding to the degree of dispersion of the classification probability distribution and the degree of cross-modal conflict. The system packages the discrete emotion classification labels together with the continuous intensity scores and uncertainty assessment scores to generate the final joint decision result, which is then provided to the business application platform for invocation. The joint decision result is the final output combination containing discrete emotion classification labels, continuous intensity scores, and uncertainty assessment scores, providing the final emotion inference profile of the target person.

[0063] S1004, in the model training phase, the system adopts a three-stage training process. In the first stage, the system pre-trains the action encoder and micro-expression encoder respectively. In the second stage, the system removes the final classification layer of the two branches, connects the cross-timescale alignment module, the bidirectional cross-modal interaction module, the adaptive gating module and the joint output module, and performs joint training using action-micro-expression samples of the same person and corresponding time. In the third stage, the system performs small learning rate fine-tuning and reliability calibration in the target video domain.

[0064] During training, the system randomly applies keypoint occlusion and jitter, consecutive frame loss, partial facial occlusion, motion blur, brightness variations, resolution degradation, head pose perturbations, and whole-modal discarding to generate motion degradation samples and micro-expression degradation samples. The system uses quality ranking constraints to decrease the reliability and gating weights of degraded modalities, increase the relative gating weights of non-degraded modalities, and increase uncertainty when both modalities degrade simultaneously. The total loss function during joint training is: ; In the formula: The objective value of the overall training loss function used for backpropagation calculation of the model; The cross-entropy classification loss function value is extracted to predict the output error based on the main task's emotion classification. The soft alignment loss function value generated for the relative temporal alignment error of the action feature sequence and the micro-expression feature sequence; The local quality assessment loss function value generated by combining the network prediction reliability score with the actual feature signal-to-noise ratio difference; The feature semantic consistency loss function value is generated to account for the differences in the mapping of cross-modal feature representations in different semantic spaces; Set fixed weight hyperparameters for the system to calculate the soft alignment loss function; Pre-determine the fixed weight hyperparameters of the system to calculate the local quality assessment loss function; A fixed weight hyperparameter is preset for the semantic consistency loss function of the control features in the system.

[0065] To aid in understanding the technical solution of this invention, an application example based on a psychological assessment interview scenario is provided below.

[0066] See attached document Figure 7 The video acquisition device acquires video of the target person at a set sampling rate. The system extracts a long action time window on the time axis and extracts the key point sequence of the human skeleton. At the same time, the face analysis module extracts a short micro-expression window within the overlapping time period and extracts the positive and negative difference features of the local facial region.

[0067] During the feature extraction stage, the system detected that the target person had a shoulder contraction movement, and the confidence of local key points was high. Based on this, the system calculated that the action feature unit had a high overall reliability. At the same time, the target person had a brow contraction and head-down movement, which caused the lower half of the face to be obscured. The system calculated that the micro-expression feature unit in the brow area maintained a high reliability, while the reliability of the micro-expression feature unit in the corner of the mouth area decreased.

[0068] In the cross-timescale soft alignment and bidirectional interaction stage, the system uses the eyebrow features as the query end to calculate the content similarity with the shoulder action features. The system combines the micro-expression window and the time distance of the action moment to generate a time bias term, and combines the reliability of the action features to generate a reliability bias term. Combining the bias term and content similarity, the system calculates the corresponding attention weight for the eyebrow features to the shoulder action. When performing reverse interaction, the system queries the micro-expression features based on the action features. Since the reliability of the corner of the mouth area is low, the logarithmic bias reduces the attention weight of this area in the normalization calculation, thereby the system inhibits local features from participating in the subsequent residual update.

[0069] In the adaptive gating and joint output stage, the system extracts the global brightness, contrast, and the proportion of facial occlusion of the target person as visual conditional features. The gating network combines the aforementioned visual conditions with the reliability of each modal feature to generate dynamic fusion weights. Due to facial occlusion, the network adaptively reduces the weight of micro-expression features and correspondingly increases the weight of action features and cross-modal interaction features. The weighted joint features are input into the output network, and the final output indicates the classification result of the target emotion category and a low model uncertainty.

[0070] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result using substantially the same method falls within the scope of the present invention.

Claims

1. A joint emotion recognition method based on action and micro-expression expressions in perceptual cross-modal interaction, characterized in that, The method includes the following steps: Acquire a continuous video stream sequence and assign a tracking identifier to the target person; Combine the tracking identifiers to extract motion data and facial data of the same target from the continuous video stream sequence; Spatial reference alignment and time axis extraction are performed on the motion data to construct the motion input sequence; Rigid motion compensation and facial region segmentation are performed on the facial data to construct a micro-expression input difference sequence; The action input sequence and the micro-expression input difference sequence are feature encoded to generate action feature units and micro-expression feature units; The reliability scores of the motion feature unit and the micro-expression feature unit are calculated based on local quality assessment. A cross-modal alignment correlation matrix is ​​constructed by combining time distance constraints, content relevance, and the aforementioned reliability score; Calculations are performed based on the alignment correlation matrix and the reliability score to generate cross-modal interaction features, updated unimodal features, and cross-modal conflict features; The visual conditional features extracted from the video window, the reliability score, and the cross-modal conflict features are combined to generate feature fusion weights; Based on the feature fusion weights, a weighted summation is performed on the updated single-modal features and the cross-modal interaction features to output a joint decision result.

2. The joint emotion recognition method based on action and micro-expression expressions according to claim 1, characterized in that, The steps of acquiring a continuous video stream sequence and assigning a tracking identifier to the target person, and combining the tracking identifier to extract motion data and facial data of the same target from the continuous video stream sequence, include: The algorithm performs human target detection and multi-target tracking, assigning continuous tracking labels across frames to target human figures in the video frame; Extract key human body points and facial image data within the identified target area; The coordinates of the predicted head center are calculated, and the spatial distance deviation is calculated by combining the spatial geometric relationship between the face detection box and the predicted head center coordinates. By combining the tracking identifier and the spatial distance deviation parameter, the action sequence and the micro-expression sequence are bound to the same target instance to obtain the action data and the face data.

3. The joint emotion recognition method based on action and micro-expression expressions according to claim 1, characterized in that, The steps of performing spatial reference alignment and time axis extraction on the motion data to construct the motion input sequence include: The hip center node or the confidence weighting center is selected as the origin of the three-dimensional coordinate system. The overall displacement vector is subtracted and the normalized key point coordinates are generated using the scale-normalized reference parameters. The difference operation is performed along the time axis to extract multi-order kinematic feature parameters. The normalized coordinates, the multi-order kinematic feature parameters, and the node missing mask are concatenated using an execution matrix to generate the action input sequence.

4. The joint emotion recognition method based on action and micro-expression expressions according to claim 1, characterized in that, The steps of performing rigid motion compensation and facial region segmentation on the facial data to construct a micro-expression input difference sequence include: Affine transformation matrix is ​​used to eliminate the interference of rigid head motion and obtain a normalized face image sequence; Within a sliding short window, locate the starting phase frame and the peak phase frame of the motion process, and divide the aligned single-frame face into multiple independent facial local blocks; The forward and backward difference matrices of each facial local block are calculated, and the peak phase feature activation tensor and the fused region location code are concatenated along the channel dimension to construct the micro-expression input difference sequence.

5. The joint emotion recognition method based on action and micro-expression expressions according to claim 1, characterized in that, The steps of performing feature encoding on the action input sequence and the micro-expression input difference sequence to generate action feature units and micro-expression feature units include: The action feature unit is output by aggregating local spatial morphological features and capturing long-term dependencies using spatial graph convolutional layers and temporal network models. Lightweight convolutional layers and region self-attention encoding modules are used to generate collaborative change association weights between various local blocks of the face, and the micro-expression feature units are output.

6. The joint emotion recognition method based on action and micro-expression expression in perceptual cross-modal interaction according to claim 1, characterized in that, The step of calculating the reliability scores of the action feature unit and the micro-expression feature unit based on local quality assessment includes: The confidence score of key point detection, trajectory continuity and signal-to-noise ratio of local joint motion are aggregated, and the reliability score corresponding to the motion feature unit is generated by mapping through motion multilayer perceptron. The confidence score of key point detection, trajectory continuity and signal-to-noise ratio of local joint motion are aggregated, and the reliability score corresponding to the motion feature unit is generated by mapping through motion multilayer perceptron. The reliability score corresponding to the micro-expression feature unit is generated by combining the visible proportion of the nine-square facial region, the face alignment residual, and the non-rigid motion signal-to-noise ratio through a micro-expression multilayer perceptron mapping. For feature units that generate invalid input states due to missing corresponding modal data, a reliability value of 0 is assigned and a global masking code is generated simultaneously.

7. The joint emotion recognition method based on action and micro-expression expressions according to claim 1, characterized in that, The step of constructing a cross-modal alignment correlation matrix by combining time distance constraints, content relevance, and the reliability score includes: The time dimension constraint bias term is calculated by substituting the difference between the relative event stage positions within the sliding window and the real physical timestamp into the Gaussian decay function. The input tensor is mapped by a linear transformation network and a dot product is performed to obtain the content correlation matrix between actions and micro-expressions. The reliability score of the read feature is extracted and the natural logarithm is performed to generate a reliability bias term. The content relevance matrix, the time dimension constraint bias term, and the reliability bias term are superimposed to generate a cross-modal soft alignment prior matrix as the alignment correlation matrix.

8. The joint emotion recognition method based on action and micro-expression expression in perceptual cross-modal interaction according to claim 1, characterized in that, The steps for generating cross-modal interaction features, updated unimodal features, and cross-modal conflict features based on the alignment correlation matrix and the reliability score include: Construct a bidirectional interaction path, and superimpose the reliability score as a bias term on the bidirectional cross-modal attention score to penalize low-quality features, and calculate and generate an interaction feature matrix as the cross-modal interaction feature; The cross-modal interaction features are written back to update the original single-modal feature sequence in residual form, forming updated action features and updated micro-expression features as the updated single-modal features; The updated action features and the updated micro-expression features are mapped to a unified semantic space to calculate and generate the cross-modal conflict features that quantify the degree of conflict.

9. The joint emotion recognition method based on action and micro-expression expressions according to claim 1, characterized in that, The step of generating feature fusion weights by combining the visual conditional features extracted from the video window, the reliability score, and the cross-modal conflict features includes: The visual condition features are constructed by extracting scene brightness, motion blur level, and occlusion ratio parameters from the current video window. The visual condition features, the aggregated reliability value of the two branches, the cross-modal conflict features, and the modal missing markers are input into the gating network. Dynamically updated action feature weights, micro-expression feature weights, and cross-modal interaction feature weights are generated by normalized exponential function mapping and used as the feature fusion weights.

10. The joint emotion recognition method based on action and micro-expression expression in perceptual cross-modal interaction according to claim 9, characterized in that, The steps of performing a weighted summation of the updated single-modal features and the cross-modal interaction features based on the feature fusion weights, and outputting the joint decision result, include: Using the corresponding feature fusion weights, a weighted summation calculation is performed to generate a joint feature representation vector; The classification decision network outputs the discrete probability distribution vector and continuous intensity score of the corresponding emotion category, and calculates the information entropy value and concatenates the cross-modal conflict feature value in the channel dimension to generate the inference uncertainty assessment score. The joint decision result is generated by combining the discrete emotion classification label, the continuous intensity score, and the inference uncertainty assessment score.

Images