A human-computer interaction-based image emotion recognition method and system

Abstract

The application relates to the technical field of image recognition, and discloses an image emotion recognition method and system based on human-computer interaction, an image emotion recognition method based on human-computer interaction, which comprises the following steps: step S101, obtaining a brow muscle region image sequence; step S102, calculating an event anchoring phase lag amount; step S103, constructing a time normalization input sequence; step S104, calculating a time sequence feature sequence; step S105, generating a global representation vector; and step S106, outputting discrete emotion category labels. The application realizes unified interception of the brow muscle region through face key point positioning and similarity transformation, and avoids interference of redundant information; in combination with a gating coefficient generated by the event anchoring phase lag amount and a Gaussian time weight window, the application realizes feature amplitude modulation and key information strengthening in time domain convolution, reduces feature fluctuation caused by individual differences, and thus improves the accuracy of emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically, to an image emotion recognition method and system based on human-computer interaction. Background Technology

[0002] With the rapid development of human-computer interaction technology, image emotion recognition has become one of the key technologies for improving interactive experience, and is widely used in smart terminals, service robots, and other scenarios. The core of emotion recognition is to accurately determine the user's emotional state by capturing key features of facial expressions. As the core area for facial emotion expression, the brow muscle's contraction and relaxation are directly related to various emotions such as confusion, pleasure, and surprise. However, existing technologies have many problems when using brow muscle features for emotion recognition.

[0003] Existing technologies largely rely on full-face feature extraction, neglecting the crucial brow muscle region. This leads to redundant information interfering with feature effectiveness. Some methods attempting to extract brow muscle features fail to address the localization issues caused by facial pose variations and distance differences. The brow muscle region shifts across different video frames and samples, making it impossible to achieve a unified feature extraction benchmark. Furthermore, emotional responses in human-computer interaction scenarios are temporally correlated with interaction events. Existing methods do not anchor the time nodes of interaction events, ignoring the phase lag characteristics of emotional responses. This results in misalignment of emotional expressions across different samples on the timeline, leading to dilution of temporal features. In addition, individual differences cause fluctuations in the amplitude of brow muscle movement features. Existing technologies lack effective feature modulation mechanisms, making it difficult to stably capture temporal dependencies. Ultimately, this results in low accuracy and insufficient stability in emotion recognition, failing to meet the demands for accuracy and robustness in human-computer interaction scenarios. Summary of the Invention

[0004] This invention provides an image emotion recognition method and system based on human-computer interaction, which solves the technical problems mentioned in the background.

[0005] This invention provides an image emotion recognition method based on human-computer interaction, comprising the following steps:

[0006] Step S101: Perform face detection and key point localization on the video frame sequence, align the face image to the standard coordinate system, and map the brow muscle mask in the standard coordinate system back to each video frame to extract the corresponding brow muscle region image sequence.

[0007] Step S102: Extract the dynamic displacement signal from the image sequence of the brow muscle region and convert it into an acceleration sequence. Locate the first global peak moment of the acceleration sequence within the analysis window starting from the interaction event, calculate the time difference between the interaction event and the first global peak moment, and obtain the event anchoring phase lag.

[0008] Step S103: Using the first global peak moment as the phase zero point, perform equidistant sampling on the image sequence of the brow muscle region before and after the phase zero point to construct a time-normalized input sequence;

[0009] Step S104: Perform frame-by-frame visual encoding on the time-normalized input sequence, generate gating coefficients based on the event anchor phase lag, modulate the visual feature amplitude, and perform temporal convolution under the action of a time-weighted window to obtain a temporal feature sequence.

[0010] Step S105: Perform center weighted aggregation with phase zero as the core on the time-series feature sequence to obtain the center representation vector, calculate the feature mean before and after phase zero respectively and construct the difference comparison vector, and fuse the center representation vector and the difference comparison vector to generate the global representation vector.

[0011] Step S106: Input the global representation vector into the classification network layer, map it, and output discrete emotion category labels.

[0012] This invention provides an image emotion recognition system based on human-computer interaction, comprising:

[0013] The brow muscle region image cropping module performs face detection and key point localization on the video frame sequence, aligns the face image to the standard coordinate system, and maps the brow muscle mask in the standard coordinate system back to each video frame to crop the corresponding brow muscle region image sequence.

[0014] The phase lag determination module extracts dynamic displacement signals from the image sequence of the brow muscle region and converts them into acceleration sequences. Within the analysis window starting from the interaction event, it locates the first global peak moment of the acceleration sequence, calculates the time difference between the interaction event and the first global peak moment, and obtains the event anchoring phase lag.

[0015] The input sequence construction module takes the first global peak moment as the phase zero point, and performs equidistant sampling on the image sequence of the brow muscle region before and after the phase zero point to construct a time-normalized input sequence.

[0016] The temporal convolution module performs frame-by-frame visual encoding on the time-normalized input sequence, generates gating coefficients based on the event anchor phase lag, modulates the amplitude of visual features, and performs temporal convolution under the action of a time-weighted window to obtain a temporal feature sequence.

[0017] The global representation vector generation module performs a central weighted aggregation with the phase zero point as the core on the temporal feature sequence to obtain a central representation vector. It calculates the feature mean before and after the phase zero point and constructs a difference comparison vector. The central representation vector and the difference comparison vector are then fused to generate a global representation vector.

[0018] The emotion category output module takes the global representation vector as input to the classification network layer, maps it, and outputs discrete emotion category labels.

[0019] The beneficial effects of this invention are as follows: This invention achieves unified extraction of the brow muscle region from different frames and samples through facial key point localization and similarity transformation, avoiding interference from redundant information; by extracting the acceleration sequence of brow muscle movement and anchoring the temporal correlation between interactive events and peak moments, a time-normalized input sequence is constructed using phase zero points, thereby overcoming the problem of temporal misalignment between emotional responses and interactive events. Combining the gating coefficients generated by event-anchored phase lag with a Gaussian time weighted window, feature amplitude modulation and key information enhancement are achieved in temporal convolution, reducing feature fluctuations caused by individual differences. Further, through center-weighted aggregation and differential contrast vector fusion, the completeness and discriminativeness of feature expression are improved. Ultimately, this improves the accuracy and stability of emotion recognition in human-computer interaction scenarios, providing more reliable emotion recognition support for applications such as smart terminals and service robots. Attached Figure Description

[0020] Figure 1 This is a flowchart of an image emotion recognition method based on human-computer interaction according to the present invention;

[0021] Figure 2 This is a schematic diagram of an image emotion recognition system based on human-computer interaction according to the present invention;

[0022] Figure 3 This is the emotion recognition confusion matrix diagram of the present invention.

[0023] In the figure: brow muscle region image cropping module 201, phase lag determination module 202, input sequence construction module 203, temporal convolution module 204, global representation vector generation module 205, emotion category output module 206. Detailed Implementation

[0024] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0025] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of the present invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of the present invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" indicate that the element or object preceding the term encompasses the elements or objects listed following the term and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0026] like Figures 1-3 As shown, an image emotion recognition method based on human-computer interaction includes the following steps:

[0027] Step S101: Perform face detection and key point localization on the video frame sequence, align the face image to the standard coordinate system, and map the brow muscle mask in the standard coordinate system back to each video frame to extract the corresponding brow muscle region image sequence.

[0028] Step S102: Extract the dynamic displacement signal from the image sequence of the brow muscle region and convert it into an acceleration sequence. Locate the first global peak moment of the acceleration sequence within the analysis window starting from the interaction event, calculate the time difference between the interaction event and the first global peak moment, and obtain the event anchoring phase lag.

[0029] Step S103: Using the first global peak moment as the phase zero point, perform equidistant sampling on the image sequence of the brow muscle region before and after the phase zero point to construct a time-normalized input sequence;

[0030] Step S104: Perform frame-by-frame visual encoding on the time-normalized input sequence, generate gating coefficients based on the event anchor phase lag, modulate the visual feature amplitude, and perform temporal convolution under the action of a time-weighted window to obtain a temporal feature sequence.

[0031] Step S105: Perform center weighted aggregation with phase zero as the core on the time-series feature sequence to obtain the center representation vector, calculate the feature mean before and after phase zero respectively and construct the difference comparison vector, and fuse the center representation vector and the difference comparison vector to generate the global representation vector.

[0032] Step S106: Input the global representation vector into the classification network layer, map it, and output discrete emotion category labels.

[0033] In one embodiment of the present invention, face detection and key point localization are performed on a video frame sequence, the face images are aligned to a standard coordinate system, and the brow muscle mask in the standard coordinate system is mapped back to each video frame to obtain the corresponding brow muscle region image sequence, including:

[0034] For each frame of image Face detection and keypoint regression are performed to obtain a keypoint set. ,in The first in the video frame sequence Frame image, This is the set of facial keypoint coordinates defined in the image coordinate system of this frame;

[0035] based on Solve similarity transformations To align to the standard coordinate system, For geometric mappings that include rotation, scaling, and translation;

[0036] The eyebrow muscle mask in the standard coordinate system Map back to the first through inverse transformation Frame, obtain the brow muscle region Its formula is:

[0037] ,in To fix the binary mask, for The inverse mapping, For the first The set of pixels corresponding to a frame;

[0038] according to For the The image of the brow muscle region is obtained by cropping the frame image. Its formula is:

[0039] ,in For the first Image of the brow muscle region in the frame. Represents a set by pixels The smallest bounding rectangle is used to cut and retain the content.

[0040] It should be noted that face alignment is achieved through keypoint regression and similarity transformation, and then the brow muscle region is locked through mask inverse mapping and cropping. This process ensures that the brow muscle localization is consistent across different frames and samples, avoiding regional offset and redundant information interference caused by pose and distance.

[0041] It should be noted that the video frame sequence represents a collection of consecutive image frames arranged chronologically, serving as the raw input data for emotion recognition and containing multi-frame facial image information. The facial landmark coordinate set represents a combination of coordinates obtained through a facial landmark regression algorithm to describe key facial locations (such as feature points of eyebrows, eyes, and nose). Each coordinate corresponds to a specific location in the image coordinate system, and all coordinates together form the key facial structure. The frame image coordinate system represents the coordinate system of each frame image itself, with the top-left corner of the image as the origin, the horizontal axis pointing to the right as the positive direction, and the vertical axis pointing downwards as the positive direction. It is used to locate the specific position of each pixel in the image. The standard coordinate system represents a pre-defined unified facial coordinate system with fixed scale, orientation, and origin. It is used to eliminate differences in pose and distance between faces in different frames, providing a unified comparison benchmark for faces in each frame.

[0042] It should be noted that the similarity transformation is used to adjust the pose, size, and position of the face in each frame of the image, aligning it with the face structure in the standard coordinate system. The fixed binary brow muscle mask represents a pre-defined binary image template in the standard coordinate system used to select the brow muscle region. Pixels belonging to the brow muscle region within the template are marked as 1, and pixels not belonging to the brow muscle region are marked as 0. The shape, size, and position of the template remain fixed. The inverse mapping of the similarity transformation represents a geometric mapping operation in the opposite direction to the similarity transformation. It is used to reverse the fixed binary brow muscle mask in the standard coordinate system to the coordinate system of each frame of the image, achieving the correspondence between the brow muscle region and the original image coordinates. The set of pixels corresponding to the brow muscle represents the set of all pixels marked as 1 obtained after mapping the fixed binary brow muscle mask to the original frame image coordinate system through the inverse mapping of the similarity transformation, corresponding to the brow muscle region in the original frame image. The minimum bounding rectangle of the pixel set represents the smallest rectangle that can completely contain the set of pixels corresponding to the brow muscle. The four sides of the rectangle are parallel to the horizontal and vertical axes of the image coordinate system, respectively, and are used to crop out the brow muscle region. The brow muscle region image represents the image obtained by cropping the original frame image according to the smallest bounding rectangle of the pixel set, and contains only the visual information of the brow muscle region.

[0043] Specifically, the similarity transformation first calculates the scale ratio between the set of facial keypoint coordinates in each frame and the corresponding set of keypoints in the standard coordinate system. The scale ratio is equal to the average distance between corresponding keypoints in the standard coordinate system divided by the average distance between facial keypoints in the original frame. Next, the rotation angle is calculated, which is determined by the angle between the direction vectors of the two sets of keypoints, and the average of the angles between the direction vectors of all corresponding keypoints is taken. Then, the translation is calculated, which is equal to the center coordinates of the corresponding set of keypoints in the standard coordinate system minus the center coordinates of the set of facial keypoints in the original frame after rotation and scaling. Finally, using the scale ratio, rotation angle, and translation obtained above, a similarity transformation is constructed to adjust the face in the original frame image.

[0044] Specifically, the inverse mapping of the similarity transformation first takes the reciprocal of the scale ratio in the similarity transformation as the scale ratio of the inverse mapping; takes the opposite of the rotation angle in the similarity transformation as the rotation angle of the inverse mapping; takes the value of the translation amount in the similarity transformation after inverse rotation (opposite of the rotation angle) and inverse scaling (reciprocal of the scale ratio), and takes its opposite as the translation amount of the inverse mapping; and constructs the inverse mapping of the similarity transformation using the scale ratio, rotation angle, and translation amount of the inverse mapping obtained above.

[0045] It should be noted that face detection and keypoint localization can be achieved using models such as FaceNet and RetinaFace. FaceNet improves the robustness of keypoint regression by learning the deep features of the face. This algorithm outputs 68 keypoints, which highly match the feature point distribution in the standard coordinate system. The keypoint regression module included with RetinaFace can output 5 or 106 keypoints. The 106-keypoint version can more finely depict the local structure of the face, including subtle feature points of the brow muscle, and is suitable for scenarios requiring precise localization of the brow muscle region. The selection of the corresponding keypoint set in the standard coordinate system needs to cover the key facial structures, and typically 68 feature points are selected. These feature points are evenly distributed in areas such as the jaw, eyebrows, eyes, nose, and mouth of the face, which will not be elaborated on here. Furthermore, in the standard coordinate system, the horizontal range of the brow muscle region is from the x-coordinate of the 17th feature point (starting point of the left eyebrow) to the x-coordinate of the 27th feature point (ending point of the right eyebrow), and the vertical range is from the y-coordinate of the 19th feature point (5 pixels above the midpoint of the left eyebrow) to the y-coordinate of the 26th feature point (3 pixels below the midpoint of the right eyebrow). Pixels within this range in the mask are marked as 1, and pixels outside the range are marked as 0.

[0046] In one embodiment of the present invention, dynamic displacement signals are extracted from the image sequence of the brow muscle region and converted into acceleration sequences. The first global peak moment of the acceleration sequence is located within an analysis window starting from an interaction event. The time difference between the interaction event and the first global peak moment is calculated to obtain the event anchoring phase lag. This includes: in the brow muscle region... Calculate optical flow ,in For horizontal displacement, For vertical displacement, the displacement sequence is obtained:

[0047] ,in For vertical displacement, For gradient intensity, The weighted average vertical displacement;

[0048] Acceleration sequence obtained from displacement sequence: ,in For frame rate, For acceleration values; set the analysis set: ;

[0049] And determine the moment of the first global peak: ,in Frames for interactive events To analyze the window length, This marks the first global peak moment.

[0050] Calculate the event anchor phase lag: ,in Anchor phase lag for the event. and These are the initial global peak time and the interaction event time, respectively. For frame rate.

[0051] It should be noted that by aggregating displacement signals with gradient intensity as weight, obtaining acceleration sequences through discrete second-order differences, and combining this with the analysis window to locate peak moments, noise interference is effectively reduced, enabling the acceleration sequences to truly reflect the movement state of the brow muscle. The phase lag can accurately reflect the temporal correlation between interactive events and brow muscle contraction, providing reliable time parameters for subsequent phase alignment and encoding, thereby enhancing the relevance and effectiveness of emotion recognition.

[0052] It should be noted that optical flow describes the movement trajectory of pixels in two adjacent frames, reflecting the positional change trend of pixels within the brow muscle region, and includes displacement information in both horizontal and vertical directions. Gradient intensity represents the degree of change in the grayscale value of image pixels; pixels with rich edges and textures within the brow muscle region have higher gradient intensities. Vertical displacement represents the distance a pixel moves along the vertical direction (up and down) of the image between two adjacent frames, with downward being the positive direction, and is associated with the downward movement of the brow muscle during contraction. The weighted average vertical displacement sequence represents a sequence formed by weighting the vertical displacement of all pixels within the brow muscle region in each frame using gradient intensity as the weight, and arranging them in frame time order, which can stably reflect the overall vertical movement trend of the brow muscle. Discrete second-order difference represents a second-order difference operation performed on the discrete weighted average vertical displacement sequence to convert the displacement signal into an acceleration signal, reflecting the rate of change of the velocity of the brow muscle movement. The acceleration sequence represents a sequence obtained through discrete second-order difference, where each element corresponds to the magnitude of acceleration of the brow muscle movement in one frame, and can capture the dynamic characteristics of rapid brow muscle contraction. The interaction event occurrence frame represents the specific video frame in which a key event occurs during human-computer interaction, serving as a time reference starting point. The set of discrete frames represents a continuous set of video frames defined by a set length, starting with the interaction event occurrence frame. The moment of the first global maximum value represents the video frame moment corresponding to the first occurrence of the global maximum value in the acceleration sequence corresponding to the set of discrete frames, marking the initiation node of the rapid contraction of the brow muscle. The frame difference represents the difference between the frame number corresponding to the moment of the first global maximum value and the frame number of the interaction event occurrence frame. The frame rate represents the number of image frames per second in the video. The event anchoring phase lag represents the actual time difference between the occurrence of the interaction event and the initiation of the rapid contraction of the brow muscle.

[0053] It should be noted that the Lucas-Cannard algorithm is used for optical flow calculation. This algorithm is suitable for displacement estimation in local motion regions and fits the local motion characteristics of the brow muscle region. In actual calculation, a 15x15 pixel rectangular window is set. The assumption that the pixel grayscale value within the window is constant is valid, which can effectively reduce noise interference. The number of iterations is set to 5. In each iteration, the displacement vector of the pixel within the window is adjusted until the displacement change is less than 0.01 pixels to ensure the stability of displacement estimation. The gradient intensity is calculated using the Sobel operator, which contains convolution kernels in both horizontal and vertical directions. The horizontal convolution kernel values ​​are -1, 0, 1 for the first row, -2, 0, 2 for the second row, and -1, 0, 1 for the third row. The vertical convolution kernel values ​​are -1, -2, -1 for the first row, 0, 0, 0 for the second row, and 1, 2, 1 for the third row. During calculation, the two convolution kernels are convolved with the image pixel grayscale value matrix to obtain the horizontal and vertical gradient components. The two gradient components are squared and summed. The square root of the sum is then taken to obtain the gradient intensity of the pixel. The image pixel grayscale values ​​range from 0 to 255 to ensure the consistency of the values ​​during the calculation process.

[0054] It should be noted that the default analysis window length is 3 seconds, with a range of 2 to 5 seconds. This value is based on statistical data from a large number of human-computer interaction scenarios. In most cases, users will exhibit a brow muscle emotional response within 1 to 3 seconds after an interaction event occurs, and the default length of 3 seconds can cover the response process in most scenarios. If the acceleration sequence shows multiple identical global maximum values ​​within the analysis discrete frame set, the rule is to select the moment corresponding to the maximum value with the smallest frame number. For example, if the analysis discrete frame set includes frames 20 to 80, and the acceleration values ​​in frames 30 and 31 are both global maximum values, then the moment corresponding to frame 30 is selected as the moment of the first global maximum value, ensuring that the earliest brow muscle rapid contraction initiation node is captured.

[0055] In one embodiment of the present invention, the first global peak moment is used as the phase zero point, and isometric sampling is performed on the image sequence of the brow muscle region before and after the phase zero point to construct a time-normalized input sequence, including: taking the first global peak moment as the phase zero point. Set as phase zero;

[0056] Let the length of the time-normalized input sequence be... For sampling location variables Pick to Define equidistant integer displacements : ,in This is the floor function;

[0057] Calculate the sampling frame position : ,in This represents the total number of frames in the video frame sequence. Indicates the real number Restricted to closed intervals Inside;

[0058] By sampling frame position Image sequence in the brow muscle region Sampling is performed to obtain the first time-normalized input sequence. element : .

[0059] It should be noted that by taking the moment of the first peak of acceleration as the phase zero point, sampling is ensured through equidistant sampling and effective frame range limitation, thus achieving alignment of different samples on the time axis, eliminating the feature dilution problem caused by phase misalignment, and thereby improving the consistency and effectiveness of temporal features.

[0060] It should be noted that the first global peak moment of the acceleration sequence represents the video frame moment corresponding to the first occurrence of the global maximum value in the acceleration sequence corresponding to the set of discrete frames being analyzed; this is the starting point of the rapid contraction of the brow muscle. The phase zero point represents a time reference point set based on the first global peak moment of the acceleration sequence, used to unify the time axis of different samples. The length of the time-normalized input sequence represents the fixed number of image frames subsequently used as model input, serving as the quantitative basis for equidistant sampling. The sampling position variable represents a variable that iterates through the length of the time-normalized input sequence, with each variable corresponding to a sampling position. The equidistant integer displacement relative to the length center represents the integer displacement value corresponding to each sampling position variable with the length center of the time-normalized input sequence as the symmetrical point, ensuring that the sampling is symmetrically distributed before and after the phase zero point. The effective frame number range of the video frame sequence represents the range of the actual effective image frame numbers contained in the video, used to limit the sampling frame position from exceeding the actual frame range of the video. The sampling frame position represents the specific video frame number obtained by superimposing the equidistant integer displacement and the phase zero point, and then limiting it to the effective range; this is the basis for extracting the brow muscle region image. The brow muscle region image sequence represents a set of brow muscle region images arranged in frame-time order. The time-normalized input sequence represents a fixed-length sequence formed by extracting images from the brow muscle region image sequence according to the sampling frame position and arranging them in order of sampling position variable, which is used for subsequent visual encoding.

[0061] It should be noted that the default length of the time-normalized input sequence is 32 frames, with a range of 16 to 64 frames. This value is based on a balance between model encoding efficiency and feature capture completeness. 32 frames can fully cover the dynamic process of the brow muscle from contraction initiation to a steady state, while avoiding computational redundancy caused by too many frames. For scenarios with high real-time requirements, 16 frames can be selected; for scenarios that need to capture more detailed motion features, 64 frames can be selected.

[0062] In one embodiment of the present invention, a frame-by-frame visual encoding is performed on the time-normalized input sequence, gating coefficients are generated based on the event anchoring phase lag, the visual feature amplitude is modulated, and temporal convolution is performed under the action of a time-weighted window to obtain a temporal feature sequence, including:

[0063] The first time-normalized input sequence element Calculate frame-level feature vectors :

[0064] ,in For frame-level visual encoders, For feature dimensions;

[0065] Based on event anchoring phase hysteresis Generate gating coefficients :

[0066] ,in For Sigmoid mapping, and For fixed parameters;

[0067] Constructing a Gaussian time weighted window : ,in For a fixed parameter on the time scale, Input length;

[0068] Perform temporal convolution under a time-weighted window and output the first element in the temporal feature sequence. Features : ,in The kernel length is a fixed parameter. For the first Each convolutional kernel weight tensor Indicates multiplication by channel. This is the bias vector.

[0069] It should be noted that by extracting high-dimensional visual features through frame-level coding, highlighting key information near the phase zero point through Gaussian time weighted windows, unifying the feature amplitude of different samples using gating coefficients, and integrating time weights and gating modulation in the temporal convolution process, the temporal dependence of brow muscle movement can be effectively captured, reducing feature fluctuations caused by individual differences. The generated temporal features have both visual information and dynamic change attributes, thereby improving the stability and discriminativeness of the features.

[0070] It should be noted that the frame-level visual encoder is used to extract features from a single frame of the brow muscle region image, converting image pixel information into high-dimensional feature vectors to capture key visual information such as texture and edges. The frame-level feature vector sequence represents the set of vectors formed by encoding each element in the time-normalized input sequence by the frame-level visual encoder and arranging them in their original order; each vector corresponds to a high-dimensional feature of one frame of the image. The linear transformation represents a linear scaling and translation of the event-anchored phase lag. The sigmoid function mapping is a non-linear mapping function with output values ​​between 0 and 1. The scalar gating coefficient represents a single value obtained from the event-anchored phase lag through the linear transformation and sigmoid function mapping, used to modulate the amplitude of the temporal convolution result. The center position of the input sequence length represents the index of the position corresponding to the midpoint of the time-normalized input sequence length, and is the center of symmetry for constructing the Gaussian time weight window.

[0071] It should be noted that the Gaussian temporal weight window represents a set of weights constructed around the center of the input sequence length. The weight values ​​follow a Gaussian distribution with increasing distance from the center, highlighting feature weights near the phase zero. The temporal weight represents a single weight value within the Gaussian temporal weight window, with one weight corresponding to each position, used to weight the feature vectors of the corresponding frame in temporal convolution. The convolution kernel represents the weight tensor used in the temporal convolution operation to extract the temporal dependencies of the frame-level feature vector sequence. Temporal convolution represents the convolution operation performed on the frame-level feature vector sequence in the time dimension, combining temporal weights to achieve selective extraction of temporal features. Amplitude modulation represents the operation of multiplying the temporal convolution result by scalar gating coefficients, used to adjust the amplitude of the feature vectors, reducing feature fluctuations caused by differences in phase lag between different samples. The bias vector represents a vector with the same dimension as the convolution result, used to offset and adjust the feature vectors after convolution and amplitude modulation. The temporal feature sequence represents the feature vector sequence obtained after frame-level coding, temporal convolution, amplitude modulation, and bias superposition, which integrates visual features and temporal dynamic information.

[0072] It should be noted that the frame-level visual encoder uses a convolutional neural network, and the network structure includes 3 convolutional layers and 2 fully connected layers. The first convolutional layer uses 32 3x3 kernels with a stride of 1 and padding of 1. The second convolutional layer uses 64 3x3 kernels with a stride of 1 and padding of 1. The third convolutional layer uses 128 3x3 kernels with a stride of 1 and padding of 1. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The first fully connected layer has a height dimension of 256, and the second layer has a dimension of 128, i.e., the feature dimension is 128. The network uses He initialization to initialize the convolutional kernels and fully connected layer weights, and the Adam optimizer is used during training with a learning rate of 0.001.

[0073] It should be noted that the two fixed parameters of the linear transformation are set to 1.2 and 0.5, respectively. These values ​​are based on a large number of statistical samples. The common range of event-anchored phase lag is 0.1 seconds to 2.0 seconds. After this linear transformation, the result range is mapped to 0.62 to 2.9, fitting the sensitive response range of the sigmoid function (0 to 3), ensuring that the scalar gating coefficients can effectively distinguish different phase lag situations. The time scale parameter of the Gaussian time weight window is set to 2.5. This value, verified through experiments, ensures that the weight at the center position is 1, and the weights at positions 3 positions away from the center decay to below 0.1. This highlights the features near the center (the neighborhood of the phase zero point) while also covering a certain time range, capturing the correlation information between consecutive frames. The convolutional kernel length is set to 3, with the dimension consistent with the frame-level feature vector dimension (128 dimensions). Each convolutional kernel contains three 128x128 weight matrices. The convolutional kernels use Xavier initialization to ensure numerical stability during forward and backward propagation, avoiding gradient vanishing or exploding. The bias vector dimension is consistent with the frame-level feature vector dimension (128 dimensions). It adopts constant initialization, with all elements initially set to 0.1, providing a stable initial offset for model training and accelerating convergence.

[0074] In one embodiment of the present invention, a central weighted aggregation with phase zero as the core is performed on the temporal feature sequence to obtain a central representation vector. The feature mean before and after the phase zero is calculated respectively, and a difference comparison vector is constructed. The central representation vector and the difference comparison vector are fused to generate a global representation vector, including:

[0075] Computational central representation vector :

[0076]

[0077] in The first in the time series feature sequence One characteristic, The corresponding Gaussian time window weights are used for the positions. Input length;

[0078] Define the weights before and after the phase zero point and :

[0079] ; ;

[0080] And calculate the weighted mean vector before and after the phase zero point. and :

[0081] ; ;

[0082] in Central index, and These are the floor function and the floor function, respectively.

[0083] Constructing the difference contrast vector : ;

[0084] Perform fixed-form fusion to generate global representation vectors : .

[0085] It should be noted that by using center-weighted aggregation to retain the core features of the phase zero point, dividing the front and back sides to calculate the mean vector and constructing the difference comparison vector, the two are finally fused; this allows the global representation vector to simultaneously contain the core features and the information on the changes in features on the front and back sides, reflecting the key attributes of the brow muscle movement and improving the feature expression ability.

[0086] It should be noted that the center representation vector is the vector obtained by weighting and aggregating the temporal feature sequences using a Gaussian time weight window. The center of the input length represents the midpoint of the time-normalized input sequence, serving as the boundary between the areas before and after the phase zero point, determining the selection range of features before and after it. The weighted sum before the phase zero point represents the sum of the time weights of all positions before the center of the input length within the Gaussian time weight window, used to normalize the weighted mean vector before the phase zero point. The weighted sum after the phase zero point represents the sum of the time weights of all positions after the center of the input length within the Gaussian time weight window, used to normalize the weighted mean vector after the phase zero point. The weighted mean vector before the phase zero point represents the vector obtained by weighting and summing the temporal features before the center of the input length using a Gaussian time weight window, and then dividing it by the corresponding weighted sum, reflecting the feature statistical information before the phase zero point. The weighted mean vector following the phase zero represents the temporal features after the input length center. After being weighted and summed using a Gaussian time window, it is divided by the corresponding weighted sum to obtain the vector, reflecting the statistical information of the features after the phase zero. The difference contrast vector is the vector obtained by subtracting the weighted mean vector following the phase zero from the weighted mean vector following the phase zero, highlighting the feature differences before and after the phase zero. The global representation vector is the vector obtained by superimposing the center representation vector and the difference contrast vector, fusing core features with information about the differences before and after the phase zero.

[0087] It should be noted that when the input length is odd, subtracting 1 from the input length and dividing by 2 gives the center position index. Positions before this index are less than the center index, and positions after it are greater than or equal to the center index. When the input length is even, subtracting 1 from the input length and dividing by 2 gives the center position index, rounded down. Positions before this index are less than the center index, and positions after it are greater than or equal to the center index. For example, if the input length is 32 (even), 32 minus 1 equals 31, 31 divided by 2 equals 15.5, rounded down to 15, with positions before it ranging from 0 to 14 and positions after it ranging from 15 to 31. If the input length is 33 (odd), 33 minus 1 equals 32, 32 divided by 2 equals 16, with positions before it ranging from 0 to 15 and positions after it ranging from 16 to 32.

[0088] In one embodiment of the present invention, inputting a global representation vector into a classification network layer, mapping and outputting discrete emotion category labels includes: calculating the real-valued vector output of the classification network layer. :

[0089] ,in For the weight matrix of the classification network layer, For bias vectors, This is a global representation vector;

[0090] Calculate the discrete emotion category probability distribution vector The Each component :

[0091]

[0092] in The first real number vector output One portion, For the number of discrete emotion categories, For exponentiation;

[0093] Output discrete emotion category labels : ,in This indicates the index operation corresponding to the maximum value.

[0094] It should be noted that the weight matrix of the classification network layer represents the core parameter matrix in the classification network layer, used to linearly map the global representation vector to the emotion category dimension. The bias vector represents the offset parameter vector in the classification network layer, used to adjust the output result after linear mapping. The real vector output of the classification network layer represents the real vector obtained by mapping the global representation vector to the weight matrix and summing the bias vector, with each component corresponding to the original response value of an emotion category. The discrete emotion category probability distribution vector represents the vector obtained by normalizing the result of the exponential operation, with each component taking a value between 0 and 1, and a sum of 1, reflecting the probability that the input belongs to the corresponding emotion category. The discrete emotion category label represents the index corresponding to the component with the highest probability in the probability distribution vector, which is the emotion category identifier of the final output.

[0095] It should be noted that the default number of discrete emotion categories is 5, specifically including confusion, calmness, joy, surprise, and anger. This classification is based on the most common user emotional feedback in human-computer interaction scenarios. The weight matrix uses Xavier initialization, randomly initializing the weight values ​​by calculating the reciprocal of the square root of the mean of the input and output dimensions to ensure numerical stability during forward propagation. The bias vector uses constant initialization, with all elements initially set to 0.01, providing a smooth initial bias for model training and accelerating convergence. Furthermore, to avoid overflow of the exponential operation results of larger components in the real number vector output, a temperature coefficient of 0.8 is introduced to scale the real number vector output. During calculation, each component of the real number vector output is first divided by the temperature coefficient before performing the exponential operation, ensuring that the values ​​are within the computer's processing range without changing the relative magnitudes of the components; this will not be elaborated further here. If there are two or more maximum probability components with the same value in the discrete emotion category probability distribution vector, the index corresponding to the component with the smallest index is selected as the discrete emotion category label; for example, if the components at index 2 and index 4 in the probability distribution vector are both the maximum value of 0.35, then the emotion category corresponding to index 2 is selected as the output result.

[0096] In one embodiment of the present invention, such as Figure 2 As shown, an image emotion recognition system based on human-computer interaction includes:

[0097] The brow muscle region image cropping module 201 performs face detection and key point localization on the video frame sequence, aligns the face image to the standard coordinate system, and maps the brow muscle mask in the standard coordinate system back to each video frame to crop and obtain the corresponding brow muscle region image sequence.

[0098] The phase lag determination module 202 extracts dynamic displacement signals from the image sequence of the brow muscle region and converts them into acceleration sequences. Within the analysis window starting from the interaction event, it locates the first global peak moment of the acceleration sequence, calculates the time difference between the interaction event and the first global peak moment, and obtains the event anchoring phase lag.

[0099] The input sequence construction module 203 takes the first global peak moment as the phase zero point, and performs equidistant sampling on the image sequence of the brow muscle region before and after the phase zero point to construct a time-normalized input sequence.

[0100] The temporal convolution module 204 performs frame-by-frame visual encoding on the time-normalized input sequence, generates gating coefficients based on the event anchor phase lag, modulates the amplitude of visual features, and performs temporal convolution under the action of a time-weighted window to obtain a temporal feature sequence.

[0101] The global representation vector generation module 205 performs a central weighted aggregation with the phase zero point as the core on the temporal feature sequence to obtain a central representation vector. It calculates the feature mean before and after the phase zero point and constructs a difference comparison vector. The central representation vector and the difference comparison vector are then fused to generate a global representation vector.

[0102] The emotion category output module 206 inputs the global representation vector into the classification network layer, maps it, and outputs discrete emotion category labels.

[0103] It should be noted that, as Figure 3 The diagram shows the confusion matrix generated by this invention on the test dataset. The horizontal axis represents the emotion category label predicted by the model, and the vertical axis represents the true category label of the sample. The values ​​on the main diagonal of the matrix represent the number of samples correctly identified as belonging to that category by the model; a larger value indicates a better recognition performance for that emotion category. The values ​​off the main diagonal represent the number of samples incorrectly predicted as other categories by the model, reflecting the feature confusion between different emotion categories. The diagram includes five emotion categories: confusion, calmness, joy, surprise, and anger.

[0104] It should be noted that this invention can be applied to primary and secondary school classroom teaching. Teachers often find it difficult to simultaneously monitor the attention of every student, and some students are prone to inattentiveness, drowsiness, and other issues that affect learning outcomes. The system can be deployed with distributed cameras at the front and sides of the classroom, covering the entire teaching area. During teacher instruction and questioning, the system captures real-time video frame sequences of students. The system automatically extracts the brow muscle region, analyzes acceleration sequences, and recognizes emotions, simultaneously identifying students' attention spans, inattentiveness, drowsiness, and restlessness, and feeding the results back to the teacher's teaching terminal in real time. Teachers can adjust the teaching pace based on the feedback, using interactive questioning and engaging explanations to reawaken the attention of inattentive students. Simultaneously, the system records students' attention fluctuations throughout the lesson, generating class and individual attention reports after class. This helps teachers understand the appeal of the teaching content and provides a basis for optimizing instructional design and providing personalized tutoring. Furthermore, when students are at home or doing homework, they are easily distracted and impatient due to external interference and the difficulty of the material, affecting the efficiency and quality of their work. The system is deployed on a smart terminal or independent camera on the student's learning desktop, capturing facial images non-contactly during the student's interaction with questions. The system recognizes the student's emotions in real time, and when it detects negative emotions such as persistent distraction, irritability, or perfunctory work, it automatically sends a reminder through the associated parent terminal. Parents can communicate in a timely manner to understand the cause and help the student regain focus through methods such as answering questions and adjusting the learning environment.

[0105] It should be noted that this invention can be applied to patient rehabilitation training. Rehabilitation patients often experience negative emotions due to long treatment cycles, high training intensity, or pain reactions, which negatively impacts rehabilitation compliance and treatment effectiveness. The system can be deployed in the rehabilitation training room, using distributed cameras to cover the training area and non-contactly capture facial images of patients during limb and speech rehabilitation training. The system identifies the patient's emotional state in real time, automatically sending a reminder to the therapist's terminal device when persistent negative emotions such as depression or irritability are detected. The therapist can then promptly pause training, alleviate the patient's emotions through communication, or adjust the training intensity and method. Simultaneously, the system automatically records the emotional fluctuation curve throughout the training process, combining it with data such as training completion and physiological indicators to provide a reference for developing personalized rehabilitation plans. For example, for stroke rehabilitation patients, by analyzing the correlation between emotions and training effects, the training rhythm can be optimized, improving patient motivation and shortening the rehabilitation cycle.

[0106] It should be noted that the interval and threshold sizes are set for ease of comparison. The size of the threshold depends on the amount of sample data and the base number set by those skilled in the art for each set of sample data, as long as it does not affect the proportional relationship between the parameter and the quantized value. Furthermore, the above formulas are all dimensionless calculations, and the formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0107] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A human-computer interaction-based image emotion recognition method, characterized in that, Includes the following steps: Step S101: Perform face detection and key point localization on the video frame sequence, align the face image to the standard coordinate system, and map the brow muscle mask in the standard coordinate system back to each video frame to extract the corresponding brow muscle region image sequence. Step S102: Extract the dynamic displacement signal from the image sequence of the brow muscle region and convert it into an acceleration sequence. Locate the first global peak moment of the acceleration sequence within the analysis window starting from the interaction event, calculate the time difference between the interaction event and the first global peak moment, and obtain the event anchoring phase lag. Displacement signals are aggregated using gradient intensities as weights, and acceleration sequences are obtained through discrete second-order difference, including those in the brow muscle region. Calculate optical flow ,in For horizontal displacement, For vertical displacement, the displacement sequence is obtained: in For vertical displacement, For gradient intensity, The weighted average vertical displacement. The frame number index of the video frame sequence; Acceleration sequence obtained from displacement sequence: ,in For frame rate, This is the acceleration value; The acceleration sequence represents the sequence obtained through discrete second-order difference, with each element corresponding to the magnitude of acceleration of the brow muscle movement in one frame; Step S103: Using the first global peak moment as the phase zero point, perform equidistant sampling on the image sequence of the brow muscle region before and after the phase zero point to construct a time-normalized input sequence; Step S104: Perform frame-by-frame visual encoding on the time-normalized input sequence, generate gating coefficients based on the event anchor phase lag, modulate the visual feature amplitude, and perform temporal convolution under the action of a time-weighted window to obtain a temporal feature sequence. Step S105: Perform center weighted aggregation with phase zero as the core on the time-series feature sequence to obtain the center representation vector, calculate the feature mean before and after phase zero respectively and construct the difference comparison vector, and fuse the center representation vector and the difference comparison vector to generate the global representation vector. Step S106: Input the global representation vector into the classification network layer, map it, and output discrete emotion category labels.

2. The image emotion recognition method based on human-computer interaction according to claim 1, characterized in that, Perform face detection and keypoint regression on each frame of the video frame sequence to obtain the set of face keypoint coordinates defined in the coordinate system of that frame image; Using the set of facial keypoint coordinates and the corresponding set of keypoints in the standard coordinate system as constraints, solve the similarity transformation including rotation, scaling and translation, and align the face of the frame to the standard coordinate system; The fixed binary eyebrow muscle mask given in the standard coordinate system is transformed back to the image coordinate system of the frame through the inverse mapping of similarity transformation, so as to obtain the set of pixels corresponding to the eyebrow muscle in the frame. Cropping and preserving the brow muscle region image is performed within the frame image based on the smallest bounding rectangle of the pixel set.

3. The image emotion recognition method based on human-computer interaction according to claim 1, characterized in that, Optical flow is calculated in the brow muscle region, and vertical displacements in the brow muscle region are aggregated with gradient intensity as weight to obtain a weighted average vertical displacement sequence. Discretize the weighted average vertical displacement sequence into a second-order difference to obtain the acceleration sequence.

4. The image emotion recognition method based on human-computer interaction according to claim 3, characterized in that, Define a set of discrete analysis frames starting from the frame where the interaction event occurs, and determine the moment when the acceleration sequence first reaches its global maximum value within this set; The event anchoring phase lag is calculated by combining the frame difference between the moment when the global maximum value is first obtained and the frame in which the interactive event occurs, and the frame rate.

5. The image emotion recognition method based on human-computer interaction according to claim 1, characterized in that, Set the first global peak moment of the acceleration sequence as the phase zero point; Set the length of the time-normalized input sequence, and for each sampling position variable within this length, calculate its equidistant integer displacement relative to the center of the length; The sampling frame position is obtained by superimposing equidistant integer displacements onto the phase zero point and limiting the result to the effective frame number range of the video frame sequence. Based on the sampling frame position, the corresponding image is extracted from the image sequence of the brow muscle region to form a time-normalized input sequence.

6. The image emotion recognition method based on human-computer interaction according to claim 1, characterized in that, Each element in the time-normalized input sequence is encoded by a frame-level visual encoder to obtain a frame-level feature vector sequence. The event anchor phase lag is linearly transformed and then input into a sigmoid function to generate scalar gating coefficients; a Gaussian time weight window is constructed around the center of the input sequence length to obtain the time weight of each position. Temporal convolution is performed on the frame-level feature vector sequence using convolution kernels. During the convolution process, the features are weighted by time weights, and the convolution result is multiplied by scalar gating coefficients for amplitude modulation. Finally, the bias vector is superimposed to output the temporal feature sequence.

7. The image emotion recognition method based on human-computer interaction according to claim 1, characterized in that, By using Gaussian time weight windows as weights, the time-series feature sequences are weighted and summed, and then divided by the sum of the weights to obtain the central representation vector.

8. The image emotion recognition method based on human-computer interaction according to claim 7, characterized in that, Using the center of the input length as the boundary, calculate the weight sum of the sides before the phase zero and the weight sum of the sides after the phase zero respectively; The time series feature sequences before and after the phase zero are weighted and summed using Gaussian time weight windows, and then divided by the corresponding weights to obtain the weighted mean vector before the phase zero and the weighted mean vector after the phase zero. Subtract the weighted mean vector behind the phase zero from the weighted mean vector before the phase zero to obtain the difference comparison vector; The global representation vector is obtained by directly adding the central representation vector to the difference comparison vector.

9. The image emotion recognition method based on human-computer interaction according to claim 1, characterized in that, Multiply the global representation vector by the weight matrix of the classification network layer and sum the bias vector to obtain the real vector output of the classification network layer; Perform an exponential operation on each component of the real number vector output, and divide by the sum of the exponential operation results of all components to obtain a discrete emotion category probability distribution vector. The index corresponding to the component with the largest value in the probability distribution vector is selected as the final discrete emotion category label for output.

10. An image emotion recognition system based on human-computer interaction, characterized in that, Performing an image emotion recognition method based on human-computer interaction as described in any one of claims 1 to 9, comprising: The brow muscle region image cropping module performs face detection and key point localization on the video frame sequence, aligns the face image to the standard coordinate system, and maps the brow muscle mask in the standard coordinate system back to each video frame to crop the corresponding brow muscle region image sequence. The phase lag determination module extracts dynamic displacement signals from the image sequence of the brow muscle region and converts them into acceleration sequences. Within the analysis window starting from the interaction event, it locates the first global peak moment of the acceleration sequence, calculates the time difference between the interaction event and the first global peak moment, and obtains the event anchoring phase lag. The input sequence construction module takes the first global peak moment as the phase zero point, and performs equidistant sampling on the image sequence of the brow muscle region before and after the phase zero point to construct a time-normalized input sequence. The temporal convolution module performs frame-by-frame visual encoding on the time-normalized input sequence, generates gating coefficients based on the event anchor phase lag, modulates the amplitude of visual features, and performs temporal convolution under the action of a time-weighted window to obtain a temporal feature sequence. The global representation vector generation module performs a central weighted aggregation with the phase zero point as the core on the temporal feature sequence to obtain a central representation vector. It calculates the feature mean before and after the phase zero point and constructs a difference comparison vector. The central representation vector and the difference comparison vector are then fused to generate a global representation vector. The emotion category output module takes the global representation vector as input to the classification network layer, maps it, and outputs discrete emotion category labels.

Images