Concentration non-inductive monitoring method and system based on multi-modal behavior analysis

Abstract

The invention relates to the technical field of concentration non-inductive monitoring, and discloses a concentration non-inductive monitoring method and system based on multi-modal behavior analysis, and the method comprises the steps: obtaining a face video stream, a body posture video stream and a screen content image of a user; the method comprises the following steps: processing a face video stream and a body posture video stream by adopting a spatial-temporal feature recursive aggregation network, extracting multi-scale behavior features, calculating the stay duration of a sight falling point in each semantic region of a screen content image based on the face video stream, calculating a visual inertia coefficient for each semantic region in combination with the content complexity of the semantic region, and obtaining a visual inertia coefficient; constructing a weighted sight transfer matrix, and extracting a main characteristic value as a visual focus stability measure; calculating and monitoring an instantaneous value of the head shaking impact degree, and when the instantaneous value exceeds a preset threshold value, adjusting model parameters of the spatial-temporal feature recursive aggregation network; and calculating the real-time concentration score of the user. According to the method, the comprehensiveness and reliability of concentration monitoring are improved by fusing multi-dimensional and deep features.

Description

Technical Field

[0001] This invention relates to the field of attention-free monitoring technology, specifically to a method and system for attention-free monitoring based on multimodal behavior analysis. Background Technology

[0002] In fields such as distance education, autonomous driving, and human-computer interaction, real-time, non-contact monitoring of user attention has significant application value. Traditional monitoring methods mainly rely on the acquisition of physiological signals such as electroencephalograms (EEGs) and electrocardiograms (ECGs), but these usually require contact devices, limiting their usability. With the development of computer vision technology, non-contact monitoring methods based on video analysis have become a research hotspot. These methods mainly capture visual cues such as facial expressions, head posture, gaze direction, and blink frequency through cameras, and infer their attention state based on these cues. However, most existing technical solutions only perform simple linear weighted fusion of these multimodal information, ignoring the complex intrinsic relationships between different behavioral modalities. This results in limited model representation capabilities of attention states, and the accuracy of monitoring results needs improvement.

[0003] Furthermore, existing visual analysis methods, when assessing user visual focus, generally only focus on the location and duration of the gaze on the screen, neglecting the specific information of the content being looked at. For example, the level of focus reflected by prolonged observation of a complex area is clearly different from that reflected by observing a blank area. This disregard for the semantics of screen content prevents models from accurately representing the stability and depth of visual attention. In addition, users' behavioral states can change abruptly in real-world scenarios, such as suddenly turning their heads or swaying their bodies due to external disturbances. Most existing monitoring models are static; when users exhibit violent or atypical movements, their feature extraction capabilities decrease, leading to significant biases in attention assessment. Summary of the Invention

[0004] This invention provides a non-intrusive attention monitoring method and system based on multimodal behavior analysis to solve the problem of low accuracy of attention monitoring results in the prior art.

[0005] In a first aspect, the attention monitoring method based on multimodal behavior analysis of the present invention includes the following steps:

[0006] Acquire user's facial video stream, body posture video stream, and screen content images; A spatiotemporal feature recursive aggregation network is used to process facial video streams and body pose video streams to extract multi-scale behavioral features. The spatiotemporal feature recursive aggregation network fuses the output features of the previous node, the input features of the current time step, and the temporal gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation. The duration of gaze placement in each semantic region of the screen content image is calculated based on the facial video stream. The visual inertia coefficient is calculated for each semantic region based on the content complexity of the semantic region. A weighted gaze transfer matrix is ​​constructed accordingly, and the main eigenvalue of the weighted gaze transfer matrix is ​​extracted as a measure of visual focus stability. The first-order time derivative of the head posture angular velocity in the facial video stream is calculated to obtain the head shaking impact. The instantaneous value of the head shaking impact is monitored, and when it exceeds a preset threshold, the model parameters of the spatiotemporal feature recursive aggregation network are adjusted to respond to sudden changes in user behavior. The user's real-time attention score is calculated by integrating multi-scale behavioral features, visual focus stability measurement, and head shaking impact.

[0007] Preferably, the step of using a spatiotemporal feature recursive aggregation network to process facial video streams and body pose video streams to extract multi-scale behavioral features includes: The spatiotemporal feature recursive aggregation network includes a parallel processing branch that extracts facial features and pose features from the facial video stream and body pose video stream, respectively. The spatiotemporal feature recursive aggregation network also includes a feature fusion attention module, which is used to perform weighted fusion of facial features and pose features to generate the multi-scale behavioral features.

[0008] Preferably, the spatiotemporal feature recursive aggregation network fuses the output features of the previous node, the input features of the current time step, and the temporal gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation, including: The operation of the aggregation node within each parallel processing branch is as follows: the output features of the previous node, the feature map extracted from the video stream at the current time step, and the temporal gradient of the feature map are concatenated along the feature dimension to form an aggregated feature vector; the aggregated feature vector is input into a linear transformation layer, the weight matrix of the linear transformation layer is constrained to an orthogonal matrix through QR decomposition, and the output features of the current node are generated through the linear transformation layer.

[0009] Preferably, the step of calculating the visual inertia coefficient for each semantic region based on the content complexity of the semantic region includes: Each semantic region of the screen content image is converted into a grayscale image, and the variance of the pixel values ​​after the Laplacian transform of the grayscale image is calculated. The variance of the pixel values ​​is used as the content complexity of the semantic region. The visual inertia coefficient is a normalized value of the content complexity.

[0010] Preferably, constructing the weighted gaze transition matrix accordingly includes: Statistically count the frequency of the gaze shifting from semantic region i to semantic region j within a preset time window. The frequency Multiplied by the visual inertia coefficient of semantic region j The weighted transfer value is obtained. Weighted transfer value The elements that make up matrix W, The calculation formula is: ; Perform on all rows of matrix W Norm normalization is performed to make the sum of the elements in each row equal to 1, thereby constructing the weighted line-of-sight transition matrix.

[0011] Preferably, the step of calculating the first-order time derivative of the head posture angular velocity in the facial video stream to obtain the head sway impact includes: The Euler angle sequence of head pose is extracted from consecutive video frames of the facial video stream, and the angular velocity is calculated. The first time derivative of the angular velocity is calculated using the backward difference method to obtain the head sway impact, which is calculated using the following formula: ; in, for The impact of head shaking at any moment, for angular velocity at time t, for angular velocity at time t, This represents the time interval between video frames.

[0012] Preferably, the preset threshold is .

[0013] Preferably, adjusting the model parameters of the spatiotemporal feature recursive aggregation network includes: Adjust the attention weights in the feature fusion attention module to allocate attention weights to the body pose video stream features. Increase the fixed adjustment value Attention weights will be assigned to facial video stream features. reduce .

[0014] Preferably, the calculation of the user's real-time attention score by integrating multi-scale behavioral features, visual focus stability measurement, and head shaking impact includes: The multi-scale behavioral features, visual focus stability measure, and head shaking impact output by the spatiotemporal feature recursive aggregation network are concatenated along the feature dimension to form a comprehensive feature vector. The comprehensive feature vector is then input into a pre-trained three-layer fully connected neural network, where the output layer of the three-layer fully connected neural network is a single neuron, and the output value is the user's real-time focus score.

[0015] Secondly, the attention-free monitoring system based on multimodal behavior analysis of the present invention includes a memory and a processor. The memory stores computer instructions, and when the processor executes the computer instructions, it implements the above-mentioned attention-free monitoring method based on multimodal behavior analysis.

[0016] The beneficial effects of this invention are as follows: Through a unique spatiotemporal feature recursive aggregation network, this invention can deeply analyze the multi-scale behavioral characteristics of users' facial and body postures over long time periods, detecting the continuous evolution of behavior. This invention innovatively integrates the semantic complexity of screen content into gaze analysis, measuring the stability of visual focus by constructing a weighted gaze transfer matrix, thus enabling the assessment of visual attention to go beyond spatial location and delve into the content level. By inputting a head-shaking impact index, it can sensitively detect sudden head movements that indicate distraction. By integrating the above-mentioned multi-dimensional and in-depth features, the comprehensiveness and reliability of attention monitoring are improved. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the attention-based non-intrusive monitoring method based on multimodal behavior analysis provided in an embodiment of the present invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0019] like Figure 1 As shown, an embodiment of the attention-based non-intrusive monitoring method based on multimodal behavior analysis provided by the present invention includes the following steps: S1 acquires the user's facial video stream, body posture video stream, and screen content image.

[0020] Specifically, a single webcam deployed directly in front of the user synchronously captures video of the user's upper body at a rate of 30 frames per second, and decomposes it into facial video streams and body posture video streams. Simultaneously, through an operating system-level screen capture application interface, the entire desktop or active window currently being operated by the user is detected in real time, generating a sequence of screen content images.

[0021] S2, a spatiotemporal feature recursive aggregation network is used to process facial video streams and body pose video streams, extracting multi-scale behavioral features. The spatiotemporal feature recursive aggregation network fuses the output features of the previous node, the input features of the current time step, and the temporal gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation.

[0022] Specifically, a pre-trained deep learning model, such as MediaPipe, is used to process the video stream: extracting 68 facial keypoint coordinates and head pose Euler angles from each frame of the facial video stream, and extracting 17 body joint coordinates, including shoulder, elbow, and wrist, from the body pose video stream. These coordinates are then concatenated into the input feature vector for the current time step t. .

[0023] Calculate the time gradient of features This represents the instantaneous change in action. At the aggregation node t of the network, the output features of the previous node are... Current input features and time gradient Vector concatenation is performed, followed by linear fusion through a fully connected layer. Finally, the fused feature vector is left-multiplied by a fixed or learnable orthogonal matrix Q to generate the output feature of the current node. This feature This refers to multi-scale behavioral features that contain long-term temporal dependencies from the beginning to the current time t.

[0024] S3. Calculate the duration of gaze landing on each semantic region of the screen content image based on the facial video stream. Combine the content complexity of the semantic region to calculate the visual inertia coefficient for each semantic region. Construct a weighted gaze transfer matrix based on this and extract the main eigenvalue of the weighted gaze transfer matrix as a measure of visual focus stability.

[0025] Specifically, a deep learning gaze estimation model is used to infer the user's gaze direction vector in real time based on eye images and head pose in the facial video stream. This vector is then geometrically calibrated and mapped to two-dimensional coordinates on the screen, i.e., the gaze placement point. Simultaneously, a pre-trained image segmentation network, such as U-Net, is used to automatically segment the screen content image into multiple semantic regions, including text regions, image regions, code regions, and blank regions. Next, the content complexity of each semantic region is calculated; for example, the complexity of the text region can be defined as character density, and the complexity of the image region is the edge density calculated using the Sobel operator. Finally, the total dwell time of the gaze placement point in each semantic region i is statistically analyzed. And calculate the visual inertia coefficient. ,in, Let's define the content complexity of this region. Finally, construct a... The weighted gaze transition matrix, where N is the number of semantic regions, is calculated by taking its diagonal elements. Set as visual inertia coefficient Off-diagonal elements Let be the frequency of gaze shifting from region i to region j. By performing eigenvalue decomposition on the weighted gaze shift matrix, the largest eigenvalue is extracted as a measure of visual focus stability.

[0026] S4. Calculate the first-order time derivative of the head posture angular velocity in the facial video stream to obtain the head shaking impact; monitor the instantaneous value of the head shaking impact, and when it exceeds a preset threshold, adjust the model parameters of the spatiotemporal feature recursive aggregation network to respond to sudden changes in user behavior; calculate the user's real-time attention score by integrating multi-scale behavioral features, visual focus stability measurement, and head shaking impact.

[0027] Specifically, the preset threshold is .

[0028] Specifically, the pitch, yaw, and roll angle sequences of the head posture are calculated in real time from the facial video stream. The head posture angular velocity vector is obtained by dividing the change in attitude angles between two consecutive frames by the inter-frame time interval. The same time-difference operation is then performed on the angular velocity vector sequence to obtain the head posture angular acceleration vector. The Euclidean norm of this acceleration vector is taken as the instantaneous head sway impact, a value that can sensitively reflect rapid head rotation or sway.

[0029] Set an impact threshold based on empirical data, for example, The system continuously calculates and monitors the instantaneous value of head-shaking impact. Once this value exceeds a set threshold, it is considered that the user has experienced a behavioral change, such as suddenly turning their head due to external interference. At this point, the state of the spatiotemporal feature recursive aggregation network is immediately adjusted. Specifically, the weight coefficient α used to fuse historical information in the network is attenuated, for example, by multiplying it by a attenuation factor less than 1. This reduces the influence of historical behavioral features on the current judgment, allowing the model to quickly adapt to new behavioral patterns.

[0030] Then, the multi-scale behavioral feature vector output by the spatiotemporal feature recursive aggregation network at the current time step, the visual focus stability measure as a scalar, and the head sway impact as a scalar are concatenated into a high-dimensional comprehensive feature vector. This comprehensive feature vector is then input into a pre-trained multilayer perceptron regression model, whose output is a continuous value between 0 and 1, which represents the user's real-time focus score at that moment, where 1 represents complete focus and 0 represents complete lack of focus.

[0031] In an optional embodiment, the step of using a spatiotemporal feature recursive aggregation network to process facial video streams and body pose video streams to extract multi-scale behavioral features includes: The spatiotemporal feature recursive aggregation network includes a parallel processing branch that extracts facial features and pose features from the facial video stream and body pose video stream, respectively. The spatiotemporal feature recursive aggregation network also includes a feature fusion attention module, which is used to perform weighted fusion of facial features and pose features to generate the multi-scale behavioral features.

[0032] The spatiotemporal feature recursive aggregation network fuses the output features of the previous node, the input features of the current time step, and the temporal gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation, including: The operation of the aggregation node within each parallel processing branch is as follows: the output features of the previous node, the feature map extracted from the video stream at the current time step, and the temporal gradient of the feature map are concatenated along the feature dimension to form an aggregated feature vector; the aggregated feature vector is input into a linear transformation layer, the weight matrix of the linear transformation layer is constrained to an orthogonal matrix through QR decomposition, and the output features of the current node are generated through the linear transformation layer.

[0033] For example, the network receives two video streams as input, such as a facial video and a pose video recording the coordinates of 18 body keypoints. The facial branch processes each frame using a convolutional neural network, outputting a 512-dimensional feature map. The pose branch processes the keypoint coordinates using a multilayer perceptron, outputting a 256-dimensional feature map.

[0034] Taking a certain aggregation node in the facial branch as an example, assuming the output feature of the previous node... It is 512-dimensional, the feature map extracted from the current frame. The time gradient is 512-dimensional, and the three dimensions are also 512-dimensional. These three are concatenated to form a 1536-dimensional aggregate vector. This vector is passed through a linear layer with orthogonal weight matrices, outputting a new 512-dimensional feature as the output of the current node.

[0035] At the end of the network, the face branch and pose branch output feature vectors respectively. and The feature fusion attention module calculates attention weights, for example, assigning a weight of 0.7 to facial features and a weight of 0.3 to pose features. The multi-scale behavioral features are then the sum of these two weighted feature vectors, i.e.: The fused vector can more comprehensively represent the user's behavioral state.

[0036] In an optional embodiment, the step of calculating the visual inertia coefficient for each semantic region based on the content complexity of the semantic region includes: Each semantic region of the screen content image is converted into a grayscale image, and the variance of the pixel values ​​after the Laplacian transform of the grayscale image is calculated. The variance of the pixel values ​​is used as the content complexity of the semantic region. The visual inertia coefficient is a normalized value of the content complexity.

[0037] For example, the screen can be divided into different semantic regions, such as a video playback area A and a code editing area B. For region A, a segment is captured. A color image of pixels is generated and converted to a grayscale image. Then, an atomizer is applied to the grayscale image. The Laplacian operator is used for convolution to enhance the edge and texture details of the image. Finally, the variance of all pixel values ​​in the transformed image is calculated; for example, the variance of video region A is 625.5, and this variance is used as the content complexity of that region.

[0038] Then, the same operation is performed on code editing area B. Because the code area content is relatively static and the texture is simple, the calculated variance is low, for example, 95.2. These two variances represent the content complexity of their respective areas. These two content complexities are then normalized, for example, using a minimum-maximum normalization method to map them to the interval between 0 and 1, yielding the visual inertia coefficient. Assuming the visual inertia coefficient of video area A is 1.0 and the visual inertia coefficient of code area B is 0.15, the coefficient values ​​reflect the ability of different areas to attract and maintain the user's visual attention.

[0039] In an optional embodiment, constructing the weighted gaze transfer matrix accordingly includes: Statistically count the frequency of the gaze shifting from semantic region i to semantic region j within a preset time window. The frequency Multiplied by the visual inertia coefficient of semantic region j The weighted transfer value is obtained. Weighted transfer value The elements that make up matrix W, The calculation formula is: ; Perform on all rows of matrix W Norm normalization is performed to make the sum of the elements in each row equal to 1, thereby constructing the weighted line-of-sight transition matrix.

[0040] For example, suppose there are two semantic regions on the screen: region 1 is the video area, and region 2 is the chat area. Their visual inertia coefficients are respectively... and Within a 5-second time window, the eye-tracking device recorded the number of times the gaze moved from region 1 to region 1. The number of times is 20, representing the number of times from region 1 to region 2. The number of times is 5, representing the number of times from region 2 to region 1. The number of times is 8, representing the number of times from region 2 to region 2. The frequency is 15 times. These data constitute the original frequency matrix N. Then, the weighted transition value is calculated according to the formula to obtain... , , , This yields matrix W. Then, each row of matrix W is normalized; the sum of the first row is 20. After normalization, it becomes 0.9. After normalization, the result is 0.1; the sum in the second row is 13.2. After normalization, it becomes 0.55. After normalization, the value is 0.45; thus, the weighted gaze shift matrix is ​​obtained. The constructed weighted gaze shift matrix reflects the probability of user attention shifts between regions of different complexity.

[0041] In an optional embodiment, calculating the first-order time derivative of the head pose angular velocity in the facial video stream to obtain the head sway impact includes: The Euler angle sequence of head pose is extracted from consecutive video frames of the facial video stream, and the angular velocity is calculated. The first time derivative of the angular velocity is calculated using the backward difference method to obtain the head sway impact, which is calculated using the following formula: ; in, for The impact of head shaking at any moment, for angular velocity at time t, for angular velocity at time t, This represents the time interval between video frames.

[0042] For example, suppose we are processing a facial video recorded at 30 frames per second, with a time interval Δt between adjacent video frames of 0.033 s. At time t, the yaw angle of the head is estimated to be 25° using a facial landmark detection algorithm. In the previous frame, at time t-Δt, the measured yaw angle was 24°. angular velocity at time t Dividing the change in angle by the time interval gives approximately equal to Assume the angular velocity calculated at time t-Δt is... for .according to The calculation formula yields an estimated head-shaking impact of approximately [value missing]. .

[0043] In an optional embodiment, adjusting the model parameters of the spatiotemporal feature recursive aggregation network includes: Adjust the attention weights in the feature fusion attention module to allocate attention weights to the body pose video stream features. Increase the fixed adjustment value Attention weights will be assigned to facial video stream features. reduce .

[0044] For example, assuming under normal circumstances, It is 0.6. The value is 0.4. Real-time monitoring of the user's head movement impact is performed. Assuming the user suddenly turns their head at a certain moment, the instantaneous impact value is calculated. The value exceeds the preset threshold. At this point, the weight adjustment mechanism will be automatically triggered. An adjustment value Δα will be added, assuming Δα = 0.1, and the new weight will become 0.5. Subtracting the adjustment value Δα, the new weight becomes 0.5. In this way, when a sudden head movement is detected, the model dynamically increases its attention to overall body posture information, because such abrupt behavior may indicate a significant shift or distraction of attention, rather than just a slight change in facial expression.

[0045] In an optional embodiment, the calculation of the user's real-time attention score by integrating multi-scale behavioral features, visual focus stability measures, and head sway impact includes: The multi-scale behavioral features, visual focus stability measure, and head shaking impact output by the spatiotemporal feature recursive aggregation network are concatenated along the feature dimension to form a comprehensive feature vector. The comprehensive feature vector is then input into a pre-trained three-layer fully connected neural network, where the output layer of the three-layer fully connected neural network is a single neuron, and the output value is the user's real-time focus score.

[0046] For example, at a certain point in time, the spatiotemporal feature recursive aggregation network outputs a 256-dimensional multi-scale behavioral feature vector representing the user's micro-expressions and posture; by analyzing the weighted gaze transfer matrix, the visual focus stability metric is calculated to be 0.88, where a higher value indicates a more stable gaze; the current head movement impact is measured to be... These three data points are concatenated into a single, comprehensive feature vector with a dimension of 258. This comprehensive feature vector is then input into a pre-trained three-layer fully connected neural network, which contains a 258-dimensional input layer, a 64-dimensional hidden layer, and a 1-dimensional output layer. The network performs a non-linear transformation on the input vector, and the neurons in the output layer ultimately output a value between 0 and 1, for example, 0.76. This value is the user's real-time attention score.

[0047] The implementation principle of the attention monitoring method based on multimodal behavior analysis in this invention is as follows: This invention employs a unique spatiotemporal feature recursive aggregation network, which can deeply analyze the multi-scale behavioral features of the user's facial and body postures over long time periods, capturing the continuous evolution of behavior. This invention innovatively integrates the semantic complexity of screen content into gaze analysis, constructing a weighted gaze transfer matrix to measure the stability of visual focus, thus extending the evaluation beyond spatial location to the content level. Furthermore, this invention introduces a head sway impact index to sensitively detect sudden head movements that may indicate distraction. By effectively integrating the above-mentioned multi-dimensional and in-depth features, this invention significantly improves the comprehensiveness and reliability of attention monitoring.

[0048] An embodiment of the attention-free monitoring system based on multimodal behavior analysis provided by the present invention includes a memory and a processor. The memory stores computer instructions, and when the processor executes the computer instructions, it implements the attention-free monitoring method based on multimodal behavior analysis in the above embodiment.

[0049] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for monitoring concentration without awareness based on multi-modal behavior analysis, characterized in that, The method comprises the following steps: obtaining a facial video stream, a body posture video stream and a screen content image of a user; The facial video stream and the body posture video stream are processed by using a spatio-temporal feature recursive aggregation network to extract multi-scale behavior features, the spatio-temporal feature recursive aggregation network fuses the output features of a previous node, the input features of a current time step and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation. The dwell time of the gaze landing point in each semantic region of the screen content image is calculated based on the facial video stream, the visual inertia coefficients of each semantic region are calculated in combination with the content complexity of the semantic regions, a weighted gaze transition matrix is constructed according to the visual inertia coefficients, and the principal eigenvalue of the weighted gaze transition matrix is extracted as the visual focus stability measure. The first-order time derivative of the head posture angular velocity in the facial video stream is calculated to obtain the head shaking impact degree, the instantaneous value of the head shaking impact degree is monitored, and when the instantaneous value exceeds a preset threshold, the model parameters of the spatio-temporal feature recursive aggregation network are adjusted to respond to the mutation of the user behavior; and the multi-scale behavior features, the visual focus stability measure and the head shaking impact degree are comprehensively considered to calculate the real-time concentration score of the user.

2. The method of claim 1, wherein the method further comprises: The facial video stream and the body posture video stream are processed by using a spatio-temporal feature recursive aggregation network to extract multi-scale behavior features, the spatio-temporal feature recursive aggregation network fuses the output features of a previous node, the input features of a current time step and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation. The spatio-temporal feature recursive aggregation network comprises parallel processing branches for extracting facial features and posture features from the facial video stream and the body posture video stream, respectively; and the spatio-temporal feature recursive aggregation network further comprises a feature fusion attention module for weighted fusion of the facial features and the posture features to generate the multi-scale behavior features. 3.The method of claim 2, wherein, The spatio-temporal feature recursive aggregation network fuses the output features of a previous node, the input features of a current time step and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation, and the method comprises the following steps: The operation of the aggregation node in each parallel processing branch is as follows: the output features of a previous node, the feature map extracted from the video stream at a current time step and the time gradient of the feature map are spliced in the feature dimension to form an aggregation feature vector; and the aggregation feature vector is input into a linear transformation layer, the weight matrix of the linear transformation layer is constrained to be an orthogonal matrix through QR decomposition, and the output features of the current node are generated through the linear transformation layer. 4.The method of claim 1, wherein, The content complexity of each semantic region is used to calculate the visual inertia coefficient of each semantic region, and the method comprises the following steps: Each semantic region of the screen content image is converted into a gray-scale image, and the pixel value variance of the gray-scale image after Laplace transformation is calculated as the content complexity of the semantic region; and the visual inertia coefficient is a normalized value of the content complexity. 5.The multi-modal behavior analysis based concentration monitoring method of claim 1, wherein, The weighted gaze transition matrix is constructed according to the visual inertia coefficients, and the method comprises the following steps: counting the frequency of the visual line falling point transferring from semantic region i to semantic region j in a preset time window ; multiplying the frequency by the visual inertia coefficient of semantic region j to obtain a weighted transfer value The weighted transfer value is an element of the matrix W, and the calculation formula is: ; all rows of the matrix W are normalized The weighted visual line transition matrix is constructed by normalizing each row element to 1. 6.The multi-modal behavior analysis based concentration monitoring method of claim 1, wherein, The first-order time derivative of the head posture angular velocity in the facial video stream is calculated to obtain the head shaking impact degree, the instantaneous value of the head shaking impact degree is monitored, and when the instantaneous value exceeds a preset threshold, the model parameters of the spatio-temporal feature recursive aggregation network are adjusted to respond to the mutation of the user behavior; and the multi-scale behavior features, the visual focus stability measure and the head shaking impact degree are comprehensively considered to calculate the real-time concentration score of the user. Euler angle sequences of head pose are extracted from consecutive video frames of the facial video stream, and angular velocities are calculated; a backward difference method is used to calculate the first-order time derivative of the angular velocity to obtain a head shaking impact degree, and the head shaking impact degree is calculated by the following formula: ； wherein is the head shaking impact degree at the moment, is the angular velocity at the moment, is the angular velocity at the moment, is the time interval between video frames.

7. The method of claim 2, wherein the method further comprises: The preset threshold is .

8. The method of claim 7, wherein the method further comprises: The model parameters of the spatio-temporal feature recursive aggregation network are adjusted, including: adjusting the attention weight in the feature fusion attention module, assigning the attention weight to the body posture video stream feature increasing the fixed adjustment value , while reducing the attention weight assigned to the facial video stream feature decreasing . 9.The multi-modal behavior analysis based concentration monitoring method of claim 1, wherein, The multi-scale behavior features, the visual focus stability measure, and the head shaking impact degree are integrated to calculate a real-time concentration score of the user, including: The multi-scale behavior features, the visual focus stability measure, and the head shaking impact degree output by the spatio-temporal feature recursive aggregation network are spliced in the feature dimension to form a comprehensive feature vector; the comprehensive feature vector is input into a pre-trained three-layer fully connected neural network, and the output layer of the three-layer fully connected neural network is a single neuron, and the output value is the real-time concentration score of the user.

10. A concentration monitoring system based on multi-modal behavioral analysis, characterized in that, The method comprises a memory and a processor, the memory stores computer instructions, and the processor executes the computer instructions to implement the concentration monitoring method based on multi-modal behavior analysis according to any one of claims 1-9. The method comprises a memory and a processor, the memory stores computer instructions, and the processor executes the computer instructions to implement the concentration monitoring method based on multi-modal behavior analysis according to any one of claims 1-9.

Images