Concentration monitoring method and system based on multi-modal behavior analysis

Abstract

The application relates to the technical field of concentration monitoring, and discloses a concentration monitoring method and system based on multi-modal behavior analysis, which comprises the following steps: acquiring a facial video stream, a body posture video stream and a screen content image of a user; processing the facial video stream and the body posture video stream by adopting a space-time feature recursive aggregation network, extracting multi-scale behavior features, calculating the staying time of a visual line landing point in each semantic region of the screen content image based on the facial video stream, calculating a visual inertia coefficient for each semantic region in combination with the content complexity of the semantic region, constructing a weighted visual line transfer matrix according to the visual inertia coefficient, extracting a main eigenvalue as a visual focus stability measure, calculating and monitoring the instantaneous value of head shaking impact, adjusting the model parameters of the space-time feature recursive aggregation network when the instantaneous value exceeds a preset threshold, and calculating a real-time concentration score of the user. The application improves the comprehensiveness and reliability of concentration monitoring by fusing multi-dimensional and deep features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of attention monitoring, and particularly relates to an attention monitoring method and system based on multi-modal behavior analysis. BACKGROUND

[0002] In the fields of distance education, intelligent driving and human-computer interaction, real-time and unobtrusive monitoring of user attention has important application value. Traditional monitoring methods mainly rely on the collection of physiological signals such as electroencephalogram and electrocardiogram, but this usually requires a contact device, which is limited in ease of use. With the development of computer vision technology, non-contact monitoring methods based on video analysis have become a research hotspot. Such methods mainly capture visual cues such as facial expressions, head poses, gaze directions and blink frequencies of users through cameras, and infer their attention states accordingly. However, most of the existing technical solutions only perform simple linear weighted fusion on these multi-modal information, ignoring the complex internal correlation between different behavior modalities, resulting in limited representation ability of the model for attention states and the accuracy of the monitoring results needs to be improved.

[0003] In addition, existing visual analysis methods generally only focus on the position and duration of the gaze on the screen when evaluating the user's visual focus, while ignoring the specific information of the content being gazed at. For example, gazing at a complex content area for a long time is obviously different from gazing at a blank area in terms of the degree of concentration. This neglect of the semantics of the screen content results in the model's inability to accurately represent the stability and depth of visual attention. In addition, the behavior state of users in real scenarios may change suddenly, such as turning their heads or body swaying due to external interference. Most existing monitoring models are static, and when users perform dramatic or atypical actions, the feature extraction ability of the model will decrease, resulting in a large deviation in attention assessment. SUMMARY

[0004] The present application provides an attention monitoring method and system based on multi-modal behavior analysis to solve the problem of low accuracy of existing attention monitoring results.

[0005] In a first aspect, the present application provides an attention monitoring method based on multi-modal behavior analysis, comprising the following steps:

[0006] obtaining a facial video stream, a body pose video stream and a screen content image of a user;

[0007] processing the facial video stream and the body pose video stream using a spatio-temporal feature recursive aggregation network to extract multi-scale behavior features, wherein the spatio-temporal feature recursive aggregation network fuses the output features of the previous node, the input features of the current time step and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation;

[0008] The gaze landing point in each semantic region of the screen content image is calculated based on the face video stream, the visual inertia coefficient of each semantic region is calculated in combination with the content complexity of the semantic region, a weighted gaze transition matrix is constructed according to the visual inertia coefficient, and the main eigenvalue of the weighted gaze transition matrix is extracted as the visual focus stability measure;

[0009] The first-order time derivative of the head posture angular velocity in the face video stream is calculated to obtain the head shaking impact degree, the instantaneous value of the head shaking impact degree is monitored, and when the instantaneous value exceeds a preset threshold, the model parameters of the spatio-temporal feature recursive aggregation network are adjusted to respond to the mutation of the user behavior; the user real-time concentration score is calculated by comprehensively considering the multi-scale behavior feature, the visual focus stability measure and the head shaking impact degree.

[0010] Preferably, the face video stream and the body posture video stream are processed by the spatio-temporal feature recursive aggregation network to extract the multi-scale behavior feature, which includes:

[0011] The spatio-temporal feature recursive aggregation network includes parallel processing branches for extracting face features and posture features from the face video stream and the body posture video stream respectively; the spatio-temporal feature recursive aggregation network further includes a feature fusion attention module for weighted fusion of the face features and the posture features to generate the multi-scale behavior feature.

[0012] Preferably, the spatio-temporal feature recursive aggregation network fuses the output feature of the previous node, the input feature of the current time step and the time gradient of the input feature at each aggregation node, and generates the output feature of the current node through orthogonal transformation, which includes:

[0013] The operation of each aggregation node in each parallel processing branch is to splice the output feature of the previous node, the feature map extracted from the video stream at the current time step and the time gradient of the feature map in the feature dimension to form an aggregation feature vector; the aggregation feature vector is input to a linear transformation layer, the weight matrix of the linear transformation layer is constrained to an orthogonal matrix through QR decomposition, and the output feature of the current node is generated through the linear transformation layer.

[0014] Preferably, the visual inertia coefficient of each semantic region is calculated in combination with the content complexity of the semantic region, which includes:

[0015] Each semantic region of the screen content image is converted into a gray image, and the pixel value variance of the gray image after Laplace transform is calculated, and the pixel value variance is taken as the content complexity of the semantic region; the visual inertia coefficient is a normalized value of the content complexity.

[0016] Preferably, the weighted gaze transition matrix is constructed according to the visual inertia coefficient, which includes:

[0017] Statistically count the frequency of the gaze shifting from semantic region i to semantic region j within a preset time window. The frequency Multiplied by the visual inertia coefficient of semantic region j The weighted transfer value is obtained. Weighted transfer value The elements that make up matrix W, The calculation formula is: ;

[0018] Perform on all rows of matrix W Norm normalization is performed to make the sum of the elements in each row equal to 1, thereby constructing the weighted line-of-sight transition matrix.

[0019] Preferably, the step of calculating the first-order time derivative of the head posture angular velocity in the facial video stream to obtain the head sway impact includes:

[0020] The Euler angle sequence of head pose is extracted from consecutive video frames of the facial video stream, and the angular velocity is calculated. The first time derivative of the angular velocity is calculated using the backward difference method to obtain the head sway impact, which is calculated using the following formula:

[0021] ;

[0022] in, for The impact of head shaking at any moment, for angular velocity at time t, for angular velocity at time t, This represents the time interval between video frames.

[0023] Preferably, the preset threshold is .

[0024] Preferably, adjusting the model parameters of the spatiotemporal feature recursive aggregation network includes:

[0025] Adjust the attention weights in the feature fusion attention module to allocate attention weights to the body pose video stream features. Increase the fixed adjustment value Attention weights will be assigned to facial video stream features. reduce .

[0026] Preferably, the calculation of the user's real-time attention score by integrating multi-scale behavioral features, visual focus stability measurement, and head shaking impact includes:

[0027] The multi-scale behavior features, the visual focus stability measure and the head shaking impact output by the spatio-temporal feature recursive aggregation network are spliced in the feature dimension to form a comprehensive feature vector; the comprehensive feature vector is input into a pre-trained three-layer fully connected neural network, and the output layer of the three-layer fully connected neural network is a single neuron, and the output value is the real-time concentration score of the user.

[0028] In a second aspect, the concentration unobtrusive monitoring system based on multi-modal behavior analysis provided by the present application comprises a memory and a processor, and the memory stores computer instructions, and the processor implements the concentration unobtrusive monitoring method based on multi-modal behavior analysis described above when executing the computer instructions.

[0029] The present application has the following advantages: the spatio-temporal feature recursive aggregation network of the present application can deeply analyze the multi-scale behavior features of the user's face and body posture in a long time sequence, and detect the continuous evolution process of the behavior. The present application innovatively integrates the semantic complexity of the screen content into the gaze analysis, measures the stability of the visual focus by constructing a weighted gaze transition matrix, so that the evaluation of the visual attention is no longer limited to the spatial position, but is deepened to the content level. By inputting the head shaking impact index, sudden head movements indicating distraction can be sensitively detected. By fusing the above-mentioned multi-dimensional and deep features, the comprehensiveness and reliability of the concentration monitoring are improved. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The flowchart of the concentration unobtrusive monitoring method based on multi-modal behavior analysis provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0032] As shown in Figure 1 The embodiment of the concentration unobtrusive monitoring method based on multi-modal behavior analysis provided by the present application comprises the following steps:

[0033] S1, obtaining a face video stream, a body posture video stream and a screen content image of a user.

[0034] Specifically, a single network camera deployed in front of the user synchronously collects a video containing the upper body of the user at a rate of 30 frames per second, and is decomposed into a face video stream and a body posture video stream. At the same time, through an operating system level screen capture application program interface, the entire desktop or active window currently operated by the user is detected in real time to generate a screen content image sequence.

[0035] S2, a spatiotemporal feature recursive aggregation network is used to process facial video streams and body pose video streams, extracting multi-scale behavioral features. The spatiotemporal feature recursive aggregation network fuses the output features of the previous node, the input features of the current time step, and the temporal gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation.

[0036] Specifically, a pre-trained deep learning model, such as MediaPipe, is used to process the video stream: extracting 68 facial keypoint coordinates and head pose Euler angles from each frame of the facial video stream, and extracting 17 body joint coordinates, including shoulder, elbow, and wrist, from the body pose video stream. These coordinates are then concatenated into the input feature vector for the current time step t. .

[0037] Calculate the time gradient of features This represents the instantaneous change in action. At the aggregation node t of the network, the output features of the previous node are... Current input features and time gradient Vector concatenation is performed, followed by linear fusion through a fully connected layer. Finally, the fused feature vector is left-multiplied by a fixed or learnable orthogonal matrix Q to generate the output feature of the current node. This feature This refers to multi-scale behavioral features that contain long-term temporal dependencies from the beginning to the current time t.

[0038] S3. Calculate the duration of gaze landing on each semantic region of the screen content image based on the facial video stream. Combine the content complexity of the semantic region to calculate the visual inertia coefficient for each semantic region. Construct a weighted gaze transfer matrix based on this and extract the main eigenvalue of the weighted gaze transfer matrix as a measure of visual focus stability.

[0039] Specifically, a deep learning gaze estimation model is used to infer the user's gaze direction vector in real time based on eye images and head pose in the facial video stream. This vector is then geometrically calibrated and mapped to two-dimensional coordinates on the screen, i.e., the gaze placement point. Simultaneously, a pre-trained image segmentation network, such as U-Net, is used to automatically segment the screen content image into multiple semantic regions, including text regions, image regions, code regions, and blank regions. Next, the content complexity of each semantic region is calculated; for example, the complexity of the text region can be defined as character density, and the complexity of the image region is the edge density calculated using the Sobel operator. Finally, the total dwell time of the gaze placement point in each semantic region i is statistically analyzed. And calculate the visual inertia coefficient. ,in, Let's define the content complexity of this region. Finally, construct a... a weighted visual line transition matrix, where N is the number of semantic regions, the diagonal elements of which are set to visual inertia coefficients , and the off-diagonal elements are set to the frequency of visual line transition from region i to region j. By performing eigenvalue decomposition on the weighted visual line transition matrix, the largest eigenvalue is extracted as the measure of visual focus stability.

[0040] S4, calculating the first-order time derivative of the head pose angular velocity in the facial video stream to obtain the head shaking impact degree; monitoring the instantaneous value of the head shaking impact degree, and adjusting the model parameters of the spatio-temporal feature recursive aggregation network when the preset threshold is exceeded in response to the mutation of the user behavior; and calculating the real-time concentration score of the user by comprehensively considering the multi-scale behavior features, the visual focus stability measure, and the head shaking impact degree.

[0041] Specifically, the preset threshold is .

[0042] Specifically, the pitch angle, yaw angle, and roll angle sequences of the head pose are calculated in real time from the facial video stream. The head pose angular velocity vector is obtained by calculating the change in the pose angle between two consecutive frames divided by the inter-frame time interval. The head pose angular acceleration vector is obtained by performing the same time difference operation on the angular velocity vector sequence. The Euclidean norm of the acceleration vector is taken as the instantaneous head shaking impact degree, which can sensitively reflect the rapid rotation or shaking of the head.

[0043] An impact degree threshold value determined according to empirical data is set, for example, The instantaneous value of the head shaking impact degree is continuously calculated and monitored. Once the value is greater than the set threshold, it is considered that the user has a behavior mutation, for example, suddenly turning his head due to external interference. At this time, the state of the spatio-temporal feature recursive aggregation network is immediately adjusted. The specific adjustment method is to attenuate the weight coefficient a in the network for fusing historical information, for example, multiplied by a decay factor less than 1, thereby reducing the influence of historical behavior features on the current judgment, so that the model can quickly adapt to the new behavior pattern.

[0044] Then, the multi-scale behavior feature vector output by the spatio-temporal feature recursive aggregation network at the current time step, the visual focus stability measure as a scalar, and the head shaking impact degree as a scalar are spliced into a high-dimensional comprehensive feature vector. Then, the comprehensive feature vector is input into a pre-trained multi-layer perceptron regression model. The output of the model is a continuous value between 0 and 1, which is the real-time concentration score of the user at that moment, where 1 represents complete concentration and 0 represents complete inattention.

[0045] ​​In an alternative embodiment, the spatio-temporal feature recursive aggregation network processes the facial video stream and the body pose video stream, extracts multi-scale behavior features, including:

[0046] The spatio-temporal feature recursive aggregation network comprises parallel processing branches that respectively extract facial features and pose features from the facial video stream and the body pose video stream; the spatio-temporal feature recursive aggregation network further comprises a feature fusion attention module for weighted fusion of the facial features and the pose features to generate the multi-scale behavior features.

[0047] The spatio-temporal feature recursive aggregation network fuses the output features of the previous node, the input features of the current time step, and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation, including:

[0048] The operation of the aggregation node in each parallel processing branch is to splice the output features of the previous node, the feature map extracted from the video stream at the current time step, and the time gradient of the feature map in the feature dimension to form an aggregation feature vector; the aggregation feature vector is input to a linear transformation layer, the weight matrix of the linear transformation layer is constrained to an orthogonal matrix through QR decomposition, and the output features of the current node are generated through the linear transformation layer.

[0049] For example, the network receives two video streams as input, such as a facial video and a pose video recording 18 body key point coordinates. The facial branch uses a convolutional neural network to process each frame of image and outputs a 512-dimensional feature map. The pose branch uses a multi-layer perceptron to process the key point coordinates and outputs a 256-dimensional feature.

[0050] Taking a certain aggregation node of the facial branch as an example, suppose the output features of the previous node are 512-dimensional, the feature map extracted from the current frame is 512-dimensional, and the time gradient is also 512-dimensional. After splicing the three, an aggregation vector of 1536 dimensions is formed. The vector is output through a linear layer with an orthogonal weight matrix, and a new 512-dimensional feature is output as the output of the current node.

[0051] At the end of the network, the facial branch and the pose branch respectively output feature vectors and . The feature fusion attention module calculates the attention weight, such as assigning a weight of 0.7 to the facial features and a weight of 0.3 to the pose features. The multi-scale behavior features are the sum of the two weighted feature vectors, i.e. The fused vector can more comprehensively represent the behavior state of the user.

[0052] In an optional embodiment, the step of calculating the visual inertia coefficient for each semantic region based on the content complexity of the semantic region includes:

[0053] Each semantic region of the screen content image is converted into a grayscale image, and the variance of the pixel values ​​after the Laplacian transform of the grayscale image is calculated. The variance of the pixel values ​​is used as the content complexity of the semantic region. The visual inertia coefficient is a normalized value of the content complexity.

[0054] For example, the screen can be divided into different semantic regions, such as a video playback area A and a code editing area B. For region A, a segment is captured. A color image of pixels is generated and converted to a grayscale image. Then, an atomizer is applied to the grayscale image. The Laplacian operator is used for convolution to enhance the edge and texture details of the image. Finally, the variance of all pixel values ​​in the transformed image is calculated; for example, the variance of video region A is 625.5, and this variance is used as the content complexity of that region.

[0055] Then, the same operation is performed on code editing area B. Because the code area content is relatively static and the texture is simple, the calculated variance is low, for example, 95.2. These two variances represent the content complexity of their respective areas. These two content complexities are then normalized, for example, using a minimum-maximum normalization method to map them to the interval between 0 and 1, yielding the visual inertia coefficient. Assuming the visual inertia coefficient of video area A is 1.0 and the visual inertia coefficient of code area B is 0.15, the coefficient values ​​reflect the ability of different areas to attract and maintain the user's visual attention.

[0056] In an optional embodiment, constructing the weighted gaze transfer matrix accordingly includes:

[0057] Statistically count the frequency of the gaze shifting from semantic region i to semantic region j within a preset time window. The frequency Multiplied by the visual inertia coefficient of semantic region j The weighted transfer value is obtained. Weighted transfer value The elements that make up matrix W, The calculation formula is: ;

[0058] Perform on all rows of matrix W Norm normalization is performed to make the sum of the elements in each row equal to 1, thereby constructing the weighted line-of-sight transition matrix.

[0059] For example, suppose there are two semantic regions on the screen: region 1 is the video area, and region 2 is the chat area. Their visual inertia coefficients are respectively... and Within a 5-second time window, the eye-tracking device recorded the number of times the gaze moved from region 1 to region 1. The number of times is 20, representing the number of times from region 1 to region 2. The number of times is 5, representing the number of times from region 2 to region 1. The number of times is 8, representing the number of times from region 2 to region 2. The frequency is 15 times. These data constitute the original frequency matrix N. Then, the weighted transition value is calculated according to the formula to obtain... , , , This yields matrix W. Then, each row of matrix W is normalized; the sum of the first row is 20. After normalization, it becomes 0.9. After normalization, the result is 0.1; the sum in the second row is 13.2. After normalization, it becomes 0.55. After normalization, the value is 0.45; thus, the weighted gaze shift matrix is ​​obtained. The constructed weighted gaze shift matrix reflects the probability of user attention shifts between regions of different complexity.

[0060] In an optional embodiment, calculating the first-order time derivative of the head pose angular velocity in the facial video stream to obtain the head sway impact includes:

[0061] The Euler angle sequence of head pose is extracted from consecutive video frames of the facial video stream, and the angular velocity is calculated. The first time derivative of the angular velocity is calculated using the backward difference method to obtain the head sway impact, which is calculated using the following formula:

[0062] ;

[0063] in, for The impact of head shaking at any moment, for angular velocity at time t, for angular velocity at time t, This represents the time interval between video frames.

[0064] For example, suppose we are processing a facial video recorded at 30 frames per second, with a time interval Δt between adjacent video frames of 0.033 s. At time t, the yaw angle of the head is estimated to be 25° using a facial landmark detection algorithm. In the previous frame, at time t-Δt, the measured yaw angle was 24°. angular velocity at time t Dividing the change in angle by the time interval gives approximately equal to Assuming that the angular velocity calculated at the t-Δt moment is According to the calculation formula , the head shaking impact degree is calculated to be about .

[0065] In an optional embodiment, the model parameters of the spatio-temporal feature recursive aggregation network are adjusted, including:

[0066] Adjusting the attention weight in the feature fusion attention module, the attention weight allocated to the body posture video stream feature is increased by a fixed adjustment value , and the attention weight allocated to the facial video stream feature is decreased by a fixed adjustment value .

[0067] For example, assuming that under normal circumstances, is 0.6, is 0.4. Real-time monitoring of the user's head shaking impact degree, assuming that at a certain moment, the user suddenly turns his head, and the calculated impact degree instantaneous value is , which exceeds the preset threshold . At this time, the weight adjustment mechanism will be automatically triggered, an adjustment value Δα will be added, assuming Δα=0.1, and the new weight becomes 0.5. Subtract the adjustment value Δα, and the new weight becomes 0.5. In this way, when detecting a violent head movement, the model will dynamically increase the attention to the overall information of the body posture, because such a sudden behavior may indicate a major shift in attention or distraction, rather than just a slight change in facial expression.

[0068] In an optional embodiment, the multi-scale behavior features, visual focus stability measures, and head shaking impact degrees are integrated to calculate the user's real-time concentration score, including:

[0069] The multi-scale behavior features, visual focus stability measures, and head shaking impact degrees output by the spatio-temporal feature recursive aggregation network are spliced in the feature dimension to form a comprehensive feature vector; the comprehensive feature vector is input into a pre-trained three-layer fully connected neural network, and the output layer of the three-layer fully connected neural network is a single neuron, and the output value is the user's real-time concentration score.

[0070] For example, at a certain time point, the spatio-temporal feature recursive aggregation network outputs a 256-dimensional multi-scale behavior feature vector representing the user's micro-expression and posture; by analyzing the weighted gaze shift matrix, the visual focus stability measure is calculated to be 0.88, and the higher the value, the more stable the gaze; the current head shaking impact degree is measured to be These three data are spliced into a single comprehensive feature vector with a dimension of 258. The comprehensive feature vector is input into a pre-trained three-layer fully connected neural network, which includes a 258-dimensional input layer, a 64-dimensional hidden layer, and a 1-dimensional output layer. The network performs a nonlinear transformation on the input vector, and finally the neuron of the output layer gives a value between 0 and 1, for example, 0.76. This value is the evaluation of the user's real-time concentration score at the current time.

[0071] The implementation principle of the multi-modal behavior analysis-based concentration unobtrusive monitoring method of the embodiment of the application is as follows: the application adopts a unique spatio-temporal feature recursive aggregation network, which can deeply analyze multi-scale behavior features of user's face and body posture in a long time sequence, and capture the continuous evolution process of behavior. The application innovatively integrates the semantic complexity of screen content into gaze analysis, measures the stability of visual focus by constructing a weighted gaze shift matrix, so that the evaluation is no longer limited to spatial position, but goes deep into the content level. In addition, the application also introduces a head shaking impact degree index to sensitively detect sudden head movements that may indicate distraction. By effectively fusing the above multi-dimensional and deep features, the application significantly improves the comprehensiveness and reliability of concentration monitoring.

[0072] The embodiment of the multi-modal behavior analysis-based concentration unobtrusive monitoring system provided by the application includes a memory and a processor, the memory stores computer instructions, and the processor executes the computer instructions to realize the multi-modal behavior analysis-based concentration unobtrusive monitoring method in the above embodiment.

[0073] Although the embodiments of the application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the application.

Claims

1. A method for monitoring concentration without awareness based on multi-modal behavior analysis, characterized in that, The method comprises the following steps: obtaining a facial video stream, a body posture video stream and a screen content image of a user; The facial video stream and the body posture video stream are processed by using a spatio-temporal feature recursive aggregation network to extract multi-scale behavior features, the spatio-temporal feature recursive aggregation network fuses the output features of a previous node, the input features of a current time step and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation. The dwell time of the gaze landing point in each semantic region of the screen content image is calculated based on the facial video stream, the visual inertia coefficients of each semantic region are calculated in combination with the content complexity of the semantic regions, a weighted gaze transition matrix is constructed according to the visual inertia coefficients, and the principal eigenvalue of the weighted gaze transition matrix is extracted as the visual focus stability measure. The first-order time derivative of the head posture angular velocity in the facial video stream is calculated to obtain the head shaking impact degree, the instantaneous value of the head shaking impact degree is monitored, and when the instantaneous value exceeds a preset threshold, the model parameters of the spatio-temporal feature recursive aggregation network are adjusted to respond to the mutation of the user behavior; and the multi-scale behavior features, the visual focus stability measure and the head shaking impact degree are comprehensively considered to calculate the real-time concentration score of the user.

2. The method of claim 1, wherein the method further comprises: The facial video stream and the body posture video stream are processed by using a spatio-temporal feature recursive aggregation network to extract multi-scale behavior features, the spatio-temporal feature recursive aggregation network fuses the output features of a previous node, the input features of a current time step and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation. The spatio-temporal feature recursive aggregation network comprises parallel processing branches for extracting facial features and posture features from the facial video stream and the body posture video stream, respectively; and the spatio-temporal feature recursive aggregation network further comprises a feature fusion attention module for weighted fusion of the facial features and the posture features to generate the multi-scale behavior features. 3.The method of claim 2, wherein, The spatio-temporal feature recursive aggregation network fuses the output features of a previous node, the input features of a current time step and the time gradient of the input features at each aggregation node, and generates the output features of the current node through orthogonal transformation, and the method comprises the following steps: The operation of the aggregation node in each parallel processing branch is as follows: the output features of a previous node, the feature map extracted from the video stream at a current time step and the time gradient of the feature map are spliced in the feature dimension to form an aggregation feature vector; and the aggregation feature vector is input into a linear transformation layer, the weight matrix of the linear transformation layer is constrained to be an orthogonal matrix through QR decomposition, and the output features of the current node are generated through the linear transformation layer. 4.The method of claim 1, wherein, The content complexity of each semantic region is used to calculate the visual inertia coefficient of each semantic region, and the method comprises the following steps: Each semantic region of the screen content image is converted into a gray image, and the pixel value variance of the gray image after Laplace transformation is calculated as the content complexity of the semantic region; and the visual inertia coefficient is a normalized value of the content complexity. 5.The multi-modal behavior analysis based concentration monitoring method of claim 1, wherein, The weighted gaze transition matrix is constructed according to the visual inertia coefficients, and the method comprises the following steps: counting the frequency of the visual line falling point transferring from semantic region i to semantic region j in a preset time window ; multiplying the frequency by the visual inertia coefficient of semantic region j to obtain a weighted transfer value The weighted transfer value is an element of the matrix W, and the calculation formula is: ; all rows of the matrix W are normalized The weighted visual line transition matrix is constructed by normalizing each row element to 1. 6.The multi-modal behavior analysis based concentration monitoring method of claim 1, wherein, The first-order time derivative of the head posture angular velocity in the facial video stream is calculated to obtain the head shaking impact degree, the instantaneous value of the head shaking impact degree is monitored, and when the instantaneous value exceeds a preset threshold, the model parameters of the spatio-temporal feature recursive aggregation network are adjusted to respond to the mutation of the user behavior; and the multi-scale behavior features, the visual focus stability measure and the head shaking impact degree are comprehensively considered to calculate the real-time concentration score of the user. Euler angle sequences of head pose are extracted from consecutive video frames of the facial video stream, and angular velocities are calculated; a backward difference method is used to calculate the first-order time derivative of the angular velocity to obtain a head shaking impact degree, and the head shaking impact degree is calculated by the following formula: ； wherein is the head shaking impact degree at the moment, is the angular velocity at the moment, is the angular velocity at the moment, is the time interval between video frames.

7. The method of claim 2, wherein the method further comprises: The preset threshold is .

8. The method of claim 7, wherein the method further comprises: The model parameters of the spatio-temporal feature recursive aggregation network are adjusted, including: adjusting the attention weight in the feature fusion attention module, assigning the attention weight to the body posture video stream feature increasing the fixed adjustment value , while reducing the attention weight assigned to the facial video stream feature decreasing . 9.The multi-modal behavior analysis based concentration monitoring method of claim 1, wherein, The multi-scale behavior features, the visual focus stability measure, and the head shaking impact degree are integrated to calculate a real-time concentration score of the user, including: The multi-scale behavior features, the visual focus stability measure, and the head shaking impact degree output by the spatio-temporal feature recursive aggregation network are spliced in the feature dimension to form a comprehensive feature vector; the comprehensive feature vector is input into a pre-trained three-layer fully connected neural network, and the output layer of the three-layer fully connected neural network is a single neuron, and the output value is the real-time concentration score of the user.

10. A concentration monitoring system based on multi-modal behavioral analysis, characterized in that, The method comprises a memory and a processor, the memory stores computer instructions, and the processor executes the computer instructions to implement the concentration monitoring method based on multi-modal behavior analysis according to any one of claims 1-9. The method comprises a memory and a processor, the memory stores computer instructions, and the processor executes the computer instructions to implement the concentration monitoring method based on multi-modal behavior analysis according to any one of claims 1-9.

Images