Attention state analysis method, system and device based on fine-grained hierarchical semantic fusion and storage medium

Abstract

The invention discloses an attention state analysis method, system and device based on fine-grained hierarchical semantic fusion and a storage medium. The method comprises the following steps: acquiring multi-modal image data of a target object; inputting the multi-modal image data into a pre-trained feature extraction mapping model, and performing hierarchical extraction to obtain fine-grained semantic features of each modal; wherein the feature extraction mapping model comprises an expression fine-grained feature extraction module, a head posture fine-grained feature extraction module and a blink fine-grained feature extraction module; inputting the fine-grained semantic features of each modal into a pre-trained feature fusion model for multi-modal feature fusion to obtain final fusion features; and based on the final fusion feature, obtaining an attention state analysis result of the target object through a pre-constructed attention state prediction model. According to the method, the accuracy and reliability of attention state analysis are remarkably improved by fusing the multi-modal fine-grained features.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent teaching technology, and specifically relates to an attention state analysis method, system, device and storage medium based on fine-grained hierarchical semantic fusion. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and computer vision technology, automated attention analysis methods based on physiological signals and behavioral characteristics have gradually emerged. More and more attention state analysis methods focus on multiple modal features of the target object to obtain more accurate analysis results.

[0003] However, most existing technologies rely on coarse-grained multimodal features, such as only recognizing basic expression categories (such as happiness, sadness, anger) or simple head posture angles (pitch, yaw, roll). These coarse-grained feature information cannot capture more subtle expression changes that are closely related to the state of attention, small adjustments in the head posture, and complex blinking patterns, which limits the accuracy and reliability of attention analysis. At the same time, most existing technologies are limited to simple feature fusion strategies. Even if some existing technologies try to fuse multimodal features, most of them use simple linear weighting or simple connection methods, which cannot effectively capture the complex interactions and nonlinear associations between different modal features. This simple fusion strategy cannot fully utilize the complementarity of multimodal data, limiting the accuracy of attention analysis. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide an attention state analysis method, system, device and storage medium based on fine-grained hierarchical semantic fusion, which significantly improves the accuracy and reliability of attention state analysis by fusing multimodal fine-grained features.

[0005] The present invention provides the following technical solutions:

[0006] In a first aspect, a method for analyzing attention state based on fine-grained hierarchical semantic fusion is provided, comprising: obtaining multimodal image data of a target object, which comprises expression image data, head posture image data and blink image data; inputting the multimodal image data into a pre-trained feature extraction mapping model, and obtaining fine-grained semantic features of each modality through hierarchical extraction; wherein the feature extraction mapping model comprises an expression fine-grained feature extraction module, a head posture fine-grained feature extraction module and a blink fine-grained feature extraction module; inputting the fine-grained semantic features of each modality into a pre-trained feature fusion model for multimodal feature fusion to obtain a final fused feature; based on the final fused feature, obtaining the attention state analysis result of the target object through a pre-constructed attention state prediction model.

[0007] As a preferred technical solution of the present invention, the multimodal image data is input into a pre-trained feature extraction mapping model, and the fine-grained semantic features of each modality are obtained through hierarchical extraction, including:

[0008] The expression fine-grained feature extraction module extracts the expression fine-grained feature vector in the expression image data, which is expressed as:

[0009] ;

[0010] in, represents the fine-grained feature vector of expression, n represents the total number of fine-grained features of expression, represents the i-th expression fine-grained feature, , represents the activation state of the i-th facial action unit, represents the activation strength of the i-th facial action unit, represents the duration of the i-th facial action unit;

[0011] The head posture fine-grained feature extraction module extracts the head posture fine-grained feature vector in the head posture image data, which is expressed as:

[0012] ;

[0013] ;

[0014] ;

[0015] in, represents the fine-grained feature vector of head posture, Represents static characteristics, represents the pitch angle, represents the yaw angle, represents the roll angle, Represents dynamic features, Represent the speed of x, y, and z axes respectively. Represent the acceleration of x, y, and z axes respectively;

[0016] The blink fine-grained feature extraction module extracts the blink fine-grained feature vector in the blink image data, which is expressed as:

[0017] ;

[0018] ;

[0019] ;

[0020] in, represents the fine-grained feature vector of eye blink, Indicates the basic characteristics of blinking, Indicates the blink frequency, Indicates the duration of blinking. Indicates the blink time interval, represents the blink pattern characteristics, represents the blink duration variance, represents the blink interval skewness, Indicates blink rhythm.

[0021] As a preferred technical solution of the present invention, the multimodal image data is input into a pre-trained feature extraction mapping model, and the fine-grained semantic features of each modality are obtained through hierarchical extraction, which also includes:

[0022] The fine-grained feature vectors of each mode are transformed nonlinearly and expressed as:

[0023] ;

[0024] ;

[0025] in, represents the features obtained by the first nonlinear transformation, , σ represents the LeakyReLU activation function, represents the first layer weight matrix, represents the fine-grained feature vector of each mode, represents the bias vector, represents the features obtained by the second nonlinear transformation, represents the second layer weight matrix, represents the bias vector;

[0026] Based on the features obtained by the second nonlinear transformation of each mode, the residual connection is introduced to obtain the integrated features of each mode, which can be expressed as:

[0027] ;

[0028] in, Represents the integrated features of each mode, and Linear represents the linear mapping layer.

[0029] As a preferred technical solution of the present invention, the multimodal image data is input into a pre-trained feature extraction mapping model, and the fine-grained semantic features of each modality are obtained through hierarchical extraction, which also includes:

[0030] Calculate the temporal attention of the integrated features of each modality, expressed as:

[0031] ;

[0032] ;

[0033] in, represents the temporal attention of each modality, represents normalization, represents the query vector, express The mapping weight of represents the key vector, express The mapping weight of represents a value vector, express The mapping weight of , T represents transpose, express Dimensions;

[0034] Based on the temporal attention of each modality, the feature attention of each modality is calculated, which is expressed as:

[0035] ;

[0036] ;

[0037] in, represents the feature attention of each modality, represents normalization, represents the query vector, express The mapping weight of represents the key vector, express The mapping weight of represents a value vector, express The mapping weight of , T represents transpose, express Dimensions;

[0038] Based on the temporal attention and feature attention of each modality, the fine-grained semantic features of each modality are generated, which can be expressed as:

[0039] ;

[0040] in, Represents the fine-grained semantic features of each modality, Representation layer normalization, represents a two-layer feed-forward neural network.

[0041] As a preferred technical solution of the present invention, the method of inputting the fine-grained semantic features of each modality into a pre-trained feature fusion model to perform multi-modal feature fusion to obtain the final fusion feature includes:

[0042] Calculating the attention weight between any two modalities is expressed as:

[0043] ;

[0044] in, Indicates The attention weights of the jth modality and the jth modality, Indicates Fine-grained semantic features of each modality, Indicates The attention weight matrix between the jth modality and the jth modality, , all represent different modes;

[0045] Based on the fine-grained semantic features of each modality, residual connections and layer normalization are introduced to obtain the enhanced features of each modality, which can be expressed as:

[0046] ;

[0047] in, represents the enhanced features of each mode, Representation layer normalization, Indicates Fine-grained semantic features of each modality, represents a linear transformation matrix;

[0048] The enhanced features of all modalities are fused to obtain the first fused feature, which is expressed as:

[0049] ;

[0050] ;

[0051] in, represents the first fusion feature, represents the multi-attention head feature fusion operation, Represent the enhanced features of the 1st, 2nd and 3rd modes respectively, Represents a splicing operation, represents the h-th attention head feature, represents a linear transformation matrix;

[0052] The final fusion feature is obtained based on the first fusion feature value, expressed as:

[0053] ;

[0054] in, represents the final fusion feature, represents a two-layer feedforward neural network, Representation layer normalization.

[0055] As a preferred technical solution of the present invention, the method of obtaining the attention state analysis result of the target object based on the final fusion feature through a pre-built attention state prediction model includes:

[0056] The attention state prediction model includes a context information encoding module, which encodes context information in the classroom into context features;

[0057] The context feature is fused and enhanced with the final fusion feature to obtain the attention state feature, which is expressed as:

[0058] ;

[0059] in, Represents the attention state characteristics, represents a multi-layer perceptron, represents the final fusion feature, and C represents the context feature;

[0060] The attention state prediction model also includes a time series modeling module, which is used to capture the dynamic change characteristics in the attention state characteristics, expressed as:

[0061] ;

[0062] in, Indicates the dynamic change characteristics of the current moment. is the dynamic change characteristic of the previous moment, Represents a gated loop operation.

[0063] As a preferred technical solution of the present invention, the attention state analysis result of the target object is obtained based on the final fusion feature through a pre-built attention state prediction model, and further includes:

[0064] The attention level of the target object is predicted, expressed as:

[0065] ;

[0066] in, Indicates the level of attention, [0, 1], sigmoid represents the activation function, represents the weight term, represents the bias term;

[0067] The attention type of the target object is predicted, expressed as:

[0068] ;

[0069] in, Indicates attention type, which includes focused, distracted, confused, and tired, represents normalization, represents the weight term, Represents the bias term.

[0070] In a second aspect, an attention state analysis system based on fine-grained hierarchical semantic fusion is provided, comprising: a data acquisition module for acquiring multimodal image data of a target object, which includes expression image data, head posture image data, and blink image data;

[0071] A feature extraction module, used to input the multimodal image data into a pre-trained feature extraction mapping model, and obtain fine-grained semantic features of each modality through hierarchical extraction; wherein the feature extraction mapping model includes an expression fine-grained feature extraction module, a head posture fine-grained feature extraction module, and a blink fine-grained feature extraction module;

[0072] A feature fusion module is used to input the fine-grained semantic features of each modality into a pre-trained feature fusion model to perform multi-modal feature fusion to obtain a final fusion feature;

[0073] The analysis module is used to obtain the attention state analysis result of the target object based on the final fusion features through a pre-built attention state prediction model.

[0074] In a third aspect, an attention state analysis device based on fine-grained hierarchical semantic fusion is provided, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

[0075] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] The present invention provides an attention state analysis method based on fine-grained hierarchical semantic fusion. The method analyzes the attention state of a target object through fusion features obtained by fine-grained feature extraction and hierarchical semantic fusion. It focuses on more subtle motion features of the target object and focuses on the complex nonlinear relationship between different fine-grained features. It achieves a more accurate and reliable analysis of human attention state, and has higher value and broader application prospects in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of an attention state analysis method based on fine-grained hierarchical semantic fusion in an embodiment of the present invention;

[0079] Figure 2 is a diagram showing the analysis result of the state of attention concentration in an embodiment of the present invention;

[0080] Figure 3 is a diagram of the analysis result of the distraction state in an embodiment of the present invention;

[0081] Figure 4 is a diagram showing the analysis result of the attention confusion state in an embodiment of the present invention;

[0082] Figure 5 4 is a diagram showing the analysis result of the attention fatigue state in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0084] Example 1

[0085] This embodiment provides an attention state analysis method based on fine-grained hierarchical semantic fusion. Figure 1 As shown, the specific steps include:

[0086] Step 1: Acquire multimodal image data of the target object, which includes expression image data, head posture image data and blinking image data.

[0087] The facial expression image data, the head posture image data and the blinking image data are all captured by a camera.

[0088] Step 2: Input the multimodal image data into a pre-trained feature extraction mapping model, and obtain fine-grained semantic features of each modality through hierarchical extraction.

[0089] The feature extraction and mapping model is divided into bottom layer, middle layer and high layer for feature extraction and mapping. The feature extraction and mapping model includes expression fine-grained feature extraction module, head posture fine-grained feature extraction module and blink fine-grained feature extraction module for bottom layer feature extraction.

[0090] 2.1. Low-level feature extraction

[0091] (1) The expression fine-grained feature extraction module extracts the expression fine-grained feature vector in the expression image data, which is expressed as:

[0092] ;

[0093] in, represents the fine-grained feature vector of expression, n represents the total number of fine-grained features of expression, represents the i-th expression fine-grained feature, , represents the activation state of the i-th facial action unit, represents the activation strength of the i-th facial action unit, represents the duration of the i-th facial action unit.

[0094] In this embodiment, the expression fine-grained feature extraction module uses the deep learning model OpenFace 2.0, which is based on facial action units (AUs) and uses facial landmark tracking technology to identify micro-expressions, such as a brief frown and a slight upward turn of the mouth corners. These fine-grained expression features can more accurately reflect the user's emotional state. For example, the co-activation and duration of AU1 and AU4 indicate confusion; the brief appearance of AU12 suggests that the target object is trying to conceal its misunderstanding.

[0095] (2) The head posture fine-grained feature extraction module is used to extract the head posture fine-grained feature vector in the head posture image data, which is expressed as:

[0096] ;

[0097] ;

[0098] ;

[0099] in, represents the fine-grained feature vector of head posture, Represents static characteristics, represents the pitch angle, represents the yaw angle, represents the roll angle, Represents dynamic features, Represent the speed of x, y, and z axes respectively. Represents the acceleration of the x, y, and z axes respectively.

[0100] In this embodiment, the head posture fine-grained feature extraction module adopts a head posture estimation model based on 3D face reconstruction. It not only extracts static head posture angles (Pitch, Yaw, Roll), but also tracks the head movement trajectory and speed. For example, frequent small swings of the head indicate that the target object is trying to concentrate, while long periods of stillness indicate distraction.

[0101] (3) The blink fine-grained feature extraction module is used to extract the blink fine-grained feature vector in the blink image data, which is expressed as:

[0102] ;

[0103] ;

[0104] ;

[0105] in, represents the fine-grained feature vector of eye blink, Indicates the basic characteristics of blinking, Indicates the blink frequency, Indicates the duration of blinking. Indicates the blink time interval, represents the blink pattern characteristics, represents the blink duration variance, represents the blink interval skewness, Indicates blink rhythm, which includes rapid blink, long blink, double blink and irregular blink.

[0106] In this embodiment, the blink fine-grained feature extraction module adopts a blink pattern analysis model based on eye key point detection. Not only the blink frequency is calculated, but also the blink duration, blink interval and blink pattern are analyzed. For example, the variance of the blink interval and the skewness of the blink duration are used to evaluate the fatigue level of the target object; frequent blinking indicates anxiety or tension.

[0107] 2.2 Mid-level feature fusion

[0108] 2.2.1. Perform nonlinear transformation on the fine-grained feature vectors of each mode, expressed as:

[0109] ;

[0110] ;

[0111] in, represents the features obtained by the first nonlinear transformation, , σ represents the LeakyReLU activation function, represents the first layer weight matrix, represents the fine-grained feature vector of each mode, represents the bias vector, represents the features obtained by the second nonlinear transformation, represents the second layer weight matrix, Represents the bias vector.

[0112] 2.2.2. Based on the features obtained by the second nonlinear transformation of each mode, the residual connection is introduced to obtain the integrated features of each mode to alleviate the gradient vanishing problem, which is expressed as:

[0113] ;

[0114] in, Represents the integrated features of each modality, and Linear represents the linear mapping layer, which is used to adjust the feature dimension.

[0115] 2.3. High-level semantic mapping, taking into account the dependencies of both temporal and feature dimensions.

[0116] 2.3.1. Calculate the temporal attention of the integrated features of each modality, expressed as:

[0117] ;

[0118] ;

[0119] in, represents the temporal attention of each modality, represents normalization, represents the query vector, express The mapping weight of represents the key vector, express The mapping weight of represents a value vector, express The mapping weight of , T represents transpose, express Dimension.

[0120] 2.3.2. Based on the temporal attention of each modality, the feature attention of each modality is calculated, which is expressed as:

[0121] ;

[0122] ;

[0123] in, represents the feature attention of each modality, represents normalization, represents the query vector, express The mapping weight of represents the key vector, express The mapping weight of represents a value vector, express The mapping weight of , T represents transpose, express Dimension.

[0124] 2.3.3. Based on the temporal attention and feature attention of each modality, the fine-grained semantic features of each modality are generated, expressed as:

[0125] ;

[0126] in, Represents the fine-grained semantic features of each modality, Representation layer normalization, represents a two-layer feed-forward neural network.

[0127] Step 3: Input the fine-grained semantic features of each modality into a pre-trained feature fusion model to perform multimodal feature fusion to obtain the final fusion feature.

[0128] In this embodiment, the feature fusion model adopts a Transformer-based cross-modal fusion model to achieve deep interaction between different modalities. Specifically, it includes:

[0129] 3.1. Calculate the attention weight between any two modalities as:

[0130] ;

[0131] in, Indicates The attention weights of the jth modality and the jth modality, Indicates Fine-grained semantic features of each modality, Indicates The attention weight matrix between the jth modality and the jth modality, , all represent different modes.

[0132] 3.2. Based on the fine-grained semantic features of each modality, residual connection and layer normalization are introduced to obtain the enhanced features of each modality, which can be expressed as:

[0133] ;

[0134] in, represents the enhanced features of each mode, Representation layer normalization, Indicates Fine-grained semantic features of each modality, represents the linear transformation matrix used to project the features of the jth mode into the same feature space as the ith mode.

[0135] 3.3. Fuse the enhanced features of all modalities to obtain the first fusion feature, expressed as:

[0136] ;

[0137] ;

[0138] in, represents the first fusion feature, represents the multi-attention head feature fusion operation, Represent the enhanced features of the 1st, 2nd and 3rd modes respectively, Represents a splicing operation, represents the h-th attention head feature, Represents a linear transformation matrix.

[0139] In this embodiment, the number of attention heads is set to 8.

[0140] 3.4. The final fusion feature is obtained based on the first fusion feature value, expressed as:

[0141] ;

[0142] in, represents the final fusion feature, represents a two-layer feedforward neural network, Representation layer normalization.

[0143] Step 4: Based on the final fusion features, the attention state analysis results of the target object are obtained through a pre-built attention state prediction model.

[0144] 4.1. In order to improve the accuracy and adaptability of prediction, the attention state prediction model includes a context information encoding module, which encodes the context information in the classroom into context features. This module takes into account the following factors: The difficulty of the course uses a predefined difficulty level (1-5) to represent the complexity of the current teaching content. The type of teaching activities is represented by one-hot encoding, such as explanation, discussion, experiment, etc. Time information is normalized to the range of [0, 1], including the duration of the course and the current time point.

[0145] The context feature is fused and enhanced with the final fusion feature to obtain the attention state feature, which is expressed as:

[0146] ;

[0147] in, Represents the attention state characteristics, represents a multi-layer perceptron, represents the final fusion feature, and C represents the context feature.

[0148] 4.2. The attention state prediction model also includes a time series modeling module, which is used to capture the dynamic change characteristics in the attention state characteristics, expressed as:

[0149] ;

[0150] in, Indicates the dynamic change characteristics of the current moment. is the dynamic change characteristic of the previous moment, Represents a gated loop operation.

[0151] 4.3. Predict the attention level of the target object, expressed as:

[0152] ;

[0153] in, Indicates the level of attention, [0, 1], sigmoid represents the activation function, represents the weight term, Represents the bias term.

[0154] 4.4. Predict the attention type of the target object, expressed as:

[0155] ;

[0156] in, Indicates attention type, which includes focused, distracted, confused, and tired, represents normalization, represents the weight term, Represents the bias term.

[0157] This embodiment predicts the attention level and attention type at the same time, which can make full use of their correlation and improve the accuracy and robustness of the prediction. Figure 2-5As shown, the timeline shows the changes in attention levels over time. Attention levels are expressed as values ​​between 0 and 1, where 0 represents no attention at all and 1 represents high concentration. By analyzing the curve chart, you can observe the fluctuations in the target object's attention level in different time periods. If the attention level curve is close to 1 during a certain period of time, and the bar chart shows that the attention type for that period is "focused", this indicates that the student is very focused during this period. If the attention level curve drops significantly during certain periods of time, and the corresponding bar chart shows that the attention type is "distracted", "confused" or "tired", this indicates that the student may be distracted by the outside world, have difficulty understanding the current content, or feel tired during these periods.

[0158] By observing the curve graph, we can find the fluctuation pattern of the target object's attention level. For example, whether there are regular valleys and peaks can help analyze in what environment or time the target object's attention is most likely to be focused or distracted.

[0159] Example 2

[0160] This embodiment provides an attention state analysis system based on fine-grained hierarchical semantic fusion, including:

[0161] The data acquisition module is used to acquire multimodal image data of the target object, including expression image data, head posture image data and blinking image data.

[0162] A feature extraction module is used to input the multimodal image data into a pre-trained feature extraction mapping model, and obtain the fine-grained semantic features of each modality through hierarchical extraction; wherein the feature extraction mapping model includes an expression fine-grained feature extraction module, a head posture fine-grained feature extraction module and a blinking fine-grained feature extraction module.

[0163] The feature fusion module is used to input the fine-grained semantic features of each modality into a pre-trained feature fusion model to perform multi-modal feature fusion to obtain the final fusion feature.

[0164] The analysis module is used to obtain the attention state analysis result of the target object based on the final fusion features through a pre-built attention state prediction model.

[0165] Example 3

[0166] This embodiment provides an attention state analysis device based on fine-grained hierarchical semantic fusion, including a processor and a storage medium. The storage medium is used to store instructions. The processor is used to operate according to the instructions to execute the steps of the attention state analysis method based on fine-grained hierarchical semantic fusion described in Example 1.

[0167] Example 4

[0168] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the attention state analysis method based on fine-grained hierarchical semantic fusion described in Embodiment 1 are implemented.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for analyzing attention states based on fine-grained hierarchical semantic fusion, characterized in that: include: Acquire multimodal image data of the target object, which includes expression image data, head posture image data, and blink image data; Input the multimodal image data into a pre-trained feature extraction mapping model, and obtain fine-grained semantic features of each modality through hierarchical extraction; wherein the feature extraction mapping model includes an expression fine-grained feature extraction module, a head posture fine-grained feature extraction module, and a blink fine-grained feature extraction module; Inputting the fine-grained semantic features of each modality into a pre-trained feature fusion model to perform multi-modal feature fusion to obtain a final fusion feature; Based on the final fusion features, the attention state analysis result of the target object is obtained through a pre-built attention state prediction model.

2. The attention state analysis method based on fine-grained hierarchical semantic fusion according to claim 1 is characterized in that: The multimodal image data is input into a pre-trained feature extraction mapping model, and the fine-grained semantic features of each modality are obtained through hierarchical extraction, including: The expression fine-grained feature extraction module extracts the expression fine-grained feature vector in the expression image data, which is expressed as: ； in, represents the fine-grained feature vector of expression, n represents the total number of fine-grained features of expression, represents the i-th expression fine-grained feature, , represents the activation state of the i-th facial action unit, represents the activation strength of the i-th facial action unit, represents the duration of the i-th facial action unit; The head posture fine-grained feature extraction module extracts the head posture fine-grained feature vector in the head posture image data, which is expressed as: ； ； ； in, represents the fine-grained feature vector of head posture, Represents static characteristics, represents the pitch angle, represents the yaw angle, represents the roll angle, Represents dynamic features, Represent the speed of x, y, and z axes respectively. Represent the acceleration of x, y, and z axes respectively; The blink fine-grained feature extraction module extracts the blink fine-grained feature vector in the blink image data, which is expressed as: ； ； ； in, represents the fine-grained feature vector of eye blink, Indicates the basic characteristics of blinking, Indicates the blink frequency, Indicates the duration of blinking. Indicates the blink time interval, represents the blink pattern characteristics, represents the blink duration variance, represents the blink interval skewness, Indicates blink rhythm.

3. The attention state analysis method based on fine-grained hierarchical semantic fusion according to claim 2 is characterized in that: The method of inputting the multimodal image data into a pre-trained feature extraction mapping model and obtaining fine-grained semantic features of each modality through hierarchical extraction also includes: The fine-grained feature vectors of each mode are transformed nonlinearly and expressed as: ； ； in, represents the features obtained by the first nonlinear transformation, ,σ represents the LeakyReLU activation function, represents the first layer weight matrix, represents the fine-grained feature vector of each mode, represents the bias vector, represents the features obtained by the second nonlinear transformation, represents the second layer weight matrix, represents the bias vector; Based on the features obtained by the second nonlinear transformation of each mode, the residual connection is introduced to obtain the integrated features of each mode, which can be expressed as: ； in, Represents the integrated features of each mode, and Linear represents the linear mapping layer.

4. The attention state analysis method based on fine-grained hierarchical semantic fusion according to claim 3 is characterized in that: The method of inputting the multimodal image data into a pre-trained feature extraction mapping model and obtaining fine-grained semantic features of each modality through hierarchical extraction also includes: Calculate the temporal attention of the integrated features of each modality, expressed as: ； ； in, represents the temporal attention of each modality, represents normalization, represents the query vector, express The mapping weight of represents the key vector, express The mapping weight of represents a value vector, express The mapping weight of , T represents transpose, express Dimensions; Based on the temporal attention of each modality, the feature attention of each modality is calculated, which is expressed as: ； ； in, represents the feature attention of each modality, represents normalization, represents the query vector, express The mapping weight of represents the key vector, express The mapping weight of represents a value vector, express The mapping weight of , T represents transpose, express Dimensions; Based on the temporal attention and feature attention of each modality, the fine-grained semantic features of each modality are generated, which can be expressed as: ； in, Represents the fine-grained semantic features of each modality, Representation layer normalization, represents a two-layer feed-forward neural network.

5. The attention state analysis method based on fine-grained hierarchical semantic fusion according to claim 1 is characterized in that: The method of inputting the fine-grained semantic features of each modality into a pre-trained feature fusion model to perform multi-modal feature fusion to obtain the final fusion features includes: Calculating the attention weight between any two modalities is expressed as: ； in, Indicates The attention weights of the jth modality and the jth modality, Indicates Fine-grained semantic features of each modality, Indicates The attention weight matrix between the jth modality and the jth modality, , all represent different modes; Based on the fine-grained semantic features of each modality, residual connections and layer normalization are introduced to obtain the enhanced features of each modality, which can be expressed as: ； in, represents the enhanced features of each mode, Representation layer normalization, Indicates Fine-grained semantic features of each modality, represents a linear transformation matrix; The enhanced features of all modalities are fused to obtain the first fused feature, which is expressed as: ； ； in, represents the first fusion feature, represents the multi-attention head feature fusion operation, Represent the enhanced features of the 1st, 2nd and 3rd modes respectively, Represents a splicing operation, represents the h-th attention head feature, represents a linear transformation matrix; The final fusion feature is obtained based on the first fusion feature value, expressed as: ； in, represents the final fusion feature, represents a two-layer feedforward neural network, Representation layer normalization.

6. The attention state analysis method based on fine-grained hierarchical semantic fusion according to claim 1 is characterized in that: The method of obtaining the attention state analysis result of the target object based on the final fusion feature through a pre-built attention state prediction model includes: The attention state prediction model includes a context information encoding module, which encodes context information in the classroom into context features; The context feature is fused and enhanced with the final fusion feature to obtain the attention state feature, which is expressed as: ； in, Represents the attention state characteristics, represents a multi-layer perceptron, represents the final fusion feature, and C represents the context feature; The attention state prediction model also includes a time series modeling module, which is used to capture the dynamic change characteristics in the attention state characteristics, expressed as: ； in, Indicates the dynamic change characteristics of the current moment. is the dynamic change characteristic of the previous moment, Represents a gated loop operation.

7. The attention state analysis method based on fine-grained hierarchical semantic fusion according to claim 6 is characterized in that: The method of obtaining the attention state analysis result of the target object based on the final fusion feature through a pre-built attention state prediction model also includes: The attention level of the target object is predicted, expressed as: ； in, Indicates the level of attention, [0, 1], sigmoid represents the activation function, represents the weight term, represents the bias term; The attention type of the target object is predicted, expressed as: ； in, Indicates attention type, which includes focused, distracted, confused, and tired, represents normalization, represents the weight term, Represents the bias term.

8. An attention state analysis system based on fine-grained hierarchical semantic fusion, characterized in that: include: A data acquisition module, used to acquire multimodal image data of a target object, including expression image data, head posture image data, and blink image data; A feature extraction module, used to input the multimodal image data into a pre-trained feature extraction mapping model, and obtain fine-grained semantic features of each modality through hierarchical extraction; wherein the feature extraction mapping model includes an expression fine-grained feature extraction module, a head posture fine-grained feature extraction module, and a blink fine-grained feature extraction module; A feature fusion module is used to input the fine-grained semantic features of each modality into a pre-trained feature fusion model to perform multi-modal feature fusion to obtain a final fusion feature; The analysis module is used to obtain the attention state analysis result of the target object based on the final fusion features through a pre-built attention state prediction model.

9. An attention state analysis device based on fine-grained hierarchical semantic fusion, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the attention state analysis method based on fine-grained hierarchical semantic fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the attention state analysis method based on fine-grained hierarchical semantic fusion described in any one of claims 1 to 7 are implemented.

Images