A student classroom motion detection method and system

Abstract

This invention relates to a student classroom action detection method and system in the field of big data processing technology, comprising the following steps: stacking and grouping aggregated features, and calculating the behavioral feature center of each individual in the aggregated features; constructing a temporal representation of each individual based on the behavioral feature center of each individual, and after evaluating the importance using a scaled dot product attention mechanism, concatenating and linearly transforming the outputs of all attention heads to obtain the final output features of multi-head attention; reconstructing the final output features to obtain reconstructed features; performing original classification prediction on the reconstructed features to obtain initial prediction results, and performing co-occurrence matrix enhancement to obtain co-occurrence enhancement results; performing nonlinear modeling based on the co-occurrence enhancement results, calculating the final loss function, and detecting individual student actions in the video to be detected, thus solving the problem that existing classroom behavior recognition methods are highly subjective and difficult to accurately identify in multi-person scenarios.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a method and system for detecting student classroom actions. Background Technology

[0002] Computer vision-based behavior recognition has shown broad application prospects in fields such as intelligent monitoring, educational assessment, and human-computer interaction. In smart education scenarios, the automatic identification and analysis of student classroom behavior is of great significance for improving teaching quality and optimizing classroom management.

[0003] Existing classroom behavior recognition methods mainly rely on the following three types of technical solutions:

[0004] First, traditional action recognition methods rely heavily on manual evaluation, recording, and coding to collect and interpret behavioral data. Researchers need to manually label and classify observed behaviors. However, these methods suffer from drawbacks such as strong coding subjectivity and limited sample size. Differences in judgment among observers lead to inconsistent coding, resulting in a lack of objectivity and stability in action recognition results. Furthermore, manual coding is easily influenced by observer experience, fatigue, and attention, making it difficult to guarantee reliable long-term, continuous data collection. When facing complex group behaviors and long-term series analysis, these methods often lack consistency and scalability, failing to meet the needs of large-scale scenarios.

[0005] Second, machine learning-based methods: Typical methods include using spatial-temporal points of interest (STIP), 3D histogram of oriented gradients (HOG3D), and optical flow. These methods construct spatial-temporal local descriptors, combine them with BoVW or Fisher Vectors for feature encoding, and finally use classifiers such as SVM and KNN to complete the recognition. However, these methods rely on the expressive power of handcrafted features, making it difficult to effectively characterize complex spatiotemporal dependencies. They also lack robustness to scene noise, occlusion, and viewpoint changes, leading to unreliable recognition results. Furthermore, their performance is limited in large-scale video data and behavior recognition in complex scenes.

[0006] Third, deep learning-based spatiotemporal feature modeling methods: Current mainstream methods are represented by 3D convolutional networks (such as C3D, I3D, SlowFast, MViT, etc.), which learn spatiotemporal features by constraining feature sampling within the bounding boxes of actors in video clips. They can model both the temporal and spatial dimensions simultaneously, thus achieving good recognition results on large-scale video understanding tasks (such as Kinetics, AVA, etc.). However, this method only relies on global spatiotemporal features and ignores the behavioral differences between individuals. At the same time, it fails to effectively model the co-occurrence relationship between different student behaviors, resulting in insufficient accuracy in recognizing fine-grained abnormal classroom behaviors such as "playing on mobile phones," "looking down," "walking around," and "talking."

[0007] Therefore, existing classroom behavior recognition methods mainly rely on manual observation and coding, which suffers from drawbacks such as strong subjectivity, low efficiency, and limited sample size, making it difficult to accurately and continuously identify abnormal student behaviors in complex scenarios with multiple participants. This invention addresses these problems by providing an automated classroom abnormal behavior recognition method based on deep learning. Through individual temporal modeling and behavior co-occurrence analysis, it achieves efficient, objective, and accurate detection and recognition of student behavior, thereby improving the reliability and application value of classroom behavior analysis. Summary of the Invention

[0008] This invention addresses the shortcomings of existing technologies by providing a method and system for detecting student classroom actions, thus solving the problems of strong subjectivity and difficulty in accurately recognizing multiple people in existing classroom behavior recognition methods.

[0009] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0010] A method for detecting student classroom actions, which performs attention modeling for the same individual across time frames and enhances the initial prediction results with a co-occurrence matrix, includes the following steps:

[0011] The original video containing the video task features is obtained, and the original video is preprocessed to obtain aggregated features;

[0012] The aggregated features are stacked and grouped along the time dimension, and the behavioral feature center of each individual in the aggregated features is calculated;

[0013] Based on the behavioral feature center of each individual, a temporal representation of each individual is constructed through multi-head attention computation. After the importance is evaluated by scaling dot product attention mechanism, the outputs of all attention heads are concatenated and linearly transformed to obtain the final output features of multi-head attention.

[0014] The final output features are reconstructed so that the spatial dimension of the final output features is restored to the original five-dimensional structure, thus obtaining the reconstructed features;

[0015] The reconstructed features are subjected to original classification prediction to obtain initial prediction results, and the initial prediction results are enhanced by co-occurrence matrix to obtain co-occurrence enhanced results;

[0016] Based on the co-occurrence enhancement results, nonlinear modeling is performed, and a final loss function is calculated. Based on the final loss function, a nonlinear model is used to detect individual student actions in the video to be detected.

[0017] Optionally, restoring the spatial dimension of the final output feature to the original five-dimensional structure includes the following steps:

[0018] The final output features are then reshaped to restore the flattened spatial dimensions to five dimensions.

[0019] The permute function is used to adjust the dimensional order of the tensors, and the final output features after the reshape operation are rearranged into a standard spatiotemporal format to obtain the reconstructed features.

[0020] Optionally, the reconstructed features are subjected to original classification prediction to obtain an initial prediction result, and the initial prediction result is enhanced by a co-occurrence matrix to obtain a co-occurrence enhanced result, including the following steps:

[0021] Candidate boxes are defined for the reconstructed features, and the region feature vector of each candidate box is extracted;

[0022] Classification prediction is performed based on the feature vectors of all the regions, and an initial prediction result is obtained;

[0023] Obtain the co-occurrence frequency normalization matrix, and adjust the influence of the co-occurrence frequency normalization matrix in the classification process by using the co-occurrence weight factor to obtain the co-occurrence enhancement result.

[0024] Optionally, based on the co-occurrence enhancement results, nonlinear modeling is performed, and a final loss function is calculated. Based on the final loss function, a nonlinear model is used to detect individual student actions in the video to be detected, including the following steps:

[0025] The co-occurrence enhancement results are mapped to a new feature space to obtain nonlinear co-occurrence enhancement features, and the nonlinear co-occurrence enhancement features are used to perform residual fusion on the initial prediction results to obtain residual fusion results;

[0026] The residual fusion result is input into the binary cross-entropy loss function to obtain the final loss function. Based on the final loss function, a nonlinear model containing the final loss function is used to detect individual student actions in the video to be detected.

[0027] Optionally, the final loss function is: , where N represents the total number of candidate boxes; It is the true label of the i-th sample to the c-th category; This represents the residual fusion result of the i-th sample with respect to the c-th category.

[0028] Optionally, the original video for obtaining video task features is preprocessed to obtain aggregated features, including the following steps:

[0029] A fixed-length video segment is extracted from the original video, and spatiotemporal features are extracted from the video segment to obtain the video spatiotemporal features;

[0030] The feature dimensions of the video spatiotemporal features are transformed into a normalized target dimension to obtain three-dimensional tensor features, and a grouping index is generated to obtain aggregate features.

[0031] A student classroom action detection system, wherein the student classroom action detection system executes the student classroom action detection method as described in any one of the above, including a preprocessing unit, a dimension processing unit, an attention mechanism unit, a feature reconstruction unit, an initial prediction unit, and a final detection unit;

[0032] The preprocessing unit is used to obtain the original video of the video task features and preprocess the original video to obtain aggregated features.

[0033] The dimension processing unit is used to stack and group the aggregated features along the time dimension, and calculate the behavioral feature center of each individual in the aggregated features;

[0034] The attention mechanism unit is used to construct the temporal representation of each individual based on the behavioral feature center of each individual through multi-head attention computation, and after evaluating the importance using a scaled dot product attention mechanism, the outputs of all attention heads are concatenated and linearly transformed to obtain the final output features of multi-head attention.

[0035] The feature reconstruction unit is used to reconstruct the final output feature so that the spatial dimension of the final output feature is restored to the original five-dimensional structure, thereby obtaining the reconstructed feature;

[0036] The initial prediction unit is used to perform original classification prediction on the reconstructed features to obtain an initial prediction result, and to perform co-occurrence matrix enhancement on the initial prediction result to obtain a co-occurrence enhancement result;

[0037] The final detection unit is used to perform nonlinear modeling based on the co-occurrence enhancement results, calculate the final loss function, and use the nonlinear model to detect individual student actions in the video to be detected based on the final loss function.

[0038] Optionally, the initial prediction unit includes a feature extraction unit, a classification prediction unit, and a co-occurrence enhancement unit;

[0039] The feature extraction unit is used to delineate candidate boxes for the reconstructed features and extract the region feature vector of each candidate box;

[0040] The classification prediction unit is used to perform classification prediction based on all the feature vectors of the regions and obtain an initial prediction result;

[0041] The co-occurrence enhancement unit is used to obtain a co-occurrence frequency normalization matrix and adjust the influence intensity of the co-occurrence frequency normalization matrix in the classification process through a co-occurrence weight factor to obtain the co-occurrence enhancement result.

[0042] Optionally, the final detection unit includes a residual processing unit and a model detection unit;

[0043] The residual processing unit is used to map the co-occurrence enhancement result to a new feature space to obtain nonlinear co-occurrence enhancement features, and use the nonlinear co-occurrence enhancement features to perform residual fusion on the initial prediction result to obtain the residual fusion result;

[0044] The model detection unit is used to input the residual fusion result into the binary cross-entropy loss function to obtain the final loss function, and based on the final loss function, use a nonlinear model containing the final loss function to detect individual student actions in the video to be detected.

[0045] A computer storage medium storing computer program instructions; when executed by a processor, the computer program instructions implement the student classroom action detection method as described above.

[0046] Compared with the prior art, the technical solution provided by this invention has the following advantages:

[0047] By performing attention modeling on the behavioral features of the same individual across time frames, the temporal consistency and stability of action detection are effectively improved, especially in recognizing continuous behaviors such as "speaking" and "walking." Secondly, by statistically analyzing the co-occurrence frequency between behavior categories and introducing a learnable relation matrix and gating mechanism, the semantic co-occurrence dependency between different behavior categories is modeled, thereby enhancing the model's ability to model complex behavior co-occurrence structures. For example, the behavior of "looking down" often co-occurs with the behavior of "using a mobile phone" in actual teaching scenarios. By explicitly modeling such statistical regularities, the model's classification robustness under multiple behavioral interferences can be significantly improved. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a student classroom action detection method proposed in Embodiment 1. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0051] Example 1

[0052] like Figure 1 As shown, a student classroom action detection method effectively improves the temporal consistency and stability of action detection by performing attention modeling on the behavioral features of the same individual across time frames. Simultaneously, by statistically analyzing the co-occurrence frequency between behavior categories and introducing a learnable relation matrix and gating mechanism, it models the semantic co-occurrence dependency between different behavior categories, thereby enhancing the model's ability to model complex behavior co-occurrence structures. The method includes the following steps: acquiring the original video of the video task features and preprocessing the original video to obtain aggregated features; specifically, extracting fixed-length video segments from the original video and extracting spatiotemporal features from the video segments to obtain video spatiotemporal features; transforming the feature dimensions of the video spatiotemporal features into a normalized target dimension to obtain three-dimensional tensor features and generating a grouping index to obtain aggregated features.

[0053] More specifically, in the video character feature processing workflow, the input is a short video clip V of a fixed length extracted from the original video. A video consists of T frames, each frame is denoted as T. After extracting spatiotemporal features through the backbone network, video features are obtained. Where B represents the batch size, C is the number of channels, T is the time step, H and W represent the height and width of the spatial dimension, respectively, and the individual matrix is... , where N is the maximum number of individuals detected per frame.

[0054] Next, the input feature dimension is transformed. The feature tensor is then transformed. The data is rearranged into a temporal-spatial tiling format, i.e., the batch and temporal dimension are flattened in chronological order. This is to make the feature tensor... With individual matrix P The feature dimensions are aligned, and the feature dimensions are unified to the target dimension through a linear projection layer to obtain a normalized feature representation. According to Equation (1), a dimension permutation operation is performed to adjust the feature tensor to... The layout, using spatial dimensions to flatten, generates Layout of three-dimensional tensor structure The three-dimensional tensor features are obtained, and the three-dimensional tensor structure is as follows:

[0055] ...Equation (1).

[0056] When generating the grouped index, for video frames at time t ,according to Extract the set of individual IDs from this frame Extract the individual IDs as id from the set of individual IDs for each frame. k The sequence number generates an ID named id. k Grouping index of individuals As shown in equation (2): ...Equation (2).

[0057] By using grouped index This allows for the aggregation of individual features in each frame into a set (i.e., aggregated features) to enable contextual modeling for individuals.

[0058] Next, the aggregated features are stacked and grouped along the time dimension, and the behavioral feature center of each individual in the aggregated features is calculated. Based on the behavioral feature center of each individual, the temporal representation of each individual is constructed by multi-head attention calculation. After the importance is evaluated by scaling dot product attention mechanism, the outputs of all attention heads are concatenated and linearly transformed to obtain the final output features of multi-head attention.

[0059] Specifically, group the indexes for each set of IDs. The following operations are performed sequentially to calculate individual attention. Following formula (3), grouped features are stacked along the time dimension, and the mean of all features within each group is calculated. This serves as the behavioral feature center for that ID. This operation effectively captures the individual's long-term behavioral patterns, eliminates transient detection noise interference, and calculates the mean of all features within the group using the following formula: ... Equation (3).

[0060] Each individual's temporal representation is constructed using multi-head attention as follows. First, the input features are processed according to formula (4). Mapped to query Q, key K, and value V: ...Equation (4), where, It is a learnable linear transformation.

[0061] Subsequently, Q, K, and V are divided into h heads, and the scoring method for each attention head is given by equation (5). The importance of features within the group is evaluated using a scaled dot product attention mechanism, where... As a feature dimension scaling factor, this attention weight simultaneously considers the feature correlation in both spatiotemporal dimensions, enabling cross-frame context modeling.

[0062] ...Equation (5).

[0063] Finally, according to formula (6), the outputs of all attention heads are concatenated along the last feature dimension (channel dimension), and mapped back to the original feature dimension through a linear transformation to obtain the final output features of multi-head attention. ...Equation (6), where, represents the parameters of a trainable linear matrix.

[0064] Next, the final output features are reconstructed so that the spatial dimension of the final output features is restored to the original five-dimensional structure, resulting in reconstructed features. Specifically, according to equation (7), the processed features are restored to the original dimensions: ... Equation (7).

[0065] By reshaping the processed features, the flattened spatial dimensions are restored to a five-dimensional structure. Then, permute is used to adjust the dimensional order of the tensors, rearranging the features into a standard (B, C, T, H, W) spatiotemporal format, thus completing the reconstruction of the output features. As the final output of the module.

[0066] This attention mechanism effectively aggregates the temporal features of each individual, captures the dependencies within individuals, and maintains the distinction between individuals, making it suitable for sequence modeling tasks involving multiple people.

[0067] Then, the reconstructed features are subjected to original classification prediction to obtain initial prediction results. The initial prediction results are then enhanced by co-occurrence matrix to obtain co-occurrence enhancement results. Specifically, the following steps are included: delineating candidate boxes for the reconstructed features and extracting the regional feature vector of each candidate box; performing classification prediction based on all the regional feature vectors to obtain initial prediction results; obtaining the co-occurrence frequency normalization matrix and adjusting the influence intensity of the co-occurrence frequency normalization matrix in the classification process through the co-occurrence weight factor to obtain the co-occurrence enhancement results.

[0068] Specifically, given input video features First, candidate bounding boxes are extracted from the feature map using RoIAlign. Features Then, the region feature vector of each candidate box is obtained. Let the set of candidate boxes be . Then the corresponding region feature vector As shown in equation (8): ... Equation (8).

[0069] Then, the region feature vector Input the standard RoI classification header to obtain the corresponding initial prediction results. As shown in equation (9): ... Equation (9).

[0070] Finally, as shown in Equation (10), stacking the prediction results of all candidate boxes yields the complete initial prediction result Z: ...Equation (10), where, Indicates the number of behavior categories; Indicates the total number of candidate boxes; It is the classification weight.

[0071] Furthermore, using the action labels from the training set, the co-occurrence of each action label in each frame is statistically analyzed to obtain a non-normalized initial co-occurrence frequency matrix U. As shown in equation (11): ...equation (11), where, This represents the number of times behavior i and behavior j occur simultaneously in the training set. This represents the number of behavior categories.

[0072] Then, each row of the initial co-occurrence frequency matrix U is normalized according to equation (12) to obtain the normalized co-occurrence frequency normalized matrix. As shown in equation (13).

[0073] ...Equation (12);

[0074] ... Equation (13).

[0075] in, It is a very small constant used to avoid division by zero; Indicates the behavior across all samples. and The normalized frequencies that appear simultaneously are used as a priori statistical matrix to represent the relationship between behaviors, and are registered as a trainable parameter in the model.

[0076] Furthermore, to adjust the influence of the co-occurrence frequency normalization matrix on the classification process, a learnable co-occurrence weight factor is introduced. Used to adjust the co-occurrence frequency normalization matrix The weights in the prediction results are as shown in Equation (14). The initial prediction result Z is multiplied by the co-occurrence frequency normalization matrix R to obtain the co-occurrence enhancement result.

[0077] ...equation (14), where, for the first... The nth sample, the nth The enhancement scores for each category are: ...Formula (15).

[0078] This process essentially achieves information fusion and context enhancement by using the co-occurrence weights of other categories on the current category.

[0079] Finally, based on the co-occurrence enhancement results, nonlinear modeling is performed, and the final loss function is calculated. Based on the final loss function, a nonlinear model is used to detect individual student actions in the video to be detected, as shown in Equation (16). The input features are mapped to a new feature space to achieve nonlinear transformation of the features, thereby enhancing the representational ability of the model. Equation (16) is expressed as: ,in, For linear layer parameters; Dropout ratio set to [value]. (Enabled only during training); Output This is a nonlinear co-occurrence enhancement feature.

[0080] Specifically, firstly through a linear transformation The input features are projected to provide a more suitable representational basis for subsequent modeling. To further enhance the nonlinear expressive power of the features, a nonlinear mapping structure consisting of a fully connected layer and a ReLU activation function is introduced. Simultaneously, Dropout is applied during the training phase to effectively alleviate overfitting and improve the model's generalization ability in complex action detection tasks.

[0081] To improve training stability and feature robustness, this embodiment introduces residual connections to process the initial prediction results. With enhancement The addition is shown in equation (17): ,in The final fused Logits are then input into the loss function for supervised optimization. This fusion method helps the model incorporate new relational information while preserving the original feature information.

[0082] Since this task is a multi-label action recognition task, the binary cross-entropy loss function combined with the Logits output is used as the final loss function, as shown in equation (18), to adapt the Logits output without activation function, where equation (18) is: , It is the true label of the i-th sample for the c-th category.

[0083] This invention effectively improves the temporal consistency and stability of action detection by performing attention modeling on the behavioral features of the same individual across time frames, showing significant advantages, especially in recognizing continuous behaviors such as "speaking" and "walking." Secondly, by statistically analyzing the co-occurrence frequency between behavior categories and introducing a learnable relation matrix and gating mechanism, it models the semantic co-occurrence dependency between different behavior categories, thereby enhancing the model's ability to model complex behavior co-occurrence structures. For example, the behavior of "looking down" often co-occurs with the behavior of "using a mobile phone" in actual teaching scenarios. By explicitly modeling such statistical regularities, the model's classification robustness under multiple behavioral interferences can be significantly improved.

[0084] Example 2

[0085] A student classroom action detection system includes a preprocessing unit, a dimension processing unit, an attention mechanism unit, a feature reconstruction unit, an initial prediction unit, and a final detection unit;

[0086] The preprocessing unit is used to obtain the original video of the video task features and preprocess the original video to obtain aggregated features;

[0087] The dimension processing unit is used to stack and group the aggregated features along the time dimension, and calculate the behavioral feature center of each individual in the aggregated features;

[0088] The attention mechanism unit is used to construct the temporal representation of each individual based on the behavioral feature center of each individual through multi-head attention computation. After the importance is evaluated by scaling dot product attention mechanism, the outputs of all attention heads are concatenated and linearly transformed to obtain the final output features of multi-head attention.

[0089] The feature reconstruction unit is used to reconstruct the final output features, so that the spatial dimension of the final output features is restored to the original five-dimensional structure, thus obtaining the reconstructed features;

[0090] The initial prediction unit is used to perform original classification prediction on the reconstructed features to obtain the initial prediction result, and to enhance the initial prediction result with a co-occurrence matrix to obtain the co-occurrence enhanced result;

[0091] The final detection unit is used to perform nonlinear modeling based on the co-occurrence enhancement results and calculate the final loss function. Based on the final loss function, the nonlinear model is used to detect individual student actions in the video to be detected.

[0092] The initial prediction unit includes a feature extraction unit, a classification prediction unit, and a co-occurrence enhancement unit. The feature extraction unit is used to delineate candidate boxes for the reconstructed features and extract the regional feature vector of each candidate box. The classification prediction unit is used to perform classification prediction based on all regional feature vectors and obtain the initial prediction result. The co-occurrence enhancement unit is used to obtain the co-occurrence frequency normalization matrix and adjust the influence strength of the co-occurrence frequency normalization matrix in the classification process through the co-occurrence weight factor to obtain the co-occurrence enhancement result.

[0093] The final detection unit includes a residual processing unit and a model detection unit. The residual processing unit maps the co-occurrence enhancement results to a new feature space to obtain nonlinear co-occurrence enhancement features, and uses the nonlinear co-occurrence enhancement features to perform residual fusion on the initial prediction results to obtain the residual fusion result. The model detection unit inputs the residual fusion result into the binary cross-entropy loss function to obtain the final loss function, and uses a nonlinear model containing the final loss function to detect individual student actions in the video to be detected.

[0094] Since the student classroom action detection system in this embodiment executes the student classroom action detection method as described in Embodiment 1, it will not be repeated in this embodiment.

[0095] A computer storage medium storing computer program instructions; when the computer program instructions are executed by a processor, they implement the student classroom action detection method as described in Example 1.

[0096] More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical cables, RF, etc., or any suitable combination thereof.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules, units, or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units, modules, or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0099] The units may or may not be physically separate. The components shown as units can be one or more physical units, meaning they can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any form or substance. It should be noted that those skilled in the art can make various improvements and additions without departing from the method of the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention. Any modifications, alterations, and equivalent changes made by those skilled in the art based on the above-disclosed technical content without departing from the spirit and scope of the present invention are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, and evolutions made to the above embodiments based on the essential technology of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for detecting student classroom actions, characterized in that, Attention modeling for the same individual across time frames and co-occurrence matrix enhancement of the initial prediction results include the following steps: The original video containing the video task features is obtained, and the original video is preprocessed to obtain aggregated features; The aggregated features are stacked and grouped along the time dimension, and the behavioral feature center of each individual in the aggregated features is calculated; Based on the behavioral feature center of each individual, a temporal representation of each individual is constructed through multi-head attention computation. After the importance is evaluated by scaling dot product attention mechanism, the outputs of all attention heads are concatenated and linearly transformed to obtain the final output features of multi-head attention. The final output features are reconstructed so that the spatial dimension of the final output features is restored to the original five-dimensional structure, thus obtaining the reconstructed features; The reconstructed features are subjected to original classification prediction to obtain initial prediction results, and the initial prediction results are enhanced by co-occurrence matrix to obtain co-occurrence enhanced results; Based on the co-occurrence enhancement results, nonlinear modeling is performed, and a final loss function is calculated. Based on the final loss function, a nonlinear model is used to detect individual student actions in the video to be detected.

2. The student classroom action detection method according to claim 1, characterized in that, Restoring the spatial dimension of the final output feature to the original five-dimensional structure includes the following steps: The final output features are then reshaped to restore the flattened spatial dimensions to five dimensions. The permute function is used to adjust the dimensional order of the tensors, and the final output features after the reshape operation are rearranged into a standard spatiotemporal format to obtain the reconstructed features.

3. The student classroom action detection method according to claim 2, characterized in that, The reconstructed features are subjected to original classification prediction to obtain initial prediction results, and the initial prediction results are then enhanced with a co-occurrence matrix to obtain co-occurrence enhanced results, including the following steps: Candidate boxes are defined for the reconstructed features, and the region feature vector of each candidate box is extracted; Classification prediction is performed based on the feature vectors of all the regions, and an initial prediction result is obtained; Obtain the co-occurrence frequency normalization matrix, and adjust the influence of the co-occurrence frequency normalization matrix in the classification process by using the co-occurrence weight factor to obtain the co-occurrence enhancement result.

4. The student classroom action detection method according to claim 3, characterized in that, Based on the co-occurrence enhancement results, nonlinear modeling is performed, and a final loss function is calculated. Based on the final loss function, a nonlinear model is used to detect individual student actions in the video to be detected, including the following steps: The co-occurrence enhancement results are mapped to a new feature space to obtain nonlinear co-occurrence enhancement features, and the nonlinear co-occurrence enhancement features are used to perform residual fusion on the initial prediction results to obtain residual fusion results; The residual fusion result is input into the binary cross-entropy loss function to obtain the final loss function. Based on the final loss function, a nonlinear model containing the final loss function is used to detect individual student actions in the video to be detected.

5. The student classroom action detection method according to claim 4, characterized in that, The final loss function is: , where N represents the total number of candidate boxes; It is the true label of the i-th sample to the c-th category; This represents the residual fusion result of the i-th sample with respect to the c-th category.

6. A method for detecting student classroom actions according to any one of claims 1-5, characterized in that, The process of obtaining aggregated features from the original video containing the video task features, and preprocessing the original video to obtain aggregated features, includes the following steps: A fixed-length video segment is extracted from the original video, and spatiotemporal features are extracted from the video segment to obtain the video spatiotemporal features; The feature dimensions of the video spatiotemporal features are transformed into a normalized target dimension to obtain three-dimensional tensor features, and a grouping index is generated to obtain aggregate features.

7. A student classroom action detection system, characterized in that, The student classroom action detection system executes the student classroom action detection method as described in any one of claims 1-6, including a preprocessing unit, a dimension processing unit, an attention mechanism unit, a feature reconstruction unit, an initial prediction unit, and a final detection unit; The preprocessing unit is used to obtain the original video of the video task features and preprocess the original video to obtain aggregated features. The dimension processing unit is used to stack and group the aggregated features along the time dimension, and calculate the behavioral feature center of each individual in the aggregated features; The attention mechanism unit is used to construct the temporal representation of each individual based on the behavioral feature center of each individual through multi-head attention computation, and after evaluating the importance using a scaled dot product attention mechanism, the outputs of all attention heads are concatenated and linearly transformed to obtain the final output features of multi-head attention. The feature reconstruction unit is used to reconstruct the final output feature so that the spatial dimension of the final output feature is restored to the original five-dimensional structure, thereby obtaining the reconstructed feature; The initial prediction unit is used to perform original classification prediction on the reconstructed features to obtain an initial prediction result, and to perform co-occurrence matrix enhancement on the initial prediction result to obtain a co-occurrence enhancement result; The final detection unit is used to perform nonlinear modeling based on the co-occurrence enhancement results, calculate the final loss function, and use the nonlinear model to detect individual student actions in the video to be detected based on the final loss function.

8. A student classroom action detection system according to claim 7, characterized in that, The initial prediction unit includes a feature extraction unit, a classification prediction unit, and a co-occurrence enhancement unit; The feature extraction unit is used to delineate candidate boxes for the reconstructed features and extract the region feature vector of each candidate box; The classification prediction unit is used to perform classification prediction based on all the feature vectors of the regions and obtain an initial prediction result; The co-occurrence enhancement unit is used to obtain a co-occurrence frequency normalization matrix and adjust the influence intensity of the co-occurrence frequency normalization matrix in the classification process through a co-occurrence weight factor to obtain the co-occurrence enhancement result.

9. A student classroom action detection system according to claim 7, characterized in that, The final detection unit includes a residual processing unit and a model detection unit; The residual processing unit is used to map the co-occurrence enhancement result to a new feature space to obtain nonlinear co-occurrence enhancement features, and use the nonlinear co-occurrence enhancement features to perform residual fusion on the initial prediction result to obtain the residual fusion result; The model detection unit is used to input the residual fusion result into the binary cross-entropy loss function to obtain the final loss function, and based on the final loss function, use a nonlinear model containing the final loss function to detect individual student actions in the video to be detected.

10. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, they implement the student classroom action detection method as described in any one of claims 1-6.

Images