A student classroom behavior recognition method and system based on cross-modal feature fusion

Abstract

The present application relates to the technical field of classroom behavior recognition, and particularly relates to a student classroom behavior recognition method and system based on cross-modal feature fusion, comprising: dividing student classroom videos into a plurality of time periods; determining the number of random samples of each time period and student classroom image data sets according to the distribution and confidence of human joint points in each time period; obtaining an artificial annotation data set; obtaining a predicted single-channel joint point heat map and a real single-channel joint point heat map according to the two data sets, and optimizing the trained HRNet model according to the difference between the two; obtaining a 3D heat map volume through the optimized HRNet model; inputting the RGB image in the student classroom video into a double-flow 3D-CNN network to obtain final fusion features, and recognizing the student classroom behavior through the final fusion features. The present application improves the accuracy of student classroom behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of classroom behavior recognition technology, and in particular to a method and system for student classroom behavior recognition based on cross-modal feature fusion. Background Technology

[0002] With the rapid development of smart education, student classroom behavior recognition has become a key link in achieving precise teaching. However, traditional methods rely on teachers' subjective observation or single-modal data (such as RGB images or pose data), which has obvious limitations: manual observation is inefficient and easily influenced by subjectivity, while single-modal methods are not robust to complex classroom environments (such as changes in lighting and occlusion) and are difficult to comprehensively capture behavioral features. Existing deep learning-based solutions (such as 3D-CNN or two-stream networks) mostly adopt simple late-stage fusion strategies, ignoring early interactions between modalities, and pose modeling often directly uses joint coordinates, lacking spatial structure representation. In addition, the lack of publicly available datasets and insufficient validation in real-world scenarios further restricts the practical application of the technology. Summary of the Invention

[0003] This invention provides a student classroom behavior recognition method and system based on cross-modal feature fusion, which solves the problems of poor robustness, imperfect fusion strategy and insufficient posture modeling in existing classroom behavior recognition methods in complex environments.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] The first aspect of this invention is to provide a method for recognizing student classroom behavior based on cross-modal feature fusion, comprising:

[0006] The HRNet model is pre-trained to obtain the trained HRNet model; a student classroom video is acquired and preprocessed; each frame of the student classroom video is input into the trained HRNet model to obtain all human joints in each frame and the confidence score of each human joint; the student classroom video is equally divided into several time periods; the number of random samples for each time period is determined by the distribution and confidence scores of human joints in each time period, and the student classroom image dataset is determined by the number of random samples.

[0007] The process involves manually annotating a dataset of student classroom images to obtain a manually annotated dataset; using this dataset, a trained HRNet model is used to obtain a predicted single-channel joint heatmap; using the manually annotated dataset, a trained HRNet model is used to obtain a true single-channel joint heatmap; based on the difference between the predicted and true single-channel joint heatmaps, the weight parameters in the trained HRNet model are optimized to obtain an optimized HRNet model; the optimized HRNet model is used to obtain a predicted limb heatmap and a new predicted single-channel joint heatmap; the new predicted single-channel joint heatmap or the predicted limb heatmap is superimposed on the original image, and then stacked along the time dimension to obtain the 3D heatmap volume.

[0008] A dual-stream 3D-CNN network is constructed. The 3D heatmap volume and the RGB images from the student classroom video are input into the dual-stream 3D-CNN network. The network is connected laterally by early feature fusion and late feature fusion to obtain the final fused features. The final fused features are then used to identify student classroom behavior.

[0009] Furthermore, the student classroom video is equally divided into several time periods; the number of random samples for each time period is determined by the distribution and confidence level of human body joints within each time period, and the student classroom image dataset is determined by the number of random samples, including:

[0010] At a preset time interval Divide the students' classroom videos evenly into several time segments;

[0011] The velocity information of each frame's joints is obtained by calculating the changes in joint position between consecutive frames; the complexity of each frame within each time period is obtained based on the average confidence of all relevant nodes in each frame within each time period, the velocity information of all relevant nodes in two adjacent frames within each time period, and the distribution of joints in adjacent frames.

[0012] The complexity of each time period is obtained by analyzing the complexity distribution of all frames within that time period. The number of random samples for each time period is determined based on the complexity of that time period. Random sampling is then performed for each time period based on the number of random samples for that time period to obtain the student classroom image dataset.

[0013] Furthermore, the step of obtaining the complexity of each frame within each time period based on the average confidence of all relevant nodes in each frame within each time period, the velocity information of all relevant nodes in two adjacent frames within each time period, and the distribution of key points in adjacent frames includes:

[0014] Based on the velocity information of all relevant nodes in two adjacent frames within each time period, the average motion amplitude of each frame in each time period is obtained.

[0015] Based on the distribution of key points in adjacent frames within each time period, the deviation of key points in each frame within each time period is obtained; based on the average confidence of all key points in each frame within each time period, the average motion amplitude and deviation of key points in each frame within each time period, the complexity of each frame within each time period is obtained.

[0016] The average motion amplitude of each frame's joint within each time period is specifically expressed by the following formula:

[0017]

[0018] In the formula, Indicates the first Within the time period, the first The first frame The speed of each joint point This indicates the total number of nodes. Indicates the first Within the time period, the first The average motion amplitude of frame joints;

[0019] The specific formula for the deviation of each frame's key point within each time period is as follows:

[0020]

[0021] In the formula, Indicates the first Within the time period, the first Frame and the The first frame The distance traveled by each joint. Indicates the first Within the time period, the first Frame and the The first frame The distance traveled by each joint. This indicates the total number of nodes. Indicates the first Within the time period, the first Frame joint deviation;

[0022] The complexity of each frame within each time period is specifically expressed by the formula:

[0023]

[0024] In the formula, Indicates the first Within the time period, the first The average confidence level of all relevant nodes in the frame. This indicates the preset first weight coefficient. This indicates a preset second weighting coefficient. Indicates the first Within the time period, the first Frame complexity, This represents the linear normalization function.

[0025] Furthermore, the complexity of each time period is obtained by analyzing the complexity distribution of all frames within that time period; the number of random samples for each time period is determined based on its complexity; and random sampling is performed for each time period using the number of random samples for that time period to obtain a student classroom image dataset, including:

[0026] The complexity of each time period is specifically expressed by the formula:

[0027]

[0028] In the formula, Indicates the first The average complexity of all frames within a given time period. Indicates the first The standard deviation of the complexity of all frames within a time period. Indicates the first The complexity of a time period Represents a linear normalization function. The contribution factor of the complexity fluctuation within a time period to the complexity of that time period is a preset value.

[0029] The specific number of random samples in each time period is expressed by the formula:

[0030]

[0031] In the formula, Indicates the basic sample number. Indicates to Perform rounding operation. Indicates the first The number of random samples within a time period; where the base sample size is set manually;

[0032] Based on the number of random samples in each time period, images are randomly collected from several frames within each time period. All images randomly collected from all time periods are combined into a dataset, denoted as the student classroom image dataset. The number of images randomly collected in each time period is the number of random samples in each time period.

[0033] Further, the process of obtaining a predicted single-channel joint heatmap using a trained HRNet model based on a student classroom image dataset; obtaining a true single-channel joint heatmap using a trained HRNet model based on a manually labeled dataset; and optimizing the weight parameters in the trained HRNet model based on the difference between the predicted and true single-channel joint heatmaps to obtain an optimized HRNet model includes:

[0034] The process of obtaining the predicted single-channel joint heatmap is as follows:

[0035]

[0036] In the formula, This represents the horizontal value of each pixel in each image of the student classroom image dataset. This represents the vertical value of each pixel in each image of the student classroom image dataset; This represents the first image in each image of the student classroom image dataset. The horizontal values ​​of individual body joints This represents the first image in each image of the student classroom image dataset. Vertical values ​​of individual joints; This represents the Gaussian kernel standard deviation. This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents an exponential function with the natural constant as its base. This represents the first image in each image of the student classroom image dataset. Human body joints at pixels The heatmap values; where the Gaussian kernel standard deviation is a preset value;

[0037] In each image of the student classroom image dataset, the first... A single image is created by combining the heatmap values ​​of all pixels for each human body joint, denoted as the [number]th [image]. A predicted single-channel joint heatmap;

[0038] The process of obtaining the real single-channel joint heatmap is as follows:

[0039]

[0040] In the formula, This represents the horizontal value of each pixel in each image of the manually labeled dataset. This represents the vertical value of each pixel in each image of the manually labeled dataset; This represents the first manually labeled image in each image of the manually labeled dataset. The horizontal values ​​of individual body joints This represents the first manually labeled image in each image of the manually labeled dataset. Vertical values ​​of individual joints; This represents the first image in each image of the manually labeled dataset. A real human joint point at a pixel Heatmap values;

[0041] In each image of the manually labeled dataset, the first... A single image is created by combining the heatmap values ​​of all pixels for each human body joint, denoted as the [number]th [image]. A real single-channel joint heat map;

[0042] Based on the predicted single-channel joint heatmap and the actual single-channel joint heatmap, the weight parameters in the trained HRNet model are optimized using the mean squared error loss function, and the optimized HRNet model is obtained using the optimized weight parameters.

[0043] Further, the process of obtaining predicted limb heatmaps and new predicted single-channel joint heatmaps using the optimized HRNet model; superimposing the new predicted single-channel joint heatmaps or predicted limb heatmaps onto the original image, and then stacking them along the time dimension to obtain the 3D heatmap volume includes:

[0044] Based on the process of obtaining the predicted single-channel joint heatmap, a new predicted single-channel joint heatmap is obtained through the optimized HRNet model.

[0045] The process of obtaining the predicted limb heatmap is as follows:

[0046]

[0047] In the formula, This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents the horizontal value of each pixel in each image of the student classroom image dataset. This represents the vertical value of each pixel in each image of the student classroom image dataset. This represents the first image in each image of the student classroom image dataset. Individual body joints and the first The limb segments formed between the joints of an individual's body. This represents the number of pixels in each image of the student classroom image dataset. To limb segments The distance between them This represents an exponential function with the natural constant as its base. This represents the limb line segments in each image of the student classroom image dataset. At pixel heatmap values, This represents the function that takes the minimum value. Indicates the Gaussian kernel standard deviation;

[0048] The heatmap values ​​of each limb segment in each image are combined into a single image, which is called the predicted limb heatmap for each limb segment.

[0049] The newly predicted single-channel joint heatmap or predicted limb heatmap is overlaid onto the original image to obtain a 2D heatmap containing all joint information; the 2D heatmap is a single frame image; consecutive frame 2D heatmaps are stacked along the time dimension to generate a size of 3D heatmap volume;

[0050] in, This indicates the total number of nodes. Indicates the length of time. Indicates altitude, Indicates the width.

[0051] Furthermore, the step of inputting the 3D heatmap volume and the RGB images from the student classroom video into a dual-stream 3D-CNN network, employing a lateral connection method of early feature fusion and late feature fusion to obtain the final fused features, and then using the final fused features to recognize student classroom behavior includes:

[0052] The dual-stream 3D-CNN network includes an RGB modality branch and a POSE modality branch;

[0053] During the forward propagation process, convolution and max pooling operations are first applied to the input RGB image and 3D heatmap volume through the paths of the RGB modality branch and the POSE modality branch, respectively, to obtain the RGB shallow feature tensor and the POSE shallow feature tensor. Then, information exchange and fusion are performed on the two shallow feature tensors through bidirectional lateral connections to obtain the information-enhanced RGB shallow features and the information-enhanced POSE shallow features.

[0054] Then, the enhanced RGB shallow features and POSE shallow features are put back into their respective paths, and convolution and non-linear activation operations are used to obtain the RGB deep feature tensor and the POSE deep feature tensor. Information fusion is then performed on the RGB deep feature tensor and the POSE deep feature tensor to obtain the final fused features.

[0055] The final fused features are passed through a fully connected layer and a Softmax classifier to output a category probability distribution. The category probability distribution is then used to predict and identify the categories of student classroom behavior.

[0056] A second aspect of the present invention is to provide a student classroom behavior recognition system based on cross-modal feature fusion, comprising:

[0057] Data acquisition module: used to pre-train the HRNet model to obtain the trained HRNet model; acquire a student classroom video and preprocess the student classroom video; input each frame of the student classroom video into the trained HRNet model to obtain all human joints in each frame and obtain the confidence score of each human joint; divide the student classroom video into several time periods equally; determine the number of random samples for each time period based on the distribution and confidence score of human joints in each time period, and determine the student classroom image dataset based on the number of random samples;

[0058] The parameter optimization module is used to manually annotate the student classroom image dataset to obtain a manually annotated dataset; based on the student classroom image dataset, it obtains a predicted single-channel joint heatmap using a trained HRNet model; based on the manually annotated dataset, it obtains a real single-channel joint heatmap using a trained HRNet model; based on the difference between the predicted and real single-channel joint heatmaps, it optimizes the weight parameters in the trained HRNet model to obtain an optimized HRNet model; it obtains a predicted limb heatmap and a new predicted single-channel joint heatmap using the optimized HRNet model; it then overlays the new predicted single-channel joint heatmap or the predicted limb heatmap onto the original image, and stacks them along the time dimension to obtain the 3D heatmap volume.

[0059] Feature fusion and extraction module: used to construct a dual-stream 3D-CNN network; inputting the 3D heatmap volume and RGB images from student classroom videos into the dual-stream 3D-CNN network, using a lateral connection method of early feature fusion and late feature fusion to obtain the final fused features, and using the final fused features to identify student classroom behavior.

[0060] A third aspect of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned student classroom behavior recognition method based on cross-modal feature fusion.

[0061] A fourth aspect of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the student classroom behavior recognition method based on cross-modal feature fusion.

[0062] Compared with existing technologies, the beneficial effects of this invention are as follows: The HRNet model is pre-trained to obtain all human joints in each frame of the image, and the confidence level of each human joint is obtained; the student classroom video is equally divided into several time periods; the distribution and confidence level of human joints within each time period are used to determine the number of random samples for each time period, and the student classroom image dataset is determined using the number of random samples, thus improving the accuracy of randomly acquiring the student classroom image dataset; the student classroom image dataset is manually annotated to obtain a manually annotated dataset; based on the student classroom image dataset, a predicted single-channel joint heatmap is obtained using the trained HRNet model; based on the manually annotated dataset, a true single-channel joint heatmap is obtained using the trained HRNet model; based on the difference between the predicted single-channel joint heatmap and the true single-channel joint heatmap... The algorithm optimizes the weight parameters in the trained HRNet model to obtain an optimized HRNet model, thus improving the accuracy of HRNet model optimization. It then obtains predicted limb heatmaps and new predicted single-channel joint heatmaps using the optimized HRNet model. These new predicted single-channel joint heatmaps or predicted limb heatmaps are superimposed on the original image and stacked along the time dimension to obtain a 3D heatmap volume, improving the accuracy of 3D heatmap volume acquisition. A dual-stream 3D-CNN network is constructed. The 3D heatmap volume and RGB images from student classroom videos are input into the dual-stream 3D-CNN network. A lateral connection method using early feature fusion and late feature fusion is employed to obtain the final fused features. These final fused features are then used for student classroom behavior recognition, improving the accuracy of student classroom behavior recognition and addressing the problems of poor robustness, imperfect fusion strategies, and insufficient posture modeling in existing classroom behavior recognition methods in complex environments. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This invention provides a flowchart illustrating the steps of a student classroom behavior recognition method based on cross-modal feature fusion.

[0065] Figure 2 This invention provides a schematic diagram of the module flow of a student classroom behavior recognition system based on cross-modal feature fusion. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0067] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0068] To address the problems existing in the background technology, this study designs a student classroom behavior recognition method and system based on cross-modal feature fusion, which has important practical significance.

[0069] like Figure 1 As shown, the first aspect of this invention is to provide a method for recognizing student classroom behavior based on cross-modal feature fusion, comprising the following steps:

[0070] Step S001: Pre-train the HRNet model to obtain all human body joints and their corresponding confidence scores in each frame of the image. Divide the student classroom video into several time periods. Determine the number of random samples for each time period based on the distribution and confidence scores of human body joints in each time period. Determine the student classroom image dataset based on the number of random samples.

[0071] It should be noted that in order to recognize students' classroom behavior, it is necessary to determine the students' posture. In the field of image processing, human posture recognition can be achieved by using the HRNet model to determine the key node information of the human body, thereby realizing the recognition of students' classroom behavior.

[0072] It should be further noted that, in order to ensure that the pre-trained model has good performance, a dataset with rich key point annotations, diverse images, and a large amount of data is selected for pre-training.

[0073] Specifically, the MS COCO dataset (Microsoft Common Objects in Context) is obtained; the HRNet model (High-Resolution Network) is pre-trained using the MS COCO dataset to obtain the trained HRNet model; the MS COCO dataset is a well-known dataset; the HRNet model is a well-known technology, and will not be described in detail here.

[0074] It should be noted that, in order to improve the generalization and randomness of the model, the video is uniformly divided into several data segments, and each segment is randomly sampled. To reduce the computational load and background interference, each frame is cropped before random sampling, retaining only the bounding box of the person, i.e., cropping out the bounding box around the person and removing most of the background.

[0075] Specifically, a student classroom video is collected, and the video is preprocessed at preset time intervals. The video was divided into several evenly spaced segments. The student classroom videos were collected with the students' consent and authorization, and therefore did not infringe on student privacy.

[0076] The specific process for preprocessing student classroom videos is as follows: Student targets are detected in video frames using Faster R-CNN; bounding boxes with a confidence level greater than a confidence threshold are retained; and each frame of the student classroom video is cropped, retaining only the bounding boxes of the figures. In this embodiment, the confidence threshold is 0.7. However, this confidence threshold is not specifically limited in this embodiment and can be determined by the implementer based on specific circumstances.

[0077] Faster R-CNN is a well-known technology and will not be described in detail here.

[0078] In this embodiment, a preset time interval is specified. seconds, where in this embodiment the preset time interval is... No specific restrictions are imposed; implementers can decide based on the specific circumstances.

[0079] It's important to note that the information density varies significantly across different time periods in classroom videos. For example, segments where students raise their hands to answer questions or participate in discussions contain rich behavioral information, while segments where students sit in uniform postures listening to lectures or taking notes are relatively static and have high redundancy. Using conventional uniform random sampling methods would result in insufficient sampling of high-information segments and missing key behavioral frames, while excessive sampling of low-information segments would generate a large amount of redundant data. To address this issue, an adaptive sampling method based on posture and action complexity is proposed. This method dynamically adjusts the number of samples according to the information density across time periods, thereby improving the representativeness and information efficiency of the dataset.

[0080] Specifically, the velocity information of each frame's joints is obtained by calculating the changes in joint position between consecutive frames; based on the velocity information of all joints in two adjacent frames within each time period, the average motion amplitude of each frame's joints within each time period is obtained; the average motion amplitude of each frame's joints within each time period is specifically expressed by the formula:

[0081]

[0082] In the formula, Indicates the first Within the time period, the first The first frame The speed of each joint point This indicates the total number of nodes. Indicates the first Within the time period, the first The average motion amplitude of frame joints.

[0083] Specifically, the greater the velocity of each frame's joint within each time period, the greater the average motion amplitude of that frame's joint; conversely, the smaller the velocity of each frame's joint within each time period, the smaller the average motion amplitude of that frame's joint.

[0084] Based on the distribution of each joint point within each time period, the deviation of each joint point in each frame within each time period (reflecting pose diversity) is obtained. The specific formula for the deviation of each joint point in each frame within each time period is as follows:

[0085]

[0086] In the formula, Indicates the first Within the time period, the first Frame and the The first frame The distance traveled by each joint. Indicates the first Within the time period, the first Frame and the The first frame The distance traveled by each joint. This indicates the total number of nodes. Indicates the first Within the time period, the first Deviation of frame joints.

[0087] Specifically, the greater the movement distance between each frame and the same joint point of the two adjacent frames, the greater the deviation of the joint point in that frame; conversely, the smaller the movement distance between each frame and the same joint point of the two adjacent frames, the smaller the deviation of the joint point in that frame.

[0088] The complexity of each frame within each time period is obtained based on the average confidence of all relevant nodes in each frame within each time period, the average motion amplitude and deviation of the key points in each frame within each time period; the complexity of each frame within each time period is specifically expressed by the formula:

[0089]

[0090] In the formula, Indicates the first Within the time period, the first The average confidence level of all relevant nodes in the frame. Indicates the first Within the time period, the first The average motion amplitude of frame joints, Indicates the first Within the time period, the first Frame keypoint deviation, This indicates the preset first weight coefficient. This indicates a preset second weighting coefficient. Indicates the first Within the time period, the first Frame complexity, This represents a linear normalization function. In this embodiment, a first weighting coefficient is preset. Preset second weighting coefficient In this embodiment, a preset first weighting coefficient is used. and preset second weighting coefficient No specific limitations are set, but it must be ensured that... The implementer can decide based on the specific circumstances.

[0091] Specifically, a larger average motion amplitude and deviation of joints per frame within each time period indicates greater complexity of the motion and posture in each frame within that time period, meaning higher complexity per frame within that time period. Conversely, a smaller average motion amplitude and deviation of joints per frame within each time period indicates less complexity of the motion and posture in each frame within that time period, meaning lower complexity per frame within that time period. A higher average confidence level of all joints in each frame within each time period indicates higher accuracy of the complexity analysis based on average motion amplitude and deviation.

[0092] Thus, the complexity of each frame within each time period is obtained using the method described above.

[0093] The complexity of each time period is obtained by analyzing the complexity distribution of all frames within that time period; the specific formula for the complexity of each time period is as follows:

[0094]

[0095] In the formula, Indicates the first The average complexity of all frames within a given time period. Indicates the first The standard deviation of the complexity of all frames within a time period. Indicates the first The complexity of a time period Represents a linear normalization function. This represents the contribution factor of complexity fluctuations over a given time period to the overall complexity over that time period. This is a preset value, in this embodiment. In this embodiment, the contribution factor No specific restrictions are imposed; implementers can decide based on the specific circumstances.

[0096] Based on the complexity and basic sample size of each time period, the number of random samples for each time period is obtained; the specific number of random samples for each time period is expressed by the formula:

[0097]

[0098] In the formula, Indicates the first The complexity of a time period This represents the base sample size (set manually). Indicates to Perform rounding operation. Indicates the first The number of random samples in each time period.

[0099] Thus, the number of random samples for each time period is obtained using the method described above.

[0100] Based on the random sampling quantity for each time period, images are randomly collected from several frames within each time period. All randomly collected images from all time periods are combined into a dataset, denoted as the student classroom image dataset. The number of images randomly collected in each time period is the random sampling quantity for that time period.

[0101] At this point, we have obtained the student classroom image dataset.

[0102] Step S002: Based on the student classroom image dataset, obtain a predicted single-channel joint heatmap using the trained HRNet model; based on the manually labeled dataset, obtain a real single-channel joint heatmap using the trained HRNet model; optimize the weight parameters in the trained HRNet model based on the difference between the predicted and real single-channel joint heatmaps to obtain an optimized HRNet model; obtain a predicted limb heatmap and a new predicted single-channel joint heatmap using the optimized HRNet model; superimpose the new predicted single-channel joint heatmap or the predicted limb heatmap onto the original image, and then stack them along the time dimension to obtain the 3D heatmap volume.

[0103] It should be noted that there may be errors when using the trained HRNet model to identify student classroom behavior. When the model training effect is poor, the accuracy of the model can be optimized by comparing the key nodes predicted by the trained HRNet model with the manually determined labeled nodes.

[0104] The student classroom image dataset was manually annotated, and the annotated dataset was denoted as the manually annotated dataset.

[0105] It should be noted that converting discrete joint coordinates into a continuous probability distribution, such as a Gaussian distribution, can smooth out positioning errors and improve the robustness of the model.

[0106] Specifically, based on the student classroom image dataset, a predicted single-channel joint heatmap is obtained using the trained HRNet model; the process of obtaining the predicted single-channel joint heatmap of human joints in the student classroom image dataset is as follows:

[0107]

[0108] In the formula, This represents the horizontal value of each pixel in each image of the student classroom image dataset. This represents the vertical value of each pixel in each image of the student classroom image dataset; This represents the first image in each image of the student classroom image dataset. The horizontal values ​​of individual body joints This represents the first image in each image of the student classroom image dataset. Vertical values ​​of individual joints; This represents the Gaussian kernel standard deviation. This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents an exponential function with the natural constant as its base. This represents the first image in each image of the student classroom image dataset. Human body joints at pixels The heatmap value. The Gaussian kernel standard deviation is a preset value; in this embodiment, the Gaussian kernel standard deviation is 7 pixel values. In this embodiment, the Gaussian kernel standard deviation is not specifically limited, and the implementer can determine it according to specific circumstances.

[0109] In each image of the student classroom image dataset, the first... A single image is created by combining the heatmap values ​​of all pixels for each human body joint, denoted as the [number]th [image]. A predictive single-channel joint heatmap.

[0110] At this point, 17 predicted single-channel joint heatmaps have been generated.

[0111] It should be noted that in order to verify the accuracy of the trained HRNet model in recognizing human nodes, it is necessary to use manually labeled data for verification and optimization.

[0112] Specifically, the process of obtaining the true single-channel joint heatmap of human joints in the manually labeled dataset is as follows:

[0113]

[0114] In the formula, This represents the horizontal value of each pixel in each image of the manually labeled dataset. This represents the vertical value of each pixel in each image of the manually labeled dataset; This represents the first manually labeled image in each image of the manually labeled dataset. The horizontal values ​​of individual body joints This represents the first manually labeled image in each image of the manually labeled dataset. Vertical values ​​of individual joints; This represents the Gaussian kernel standard deviation. This represents an exponential function with the natural constant as its base. This represents the first image in each image of the manually labeled dataset. Human body joints at pixels Heatmap values.

[0115] In each student classroom image of the manually labeled dataset, the first... A graph is created by combining the heatmap values ​​of each real human joint point across all pixels, denoted as the i-th. A real human joint point heat map;

[0116] At this point, 17 real single-channel joint heatmaps have been generated.

[0117] Based on the predicted single-channel joint heatmap and the actual single-channel joint heatmap, the weight parameters in the trained HRNet model are optimized using the mean squared error loss function, and the optimized HRNet model is obtained using the optimized weight parameters.

[0118] The mean squared error loss function is a well-known technique and will not be described in detail here.

[0119] It should be noted that supplementing the connection information between joints can better represent limb movements (such as raising an arm) and enhance the ability to capture continuous movements.

[0120] Specifically, the process of obtaining predicted limb heatmaps from student classroom image datasets is as follows:

[0121]

[0122] In the formula, This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents the horizontal value of each pixel in each image of the student classroom image dataset. This represents the vertical value of each pixel in each image of the student classroom image dataset. This represents the first image in each image of the student classroom image dataset. Individual body joints and the first The limb segments formed between the joints of an individual's body. This represents the number of pixels in each image of the student classroom image dataset. To limb segments The distance between them This represents an exponential function with the natural constant as its base. This represents the limb line segments in each image of the student classroom image dataset. At pixel heatmap values, This represents the Gaussian kernel standard deviation. This represents the function that takes the minimum value.

[0123] In each image, each limb segment is analyzed at all pixels. The heatmap values ​​are combined into a graph, which is recorded as the predicted limb heatmap for each limb segment.

[0124] At this point, multiple predicted limb heatmaps have been generated (the number depends on the defined limb connection method).

[0125] Based on the process of obtaining the predicted single-channel joint heatmap, a new predicted single-channel joint heatmap is obtained through the optimized HRNet model.

[0126] The newly predicted single-channel joint heatmap or predicted limb heatmap is overlaid onto the original image to obtain a 2D heatmap (single frame) containing all joint information; consecutive frame 2D heatmaps are stacked along the time dimension to generate a size of [size missing]. The volume of the 3D heatmap; among which, This indicates the total number of nodes. Indicates the length of time. Indicates altitude, Indicates the width.

[0127] Thus, the volume of the 3D heat map obtained through the above method is obtained.

[0128] Step S003: Construction of the dual-stream 3D-CNN network.

[0129] It should be noted that, in order to extract the temporal and spatial features from the image, a two-stream network is designed to extract the temporal and spatial features separately.

[0130] It should be further explained that the RGB slow stream focuses on high-resolution appearance information, capturing detailed semantics (such as object texture); the POSE fast stream utilizes high frame rate pose dynamics to depict action trends (such as joint movements). The dual-stream parallel processing aims to address the limitations of single-modality processing (such as RGB's sensitivity to lighting and POSE's lack of scene context), while explicitly modeling spatiotemporal relationships through 3D convolution, ultimately providing subsequent fusion modules with feature representations that combine semantics and dynamics, laying a robust feature foundation for behavior recognition in complex classroom scenarios.

[0131] Specifically, a dual-stream 3D-CNN (3-Dimensional Convolutional Neural Network) is designed, comprising an RGB modality branch and a POSE modality branch; both the RGB and POSE modality branches are neural network structures. The RGB modality branch is a ResNet-based 3D-CNN with a neural network structure containing four residual modules, used to focus on detailed texture information. The POSE modality branch is a lightweight 3D-CNN (ResNet3dSlowOnly) neural network structure, with the backbone network using ResNet3dSlowOnly as the base network, including one convolutional layer and three ResNet layers, reducing the number of channels to 32 and lowering computational cost. Here, RGB stands for Red, Green, and Blue; POSE refers to pose; ResNet is a residual network; and ResNet3dSlowOnly is a 3D residual network 3D slow-only pathway.

[0132] Specifically, the SGD algorithm is used to optimize the parameters in the RGB modality branch; the POSE modality branch is trained using pose data compression and downsampling techniques during the training phase. The SGD algorithm, pose data compression, and downsampling techniques are all well-known and will not be described in detail here.

[0133] The RGB modality branch takes the original image from the student classroom image dataset as input and outputs a high-dimensional feature vector; the POSE modality branch takes the 3D heatmap volume as input and outputs a high-dimensional feature vector.

[0134] This completes the construction of the dual-stream 3D-CNN network.

[0135] Step S004: Input the 3D heatmap volume and the RGB image from the student classroom video into the dual-stream 3D-CNN network, and use the lateral connection method of early feature fusion and late feature fusion to obtain the final fused features.

[0136] It should be noted that the strategy of combining early fusion (bidirectional lateral connections) and late fusion (feature splicing) aims to achieve deep complementarity and synergistic enhancement between RGB appearance features and POSE pose features. Early fusion promotes modal information interaction between shallow networks, guiding the feature extraction process; late fusion integrates high-level semantics from both streams, forming a unified feature representation with stronger discriminative power. The ultimate goal is to overcome the perceptual limitations of a single modality (such as RGB's sensitivity to lighting and POSE's lack of texture details), and improve the model's robustness in decision-making in complex classroom environments and the accuracy of behavior classification.

[0137] Specifically, to facilitate feature fusion between RGB and POSE modalities, during the forward propagation process, convolution and max pooling operations are first applied to the input RGB image and 3D heatmap volume respectively through the paths of the RGB modal branch and the POSE modal branch to obtain RGB shallow feature tensors and POSE shallow feature tensors. Subsequently, information exchange and fusion are performed on the two shallow feature tensors through bidirectional lateral connections to obtain information-enhanced RGB shallow features and information-enhanced POSE shallow features.

[0138] The purpose of this is to enable each path to obtain information from the other path, thereby facilitating the fusion of RGB and POSE information.

[0139] Then, the enhanced RGB shallow features and POSE shallow features are put back into their respective paths, and convolution and non-linear activation operations are performed to obtain the RGB deep feature tensor and the POSE deep feature tensor. Finally, the RGB deep feature tensor and the POSE deep feature tensor are fused to obtain the final fused features.

[0140] To avoid overfitting, the two paths are trained using the cross-entropy loss function, where the number of categories in the classification task is... The cross-entropy loss function is a well-known technique and will not be described in detail here.

[0141] A lateral connection approach combining early feature fusion (shallow features after fusion) and late feature fusion (deep features) yields a final fused feature compared to using only late feature fusion. In the experiments, RGB and POSE features were fused at a 1:1 ratio and then input into a fully connected layer, where a classifier using the Softmax function was used for classification.

[0142] This design can extract features from different sources and connect and fuse RGB and POSE features at different levels to obtain a richer and more comprehensive feature representation.

[0143] Step S005: Identify student classroom behavior through final fusion features.

[0144] The final fused features are passed through a fully connected layer and a Softmax classifier to output a class probability distribution. This class probability distribution is then used to predict and identify student classroom behavior categories. The Softmax classifier is a well-known technique and will not be described in detail here.

[0145] Finally, the classification results and performance evaluation are verified, including the following data:

[0146] The student classroom image dataset is divided into seven behavioral categories: drinking water, listening to lectures, using mobile phones, raising hands, reading, sleeping, and writing. Specifically, the number of instances of drinking water is 32, listening to lectures is 20, using mobile phones is 42, raising hands is 33, reading is 29, sleeping is 45, and writing is 25. The labels for drinking water are 0, listening to lectures is 1, using mobile phones is 2, raising hands is 3, reading is 4, sleeping is 5, and writing is 6.

[0147] The experimental results show that the Top-1 accuracy of RGB+POSE fusion is 85.29%, which is significantly better than that of single-modality (RGB is 83.89% and POSE is 82.4%).

[0148] Computational efficiency: The increase in pose estimation time in multi-person scenarios is controllable (2s video: 5s for a single person → 6s for three people).

[0149] Practicality test: The highest recognition accuracy (77.78%) was achieved when the video was segmented into 48 frames.

[0150] This concludes the embodiment.

[0151] like Figure 2 As shown, a second aspect of the present invention is to provide a student classroom behavior recognition system based on cross-modal feature fusion, comprising:

[0152] Data acquisition module 101: used to pre-train the HRNet model to obtain the trained HRNet model; acquire a student classroom video and preprocess the student classroom video; input each frame of the student classroom video into the trained HRNet model to obtain all human joints in each frame and obtain the confidence of each human joint; divide the student classroom video into several time periods equally; determine the number of random samples in each time period based on the distribution and confidence of human joints in each time period, and determine the student classroom image dataset based on the number of random samples;

[0153] Parameter optimization module 102: This module manually annotates the student classroom image dataset to obtain an annotated dataset; based on the student classroom image dataset, it uses a trained HRNet model to obtain a predicted single-channel joint heatmap; based on the annotated dataset, it uses a trained HRNet model to obtain a true single-channel joint heatmap; based on the difference between the predicted and true single-channel joint heatmaps, it optimizes the weight parameters in the trained HRNet model to obtain an optimized HRNet model; it uses the optimized HRNet model to obtain a predicted limb heatmap and a new predicted single-channel joint heatmap; it overlays the new predicted single-channel joint heatmap or the predicted limb heatmap onto the original image, and then stacks them along the time dimension to obtain the 3D heatmap volume.

[0154] Feature fusion extraction module 103: used to construct a dual-stream 3D-CNN network; inputting the 3D heat map volume and the RGB images in the student classroom video into the dual-stream 3D-CNN network, using the lateral connection method of early feature fusion and late feature fusion to obtain the final fused features, and using the final fused features to identify student classroom behavior.

[0155] A third aspect of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a student classroom behavior recognition method based on cross-modal feature fusion.

[0156] A fourth aspect of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a student classroom behavior recognition method based on cross-modal feature fusion.

[0157] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0158] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for recognizing student classroom behavior based on cross-modal feature fusion, characterized in that, include: Pre-train the HRNet model to obtain the trained HRNet model; Acquire a student's classroom video and preprocess it; Each frame of the student's classroom video is input into the trained HRNet model to obtain all human body joints in each frame and the confidence score of each human body joint. The student classroom videos are equally divided into several time segments; the number of random samples for each time segment is determined by the distribution and confidence level of human body joints within each time segment, and the student classroom image dataset is determined by the number of random samples, including: At a preset time interval The student classroom video is evenly divided into several time segments. Velocity information of each frame's joints is obtained by calculating the changes in joint position between consecutive frames. The complexity of each frame within each time segment is obtained based on the average confidence of all joints in each frame, the velocity information of all joints in two adjacent frames within each time segment, and the distribution of joints in adjacent frames. The complexity of each time segment is obtained by analyzing the distribution of complexity across all frames within each time segment. The number of random samples for each time segment is determined based on its complexity, and random sampling is performed for each time segment using this number of samples to obtain the student classroom image dataset. The student classroom image dataset was manually annotated to obtain a manually annotated dataset. Based on a student classroom image dataset, a predicted single-channel joint heatmap is obtained using a trained HRNet model. A true single-channel joint heatmap is obtained using a manually labeled dataset and the trained HRNet model. The weight parameters in the trained HRNet model are optimized based on the differences between the predicted and true single-channel joint heatmaps to obtain an optimized HRNet model. A predicted limb heatmap and a new predicted single-channel joint heatmap are obtained using the optimized HRNet model. The new predicted single-channel joint heatmap or the predicted limb heatmap is then overlaid onto the original image and stacked along the time dimension to obtain the 3D heatmap volume. A dual-stream 3D-CNN network is constructed. The 3D heatmap volume and the RGB images from the student classroom video are input into the dual-stream 3D-CNN network. The network is connected laterally by early feature fusion and late feature fusion to obtain the final fused features. The student classroom behavior is then recognized using the final fused features.

2. The student classroom behavior recognition method based on cross-modal feature fusion according to claim 1, characterized in that, The complexity of each frame within each time period is obtained based on the average confidence of all relevant nodes in each frame within each time period, the velocity information of all relevant nodes in two adjacent frames within each time period, and the distribution of key points in adjacent frames. This includes: Based on the velocity information of all relevant nodes in two adjacent frames within each time period, the average motion amplitude of each frame in each time period is obtained. Based on the distribution of key points in adjacent frames within each time period, the deviation of key points in each frame within each time period is obtained; based on the average confidence of all key points in each frame within each time period, the average motion amplitude and deviation of key points in each frame within each time period, the complexity of each frame within each time period is obtained. The average motion amplitude of each frame's joint within each time period is specifically expressed by the following formula: In the formula, Indicates the first Within the time period, the first The first frame The speed of each joint point This indicates the total number of nodes. Indicates the first Within the time period, the first The average motion amplitude of frame joints; The specific formula for the deviation of each frame's key point within each time period is as follows: In the formula, Indicates the first Within the time period, the first Frame and the The first frame The distance traveled by each joint. Indicates the first Within the time period, the first Frame and the The first frame The distance traveled by each joint. This indicates the total number of nodes. Indicates the first Within the time period, the first Frame joint deviation; The complexity of each frame within each time period is specifically expressed by the formula: In the formula, Indicates the first Within the time period, the first The average confidence level of all relevant nodes in the frame. This indicates the preset first weight coefficient. This indicates a preset second weighting coefficient. Indicates the first Within the time period, the first Frame complexity, This represents the linear normalization function.

3. The student classroom behavior recognition method based on cross-modal feature fusion according to claim 1, characterized in that, The complexity of each time period is obtained by analyzing the complexity distribution of all frames within that time period. Based on the complexity of each time period, the number of random samples for each time period is determined. Random sampling is then performed for each time period using this number of samples to obtain the student classroom image dataset, which includes: The complexity of each time period is specifically expressed by the formula: In the formula, Indicates the first The average complexity of all frames within a given time period. Indicates the first The standard deviation of the complexity of all frames within a time period. Indicates the first The complexity of a time period Represents a linear normalization function. The contribution factor of the complexity fluctuation within a time period to the complexity of that time period is a preset value. The specific number of random samples in each time period is expressed by the formula: In the formula, Indicates the basic sample number. Indicates to Perform rounding operation. Indicates the first The number of random samples within a time period; where the base sample size is set manually; Based on the number of random samples in each time period, images are randomly collected from several frames within each time period. All images randomly collected from all time periods are combined into a dataset, denoted as the student classroom image dataset. The number of images randomly collected in each time period is the number of random samples in each time period.

4. The student classroom behavior recognition method based on cross-modal feature fusion according to claim 1, characterized in that, The process involves obtaining a predicted single-channel joint heatmap using a trained HRNet model based on a student classroom image dataset; obtaining a true single-channel joint heatmap using a trained HRNet model based on a manually labeled dataset; and optimizing the weight parameters in the trained HRNet model based on the difference between the predicted and true single-channel joint heatmaps to obtain an optimized HRNet model. This includes: The process of obtaining the predicted single-channel joint heatmap is as follows: In the formula, This represents the horizontal value of each pixel in each image of the student classroom image dataset. This represents the vertical value of each pixel in each image of the student classroom image dataset; This represents the first image in each image of the student classroom image dataset. The horizontal values ​​of individual body joints, This represents the first image in each image of the student classroom image dataset. Vertical values ​​of individual joints; This represents the Gaussian kernel standard deviation. This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents an exponential function with the natural constant as its base. This represents the first image in each image of the student classroom image dataset. Human body joints at pixels The heatmap values; where the Gaussian kernel standard deviation is a preset value; In each image of the student classroom image dataset, the first... A single image is created by combining the heatmap values ​​of all pixels for each human body joint, denoted as the [number]th [image]. A predicted single-channel joint heatmap; The process of obtaining the real single-channel joint heatmap is as follows: In the formula, This represents the horizontal value of each pixel in each image of the manually labeled dataset. This represents the vertical value of each pixel in each image of the manually labeled dataset; This represents the first manually labeled image in each image of the manually labeled dataset. The horizontal values ​​of individual body joints, This represents the first manually labeled image in each image of the manually labeled dataset. Vertical values ​​of individual joints; This represents the first image in each image of the manually labeled dataset. A real human joint point at a pixel Heatmap values; In each image of the manually labeled dataset, the first... A single image is created by combining the heatmap values ​​of all pixels for each human body joint, denoted as the [number]th [image]. A real single-channel joint heat map; Based on the predicted single-channel joint heatmap and the actual single-channel joint heatmap, the weight parameters in the trained HRNet model are optimized using the mean squared error loss function, and the optimized HRNet model is obtained using the optimized weight parameters.

5. The student classroom behavior recognition method based on cross-modal feature fusion according to claim 4, characterized in that, The process involves obtaining predicted limb heatmaps and newly predicted single-channel joint heatmaps using the optimized HRNet model; overlaying the newly predicted single-channel joint heatmap or predicted limb heatmap onto the original image, and then stacking them along the time dimension to obtain the 3D heatmap volume, including: Based on the process of obtaining the predicted single-channel joint heatmap, a new predicted single-channel joint heatmap is obtained through the optimized HRNet model. The process of obtaining the predicted limb heatmap is as follows: In the formula, This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents the first image in each image of the student classroom image dataset. Confidence level of individual body joints. This represents the horizontal value of each pixel in each image of the student classroom image dataset. This represents the vertical value of each pixel in each image of the student classroom image dataset. This represents the first image in each image of the student classroom image dataset. Individual body joints and the first The limb segments formed between the joints of an individual's body. This represents the number of pixels in each image of the student classroom image dataset. To limb segments The distance between them This represents an exponential function with the natural constant as its base. This represents the limb line segments in each image of the student classroom image dataset. At pixel heatmap values, This represents the function that takes the minimum value. Indicates the Gaussian kernel standard deviation; The heatmap values ​​of each limb segment in each image are combined into a single image, which is called the predicted limb heatmap for each limb segment. The newly predicted single-channel joint heatmap or predicted limb heatmap is overlaid onto the original image to obtain a 2D heatmap containing all joint information; the 2D heatmap is a single frame image; consecutive frame 2D heatmaps are stacked along the time dimension to generate a size of 3D heatmap volume; in, This indicates the total number of nodes. Indicates the length of time. Indicates altitude, Indicates the width.

6. The student classroom behavior recognition method based on cross-modal feature fusion according to claim 1, characterized in that, The process involves inputting the 3D heatmap volume and RGB images from student classroom videos into a dual-stream 3D-CNN network. A lateral connection approach using early and late feature fusion is employed to obtain the final fused features. These final fused features are then used for student classroom behavior recognition, including: The dual-stream 3D-CNN network includes an RGB modality branch and a POSE modality branch; During the forward propagation process, convolution and max pooling operations are first applied to the input RGB image and 3D heatmap volume through the paths of the RGB modality branch and the POSE modality branch, respectively, to obtain the RGB shallow feature tensor and the POSE shallow feature tensor. Then, information exchange and fusion are performed on the two shallow feature tensors through bidirectional lateral connections to obtain the information-enhanced RGB shallow features and the information-enhanced POSE shallow features. Then, the enhanced RGB shallow features and POSE shallow features are put back into their respective paths, and convolution and non-linear activation operations are used to obtain the RGB deep feature tensor and the POSE deep feature tensor. Information fusion is then performed on the RGB deep feature tensor and the POSE deep feature tensor to obtain the final fused features. The final fused features are passed through a fully connected layer and a Softmax classifier to output a category probability distribution. The category probability distribution is then used to predict and identify the categories of student classroom behavior.

7. A student classroom behavior recognition system based on cross-modal feature fusion, characterized in that, include: Data acquisition module: used to pre-train the HRNet model to obtain the trained HRNet model; Acquire a student classroom video and preprocess it; input each frame of the student classroom video into the trained HRNet model to obtain all human body joints in each frame and obtain the confidence score of each human body joint. The student classroom videos are equally divided into several time segments; the number of random samples for each time segment is determined by the distribution and confidence level of human body joints within each time segment, and the student classroom image dataset is determined by the number of random samples, including: At a preset time interval The student classroom video is evenly divided into several time segments. Velocity information of each frame's joints is obtained by calculating the changes in joint position between consecutive frames. The complexity of each frame within each time segment is obtained based on the average confidence of all joints in each frame, the velocity information of all joints in two adjacent frames within each time segment, and the distribution of joints in adjacent frames. The complexity of each time segment is obtained by analyzing the distribution of complexity across all frames within each time segment. The number of random samples for each time segment is determined based on its complexity, and random sampling is performed for each time segment using this number of samples to obtain the student classroom image dataset. Parameter optimization module: used to manually annotate student classroom image datasets to obtain manually annotated datasets; Based on a student classroom image dataset, a predicted single-channel joint heatmap is obtained using a trained HRNet model. A true single-channel joint heatmap is obtained using a manually labeled dataset and the trained HRNet model. The weight parameters in the trained HRNet model are optimized based on the differences between the predicted and true single-channel joint heatmaps to obtain an optimized HRNet model. A predicted limb heatmap and a new predicted single-channel joint heatmap are obtained using the optimized HRNet model. The new predicted single-channel joint heatmap or the predicted limb heatmap is then overlaid onto the original image and stacked along the time dimension to obtain the 3D heatmap volume. Feature fusion and extraction module: used to construct a dual-stream 3D-CNN network; inputting the 3D heatmap volume and RGB images from student classroom videos into the dual-stream 3D-CNN network, using a lateral connection method of early feature fusion and late feature fusion to obtain the final fused features, and using the final fused features to identify student classroom behavior.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the student classroom behavior recognition method based on cross-modal feature fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the student classroom behavior recognition method based on cross-modal feature fusion as described in any one of claims 1-6.

Images