Student behavior detection method based on self-supervised learning
Through self-supervised learning method in student behavior detection, the high cost and low efficiency problems of manual annotation in the existing technology are solved through self-supervised learning methods, and efficient and accurate student behavior detection and personality are achieved. Support for teaching.
Patent Information
- Application Number
- CN202410948329.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing student behavior detection methods rely on a large amount of manual labeling data, resulting in high cost, low efficiency, inconsistent data quality, and privacy and ethical problems, making it difficult to apply on a large scale in actual teaching environments.
Using a self-supervised learning method, automatically learn features through labeled data without labeled data, combining time consistency loss and comparison loss, reducing dependence on labeled data, improving training efficiency and feature extraction accuracy, realizing student behavior detection, and supporting personalization teaching.
Significantly reduces the cost and time of labeling, improves the accuracy and robustness of behavioral detection, monitors students' attention and participation in real time, helps teachers adjust teaching strategies, improves teaching quality and student learning effects, and solves data privacy and ethical problems. .
Smart Images

Figure CN119007095A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning, and in particular to a student behavior detection method based on self-supervised learning. Background Art
[0002] With the development of artificial intelligence, the education field is gradually applying AI technology to student behavior detection to achieve more intelligent and personalized teaching methods. Student behavior detection is a key link in educational data analysis. It can provide real-time feedback on students' attention, participation, and emotional state, helping teachers to adjust teaching strategies in a timely manner. However, existing student behavior detection methods mainly rely on a large amount of manually annotated data, which brings many challenges and defects.
[0003] First, manual data labeling requires a lot of manpower and time. Collecting and labeling student behavior data is not only a cumbersome process, but also requires professionals to process and verify the data, resulting in high overall costs. Especially in large-scale application scenarios, the labeling workload is huge, and it is difficult to achieve continuous and effective labeling. In addition, the quality of manually labeled data is also uneven. Due to subjective factors and human errors, the labeling results may be inconsistent and biased, which further affects the accuracy and robustness of model training. Secondly, there are privacy and ethical issues in the acquisition of manually labeled data. Student behavior data is sensitive information. During the collection and use process, relevant laws, regulations and ethical standards must be strictly observed to ensure the protection of student privacy. These restrictions increase the difficulty of data collection and make it more difficult to obtain high-quality labeled data on a large scale in actual teaching environments. .
[0004] Therefore, a student behavior detection method based on self-supervised learning is proposed to solve the above problems. The self-supervised learning method not only greatly reduces the dependence on labeled data, improves training efficiency and the accuracy of feature representation, but also solves data privacy and ethical issues, making it easier to deploy and apply in actual teaching environments. With this method, educators can monitor students' attention and participation in real time, adjust teaching strategies in a timely manner, provide personalized teaching support, and ultimately improve teaching quality and student learning outcomes. Summary of the invention
[0005] The purpose of the present invention is to provide a student behavior detection method based on self-supervised learning, which reduces the dependence on a large amount of labeled data by automatically learning features from unlabeled data, improves training efficiency and the accuracy of feature extraction, and combines temporal consistency loss and contrast loss. The model can better capture behavioral dynamics, achieve accurate student behavior detection, and support personalized teaching.
[0006] To achieve the above object, the present invention provides a student behavior detection method based on self-supervised learning, comprising the following steps:
[0007] Step S1, preprocessing the unlabeled input data and performing data enhancement to obtain two enhanced views, and dividing the two enhanced views into time windows of fixed length;
[0008] Step S2, extracting features from the time windows of the two enhanced views respectively;
[0009] Step S3: Calculate the temporal consistency loss of the two enhanced views to encourage the feature representations of adjacent time steps to remain similar in the feature space;
[0010] Step S4, calculating a contrast loss function to enhance the feature difference between two enhanced views in different time windows while maintaining the feature consistency within the same time window;
[0011] Step S5: weighted sum of the temporal consistency loss and the contrast loss to form a final comprehensive loss function to train the model;
[0012] Step S6: Use the pre-trained feature extractor in combination with the classification head to perform fine-tuning on a small amount of labeled data, ultimately achieving accurate detection of student behavior.
[0013] Preferably, in step S1, the unlabeled input data is preprocessed, including video frame extraction, inter-frame deduplication, image scaling, denoising, color adjustment and image normalization.
[0014] Preferably, performing data enhancement on the unlabeled input data in step S1 includes:
[0015] (1) Traditional data enhancement: contrast enhancement, grayscale change, noise addition, flipping, and translation;
[0016] (2) Timing information transformation: adjusting the video playback speed, inter-frame interpolation, and inter-frame deletion;
[0017] (3) Data enhancement based on hue change: simulate hue changes by randomly disrupting the order of image channels;
[0018] (4) Generative-based video data enhancement: video generation model, video super-resolution enhancement, and video denoising and noise reduction.
[0019] Preferably, the feature extraction in step S2 is as follows:
[0020] (1) Postural characteristics: students’ body posture and limb movements, including sitting, standing, and gestures;
[0021] (2) Facial expression characteristics: students’ facial expressions and emotional states, including smiling, frowning, concentrating, and sleepiness;
[0022] (3) Action and behavior characteristics: students’ specific actions within the time window, including writing, reading, raising hands, and yawning;
[0023] (4) Temporal characteristics: behavioral changes and dependencies between time steps, reflecting the temporal series pattern of behavior;
[0024] (5) Spatial features: local information in the image frame, including the position of the head, the position of the hands, and the relative positions of various parts of the body;
[0025] (6) Motion characteristics: changes between frames and motion trajectories, including motion direction, speed, and acceleration.
[0026] Preferably, the specific process of calculating the time consistency loss in step S3 includes:
[0027] (1) For input data X containing N samples, the two enhanced views after data enhancement are X1 and X2. The time window length is T. Each enhanced view is divided into t time windows, and we get in, and represents the tth window of the ith video;
[0028] (2) After feature extraction, the feature representation is Where f(·) is the feature extraction function, and is the extracted feature;
[0029] (3) Calculate the temporal consistency loss of the two enhanced views to encourage the feature representations of adjacent time steps to remain similar in the feature space. The loss function is as follows:
[0030]
[0031] in, is the L2 norm of a and b.
[0032] Preferably, the calculation formula of the contrast loss function in step S4 is as follows:
[0033]
[0034] Where t≠t', t' represents a time window different from t, sim(x,y)=x T y / ||x|||y|| represents the cosine similarity between x and y, and τ represents the temperature coefficient.
[0035] Preferably, the expression of the comprehensive loss function in step S5 is as follows:
[0036] L total =αL time +(1-α)L contrast
[0037] Among them, α is the weighting coefficient.
[0038] Therefore, the present invention adopts the above-mentioned student behavior detection method based on self-supervised learning, which has the following beneficial effects:
[0039] (1) The present invention reduces the reliance on a large amount of labeled data by automatically learning features, thus solving the problem of time-consuming, costly and inconsistent data labeling in traditional methods. By using data enhancement technology, multiple enhanced views are generated and feature extraction is performed on the time windows of these views. The model can capture more useful information from unlabeled data.
[0040] (2) The present invention introduces temporal consistency loss and contrast loss to ensure that the model can not only maintain the continuity of features of adjacent time steps, but also enhance the feature differences between different time windows. This design enables the model to better capture the temporal dynamic characteristics of video data, thereby improving the accuracy and robustness of behavior detection.
[0041] (3) The present invention is highly practical. Student behavior detection in the field of education requires processing a large amount of unlabeled video data. The self-supervised learning method significantly reduces the labeling cost and time, and monitors students' attention, participation, and emotional state in real time, enabling teachers to adjust teaching strategies in a timely manner, provide personalized support, and improve teaching effectiveness. In addition, this method can also help educational institutions optimize resource allocation and improve teaching quality.
[0042] (4) The present invention not only solves the bottleneck problem of data labeling, but also improves the practicality and accuracy of the model through efficient feature extraction and behavior detection, and is expected to play an important role in future educational practice.
[0043] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 The present invention is a flowchart of an embodiment of a method for detecting student behavior based on self-supervised learning. DETAILED DESCRIPTION
[0045] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.
[0046] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0047] Example
[0048] See also Figure 1 The present invention provides a student behavior detection method based on self-supervised learning, comprising the following steps:
[0049] Step S1: preprocess the unlabeled input data and perform data enhancement to obtain two enhanced views, and divide the two enhanced views into time windows of fixed length.
[0050] In the above step S1, the unlabeled input data is preprocessed, including video frame extraction, inter-frame deduplication, image scaling, denoising, color adjustment and image normalization.
[0051] In the above step S1, data enhancement is performed on the unlabeled input data, including but not limited to:
[0052] (1) Traditional data enhancement: contrast enhancement, grayscale change, noise addition, flipping, and translation;
[0053] (2) Timing information transformation: adjusting the video playback speed, inter-frame interpolation, and inter-frame deletion;
[0054] (3) Data enhancement based on hue change: simulate hue changes by randomly disrupting the order of image channels;
[0055] (4) Generative-based video data enhancement: video generation model, video super-resolution enhancement, and video denoising and noise reduction.
[0056] Step S2: extract features from the time windows of the two enhanced views respectively.
[0057] The features extracted in step S2 above are as follows:
[0058] (1) Posture characteristics: students’ body posture and limb movements, including sitting, standing, gestures, etc.;
[0059] (2) Facial expression characteristics: students’ facial expressions and emotional states, including smiling, frowning, concentrating, and sleepy;
[0060] (3) Action and behavior characteristics: students’ specific actions within the time window, including writing, reading, raising hands, yawning, etc.;
[0061] (4) Temporal characteristics: behavioral changes and dependencies between time steps, reflecting the temporal series patterns of behaviors, etc.
[0062] (5) Spatial features: local information in the image frame, including the position of the head, the position of the hands, the relative positions of various parts of the body, etc.;
[0063] (6) Motion characteristics: changes between frames and motion trajectories, including motion direction, speed, acceleration, etc.
[0064] Step S3: Calculate the temporal consistency loss of the two enhanced views to encourage the feature representations of adjacent time steps to remain similar in the feature space.
[0065] The calculation of the temporal consistency loss in step S3 above refers to minimizing the distance between feature representations of adjacent time steps. The specific process includes:
[0066] (1) For input data X containing N samples, the two enhanced views after data enhancement are X1 and X2. The time window length is T. Each enhanced view is divided into t time windows, and we get in, and represents the tth window of the ith video;
[0067] (2) After feature extraction, the feature representation is Where f(·) is the feature extraction function, and is the extracted feature;
[0068] (3) Calculate the temporal consistency loss of the two enhanced views to encourage the feature representations of adjacent time steps to remain similar in the feature space. The loss function is as follows:
[0069]
[0070] in, is the L2 norm of a and b.
[0071] Step S4: Calculate the contrast loss function to enhance the feature difference between the two enhanced views in different time windows while maintaining the feature consistency within the same time window.
[0072] The calculation of the contrast loss function in the above step S4 refers to calculating the cosine similarity of the two enhanced view feature representations so that the similarity of the same time window is larger and the similarity of different time windows is smaller. The calculation formula is as follows:
[0073]
[0074] Where t≠t', t' represents a time window different from t, sim(x,y)=x T y / ||x|||y|| represents the cosine similarity between x and y, and τ represents the temperature coefficient.
[0075] Step S5: weighted sum of the temporal consistency loss and the contrast loss to form the final comprehensive loss function to train the model.
[0076] The expression of the comprehensive loss function in the above step S5 is as follows:
[0077] L total =αL time +(1-α)L contrast
[0078] Among them, α is the weighting coefficient.
[0079] Step S6: Use the pre-trained feature extractor in combination with the classification head to perform fine-tuning on a small amount of labeled data, ultimately achieving accurate detection of student behavior.
[0080] This embodiment demonstrates a student behavior detection framework based on self-supervised learning. The characteristics of student behavior video data include insufficient labels and complex data distribution. In the training phase, the system preprocesses and enhances the unlabeled input data to obtain two enhanced views and divides them into time windows of fixed length. Subsequently, feature extraction is performed on the time windows of the two enhanced views respectively.
[0081] The system calculates the temporal consistency loss of the two views to encourage the feature representations of adjacent time steps to remain similar in the feature space, and calculates the contrast loss function to enhance the feature differences between the two views in different time windows while maintaining the feature consistency within the same time window. The temporal consistency loss and contrast loss are weighted and summed to form the final comprehensive loss function to train the model.
[0082] In the application phase, the system first fine-tunes the pre-trained model using a small amount of labeled local data. Subsequently, the test data is input into the fine-tuned model to obtain the detection results of the student's behavior. The detection process is fast and simple, and its accuracy and reliability do not rely on human knowledge and experience, ensuring the efficiency and objectivity of the detection process.
[0083] In specific application, the operation steps are as follows:
[0084] 1. Data collection: Install high-definition cameras in the classroom to record students' classroom videos in real time. These video data will be automatically collected by the system and stored on the local server.
[0085] 2. Data preprocessing and enhancement: The system preprocesses the collected video data, including denoising, cropping and enhancement, to generate multiple enhanced views. These views are divided into time windows of fixed length for subsequent processing.
[0086] 3. Feature extraction and time window division: Extract features from the time window of each enhanced view to obtain the behavioral features of students at different time steps. These feature representations will be used to train the model.
[0087] 4. Model training: Calculate the temporal consistency loss and contrast loss, and train the model to capture the temporal dynamic characteristics and behavioral differences of video data by optimizing these loss functions. After training, the model can generate high-quality feature representations.
[0088] 5. Model fine-tuning: In practical applications, each school or classroom can use a small amount of labeled local data to fine-tune the pre-trained model to further improve the local adaptability of the model.
[0089] 6. Real-time detection and feedback: The fine-tuned model is used to detect student behavior in classroom videos in real time. The detection results are fed back to teachers in real time through a visual interface, including information such as students' attention, engagement, and emotional state. Teachers can adjust teaching strategies in a timely manner and provide personalized support based on this feedback.
[0090] 7. Teaching strategy optimization: Based on the student behavior analysis results provided by the system, school administrators and teachers can optimize teaching strategies, improve classroom management, and enhance overall teaching effectiveness.
[0091] Therefore, the present invention adopts the above-mentioned student behavior detection method based on self-supervised learning, which realizes efficient and accurate student behavior detection by utilizing a large amount of unlabeled data and a small amount of labeled data for learning, and improving the model performance through feature consistency and comparative learning.
[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A student behavior detection method based on self-supervised learning, characterized in that: The following steps are involved: Step S1, preprocessing the unlabeled input data and performing data enhancement to obtain two enhanced views, and dividing the two enhanced views into time windows of fixed length; Step S2, extracting features from the time windows of the two enhanced views respectively; Step S3: Calculate the temporal consistency loss of the two enhanced views to encourage the feature representations of adjacent time steps to remain similar in the feature space; Step S4, calculating a contrast loss function to enhance the feature difference between two enhanced views in different time windows while maintaining the feature consistency within the same time window; Step S5: weighted sum of the temporal consistency loss and the contrast loss to form a final comprehensive loss function to train the model; Step S6: Use the pre-trained feature extractor in combination with the classification head to perform fine-tuning on a small amount of labeled data, ultimately achieving accurate detection of student behavior.
2. The method for detecting student behavior based on self-supervised learning according to claim 1, characterized in that: In the step S1, the unlabeled input data is preprocessed, including video frame extraction, inter-frame deduplication, image scaling, denoising, color adjustment and image normalization.
3. A student behavior detection method based on self-supervised learning according to claim 2, characterized in that: In step S1, data enhancement is performed on the unlabeled input data, including: (1) Traditional data enhancement: contrast enhancement, grayscale change, noise addition, flipping, and translation; (2) Timing information transformation: adjusting the video playback speed, inter-frame interpolation, and inter-frame deletion; (3) Data enhancement based on hue change: simulate hue changes by randomly disrupting the order of image channels; (4) Generative-based video data enhancement: video generation model, video super-resolution enhancement, and video denoising and noise reduction.
4. A student behavior detection method based on self-supervised learning according to claim 3, characterized in that: The feature extraction in step S2 is as follows: (1) Postural characteristics: students’ body posture and limb movements, including sitting, standing, and gestures; (2) Facial expression characteristics: students’ facial expressions and emotional states, including smiling, frowning, concentrating, and sleepiness; (3) Action and behavior characteristics: students’ specific actions within the time window, including writing, reading, raising hands, and yawning; (4) Temporal characteristics: behavioral changes and dependencies between time steps, reflecting the temporal series pattern of behavior; (5) Spatial features: local information in the image frame, including the position of the head, the position of the hands, and the relative positions of various parts of the body; (6) Motion characteristics: changes between frames and motion trajectories, including motion direction, speed, and acceleration.
5. A student behavior detection method based on self-supervised learning according to claim 4, characterized in that: The specific process of calculating the time consistency loss in step S3 includes: (1) For input data X containing N samples, the two enhanced views after data enhancement are X1 and X2. The time window length is T. Each enhanced view is divided into t time windows, and we get in, and represents the tth window of the ith video; (2) After feature extraction, the feature representation is Where f(·) is the feature extraction function, and is the extracted feature; (3) Calculate the temporal consistency loss of the two enhanced views to encourage the feature representations of adjacent time steps to remain similar in the feature space. The loss function is as follows: in, is the L2 norm of a and b.
6. A student behavior detection method based on self-supervised learning according to claim 5, characterized in that: The calculation formula of the contrast loss function in step S4 is as follows: Where t≠t', t' represents a time window different from t, sim(x,y)=x T y / ||x|||y|| represents the cosine similarity between x and y, and τ represents the temperature coefficient.
7. A student behavior detection method based on self-supervised learning according to claim 5, characterized in that: The expression of the comprehensive loss function in step S5 is as follows: L total =αL time +(1-α)L contrast Among them, α is the weighting coefficient.
Citation Information
Cited By
Robot teaching learning method and system and electronic equipment
CN120409535A
