Student behavior detection method based on multi-scale statistical fusion domain self-adaption

Abstract

The invention discloses a student behavior detection method based on multi-scale statistical fusion domain self-adaption, and relates to the technical field of behavior detection, and the method comprises the steps: collecting student behavior data at a school end, carrying out the preprocessing, carrying out the data marking through behavior types, and constructing a local data set; initializing a behavior detection model and the weight of each statistical loss function; an HICO-DET data set is used as a source domain data set, a local data set is used as a target domain data set, a multi-scale statistical fusion domain adaptive method is adopted to carry out inter-domain feature zooming, a behavior detection model and each statistical loss function weight are updated, and knowledge migration is completed; repeating the previous step until the algorithm converges to obtain a student behavior detection model adapted to local data; and deploying the obtained student behavior detection model at a school end, and detecting and feeding back student behaviors in real time. Therefore, by the adoption of the method, behavior detection knowledge migration can be effectively achieved through the domain self-adaption method, the problem that labeled samples are insufficient is solved, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of behavior detection technology, and in particular to a student behavior detection method based on multi-scale statistical fusion domain adaptation. Background Art

[0002] With the in-depth integration of artificial intelligence in educational technology, student behavior analysis has become a key tool for optimizing teaching strategies. By capturing students' concentration, interaction level, and emotional response, AI systems can provide instant feedback, allowing educators to flexibly adjust teaching methods to improve learning outcomes. However, the current advancement of student behavior analysis algorithms is severely limited by the acquisition of high-quality labeled data, and this bottleneck has caused multiple problems.

[0003] On the one hand, creating a detailed and accurate manually annotated database is a time-consuming and labor-intensive task. This not only requires expert intervention to ensure the accuracy of the data, but also comes with high time and economic costs. Especially when faced with massive amounts of data, maintaining the efficiency and consistency of annotation becomes a major challenge. On the other hand, subjective judgment and human error inevitably introduce data noise, which may lead to diversity and inaccuracy of labels, thereby weakening the accuracy and reliability of model training, and may produce misleading results in practical applications.

[0004] In summary, although AI provides unprecedented insights into student behavior analysis, the difficulty of building reliable annotated datasets limits its full potential, and innovative solutions are urgently needed to overcome these obstacles. Summary of the invention

[0005] The purpose of the present invention is to provide a student behavior detection method based on multi-scale statistical fusion domain adaptation, which can realize the effective transfer of knowledge, detect student behavior using a small amount of unlabeled local samples, and solve the problem of insufficient labeled samples on the school side.

[0006] To achieve the above object, the present invention provides a student behavior detection method based on multi-scale statistical fusion domain adaptation, comprising the following steps:

[0007] S1. Collect student behavior data at the school end, pre-process the collected data, and use behavior categories to annotate the data to build a local data set;

[0008] S2, initialize the behavior detection model and the weights of each statistical loss function;

[0009] S3, using the HICO-DET dataset as the source domain dataset and the local dataset as the target domain dataset, adopting the multi-scale statistical fusion domain adaptation method to pull the inter-domain features closer, updating the behavior detection model and the weights of each statistical loss function, and completing the knowledge transfer;

[0010] S4, repeat step S3 until the algorithm converges and obtains a student behavior detection model adapted to local data;

[0011] S5. Deploy the obtained student behavior detection model at the school end to detect and provide feedback on student behavior in real time.

[0012] Preferably, in step S1, the student behavior data includes image data of the student's movements, demeanor, and expressions, and the preprocessing includes scaling, denoising, and standardizing the image data.

[0013] Preferably, in step S2, the loss function includes a cross entropy loss function, a maximum mean difference loss function, a joint maximum mean difference loss function and a correlation alignment loss function, which are specifically as follows:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019] Where, L MMD is the maximum mean difference loss function, L JMMD is the joint maximum mean difference loss function, L CORAL is the related alignment loss function, L CE is the cross entropy loss function, B is the batch size, is the source domain sample, is the target domain sample, is the source domain feature, is the target domain feature, f k is the kth kernel function, M is the total number of kernel functions, d is the feature dimension, C src , C tgt are the covariance matrices of source domain features and target domain features, is the i-th sample in the local dataset, is the category label corresponding to the i-th sample, N is the number of behavior categories, 1 is the indicator function, is the output probability distribution vector The probability value of category c in .

[0020] Preferably, step S3 includes updating the behavior detection model and the weights of each statistical loss function based on the maximum mean difference loss function, the joint maximum mean difference loss function and the related alignment loss function, and updating the encoder and classifier in the behavior detection model using the cross entropy loss function;

[0021] The update process within a single batch is as follows:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] In the formula, E t ′、C t ′ are the updated encoder and classifier in a single batch, α t ′、β t ′、γ t ′ are the weights of the updated maximum mean difference loss function, the joint maximum mean difference loss function and the related alignment loss function, η is the learning rate, E t , C t are the encoder and classifier trained in the tth round in a single batch, respectively, t , β t , γ t are the weights of the maximum mean difference loss function, the joint maximum mean difference loss function, and the related alignment loss function in the t-th round of training within a single batch.

[0028] Preferably, the model is updated for a single batch of source domain samples and all batches of target domain samples, and the operation is repeated until all batches of source domain samples are updated to ensure sufficient distribution alignment and knowledge transfer.

[0029] Therefore, the present invention adopts the above-mentioned student behavior detection method based on multi-scale statistical fusion domain adaptation, which has the following technical effects:

[0030] (1) Using domain adaptation methods, we transfer behavior detection knowledge from a source domain dataset with sufficient labeled samples to a target domain dataset with insufficient labeled samples, effectively overcoming the problem of insufficient labeled samples.

[0031] (2) Three statistical loss functions are used to respectively narrow the mean, joint distribution, and covariance of the source domain and target domain features, and their weights are adaptively adjusted by updating the weights by gradient descent to obtain a behavior detection model that matches the local data distribution, thereby improving the accuracy of the model.

[0032] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of a student behavior detection method based on multi-scale statistical fusion domain adaptation. DETAILED DESCRIPTION

[0034] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0035] like Figure 1 As shown, the present invention provides a student behavior detection method based on multi-scale statistical fusion domain adaptation, comprising the following steps:

[0036] S1. The student behavior data collected at the school end includes image data such as students' movements, demeanor, and expressions; the collected image data is preprocessed, including image scaling, denoising, and standardization; and the data is labeled using known behavior categories (such as raising hands, lowering heads, using mobile phones, looking at the blackboard, etc.) to build a local data set.

[0037] S2. Initialize the behavior detection model and the weights of each statistical loss function, as follows:

[0038] The behavior detection model includes an encoder for image feature extraction and a classifier for classifying the extracted features. t and C t To represent. The training process includes four loss functions: cross entropy loss function L CE , maximum mean difference loss function L MMD , joint maximum mean difference loss function L JMMD And the related alignment loss function L CORAL .

[0039] Among them, the cross entropy loss function L CE The receiving model output calculates the loss value, and its weight is fixed to 1; the maximum mean difference loss function L MMD , joint maximum mean difference loss function L JMMD And the related alignment loss function L CORALReceive source domain features and target domain features to calculate the loss value. At training round t, the weights of the three are α t , β t With γ t , all three are values ​​between 0 and 1 and their sum is one.

[0040] S3. Using the HICO-DET dataset as the source domain dataset and the local dataset as the target domain dataset, a multi-scale statistical fusion domain adaptation method is used to pull the inter-domain features closer, update the behavior detection model and the weights of each statistical loss function, complete the knowledge transfer, and obtain a student behavior detection model that adapts to the local data distribution. Among them, the HICO-DET dataset covers 600 public image datasets of different behavior categories.

[0041] In each training round, the source domain samples in each batch are And the target domain samples are Passing through encoder E t Get the endogenous domain features of this batch and target domain features Where B is the batch size. The update of the behavior detection model and the weights of each statistical loss function in a single batch includes:

[0042] First, calculate the maximum mean difference loss function L respectively MMD , joint maximum mean difference loss function L JMMD And the related alignment loss function L CORAL , as follows:

[0043]

[0044]

[0045]

[0046]

[0047] In the formula, f k is the kth kernel function, M is the total number of kernel functions, d is the feature dimension, C src With C tgt are the covariance matrices of the source domain and target domain features, respectively.

[0048] Then, calculate the cross entropy loss function L CE , the calculation process is as follows:

[0049]

[0050] In the formula, is the sample image corresponding to sample number i in the local dataset, is the corresponding category label, N is the number of behavior categories, 1 is the indicator function, is the probability distribution vector output by the model The probability value corresponding to category c.

[0051] Finally, according to the maximum mean difference loss function L MMD , joint maximum mean difference loss function L JMMD , related alignment loss function L CORAL The weighted sum of the values ​​of t and loss function weight α t , β t With γ t , and use the cross entropy loss function L CE To update the encoder E t With classifier C t , the updates within a single batch are as follows:

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] Among them, η is the learning rate, E t ′、C t ′ are the updated encoder and classifier in a single batch, α t ′、β t ′、γ t ′ are the weights of the updated maximum mean difference loss function, joint maximum mean difference loss function and related alignment loss function, respectively.

[0058] During the entire training process, after the loss value calculation and model update are completed for a single batch of source domain samples and all batches of target domain samples, the next batch of source domain samples are switched to calculate the loss value and update the model again with all batches of target domain samples until all batches of source domain samples have completed training, ensuring sufficient distribution alignment and knowledge transfer.

[0059] S4. Repeat step S3 until the algorithm converges and obtains a student behavior detection model that adapts to local data, thereby improving the generalization ability of the student behavior detection model and the target detection accuracy.

[0060] S5. Deploy the trained model at the school end to detect and provide feedback on student behavior in real time.

[0061] Therefore, the present invention adopts the above-mentioned student behavior detection method based on multi-scale statistical fusion domain adaptation, and uses the domain adaptation method to migrate the behavior detection knowledge from the source domain dataset with sufficient labeled samples to the target domain dataset with insufficient labeled samples, thereby training a behavior detection model that matches the local data distribution, effectively overcoming the problem of insufficient labeled samples.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A student behavior detection method based on multi-scale statistical fusion domain adaptation, characterized in that: The following steps are involved: S1. Collect student behavior data at the school end, pre-process the collected data, and use behavior categories to annotate the data to build a local data set; S2, initialize the behavior detection model and the weights of each statistical loss function; S3, using the HICO-DET dataset as the source domain dataset and the local dataset as the target domain dataset, adopting the multi-scale statistical fusion domain adaptation method to pull the inter-domain features closer, updating the behavior detection model and the weights of each statistical loss function, and completing the knowledge transfer; S4, repeat step S3 until the algorithm converges and obtains a student behavior detection model adapted to local data; S5. Deploy the obtained student behavior detection model at the school end to detect and provide feedback on student behavior in real time.

2. According to claim 1, a method for detecting student behavior based on multi-scale statistical fusion domain adaptation is characterized in that: In step S1, the student behavior data includes the student's movements, demeanor, and expression image data, and the preprocessing includes scaling, denoising, and standardizing the image data.

3. The student behavior detection method based on multi-scale statistical fusion domain adaptation according to claim 1 is characterized in that: In step S2, the loss functions include the cross entropy loss function, the maximum mean difference loss function, the joint maximum mean difference loss function and the correlation alignment loss function, as follows: Where, L MMD is the maximum mean difference loss function, L JMMD is the joint maximum mean difference loss function, L CORAL is the related alignment loss function, L CE is the cross entropy loss function, B is the batch size, is the source domain sample, is the target domain sample, is the source domain feature, is the target domain feature, f k is the kth kernel function, M is the total number of kernel functions, d is the feature dimension, C src , C tgt are the covariance matrices of source domain features and target domain features, is the i-th sample in the local dataset, is the category label corresponding to the i-th sample, N is the number of behavior categories, 1 is the indicator function, is the output probability distribution vector The probability value of category c in .

4. The student behavior detection method based on multi-scale statistical fusion domain adaptation according to claim 3 is characterized in that: In step S3, the behavior detection model and the weights of each statistical loss function are updated based on the maximum mean difference loss function, the joint maximum mean difference loss function and the related alignment loss function, and the encoder and classifier in the behavior detection model are updated using the cross entropy loss function; The update process within a single batch is as follows: In the formula, E t ′、C t ′ are the updated encoder and classifier in a single batch, α t ′、β t ′、γ t ′ are the weights of the updated maximum mean difference loss function, the joint maximum mean difference loss function and the related alignment loss function, η is the learning rate, E t , C t are the encoder and classifier trained in the tth round in a single batch, respectively, t , β t , γ t are the weights of the maximum mean difference loss function, the joint maximum mean difference loss function, and the related alignment loss function in the t-th round of training within a single batch.

5. The method for detecting student behavior based on multi-scale statistical fusion domain adaptation according to claim 4 is characterized in that: This includes updating the model for a single batch of source domain samples and all batches of target domain samples, and repeating the operation until all batches of source domain samples are trained.

Images