Classroom scene cross-domain multi-label behavior identification method based on meta-learning prototype alignment
By using a meta-learning prototype alignment method, the domain offset and occlusion problems of multi-label video behavior recognition in classroom scenarios are solved, achieving cross-domain knowledge transfer and effective recognition of multi-label features, thus improving the behavior recognition performance in classroom scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-16
- Publication Date
- 2026-03-27
AI Technical Summary
Existing multi-label video behavior recognition methods suffer from domain offset issues, label space inconsistencies, insufficient processing of multi-label characteristics, and lack of explicit class semantic modeling in classroom scenarios, leading to a decline in model performance.
We employ a meta-learning prototype alignment method, which extracts features using a pre-trained VideoMAE model, repairs occluded regions using a meta-learning occlusion repair model, and aligns class prototypes using a graph convolutional network for multi-label classification.
It effectively solves the occlusion problem in classroom scenarios, realizes cross-domain knowledge transfer, and improves the performance and generalization ability of multi-label behavior recognition.
Smart Images
Figure CN121746786A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer vision and pattern recognition technology, and in particular to a method for cross-domain multi-label behavior recognition in classroom scenes based on meta-learning prototype alignment. Background Technology
[0002] With the development of smart education and video surveillance technology, the demand for automatic identification, statistics, and analysis of student and teacher behavior in classroom settings is increasing. For example, multi-label recognition of behaviors such as "sitting," "answering questions," "reading," and "raising hands" can help with teaching quality assessment, classroom discipline management, and learning status analysis.
[0003] Existing multi-label video action recognition methods are mostly based on deep convolutional neural networks or spatiotemporal Transformers, trained on large-scale public datasets (such as general everyday scene video datasets), and exhibit good performance in the same or similar scenes. However, when these models are directly transferred to classroom scenes, the performance of the models deteriorates significantly due to challenges such as subtle motion differences, dense object involvement, significant scale differences, different shooting angles, and visual occlusion in classroom datasets. The following problems exist:
[0004] 1. Severe domain offset problem: There are significant differences between the source domain (public dataset) and the target domain (classroom dataset) in terms of background, human pose, camera perspective, etc., resulting in inconsistent feature distribution.
[0005] 2. Inconsistent label space: The source domain usually contains dozens to hundreds of action categories, while the classroom scene only involves a dozen or so behaviors with educational semantics. Furthermore, there is no one-to-one correspondence between the source and target domain categories, making it difficult to directly apply traditional alignment methods based on isomorphic label spaces.
[0006] 3. Insufficient handling of multi-label features: In classroom scenarios, multiple behaviors often occur simultaneously, such as "sitting + taking notes" or "raising hands + speaking". Simple single-label transfer or one-to-many classification heads are difficult to model complex co-occurrence relationships.
[0007] 4. Traditional methods lack explicit class semantic modeling: Most transfer learning methods only align distributions at the sample feature level, without explicitly constructing interpretable "class prototypes" and utilizing them for cross-domain alignment, making it difficult to fully leverage the rich prior knowledge of the source domain.
[0008] Therefore, there is an urgent need for a recognition method that can perform cross-domain knowledge transfer in a heterogeneous label space, adapt to the multi-label characteristics of classroom scenarios, and has good generalization ability and interpretability. Summary of the Invention
[0009] This application provides a cross-domain multi-label behavior recognition method for classroom scenes based on meta-learning prototype alignment, aiming to solve the significant semantic differences between source and target domains and the occlusion problem in classroom scenes. The method includes:
[0010] Obtain source domain video samples and target domain video samples, and preprocess the source domain video samples and target domain video samples;
[0011] The pre-trained VideoMAE model is used to extract features from the pre-processed source domain video samples and target domain video samples respectively, to obtain the first source domain features and the first target domain features.
[0012] The occlusion repair model trained by the meta-learning model repairs the occlusion regions of the source domain video samples and the target domain video samples respectively, and obtains the repaired second source domain features and second target domain features.
[0013] The first source domain features and the second source domain features are weighted and fused to obtain the first fused feature; the first target domain features and the second target domain features are weighted and fused to obtain the second fused feature.
[0014] The source domain class prototype set is determined based on the first fusion feature, and the target domain class prototype set is determined based on the second fusion feature.
[0015] Align the source domain class prototype and the target domain class prototype using a graph convolutional network to obtain the aligned class prototype;
[0016] Multi-label classification is performed based on the similarity between the aligned class prototype and video features to obtain behavior recognition results.
[0017] Optionally, the occlusion repair model trained on the meta-learning model repairs the occluded regions of the source domain video samples and the target domain video samples respectively, to obtain repaired second source domain features and second target domain features, including:
[0018] The YOLO object detection model is used to detect source domain video samples and target domain video samples, identify occluded regions in the source domain video samples and target domain video samples, and annotate the identified occluded regions.
[0019] Generate corresponding task category labels for the occluded area based on the size of the occluded area;
[0020] In each task category, the K-nearest neighbor method is used to perform neighborhood learning restoration on the occluded region to obtain the restored features. The restored features are then weighted and fused with the features of other regions extracted by ResNet to obtain the final restored image feature representation.
[0021] Optionally, the training process of the meta-learning occlusion repair model includes:
[0022] Construct a support set and a query set, wherein the support set is used for inner loop training and the query set is used for outer loop training;
[0023] The meta-learning occlusion repair model is trained using support sets and query sets. The parameters of the meta-learning occlusion repair model are optimized based on the inner loop training loss and the outer loop training loss to obtain the trained meta-learning occlusion repair model.
[0024] Optionally, determining the source domain class prototype set based on the first fusion feature and determining the target domain class prototype set based on the second fusion feature includes:
[0025] The first fusion feature of the source domain video samples is used to determine the source domain class prototype set by clustering or feature mean calculation;
[0026] The second fusion feature of the target domain video samples is used to determine the target domain class prototype set by clustering or feature mean calculation.
[0027] Optionally, aligning the source domain class prototype and the target domain class prototype using a graph convolutional network includes:
[0028] Text features of the source and target domains are extracted using the CLIP text encoder;
[0029] By utilizing the textual features of the source and target domains, we construct the intra-domain similarity matrix, the intra-domain similarity matrix, and the cross-domain similarity matrix between the source and target domains.
[0030] The similarity matrix within the source domain, the similarity matrix within the target domain, and the cross-domain similarity matrix between the source and target domains are combined into a global adjacency matrix, and the adjacency matrix is normalized.
[0031] Graph convolutional networks use the normalized adjacency matrix to update and propagate the prototype feature matrix, satisfying the following formula:
[0032]
[0033] in, It is a non-linear activation function. The normalized adjacency matrix, For the first The prototype feature matrix of a layered graph convolutional network. For the corresponding learnable weight matrix, This represents the total number of layers in the graph convolutional network.
[0034] Optionally, the multi-label classification based on the aligned class prototype includes:
[0035] By calculating the cosine similarity between each video feature and the target domain class prototype, the predicted label value for each class is obtained.
[0036] The sigmoid function is used to perform binary classification of the labels for each class to obtain the multi-label prediction probability, and the class-wise binary cross-entropy is used as the multi-label supervision loss.
[0037] This application provides a cross-domain multi-label behavior recognition method for classroom scenes based on meta-learning prototype alignment. The method acquires source domain video samples and target domain video samples, and preprocesses them. A pre-trained VideoMAE model is used to extract features from the preprocessed source and target domain video samples, obtaining first source domain features and first target domain features. A trained meta-learning occlusion repair model is used to repair occluded regions in the source and target domain video samples, obtaining repaired second source domain features and second target domain features. The first and second source domain features are weighted and fused to obtain a first fused feature, and the first and second target domain features are weighted and fused to obtain a second fused feature. A source domain class prototype set is determined based on the first fused feature, and a target domain class prototype set is determined based on the second fused feature. A graph convolutional network is used to align the source and target domain class prototypes, obtaining aligned class prototypes. Multi-label classification is performed based on the similarity between the aligned class prototypes and video features to obtain the behavior recognition result. This method can effectively solve the occlusion problem in classroom scenarios and perform efficient cross-domain transfer between the source and target domains, thereby improving the performance of the target domain in multi-label behavior recognition tasks. Attached Figure Description
[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0039] Figure 1 A flowchart illustrating the cross-domain multi-label behavior recognition method for classroom scenes based on meta-learning prototype alignment provided in this application embodiment;
[0040] Figure 2 A schematic diagram illustrating the processing flow of the meta-learning occlusion repair model provided in this application embodiment;
[0041] Figure 3 This is a schematic diagram of a cross-domain multi-label behavior recognition model for classroom scenes based on meta-learning prototype alignment, provided in an embodiment of this application.
[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0044] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.
[0045] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0046] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0047] Figure 1 This is a flowchart illustrating the cross-domain multi-label behavior recognition method for classroom scenes based on meta-learning prototype alignment, provided in an embodiment of this application. Figure 1 As shown in this embodiment, the cross-domain multi-label behavior recognition method for classroom scenes based on meta-learning prototype alignment includes:
[0048] S1: Obtain source domain video samples and target domain video samples, and preprocess the source domain video samples and target domain video samples.
[0049] Specifically, data augmentation processing is performed on the acquired source domain video samples and target domain video samples. Data augmentation includes operations such as rotation, scaling, cropping, and flipping.
[0050] S2: Use the pre-trained VideoMAE model to extract features from the pre-processed source domain video samples and target domain video samples respectively, to obtain the first source domain features and the first target domain features.
[0051] Specifically, the VideoMAE model pre-trained on the Kinetics-400 dataset is used to extract the spatiotemporal features of source domain video samples and target domain video samples.
[0052] S3: The occlusion regions of the source domain video samples and the target domain video samples are repaired by a pre-trained meta-learning occlusion repair model, respectively, to obtain the repaired second source domain features and second target domain features.
[0053] Figure 2 This is a schematic diagram of the processing flow of the meta-learning occlusion repair model provided in this application embodiment. Specifically, the pre-trained meta-learning occlusion repair model repairs the occluded regions of the source domain video samples and the target domain video samples respectively, obtaining the repaired second source domain features and second target domain features, including the following steps:
[0054] S31: The YOLO object detection model is used to detect source domain video samples and target domain video samples, identify occluded regions in the source domain video samples and target domain video samples, and label the identified occluded regions.
[0055] Specifically, a trained YOLO object detection model is used to detect and label occluded regions in video frames. The occlusion labels generated by the YOLO model include the object category and bounding box.
[0056] S32: Generate corresponding task tags for the occluded area based on the size of the occluded area.
[0057] For example, the occluded areas in the video are classified into small-scale occlusion, medium-scale occlusion, large-scale occlusion, and complete occlusion, and an independent task is generated for each task category.
[0058] Specifically, occlusion is categorized as follows: Small-area occlusion: The occluded area occupies less than 10% of the total area. Medium-area occlusion: The occupied area occupies 10%-30% of the total area. Large-area occlusion: The occupied area occupies 30%-60% of the total area. Complete occlusion: The occupied area occupies more than 60% of the total area.
[0059] S33: In each task category, the K-nearest neighbor method is used to perform neighborhood learning repair on the occluded region to obtain the repaired features. The repaired features are then weighted and fused with the features of other regions extracted by ResNet to obtain the final repaired image feature representation.
[0060] Understandably, the first step is to use ResNet to extract features from video frames, obtaining a deep feature representation for each region. ResNet is a pre-trained network architecture for image feature extraction, capable of effectively extracting both global and local features. Then, in the occluded region, the K-nearest neighbor method is used to find similar features in the unoccluded regions, and these similar features are used to infer the features of the occluded region. The restored features are consistent with the features of the surrounding areas, ensuring a natural restoration effect.
[0061] For example, suppose there is a video frame that needs to be repaired. and a set of training samples with known features and its corresponding tags The goal of the K-nearest neighbor method is to find the nearest neighbors to the region to be repaired. Find the K nearest neighbors. The steps are as follows:
[0062] Calculate the sample to be repaired. With each sample in the training set The distance between them. Euclidean distance is typically used to measure distance.
[0063]
[0064] in, Area to be repaired In the Features in each dimension Training set samples In the Features in each dimension It is the dimension of the feature space.
[0065] Selecting the K nearest neighbors: After calculating the distances between all training samples and the sample to be repaired, select the K neighbors with the smallest distances.
[0066]
[0067] in, express The K nearest neighbors.
[0068] Weighted Inpainting: Once K neighbors are selected, the features of these neighbors can be used to inpaint the occluded area. A weighted average is typically used for inpainting, with the weights adjusted based on distance. Closer neighbors contribute more to the inpainted features.
[0069]
[0070] in, This refers to the weight of neighbors, typically using inverse distance weighting, where closer neighbors have higher weights. It is the first The feature values of each neighbor represent the label or feature of that neighbor. These are the predicted values for the area to be repaired. The features are obtained after weighted averaging. The repaired occluded area features were obtained. These features are consistent with the features of the surrounding unoccluded areas, thus completing the repair of the occluded area.
[0071] In an optional embodiment, the training process of the meta-learning occlusion repair model includes: constructing a support set and a query set, wherein the support set is used for inner loop training and the query set is used for outer loop training; training the meta-learning occlusion repair model using the support set and the query set; optimizing the parameters of the meta-learning occlusion repair model based on the inner loop training loss and the outer loop training loss to obtain the trained meta-learning occlusion repair model.
[0072] Understandably, in step S32, the corresponding task labels generated for the occluded area based on the size of the occluded area can be used to train the meta-learning occlusion repair model. The task labels are randomly divided into a support set and a query set, and each task (e.g., small-scale occlusion, large-scale occlusion, etc.) is trained as an independent task.
[0073] Specifically, the goal of the inner loop is to enable the model to be quickly fine-tuned to adapt to tasks with a support set. For each task (e.g., small-area occlusion, medium-area occlusion, etc.), the model is trained with a small amount of labeled data (support set) to quickly adjust the model's parameters. The key to the inner loop is to optimize the loss function for each task, allowing the model to be fine-tuned for each task. The training loss formula is:
[0074]
[0075]
[0076] in, These are the parameters of the model. It's the learning rate. In the inner loop, L2 loss is used to measure the difference between the repaired video frame and the original video frame (with the occluded area). , It refers to the repaired image or features (the image with the occluded area repaired). It is the original image or feature (image with occluded areas), and T is the total number of video frames. It supports sets containing a small amount of labeled data. It is achieved by supporting the fine-tuned model parameters.
[0077] The outer loop evaluates the model's generalization ability after fine-tuning on the support set using the query set. The outer loop helps optimize the model's initialization parameters, enabling the model to quickly adapt and perform occlusion repair when encountering new tasks. The training loss formula is:
[0078]
[0079] in, It is the meta-learning loss of the outer loop, representing the model's loss on the query set. These are the model parameters after fine-tuning the inner loop. It is a query set containing a small amount of unlabeled data to evaluate the model's adaptability to new tasks.
[0080] For samples in the query set, calculate the repair loss and similarity between the model's predictions and the true labels, and evaluate the model's repair performance and cross-domain adaptability, satisfying the following formula:
[0081]
[0082] The final total loss function satisfies the following formula:
[0083]
[0084] in, For internal circulation losses, For external circulation losses, Its weighting coefficient.
[0085] S4: Perform weighted fusion of the first source domain features and the second source domain features to obtain the first fused feature; perform weighted fusion of the first target domain features and the second target domain features to obtain the second fused feature.
[0086] Specifically, a weighted average method is used to fuse the restored occluded region features with the spatiotemporal features extracted by VideoMAE, satisfying the following formula:
[0087]
[0088] in, It is the fusion feature of frame t. These are the weighting coefficients for the t-th frame, used to control the fusion ratio of spatiotemporal features and repaired features. It is the spatiotemporal feature of frame t. The features of the repaired frame t.
[0089] S5: Determine the source domain class prototype set based on the first fusion feature, and determine the target domain class prototype set based on the second fusion feature.
[0090] Specifically, the first fusion feature of the source domain video samples is used to determine the source domain class prototype set through clustering or feature mean calculation; the second fusion feature of the target domain video samples is used to determine the target domain class prototype set through clustering or feature mean calculation.
[0091] For example, based on samples of each class in the source domain dataset, the representation of the source domain class prototype is obtained by calculating the prototype of each class. The calculation formula is as follows:
[0092]
[0093] in, This represents the set of samples whose source domain belongs to the corresponding category. For the sample size, It represents the video features of each sample in the source domain.
[0094] Similarly, the prototype of the target domain class is calculated using samples from the target domain dataset, resulting in a representation of the target domain class prototype. The calculation formula is as follows:
[0095]
[0096] in, This represents the set of samples belonging to the corresponding category within the target domain. For the sample size, It represents the video features of each sample in the target domain.
[0097] S6: Align the source domain class prototype and the target domain class prototype using a graph convolutional network to obtain the aligned class prototype.
[0098] Specifically, text features of the source and target domains are extracted using the CLIP text encoder; similarity matrices within the source and target domains, as well as cross-domain similarity matrices between the source and target domains, are constructed using these text features; the similarity matrices within the source and target domains, as well as the cross-domain similarity matrices between the source and target domains, are combined into an overall adjacency matrix, and the adjacency matrix is normalized; the graph convolutional network uses the normalized adjacency matrix to update and propagate the prototype feature matrix.
[0099] For example, the text features of the source and target domains are first extracted using the CLIP text encoder. Each category label is encoded using the CLIP model to obtain a text feature representation. Source domain label text features: denoted as... Indicates source domain category Corresponding text features. Target domain label text features: denoted as... Indicates source domain category The corresponding text features.
[0100] Furthermore, three different similarity matrices are constructed using text features from the source and target domains: a similarity matrix within the source domain and a similarity matrix within the target domain. The similarity matrix between the source domain and the target domain and the similarity matrix within the target domain .
[0101]
[0102] in, This represents the cosine similarity. Similarly:
[0103]
[0104] For any source domain category With target domain category The cross-domain similarity between the visual prototype and the text prototype is calculated, and the edge weights are obtained through attention units:
[0105]
[0106] in, It is a scoring function composed of a multi-layer feedforward network and an attention mechanism, used to learn the correlation strength between different source-target categories;
[0107] Furthermore, the source domain, target domain, and cross-domain adjacency matrices are combined into a global adjacency matrix according to a block structure:
[0108]
[0109] Add an identity matrix on the diagonal. To preserve node self-connections, the degree matrix is then calculated:
[0110]
[0111] in, Symmetric normalization of the adjacency matrix yields:
[0112]
[0113] Furthermore, an L-layer graph convolutional network is used to update and propagate the overall prototype matrix, with each layer updating in the following manner:
[0114]
[0115] in, For the first The prototype feature matrix of the layer, For the corresponding learnable weight matrix, For non-linear activation functions, preferably, the number of updated layers is... Two to three layers are chosen to balance expressive power and computational efficiency.
[0116] Understandably, aligning the class prototypes of the source and target domains using graph convolutional networks can reduce the differences between them, thereby enhancing the performance of the target domain. This cross-domain prototype alignment strategy improves the adaptability of the target domain when faced with differences in labels between the source and target domains.
[0117] S7: Perform multi-label classification based on the similarity between the aligned class prototype and video features to obtain behavior recognition results.
[0118] Specifically, the predicted label values for each class are obtained by calculating the cosine similarity between each video feature and the target domain class prototype; the label of each class is binary classified using the sigmoid function to obtain the multi-label prediction probability; and the class-wise binary cross-entropy is used as the multi-label supervision loss.
[0119] Understandably, adopting a multi-label classification mechanism improves the flexibility and accuracy of classification by accurately identifying multiple behavioral categories in videos.
[0120] In an optional embodiment, a cross-domain multi-label behavior recognition model for classroom scenarios based on meta-learning prototype alignment is also provided, such as... Figure 3 As shown, this method is used to implement the aforementioned cross-domain multi-label behavior recognition method for classroom scenes based on meta-learning prototype alignment. The method further includes: designing a comprehensive loss function, comprising: 1) a class prototype alignment loss to optimize the alignment of class prototypes between the source and target domains; 2) a multi-label classification loss to perform multi-label classification through similarity calculation; and 3) a comprehensive loss function that weightedly fuses the class prototype alignment loss and the multi-label classification loss to optimize model performance.
[0121] Understandably, the goal of class prototype alignment loss is to optimize the class prototypes of the source and target domains so that they are aligned in the same feature space, thereby reducing the class differences between the source and target domains. This loss function helps the model learn common feature representations between the source and target domains, enabling effective task transfer on target domain data. The calculation formula is as follows:
[0122]
[0123] in, It is the prototype of the source domain class. It is the prototype of the target domain class. The alignment loss function aims to optimize the class prototypes of the source and target domains through graph convolutional networks or other methods, aligning them in the same feature space. By minimizing this loss, the model can learn the class relationships between the source and target domains and provide appropriate class prototype representations for the target domain task.
[0124] Understandably, multi-label classification is the core objective of multi-label behavior recognition. It involves calculating the similarity between the features of each video frame and the class prototype to perform multi-label classification. Each video sample may correspond to multiple labels, requiring the output of a probability for each label.
[0125] Optimization is achieved using binary cross-entropy loss, with sigmoid activation applied independently to each category to obtain multi-label prediction probabilities.
[0126] , ,
[0127] in, For the Sigmoid function, For the first The source domain sample pairs of the first The predicted probability of each source domain category. For its corresponding logit score, For the first The target area sample pairs the first The predicted probability of each target domain category. For its corresponding logit score,
[0128] Then, class-wise binary cross-entropy is used as the multi-label supervision loss: for the source domain, let the true multi-label be... ,but
[0129] ;
[0130] Among them, B S N represents the number of samples in a mini-batch of the source domain; S The number of label categories for the source domain samples;
[0131] For the target domain, let the real multiple tags be... ,but
[0132]
[0133] Among them, B t N represents the number of samples in a mini-batch of the target domain. t The number of label categories for the target domain samples;
[0134] The overall loss function is a weighted sum of multiple losses, satisfying the following formula:
[0135]
[0136] in, These are the weighting coefficients for the alignment loss.
[0137] This application provides a cross-domain multi-label behavior recognition method for classroom scenes based on meta-learning prototype alignment. The method acquires source domain video samples and target domain video samples, and preprocesses them. A pre-trained VideoMAE model is used to extract features from the preprocessed source and target domain video samples, obtaining first source domain features and first target domain features. A trained meta-learning occlusion repair model is used to repair occluded regions in the source and target domain video samples, obtaining repaired second source domain features and second target domain features. The first and second source domain features are weighted and fused to obtain a first fused feature; the first and second target domain features are also weighted and fused to obtain a second fused feature. A source domain class prototype set is determined based on the first fused feature, and a target domain class prototype set is determined based on the second fused feature. A graph convolutional network is used to align the source and target domain class prototypes, obtaining aligned class prototypes. Multi-label classification is performed based on the similarity between the aligned class prototypes and video features to obtain the behavior recognition result. This method can effectively solve the occlusion problem in classroom scenarios and perform efficient cross-domain transfer between the source and target domains, thereby improving the performance of the target domain in multi-label behavior recognition tasks.
[0138] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0139] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A method for cross-domain multi-label behavior recognition in classroom scenes based on meta-learning prototype alignment, characterized in that, The method includes: Obtain source domain video samples and target domain video samples, and preprocess the source domain video samples and target domain video samples; The pre-trained VideoMAE model is used to extract features from the pre-processed source domain video samples and target domain video samples respectively, to obtain the first source domain features and the first target domain features. The occlusion repair model trained by the meta-learning model repairs the occlusion regions of the source domain video samples and the target domain video samples respectively, and obtains the repaired second source domain features and second target domain features. The first source domain features and the second source domain features are weighted and fused to obtain the first fused feature; the first target domain features and the second target domain features are weighted and fused to obtain the second fused feature. The source domain class prototype set is determined based on the first fusion feature, and the target domain class prototype set is determined based on the second fusion feature. Align the source domain class prototype and the target domain class prototype using a graph convolutional network to obtain the aligned class prototype; Multi-label classification is performed based on the similarity between the aligned class prototype and video features to obtain behavior recognition results.
2. The method according to claim 1, characterized in that, The trained meta-learning occlusion repair model repairs the occluded regions of the source domain video samples and the target domain video samples, respectively, to obtain repaired second source domain features and second target domain features, including: The YOLO object detection model is used to detect source domain video samples and target domain video samples, identify occluded regions in the source domain video samples and target domain video samples, and annotate the identified occluded regions. Generate corresponding task category labels for the occluded area based on the size of the occluded area; In each task category, the K-nearest neighbor method is used to perform neighborhood learning restoration on the occluded region to obtain the restored features. The restored features are then weighted and fused with the features of other regions extracted by ResNet to obtain the final restored image feature representation.
3. The method according to claim 1, characterized in that, The training process of the meta-learning occlusion repair model includes: Construct a support set and a query set, wherein the support set is used for inner loop training and the query set is used for outer loop training; The meta-learning occlusion repair model is trained using support sets and query sets. The parameters of the meta-learning occlusion repair model are optimized based on the inner loop training loss and the outer loop training loss to obtain the trained meta-learning occlusion repair model.
4. The method according to claim 1, characterized in that, The step of determining the source domain class prototype set based on the first fusion feature and the target domain class prototype set based on the second fusion feature includes: The first fusion feature of the source domain video samples is used to determine the source domain class prototype set by clustering or feature mean calculation; The second fusion feature of the target domain video samples is used to determine the target domain class prototype set by clustering or feature mean calculation.
5. The method according to claim 1, characterized in that, The alignment of the source domain class prototype and the target domain class prototype using a graph convolutional network includes: Text features of the source and target domains are extracted using the CLIP text encoder; By utilizing the textual features of the source and target domains, we construct the intra-domain similarity matrix, the intra-domain similarity matrix, and the cross-domain similarity matrix between the source and target domains. The similarity matrix within the source domain, the similarity matrix within the target domain, and the cross-domain similarity matrix between the source and target domains are combined into a global adjacency matrix, and the adjacency matrix is normalized. Graph convolutional networks use the normalized adjacency matrix to update and propagate the prototype feature matrix, satisfying the following formula: ； in, It is a non-linear activation function. The normalized adjacency matrix, For the first The prototype feature matrix of a layered graph convolutional network. For the corresponding learnable weight matrix, This represents the total number of layers in the graph convolutional network.
6. The method according to claim 1, characterized in that, The multi-label classification based on the aligned class prototype includes: By calculating the cosine similarity between each video feature and the target domain class prototype, the predicted label value for each class is obtained. The sigmoid function is used to perform binary classification of the labels for each class to obtain the multi-label prediction probability, and the class-wise binary cross-entropy is used as the multi-label supervision loss.