Classroom zero-shot combined action recognition method based on double attribute semantic cross-alignment

Abstract

This invention discloses a zero-shot combined action recognition method for classrooms based on dual-attribute semantic cross-alignment, implemented according to the following steps: Step 1, constructing an offline enhanced noun prototype library; Step 2, preprocessing the classroom video dataset; Step 3, constructing a recognition model, using the preprocessed classroom video dataset and the offline enhanced noun prototype library as inputs to train the recognition model, obtaining a trained recognition model; Step 4, using the trained recognition model for action recognition. This invention solves the problem in existing technologies where the semantic representation of entity nouns is singular, and a single label vector cannot represent the visual feature differences brought about by changes in object shape, directly leading to the failure of zero-shot action recognition for deformation-related actions.

Description

Technical Field

[0001] This invention belongs to the field of classroom action recognition technology, and relates to a zero-sample combined action recognition method for classroom based on dual-attribute semantic cross-alignment. Background Technology

[0002] Human behavior in classroom settings exhibits typical combinatorial characteristics, with the vast majority of actions consisting of an action verb followed by a noun, such as "flipping through a textbook, picking up chalk, writing on the blackboard, folding a workbook, tearing paper, and wiping the blackboard." The number of these action combinations is enormous, and the teaching environment is highly open: as teaching tools and interactive methods are constantly updated, entirely new verb-noun combinations never before seen in the training dataset will continue to emerge. Traditional closed-set action recognition models can only recognize action categories labeled in the training set and cannot adapt to such open scenarios. Zero-shot combinatorial action recognition has become a core technical challenge in classroom visual analysis.

[0003] Current zero-sample combined action recognition technologies still have several key shortcomings that fail to meet the needs of practical classroom implementation. These shortcomings can be summarized as follows: First, the semantic representation of entity nouns is too simplistic. Existing methods rely solely on noun text tags for word embedding encoding of classroom items such as textbooks and chalk, retaining only category identification information and completely ignoring static visual attributes such as shape, material, size, and physical structure, as well as dynamic functional attributes such as readability, tearability, writability, and bendability. Classroom items undergo shape changes during use; for example, a flat textbook may become folded, or a piece of chalk may break. A single tag vector cannot represent the visual feature differences caused by these shape changes, directly leading to the model's failure to recognize zero-sample actions involving deformation.

[0004] Second, the visual and semantic feature matching methods are coarse, lacking fine-grained alignment capabilities. Mainstream solutions generally use global features of the entire image to match semantic prototypes of nouns, without one-to-one association of local image regions and detailed semantic attributes. Key features of classroom actions are often concentrated in small areas such as hands and object parts. Global features dilute effective local information, and when there are changes in lighting, shooting angle shifts, or partial occlusion, the model features drift severely, and the generalization ability drops significantly.

[0005] Third, the computational overhead during the inference phase is high, making it difficult to adapt to edge devices. Some improvement solutions choose to call large language models in real time during the inference phase to generate object attributes. While this enriches semantic information, deploying large models requires high-memory and high-computing-power hardware. However, devices such as smart classroom cameras, embedded edge terminals, and low-cost interactive whiteboards have limited computing resources and cannot support large models. This means that these technologies can only run on cloud servers and cannot be deployed locally at the front end.

[0006] Fourth, multimodal features lack unified constraints. Most existing visual-semantic fusion schemes only perform feature concatenation without designing a dedicated loss function to constrain the spatial distribution of the two types of features. This results in visual features and semantic features being independent of each other in the feature space with weak correlation. The model cannot effectively utilize semantic priors to assist visual recognition, further reducing the recognition reliability in zero-shot scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide a zero-sample combined action recognition method for classrooms based on dual-attribute semantic cross-alignment. This method solves the problem that existing technologies have a single semantic representation form for entity nouns, and a single label vector cannot represent the visual feature differences brought about by changes in object shape, which directly leads to the failure of zero-sample action recognition for deformation-type objects.

[0008] The technical solution adopted in this invention is a zero-shot combined action recognition method for classrooms based on dual-attribute semantic cross-alignment, which is implemented according to the following steps: Step 1: Build an offline enhanced noun prototype library; Step 2: Preprocessing the classroom video dataset; Step 3: Build a recognition model. Use the preprocessed classroom video dataset and the offline enhanced noun prototype library as input to train the recognition model and obtain a trained recognition model.

[0009] Step 4: Use the trained recognition model to perform action recognition.

[0010] Preferably, step 1 specifically includes: Step 1.1: Construct a standardized noun set and design static and dynamic prompt words for each noun in the standardized noun set; Step 1.2: Concatenate the static and dynamic prompt words corresponding to each noun as input to the pre-trained large language model to obtain the static and dynamic attribute text corresponding to the noun; Step 1.3: Use a pre-trained text encoder to vectorize each noun in the standardized noun set and its corresponding static and dynamic attribute text, obtaining noun label vectors respectively. Static attribute vectors and dynamic attribute vectors; Step 1.4: Perform K-means clustering on the static attribute vector and dynamic attribute vector respectively to cluster them into multiple static attribute clusters and dynamic attribute clusters. For the attribute vector in each cluster, calculate its cosine similarity with the noun label vector. Select the cluster representative attribute in each static attribute cluster and the cluster representative attribute in each dynamic attribute cluster based on the cosine similarity to form a refined noun dual attribute set. Step 1.5: Obtain the noun attribute mean vector based on the noun refinement dual attribute set; Step 1.6: Construct an offline enhanced noun prototype library based on the attribute mean vectors of all nouns.

[0011] Preferably, step 1.1 specifically includes: Summarize all target items in classroom teaching scenarios and construct a standardized terminology set. , The total number of nouns corresponding to the name of the target item; For each noun in the standardized noun set , i = 1-m Static visual cues and dynamic functional cues were designed separately. The static visual cues were "descriptive nouns". List the static visual characteristics of shape, material, size, and physical structure one by one; the dynamic functional prompts are "descriptions of human-readable nouns". List the interactive actions performed and the functional attributes of the object one by one; The prompt words are formatted and ambiguous sentences are removed to obtain a set of standardized nouns. And the corresponding set of static visual cue words and set of dynamic functional cue words; Step 1.2 specifically involves: The process involves concatenating a single noun with all its corresponding static cue words and inputting the result into a pre-trained large language model. The output is the static attribute text corresponding to that noun. The process involves concatenating a single noun with all its corresponding dynamic prompts and inputting the concatenation into a pre-trained large language model. The output is the dynamic attribute text corresponding to that noun. ; Then iterate through the set of standardized nouns. All nouns in the text are collected, and all static attribute text is used to construct a global static primitive attribute pool. All dynamic attribute text is collected to construct a global dynamic primitive attribute pool. The global static primitive attribute pool and the global dynamic primitive attribute pool are then cleaned to remove duplicate text and invalid descriptions that are irrelevant to static visuals and dynamic functions.

[0012] Preferably, step 1.3 specifically includes: A pre-trained text encoder is used to vectorize each noun in the standardized noun set and its corresponding static and dynamic attribute text, resulting in noun label vectors. Static attribute vectors and dynamic attribute vectors: (1) in, It is the BERT word segmentation function, used to convert natural text into recognizable token sequences; Indicates the extraction encoder The output vector of the position. For text encoders; Similarly, according to formula (1), the static attribute text and dynamic attribute text As input, static attribute vectors are obtained respectively. Dynamic attribute vectors ; Step 1.4 specifically involves: Perform K-means clustering on m static attribute vectors and m dynamic attribute vectors respectively, and cluster the m static attribute vectors into K static attribute clusters. Cluster m dynamic attribute vectors into K dynamic attribute clusters. ; For K static attribute clusters: For any static attribute cluster For all static attribute vectors within the vector, calculate the relationship between each static attribute vector and its corresponding noun tag vector. Cosine similarity: (2) in, Noun label vector The cosine similarity between the vector and the corresponding static attribute vector; Based on cosine similarity from static attribute clusters The three static attribute vectors with the highest cosine similarity are selected as the static attribute cluster. The cluster represents a static attribute vector; For K dynamic attribute clusters, the same formula (2) is used to calculate any dynamic attribute cluster. Each dynamic attribute vector and its corresponding noun label vector The cosine similarity is used to determine the dynamic attribute cluster. The three dynamic attribute vectors with the highest cosine similarity are selected as the dynamic attribute cluster. The cluster represents a dynamic attribute vector; The selected cluster-representative attribute vectors are merged to form a set, resulting in a refined noun dual-attribute set. : (3) in, This indicates the filtering results for static attribute clusters. This indicates the filtering results for dynamic attribute clusters. noun A refined set of dual attributes, containing M attribute vectors; Preferably, step 1.5 specifically includes: For nouns The arithmetic mean of all attribute vectors in the refined dual-attribute set is used to obtain the term. Corresponding attribute mean vector: (4) in, To refine a single attribute vector in a dual-attribute set; Step 1.6 specifically involves: noun The noun label vector and the attribute mean vector are concatenated along the channel dimension to obtain the fused vector: (5) In the formula, This involves concatenating operations along the channel dimension. The dimension is the sum of the dimensions of the noun label vector and the attribute mean vector; The concatenated fusion vector is input into a two-layer fully connected fusion network (MLP) for feature dimensionality reduction and fusion to generate the final enhanced noun prototype. : (6) In the formula, It is the first fully connected layer; It is a non-linear activation function. It is the second fully connected layer; The dimensions are consistent with the original noun label vector. Iterate through all nouns to generate a complete set of augmented noun prototypes. The data is stored offline to the local hard drive as a .pth file, resulting in an offline enhanced noun prototype library.

[0013] Preferably, step 2 specifically includes: Step 2.1: Divide the public classroom RGB video dataset into a training set and a validation set according to a certain ratio; Step 2.2: Divide the original videos in the training and validation sets into continuous video frame sequences according to the frame rate. ,in The number of frames a single video is divided into; Step 2.3: Perform random data augmentation on the original videos in the training set. Specifically, randomly flip and crop the video frames of the original videos to form a new video frame sequence. Merge the new video frame sequence into the training set and normalize the video frame sequence in the training set.

[0014] Preferably, the specific process of the recognition model processing the input data in step 3 is as follows: Step 3.1: Input the video frame sequence from the training set into the pre-trained video encoder to obtain the corresponding spatiotemporal feature tensor. ; Step 3.2, convert the spatiotemporal feature tensor Perform global average pooling (T-pooling) along the temporal dimension to obtain pure spatial feature maps. ; Step 3.3, convert the pure spatial feature map Divide the image into N independent blocks, expand the features of all blocks, and concatenate them to generate a multi-head cross-attention query matrix. ; Step 3.4: Project the refined attribute vector set along the feature dimension to generate the attention key matrix K and the value matrix V; Step 3.5, query the multi-head cross-attention matrix Multi-head scaling dot product cross-attention computation is performed on the attention key matrix K, value matrix V, and refined attribute vector set to achieve cross-modal alignment and obtain multi-head attention fusion features. ; Step 3.6: Fuse features with multi-head attention Multi-head cross-attention query matrix After performing residual connections, global average pooling is then applied to obtain the visual features of the nouns. ; Step 3.7, based on the visual characteristics of nouns Calculate noun prediction scores using an offline enhanced noun prototype library; Step 3.8, calculate the prediction score for combined actions; Step 3.9, using visual features of nouns Attribute mean vector The combined action prediction score and the real verb and noun combination labels corresponding to the training set samples are used to construct a multi-combination loss function. Training parameters are set, and training is performed to obtain the trained recognition model.

[0015] Preferably, in step 3.1, the video encoder uses a pre-trained Video Swin Transformer encoder to extract the spatiotemporal fusion features of the video. The encoder is configured as follows: 7×7 window size, 4 layers of Transformer blocks, inputting a single batch of video frame sequences, and outputting a four-dimensional spatiotemporal feature tensor. (7) in, For training batch size, The length of the video frame sequence. These are the height and width of a single video frame image, respectively. The number of feature channels, i.e., the hidden layer dimension of the video encoder. For VideoSwin Transformer encoder; Pure spatial feature map in step 3.2 Specifically: (8) In the formula, This is a temporal average pooling operation that takes the average value of features along the temporal dimension; Step 3.3 specifically involves: Pure spatial feature map The image is divided into N independent image blocks using a grid partitioning function, with each block's features flattened and then concatenated to generate a multi-head cross-attention query matrix. As shown in formula (9): (9) in, The grid partitioning function divides the spatial feature map into N=16 equal blocks using a 4×4 grid. For flattening functions, Features encompassing all N image blocks; Step 3.4 specifically involves: A dimensional projection network module is constructed, consisting of two fully connected Linear layers and an intermediate ReLU nonlinear activation function. Based on the noun category labeled in the current input video sample, the refined attribute vector set of the corresponding noun is retrieved from the global refined attribute vector library generated by offline preprocessing. As input to the dimensional projection network module; Dimensional projection network module for refining attribute vector sets Each attribute vector within the matrix undergoes an independent linear dimension mapping, and the output features are used as the key matrix for cross-attention. AND-value matrix : (10) In the formula, A set of refined attribute vectors corresponding to a single noun, containing Each independent attribute vector; Dimensional projection networks with shared weights, key matrix AND-value matrix Both are generated by mapping the same set of attributes using the same dimension projection network module, and their dimensions are consistent. It is the first fully connected layer. It is a non-linear activation function. It is the second fully connected layer. The total number of attribute vectors in a single noun's refined attribute set; Step 3.5 specifically involves: Multi-head cross-attention query matrix The attention key matrix K and value matrix V are dimensionally reshaped, splitting the feature dimensions into multi-head dimensions to obtain the reshaped query features. Reshaped bond features Reshaped value characteristics : (11) (12) (13) in, Operations for reshaping tensor dimensions To ensure the training batch size is consistent with the batch dimension output by the video encoder; For the number of attention heads, The total number of image blocks, For single attention head feature dimensions, The number of feature channels; Scaling is applied to the reshaped query features: (14) In the formula, The reshaped query features output by formula (11); For single attention head feature dimensions, The scaling factor represents the scaled query features. Dimensions and Maintain consistency; For pure spatial feature maps any image block in Calculate image blocks n The semantic relevance with all attribute features is used to obtain the corresponding attention weights: (15) In the formula, This is the index of the refined attribute, with a value range of 1. M,; For the first The scaled query features of each image block are extracted from the overall query features of formula (14) by block index. For the first The reshaped key features of the refined attribute are obtained by formulas (10) and (12) for the first key feature. Each individual attribute vector is independently projected and then reshaped in dimension; This is an exponential operation used to amplify the weights of highly correlated features; As a normalization term, it ensures that the sum of the attention weights of all individual attribute vectors is 1; For the first The image block and the first Attention weights between individual attribute vectors represent the strength of their semantic association. Using the calculated attention weights, the value matrix is ​​weighted and summed to obtain the attribute enhancement features of each image block, as shown in formula (15): (16) In the formula, The attention weights are calculated using formula (15); For the first The reshaped value characteristics of a refined attribute; For the first The attention-weighted fusion feature of each image block integrates the association information between the block and all attributes; Fusion features of all image blocks By splicing the features together, a complete attention fusion feature is obtained. ; The attention fusion features of each attention head are concatenated to restore the original feature dimensions, as shown in formula (17): (17) In the formula, To reshape the tensor dimension operation, the multi-head dimension and feature dimension are merged and restored to the format of batch × number of blocks × feature dimension; The number of feature channels should be consistent with the feature dimension of the original image block. This represents the multi-head attention fusion feature after splicing; Step 3.6 specifically involves: Multi-head attention fusion feature Multi-head cross-attention query matrix Perform residual connections to obtain the fused features after residual connections: (18) In the formula, The multi-head attention fusion feature output by equation (17); The multi-head cross-attention query matrix output by equation (9) ; The fusion features are those resulting from residual connections; Global average pooling is performed along the block dimension on the fused features after residual concatenation to compress the spatial dimension and obtain the final noun visual features. For example, in formula (19): (19) In the formula, For spatial average pooling, the feature is averaged along the image block dimension; This represents the total number of image blocks. The visual features of nouns for a single batch of samples are represented by the batch × feature dimension. Preferably, step 3.7 specifically includes: Based on the visual characteristics of nouns The corresponding noun prototypes in the offline enhanced noun prototype library are used to calculate the cosine similarity between the visual features and each enhanced noun prototype, which is used as the noun prediction score, as shown in formula (20): (20) In the formula The visual features of the nouns obtained in step 3.6 are the corresponding video samples; To enhance the noun prototypes extracted from the noun prototype set; Cosine similarity is used to measure the semantic association strength between visual features and prototype features. Predicting scores for nouns; Step 3.8 specifically involves: The final combined action prediction score is obtained by weighting the scores, as shown in formula (21): (twenty one) In the formula, This is the prediction score for the verb branch. The verb and noun branches share the same input and video encoder backbone. The input is a standardized sequence of RGB video frames from the classroom. The video frames are uniformly fed into the video encoder to extract the global spatiotemporal feature tensor. The verb branch directly reuses the original global spatiotemporal feature tensor output by the encoder, without performing temporal pooling, spatial grid partitioning, or cross-attention alignment operations. The original spatiotemporal features are input into an independent fully connected classification head to calculate the verb category prediction score. ; The predicted score for the noun branch; As the score weight; Step 3.9 Multiple combination loss function for: (20) in, Weights for cross-modal contrast loss; Weights for the action classification loss; For cross-modal contrast loss: (twenty one) in, This refers to the number of samples in a single batch. For the first The visual features of nouns obtained from each video sample after steps 3.1-3.6; For the first The attribute mean vector of each video sample corresponding to a noun; L2 loss is used to measure the distance between visual features and attribute features; Loss for classifying actions: (twenty two) The predicted score vector for combined actions is obtained by fusing the verb branch score and the noun branch score; The sample contains real verb-noun combination labels; The total loss for network training is used for backpropagation to update network parameters. During training, video frame sequences from the training set are sequentially input into the recognition model. The initial learning rate is set to 1e-4, and the weight decay is set to 1e-5. A cosine annealing learning rate strategy is adopted, and the learning rate is adjusted after each iteration. Backpropagation updates all network parameters of the video encoder, dimensionality projection MLP, and multi-head cross-attention. The parameters of the text encoder and augmented noun prototypes are frozen. After each iteration, the recognition accuracy of zero-sample combined actions is tested on the validation set. When the validation accuracy no longer improves for 5 consecutive iterations, an early stopping strategy is triggered, and training is stopped. The trained network weight file is saved, and the complete trained network weight file is finally output.

[0016] Preferably, step 4 specifically involves: dividing the collected test video into a video frame sequence according to the method in step 2.2, normalizing the video frame sequence, and then inputting it into the trained recognition model to obtain the combined action prediction score. The verb-noun combination corresponding to the maximum score is selected as the final recognition result, and the zero-sample combined action judgment of verb plus noun is completed. Finally, the final recognition result of classroom combined actions is output.

[0017] The beneficial effects of this invention are: (i) Enhanced semantic attributes in two dimensions to address the deficiency of the single semantic representation of entity nouns. This invention drives a large language model with differentiated prompt words to generate static visual attributes and dynamic functional attributes of classroom objects offline. After K-means clustering and cosine similarity filtering, a refined dual-attribute set is obtained, which supplements static features such as object shape, material, and structure, as well as interactive functional features such as foldability, writability, and bendability. This solves the problems of existing methods that rely only on noun label encoding, lack essential object attributes, and cannot adapt to changes in object shape. It enables the model to understand the physical characteristics and functions of classroom objects, and significantly improves the ability to recognize zero-sample combination actions of deformation such as book bending and chalk breaking.

[0018] (ii) Fine-grained cross-attention alignment addresses the shortcomings of coarse matching of visual and semantic features and lack of fine-grained alignment capabilities.

[0019] This invention divides the spatial feature map into multiple image blocks equally, using block features as queries and attribute features as keys. Through multi-head scaling, dot product, and cross-attention, it achieves a one-to-one association between local image blocks and subdivided semantic attributes. This solves the problems of existing global feature matching methods such as diluting local key information, being susceptible to changes in lighting, shooting angle shifts, and interference from local occlusion. It accurately captures the correspondence between key areas such as hands and object parts and attributes, effectively reduces feature drift caused by changes in the image, and significantly improves the model's generalization ability and robustness in complex classroom scenarios.

[0020] (iii) Fully offline semantic preprocessing architecture to solve the defects of high computing power consumption and difficulty in adapting to edge devices during the inference stage.

[0021] This invention completes the entire process of calling the large language model, generating and filtering attributes, and constructing enhanced noun prototypes offline. The training and inference phases do not require real-time calling of the large language model; only a lightweight visual network is run. This solves the problems of existing solutions that rely on large models and have high requirements for GPU memory and computing power during the inference phase. It does not require cloud server support and can be directly deployed on low-computing-power devices such as smart classroom cameras, embedded terminals, and low-cost teaching all-in-one machines, meeting the local deployment needs of routine classroom monitoring.

[0022] (iv) Multi-loss joint constraint mechanism to solve the defects of lack of unified constraints and weak spatial correlation of multimodal features.

[0023] This invention constructs a joint loss function that combines cross-modal contrastive loss and classification loss. By using contrastive loss, visual features and attribute semantic features are forced to be close to each other in the feature space. Combined with classification loss, the accuracy of category recognition is guaranteed. This solves the problems of existing methods that simply concatenate features, have independent spatial distribution of multimodal features, and cannot effectively assist visual recognition with semantic priors. It strengthens the spatial consistency between visual features and semantic features, gives full play to the guiding role of attribute priors, and further improves the reliability and accuracy of zero-sample combined action recognition. Attached Figure Description

[0024] Figure 1 This is a flowchart of the offline enhanced noun prototype library graph in the classroom zero-sample combined action recognition method based on dual-attribute semantic cross-alignment of the present invention; Figure 2 This is a flowchart of the visual attribute cross-alignment module of the present invention. Detailed Implementation

[0025] The following detailed description is provided in conjunction with specific implementation methods.

[0026] Example 1 This invention relates to a zero-shot combined action recognition method for classrooms based on dual-attribute semantic cross-alignment, which is implemented according to the following steps: Step 1: Build an offline enhanced noun prototype library; Step 2: Preprocessing the classroom video dataset; Step 3: Build a recognition model. Use the preprocessed classroom video dataset and the offline enhanced noun prototype library as input to train the recognition model and obtain a trained recognition model.

[0027] Step 4: Use the trained recognition model to perform action recognition.

[0028] Example 2 Based on Example 1, the process of step 1 is as follows: Figure 1 As shown, specifically: Step 1.1: Construct a standardized noun set and design static and dynamic prompt words for each noun in the standardized noun set; specifically: Summarize all target items in classroom teaching scenarios and construct a standardized terminology set. , The total number of nouns corresponding to the name of the target item; For each noun in the standardized noun set , i = 1-m Static visual cues and dynamic functional cues were designed separately. The static visual cues were "descriptive nouns". List the static visual characteristics of shape, material, size, and physical structure one by one; the dynamic functional prompts are "descriptions of human-readable nouns". List the interactive actions performed and the functional attributes of the object one by one; The prompt words are formatted and ambiguous sentences are removed to obtain a set of standardized nouns. And the corresponding set of static visual cue words and set of dynamic functional cue words; Step 1.2 involves concatenating the static and dynamic cue words corresponding to each noun and using them as input to the pre-trained large language model to obtain the static and dynamic attribute text corresponding to that noun; specifically: The process involves concatenating a single noun with all its corresponding static cue words and inputting the result into a pre-trained large language model. The output is the static attribute text corresponding to that noun. The process involves concatenating a single noun with all its corresponding dynamic prompts and inputting the concatenation into a pre-trained large language model. The output is the dynamic attribute text corresponding to that noun. ; Then iterate through the set of standardized nouns. All nouns in the text are collected, and all static attribute text is used to construct a global static raw attribute pool. All dynamic attribute text is collected to construct a global dynamic raw attribute pool. The global static raw attribute pool and the global dynamic raw attribute pool are then cleaned to remove duplicate text and invalid descriptions that are irrelevant to static visuals and dynamic functions. Step 1.3: Use a pre-trained text encoder to vectorize each noun in the standardized noun set and its corresponding static and dynamic attribute text, obtaining noun label vectors respectively. Static attribute vectors and dynamic attribute vectors; specifically: A pre-trained text encoder is used to vectorize each noun in the standardized noun set and its corresponding static and dynamic attribute text, resulting in noun label vectors. Static attribute vectors and dynamic attribute vectors: (1) in, It is the BERT word segmentation function, used to convert natural text into recognizable token sequences; Indicates the extraction encoder The output vector of the position. For text encoders, ; Similarly, according to formula (1), the static attribute text is... and dynamic attribute text As input, static attribute vectors are obtained respectively. Dynamic attribute vectors ; Step 1.4: Perform K-means clustering on the static and dynamic attribute vectors respectively, clustering them into multiple static attribute clusters and dynamic attribute clusters. For each attribute vector within a cluster, calculate its cosine similarity with the noun label vector. Based on the cosine similarity, select the cluster representative attribute from each static attribute cluster and the cluster representative attribute from each dynamic attribute cluster to form a refined noun dual-attribute set; specifically: Perform K-means clustering on m static attribute vectors and m dynamic attribute vectors respectively, and cluster the m static attribute vectors into K=3 static attribute clusters. Cluster m dynamic attribute vectors into K dynamic attribute clusters. ; For the three static attribute clusters: For any static attribute cluster For all static attribute vectors within the vector, calculate the relationship between each static attribute vector and its corresponding noun tag vector. Cosine similarity: (2) in, Noun label vector The cosine similarity with the corresponding static attribute vector. ; Based on cosine similarity from static attribute clusters The three static attribute vectors with the highest cosine similarity are selected as the static attribute cluster. The cluster represents a static attribute vector; For the three dynamic attribute clusters, the same formula (2) is used to calculate any dynamic attribute cluster. Each dynamic attribute vector and its corresponding noun label vector The cosine similarity is used to determine the dynamic attribute cluster. The three dynamic attribute vectors with the highest cosine similarity are selected as the dynamic attribute cluster. The cluster represents a dynamic attribute vector; The selected cluster-representative attribute vectors are merged to form a set, resulting in a refined noun dual-attribute set. : (3) in, This indicates the filtering results for static attribute clusters. This indicates the filtering results for dynamic attribute clusters. noun A refined set of dual attributes, containing M attribute vectors, where M=2K=6; Step 1.5: Obtain the noun attribute mean vector based on the refined noun dual attribute set; specifically: For nouns The arithmetic mean of all attribute vectors in the refined dual-attribute set is used to obtain the term. Corresponding attribute mean vector: (4) in, To refine the single attribute vector in a dual-attribute set, The dimensions are consistent with the attribute vector and noun label vector, both being 768 dimensions. Step 1.6: Construct an offline enhanced noun prototype library based on the attribute mean vectors of all nouns, specifically as follows: noun The noun label vector and the attribute mean vector are concatenated along the channel dimension to obtain the fused vector: (5) In the formula, This involves concatenating operations along the channel dimension. The dimension is the sum of the dimensions of the noun label vector and the attribute mean vector. ; The concatenated fusion vector is input into a two-layer fully connected fusion network (MLP) for feature dimensionality reduction and fusion to generate the final enhanced noun prototype. : (6) In the formula, The first fully connected layer maps a 1536-dimensional vector to a 768-dimensional vector; This is a nonlinear activation function used to introduce characteristic nonlinearity; This is the second fully connected layer, which maps the 768-dimensional vector to the final 768-dimensional vector. The dimension is consistent with the original noun tag vector, which facilitates subsequent similarity calculation; Iterate through all nouns to generate a complete set of augmented noun prototypes. The data is stored offline to the local hard drive as a .pth file for permanent preservation. Subsequent training and inference can directly call upon the data without repeated calculations, resulting in an offline enhanced noun prototype library.

[0029] Example 3 Based on Example 2, step 2 specifically includes: Step 2.1: Divide the public classroom RGB video dataset (containing various known verb-noun combination action samples) into a training set and a validation set in an 8:2 ratio; Step 2.2: Divide the original videos in the training and validation sets into continuous video frame sequences according to the frame rate. ,in To determine the number of frames a single video is divided into, this embodiment divides the original video into a sequence of consecutive video frames at a frame rate of 25 FPS. The size of each frame has been uniformly adjusted to 256×256 pixels, among which =16; Step 2.3: Perform random data augmentation on the original videos in the training set. Specifically, randomly flip the video frames of the original videos horizontally (probability set to 0.5) and randomly crop them (randomly crop a 224×224 region from the 256×256 frames and then scale it back to 224×224) to form a new video frame sequence. Merge the new video frame sequence into the training set and normalize the video frame sequence in the training set (using ImageNet pre-trained mean and variance for normalization, with a mean of [0.485, 0.456, 0.406] and a variance of [0.229, 0.224, 0.225]).

[0030] Example 4 Based on Example 3, the specific process of the recognition model processing the input data in step 3 is as follows: The processing steps 3.1-3.6 are integrated into the visual attribute cross-alignment module, and the process is as follows: Figure 2 As shown, specifically: Step 3.1: Input the video frame sequence from the training set into the pre-trained video encoder to obtain the corresponding spatiotemporal feature tensor. Specifically: In step 3.1, the video encoder uses a pre-trained Video Swin Transformer encoder to extract the spatiotemporal fusion features of the video. The encoder is configured as follows: 7×7 window size, 4 layers of Transformer blocks, inputting a single batch of video frame sequences, and outputting a four-dimensional spatiotemporal feature tensor. (7) in, For training batch size, The length of the video frame sequence. These are the height and width of a single video frame image, respectively. The number of feature channels, That is, the hidden layer dimension of the video encoder. For Video Swin Transformer encoder; Step 3.2, convert the spatiotemporal feature tensor Perform global average pooling (T-pooling) along the temporal dimension to obtain pure spatial feature maps. : (8) In the formula, This is a temporal average pooling operation, which takes the mean value of the features along the temporal dimension. The time-series dimension is removed, and only the batch, channel, and spatial dimensions are retained.

[0031] Step 3.3, convert the pure spatial feature map Divide the image into N independent blocks, expand the features of all blocks, and concatenate them to generate a multi-head cross-attention query matrix. Specifically: Pure spatial feature map The image is divided into N=16 independent image blocks using a grid partitioning function, with each block's features flattened and then concatenated to generate a multi-head cross-attention query matrix. As shown in formula (9): (9) in, The grid partitioning function divides the spatial feature map into N=16 equal blocks using a 4×4 grid. For flattening functions, Features containing all N image blocks The dimension is batch × number of blocks × feature dimension; Step 3.4 involves projecting the refined attribute vector set along its feature dimensions to generate the attention key matrix K and the value matrix V; specifically: A dimensional projection network module is constructed, consisting of two fully connected Linear layers and an intermediate ReLU nonlinear activation function. Based on the noun category labeled in the current input video sample, the refined attribute vector set of the corresponding noun is retrieved from the global refined attribute vector library generated by offline preprocessing. As input to the dimensional projection network module; Dimensional projection network module for refining attribute vector sets Each attribute vector within the matrix undergoes an independent linear dimension mapping, and the output features are used as the key matrix for cross-attention. AND-value matrix : (10) In the formula, A set of refined attribute vectors corresponding to a single noun, containing Each independent attribute vector; Dimensional projection networks with shared weights, key matrix AND-value matrix Both are generated by mapping the same set of attributes to the same dimension projection network module. They have the same dimension and respectively undertake different functions of similarity calculation and feature weighted aggregation in the subsequent cross-attention calculation. The first fully connected layer maps the 768-dimensional attribute vector to 768 dimensions; It is a non-linear activation function. This is the second fully connected layer, which outputs 768-dimensional features. This represents the total number of attribute vectors in a single noun's refined attribute set. It is 6; Step 3.5, query the multi-head cross-attention matrix Multi-head scaling dot product cross-attention computation is performed on the attention key matrix K, value matrix V, and refined attribute vector set to achieve cross-modal alignment and obtain multi-head attention fusion features. Specifically: Multi-head cross-attention query matrix The attention key matrix K and value matrix V are dimensionally reshaped, splitting the feature dimensions into multi-head dimensions to obtain the reshaped query features. Reshaped bond features Reshaped value characteristics : (11) (12) (13) in, For tensor dimension reshaping operations, used to adjust the dimensional arrangement of features to fit the multi-head attention computation format; To ensure the training batch size is consistent with the batch dimension output by the video encoder; In this embodiment, the number of attention heads is set to 4. The total number of image blocks, For single attention head feature dimensions, If the number of feature channels is 768, then This ensures dimensional consistency in multi-head computation; The reshaped query features are scaled to avoid numerical instability caused by high-dimensional dot products. (14) In the formula, The reshaped query features output by formula (11); For single attention head feature dimensions, This is a scaling factor used to normalize the variance of the dot product to 1, preventing the Softmax function from entering the saturation region and ensuring gradient stability during training; the scaled query features Dimensions and Maintain consistency, that is ; Attention weights are calculated per image block for pure spatial feature maps. any image block in Calculate image blocks n The attention weights are derived from the semantic correlation with all attribute features: (15) In the formula, This is the index of the refined attribute, with a value range of 1. 6; For the first The scaled query features of each image block are extracted from the overall query features of formula (14) by block index. For the first The reshaped key features of the refined attribute are obtained by formulas (10) and (12) for the first key feature. Each individual attribute vector is independently projected and then reshaped in dimension; This is an exponential operation used to amplify the weights of highly correlated features; As a normalization term, it ensures that the sum of the attention weights of all individual attribute vectors is 1; For the first The image block and the first Attention weights between individual attribute vectors represent the strength of their semantic association. Using the calculated attention weights, the value matrix is ​​weighted and summed to obtain the attribute enhancement features of each image block, as shown in formula (15): (16) In the formula, The attention weights are calculated using formula (15); For the first The reshaped value characteristics of a refined attribute; For the first The attention-weighted fusion feature of each image block integrates the association information between the block and all attributes; Fusion features of all image blocks By splicing the features together, a complete attention fusion feature is obtained. ; The attention fusion features of each attention head are concatenated to restore the original feature dimensions, as shown in formula (17): (17) In the formula, To reshape the tensor dimension operation, the multi-head dimension and feature dimension are merged and restored to the format of batch × number of blocks × feature dimension; The number of feature channels should be consistent with the feature dimension of the original image block. This represents the multi-head attention fusion feature after splicing; Step 3.6: Fuse features with multi-head attention Multi-head cross-attention query matrix After performing residual connections, global average pooling is then applied to obtain the visual features of the nouns. Specifically: Multi-head attention fusion feature Multi-head cross-attention query matrix Perform residual connections to obtain the fused features after residual connections: (18) In the formula, The multi-head attention fusion feature output by equation (17); The multi-head cross-attention query matrix output by equation (9) ; The fusion features are those resulting from residual connections; Global average pooling is performed along the block dimension on the fused features after residual concatenation to compress the spatial dimension and obtain the final noun visual features. As in formula (19): (19) In the formula, For spatial average pooling, the feature is averaged along the image block dimension; This represents the total number of image blocks. The visual features of nouns for a single batch of samples are represented by the batch × feature dimension. Step 3.7, based on the visual characteristics of nouns 1. Calculate noun prediction scores using an offline enhanced noun prototype library; specifically: Based on the visual characteristics of nouns The corresponding noun prototypes in the offline enhanced noun prototype library are used to calculate the cosine similarity between the visual features and each enhanced noun prototype, which is used as the noun prediction score, as shown in formula (20): (20) In the formula The visual features of the nouns obtained in step 3.6 are the corresponding video samples; To enhance the noun prototypes extracted from the noun prototype set; Cosine similarity is used to measure the semantic association strength between visual features and prototype features. Predicting scores for nouns; Step 3.8, calculate the prediction score for combined actions; specifically: The final combined action prediction score is obtained by weighting the scores, as shown in formula (21): (twenty one) In the formula, For the verb branch prediction score, the verb branch and the noun branch share the same input and video encoder backbone. The input is a normalized sequence of RGB video frames from the classroom, and the video frames are uniformly fed into the video encoder to extract the global spatiotemporal feature tensor. The verb branch directly reuses the original global spatiotemporal feature tensor output by the encoder, without performing temporal pooling, spatial grid partitioning, or cross-attention alignment operations. The original spatiotemporal features are input into an independent fully connected classification head to calculate the verb category prediction score. ; The predicted score for the noun branch; As a weighting factor for the scores, all values ​​are set to 0.5; Step 3.9, using visual features of nouns Attribute mean vector The combined action prediction score and the real verb and noun combination labels corresponding to the training set samples are used to construct a multi-combination loss function. Training parameters are set, and training is performed to obtain the trained recognition model. Among them, the multi-combination loss function Used for backpropagation to update network parameters: (20) in, In this embodiment, a weight of 0.4 is selected for the cross-modal contrast loss; In this embodiment, a weight of 0.6 is selected for the action classification loss. This is a cross-modal contrastive loss, used to constrain the spatial distribution of visual features and semantic attribute features to be consistent, thus narrowing the distance between the visual features and attribute features of similar samples. (twenty one) in, This refers to the number of samples in a single batch. For the first The visual features of nouns obtained from each video sample after steps 3.1-3.6; For the first The attribute mean vector of each video sample corresponding to a noun; L2 loss is used to measure the distance between visual features and attribute features; The action classification loss is used to constrain the combined action classification results to be consistent with the true labels: (twenty two) The predicted score vector for combined actions is obtained by fusing the verb branch score and the noun branch score; The sample contains real verb-noun combination labels. This is the cross-entropy loss function, used to measure the difference between the predicted distribution and the true distribution; The total loss for network training is used for backpropagation to update network parameters. During training, video frame sequences from the training set are sequentially input into the recognition model. The initial learning rate is set to 1e-4, and the weight decay is set to 1e-5. A cosine annealing learning rate strategy is adopted, and the learning rate is adjusted after each iteration. Backpropagation updates all network parameters of the video encoder, dimensionality projection MLP, and multi-head cross-attention. The parameters of the text encoder and augmented noun prototypes are frozen. After each iteration, the recognition accuracy of zero-sample combined actions is tested on the validation set. When the validation accuracy no longer improves for 5 consecutive iterations, an early stopping strategy is triggered, and training is stopped. The trained network weight file is saved, and the complete trained network weight file is finally output.

[0032] Example 5 Based on Example 4, step 4 specifically involves: dividing the collected test video into a video frame sequence according to the method in step 2.2, normalizing the video frame sequence, inputting it into the trained recognition model, turning off the training mode, obtaining the combined action prediction score, selecting the verb-noun combination corresponding to the maximum score as the final recognition result, completing the zero-sample combined action judgment of verb plus noun, and finally outputting the final recognition result of classroom combined actions.

[0033] Example 6: Application in regular classrooms of primary and secondary schools This embodiment uses a typical offline smart classroom in a primary or secondary school as the application scenario to verify the recognition of combined actions of conventional teaching aids such as textbooks, chalk, blackboard erasers, pencil cases, and paper. The core verification is the zero-sample recognition performance and edge device adaptation capability of this invention in conventional teaching scenarios.

[0034] Existing zero-shot combined action recognition methods have significant limitations in this scenario: First, they rely solely on noun label vectors as semantic representations, lacking prior knowledge of the object's physical form and interactive functions. When faced with zero-shot actions involving changes in object form, such as "bending a textbook, breaking chalk, or crumpling paper," the matching degree between visual features and semantic prototypes drops significantly, leading to recognition failure. Second, some schemes that rely on real-time reasoning using large language models have high requirements for hardware computing power and video memory, making them unsuitable for local deployment on low-computing-power devices such as existing ordinary surveillance cameras and embedded terminals in classrooms. They can only be processed via cloud transmission, resulting in high latency and network dependence.

[0035] This embodiment is implemented according to the following process: (1) Offline semantic preprocessing The target noun set is defined as textbook, chalk, blackboard eraser, pencil case, and paper. For each noun, static visual cues and dynamic functional cues are input to drive a pre-trained large language model to generate corresponding attribute text. After text encoding, K-means clustering, and cosine similarity optimization, a refined dual-attribute set for each noun is obtained. The attribute mean vector is calculated, concatenated with the noun label vector, and then fed into a two-layer fully connected fusion network to generate enhanced noun prototypes. All prototypes are stored offline in file format to build an offline enhanced noun prototype library.

[0036] (2) Video data preprocessing Classroom surveillance videos were collected at a uniform frame rate of 25 FPS, and the size of each frame image was standardized to 256×256. The training set and validation set were divided into an 8:2 ratio. The training set contained only known verb-noun combination samples, while the validation set included some zero-sample combinations. Data augmentation operations such as random horizontal flipping, random cropping, and pixel value normalization were performed on the training samples.

[0037] (3) Model training VideoSwin was used as the video encoder to extract spatiotemporal features, and the number of heads in the multi-head cross-attention module was set to 4. The total training loss was composed of a weighted average of cross-modal contrast loss and classification loss, with weights of 0.4 and 0.6, respectively. AdamW was selected as the optimizer, with an initial learning rate of 1e-4, and a cosine annealing strategy was used to adjust the learning rate. Training was stopped after iterative training until the zero-sample recognition accuracy on the validation set was continuously stable, and the complete network weights were saved.

[0038] (4) Online reasoning Real-time classroom video streams are captured by front-end surveillance cameras in the classroom. After frame extraction and format standardization according to standard preprocessing rules, the data is input into the trained model. The network sequentially performs spatiotemporal feature extraction, temporal pooling, spatial segmentation, and cross-attention alignment, and outputs visual features of nouns. The locally stored offline enhanced noun prototype library is loaded, and the noun matching score is obtained by calculating cosine similarity. Combined with the output results of the verb branch, the final action recognition of the verb-noun combination is completed.

[0039] Implementation Results: The model trained in this embodiment can accurately recognize known combinations of actions included in the training set, such as "flipping through a textbook, picking up chalk, and wiping the blackboard eraser." It also demonstrates stable recognition capabilities for zero-sample combinations of deformation actions, such as "bending a textbook, breaking chalk, and crumpling paper," which were never seen in the training set. This effectively solves the problem of deformation action recognition failure caused by insufficient semantic representation of a single label. Furthermore, the entire inference process does not require calling a large language model or cloud computing power, and can be run locally on existing low-computing-power monitoring terminals in primary and secondary school classrooms. The recognition latency meets real-time monitoring requirements, verifying the zero-sample recognition accuracy advantage and edge deployment adaptability of this invention.

Claims

1. A zero-shot combined action recognition method for classrooms based on dual-attribute semantic cross-alignment, characterized in that, The specific steps are as follows: Step 1: Build an offline enhanced noun prototype library; Step 2: Preprocessing the classroom video dataset; Step 3: Build a recognition model. Use the preprocessed classroom video dataset and the offline enhanced noun prototype library as input to train the recognition model and obtain a trained recognition model. Step 4: Use the trained recognition model to perform action recognition.

2. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 1, characterized in that, Step 1 specifically involves: Step 1.1: Construct a standardized noun set and design static and dynamic prompt words for each noun in the standardized noun set; Step 1.2: Concatenate the static and dynamic prompt words corresponding to each noun as input to the pre-trained large language model to obtain the static and dynamic attribute text corresponding to the noun; Step 1.3: Use a pre-trained text encoder to vectorize each noun in the standardized noun set and its corresponding static and dynamic attribute text, obtaining noun label vectors respectively. Static attribute vectors and dynamic attribute vectors; Step 1.4: Perform K-means clustering on the static attribute vector and dynamic attribute vector respectively to cluster them into multiple static attribute clusters and dynamic attribute clusters. For the attribute vector in each cluster, calculate its cosine similarity with the noun label vector. Select the cluster representative attribute in each static attribute cluster and the cluster representative attribute in each dynamic attribute cluster based on the cosine similarity to form a refined noun dual attribute set. Step 1.5: Obtain the noun attribute mean vector based on the noun refinement dual attribute set; Step 1.6: Construct an offline enhanced noun prototype library based on the attribute mean vectors of all nouns.

3. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 2, characterized in that, Step 1.1 specifically includes: Summarize all target items in classroom teaching scenarios and construct a standardized terminology set. , The total number of nouns corresponding to the name of the target item; For each noun in the standardized noun set , i = 1-m Static visual cues and dynamic functional cues were designed separately. The static visual cues were "descriptive nouns". The static visual characteristics of shape, material, size, and physical structure are listed one by one. The dynamic functional prompts are "descriptions of human-readable nouns." List the interactive actions performed and the functional attributes of the object one by one; The prompt words are formatted and ambiguous sentences are removed to obtain a set of standardized nouns. And the corresponding set of static visual cue words and set of dynamic functional cue words; Step 1.2 specifically includes: The process involves concatenating a single noun with all its corresponding static cue words and inputting the result into a pre-trained large language model. The output is the static attribute text corresponding to that noun. The process involves concatenating a single noun with all its corresponding dynamic prompts and inputting the concatenation into a pre-trained large language model. The output is the dynamic attribute text corresponding to that noun. ; Then iterate through the set of standardized nouns. All nouns in the text are collected, and all static attribute text is used to construct a global static primitive attribute pool. All dynamic attribute text is collected to construct a global dynamic primitive attribute pool. The global static primitive attribute pool and the global dynamic primitive attribute pool are then cleaned to remove duplicate text and invalid descriptions that are irrelevant to static visuals and dynamic functions.

4. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 3, characterized in that, Step 1.3 specifically includes: A pre-trained text encoder is used to vectorize each noun in the standardized noun set and its corresponding static and dynamic attribute text, resulting in noun label vectors. Static attribute vectors and dynamic attribute vectors: （1） in, It is the BERT word segmentation function, used to convert natural text into recognizable token sequences; Indicates the extraction encoder The output vector of the position. For text encoders; Similarly, according to formula (1), the static attribute text and dynamic attribute text As input, static attribute vectors are obtained respectively. Dynamic attribute vectors ; Step 1.4 specifically includes: Perform K-means clustering on m static attribute vectors and m dynamic attribute vectors respectively, and cluster the m static attribute vectors into K static attribute clusters. Cluster m dynamic attribute vectors into K dynamic attribute clusters. ; For K static attribute clusters: For any static attribute cluster For all static attribute vectors within the vector, calculate the relationship between each static attribute vector and its corresponding noun tag vector. Cosine similarity: （2） in, Noun label vector The cosine similarity between the vector and the corresponding static attribute vector; Based on cosine similarity from static attribute clusters The three static attribute vectors with the highest cosine similarity are selected as the static attribute cluster. The cluster represents a static attribute vector; For K dynamic attribute clusters, the same formula (2) is used to calculate any dynamic attribute cluster. Each dynamic attribute vector and its corresponding noun label vector The cosine similarity is used to determine the dynamic attribute cluster. The three dynamic attribute vectors with the highest cosine similarity are selected as the dynamic attribute cluster. The cluster represents a dynamic attribute vector; The selected cluster-representative attribute vectors are merged to form a set, resulting in a refined noun dual-attribute set. : （3） in, This indicates the filtering results for static attribute clusters. This indicates the filtering results for dynamic attribute clusters. noun A refined set of dual attributes, containing M attribute vectors.

5. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 4, characterized in that, Step 1.5 specifically involves: For nouns The arithmetic mean of all attribute vectors in the refined dual-attribute set is used to obtain the term. Corresponding attribute mean vector: （4） in, To refine a single attribute vector in a dual-attribute set; Step 1.6 specifically includes: noun The noun label vector and the attribute mean vector are concatenated along the channel dimension to obtain the fused vector: （5） In the formula, This involves concatenating operations along the channel dimension. The dimension is the sum of the dimensions of the noun label vector and the attribute mean vector; The concatenated fusion vector is input into a two-layer fully connected fusion network (MLP) for feature dimensionality reduction and fusion to generate the final enhanced noun prototype. : （6） In the formula, It is the first fully connected layer; It is a non-linear activation function. It is the second fully connected layer; The dimensions are consistent with the original noun label vector. Iterate through all nouns to generate a complete set of augmented noun prototypes. The data is stored offline to the local hard drive as a .pth file, resulting in an offline enhanced noun prototype library.

6. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 5, characterized in that, Step 2 specifically involves: Step 2.1: Divide the public classroom RGB video dataset into a training set and a validation set according to a certain ratio; Step 2.2: Divide the original videos in the training and validation sets into continuous video frame sequences according to the frame rate. ,in The number of frames a single video is divided into; Step 2.3: Perform random data augmentation on the original videos in the training set. Specifically, randomly flip and crop the video frames of the original videos to form a new video frame sequence. Merge the new video frame sequence into the training set and normalize the video frame sequence in the training set.

7. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 6, characterized in that, The specific process of the recognition model processing the input data in step 3 is as follows: Step 3.1: Input the video frame sequence from the training set into the pre-trained video encoder to obtain the corresponding spatiotemporal feature tensor. ; Step 3.2, convert the spatiotemporal feature tensor Perform global average pooling (T-pooling) along the temporal dimension to obtain pure spatial feature maps. ; Step 3.3, convert the pure spatial feature map Divide the image into N independent blocks, expand the features of all blocks, and concatenate them to generate a multi-head cross-attention query matrix. ; Step 3.4: Project the refined attribute vector set along the feature dimension to generate the attention key matrix K and the value matrix V; Step 3.5, query the multi-head cross-attention matrix Multi-head scaling dot product cross-attention computation is performed on the attention key matrix K, value matrix V, and refined attribute vector set to achieve cross-modal alignment and obtain multi-head attention fusion features. ; Step 3.6: Fuse features with multi-head attention Multi-head cross-attention query matrix After performing residual connections, global average pooling is then applied to obtain the visual features of the nouns. ; Step 3.7, based on the visual characteristics of nouns Calculate noun prediction scores using an offline enhanced noun prototype library; Step 3.8, calculate the prediction score for combined actions; Step 3.9, using visual features of nouns Attribute mean vector The combined action prediction score and the real verb and noun combination labels corresponding to the training set samples are used to construct a multi-combination loss function. Training parameters are set, and training is performed to obtain the trained recognition model.

8. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 7, characterized in that, In step 3.1, the video encoder uses a pre-trained Video Swin Transformer encoder to extract the spatiotemporal fusion features of the video. The encoder is configured as follows: 7×7 window size, 4 layers of Transformer blocks, inputting a single batch of video frame sequences, and outputting a four-dimensional spatiotemporal feature tensor. （7） in, For training batch size, The length of the video frame sequence. These are the height and width of a single video frame image, respectively. The number of feature channels, i.e., the hidden layer dimension of the video encoder. For Video SwinTransformer encoder; Pure spatial feature map in step 3.2 Specifically: （8） In the formula, This is a temporal average pooling operation that takes the average value of features along the temporal dimension; Step 3.3 specifically involves: Pure spatial feature map The image is divided into N independent image blocks using a grid partitioning function, with each block's features flattened and then concatenated to generate a multi-head cross-attention query matrix. As shown in formula (9): （9） in, The grid partitioning function divides the spatial feature map into N=16 equal blocks using a 4×4 grid. For flattening functions, Features encompassing all N image blocks; Step 3.4 specifically involves: A dimensional projection network module is constructed, consisting of two fully connected Linear layers and an intermediate ReLU nonlinear activation function. Based on the noun category labeled in the current input video sample, the refined attribute vector set of the corresponding noun is retrieved from the global refined attribute vector library generated by offline preprocessing. As input to the dimensional projection network module; Dimensional projection network module for refining attribute vector sets Each attribute vector within the matrix undergoes an independent linear dimension mapping, and the output features are used as the key matrix for cross-attention. AND-value matrix : （10） In the formula, A set of refined attribute vectors corresponding to a single noun, containing Each independent attribute vector; Dimensional projection networks with shared weights, key matrix AND-value matrix Both are generated by mapping the same set of attributes using the same dimension projection network module, and their dimensions are consistent. It is the first fully connected layer. It is a non-linear activation function. It is the second fully connected layer. The total number of attribute vectors in a single noun's refined attribute set; Step 3.5 specifically involves: Multi-head cross-attention query matrix The attention key matrix K and value matrix V are dimensionally reshaped, splitting the feature dimensions into multi-head dimensions to obtain the reshaped query features. Reshaped bond features Reshaped value characteristics : （11） （12） （13） in, Operations for reshaping tensor dimensions To ensure the training batch size is consistent with the batch dimension output by the video encoder; For the number of attention heads, The total number of image blocks, For single attention head feature dimensions, The number of feature channels; Scaling is applied to the reshaped query features: （14） In the formula, The reshaped query features output by formula (11); For single attention head feature dimensions, The scaling factor represents the scaled query features. Dimensions and Maintain consistency; For pure spatial feature maps any image block in Calculate image blocks n The attention weights are derived from the semantic correlation with all attribute features: （15） In the formula, This is the index of the refined attribute, with a value range of 1. M,; For the first The scaled query features of each image block are extracted from the overall query features of formula (14) by block index. For the first The reshaped key features of the refined attribute are obtained by formulas (10) and (12) for the first key feature. Each individual attribute vector is independently projected and then reshaped in dimension; This is an exponential operation used to amplify the weights of highly correlated features; As a normalization term, it ensures that the sum of the attention weights of all individual attribute vectors is 1; For the first The image block and the first Attention weights between individual attribute vectors represent the strength of their semantic association. Using the calculated attention weights, the value matrix is ​​weighted and summed to obtain the attribute enhancement features of each image block, as shown in formula (15): （16） In the formula, The attention weights are calculated using formula (15); For the first The reshaped value characteristics of a refined attribute; For the first The attention-weighted fusion feature of each image block integrates the association information between the block and all attributes; Fusion features of all image blocks By splicing the features together, a complete attention fusion feature is obtained. ; The attention fusion features of each attention head are concatenated to restore the original feature dimensions, as shown in formula (17): （17） In the formula, To reshape the tensor dimension operation, the multi-head dimension and feature dimension are merged and restored to the format of batch × number of blocks × feature dimension; The number of feature channels should be consistent with the feature dimension of the original image block. This represents the multi-head attention fusion feature after splicing; Step 3.6 specifically involves: Multi-head attention fusion feature Multi-head cross-attention query matrix Perform residual connections to obtain the fused features after residual connections: （18） In the formula, The multi-head attention fusion feature output by equation (17); The multi-head cross-attention query matrix output by equation (9) ; The fusion features are those resulting from residual connections; Global average pooling is performed along the block dimension on the fused features after residual concatenation to compress the spatial dimension and obtain the final noun visual features. As in formula (19): （19） In the formula, For spatial average pooling, the feature is averaged along the image block dimension; This represents the total number of image blocks. The visual features of nouns for a single batch of samples are represented by the batch × feature dimension.

9. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 8, characterized in that, Step 3.7 specifically involves: Based on the visual characteristics of nouns The corresponding noun prototypes in the offline enhanced noun prototype library are used to calculate the cosine similarity between the visual features and each enhanced noun prototype, which is used as the noun prediction score, as shown in formula (20): （20） In the formula The visual features of the nouns obtained in step 3.6 are the corresponding video samples; To enhance the noun prototypes extracted from the noun prototype set; Cosine similarity is used to measure the semantic association strength between visual features and prototype features. Predicting scores for nouns; Step 3.8 specifically involves: The final combined action prediction score is obtained by weighting the scores, as shown in formula (21): （21） In the formula, The predicted score for the verb branch; The predicted score for the noun branch; As the score weight; The multi-combination loss function in step 3.9 for: （20） in, Weights for cross-modal contrast loss; Weights for the action classification loss; For cross-modal contrast loss: （21） in, This refers to the number of samples in a single batch. For the first The visual features of nouns obtained from each video sample after steps 3.1-3.6; For the first The attribute mean vector of each video sample corresponding to a noun; L2 loss is used to measure the distance between visual features and attribute features; Loss for classifying actions: （22） The predicted score vector for combined actions is obtained by fusing the verb branch score and the noun branch score; The sample contains real verb-noun combination labels; The total loss for network training is used for backpropagation to update network parameters. During training, video frame sequences from the training set are sequentially input into the recognition model. The initial learning rate is set to 1e-4, and the weight decay is set to 1e-5. A cosine annealing learning rate strategy is adopted, and the learning rate is adjusted after each iteration. Backpropagation updates all network parameters of the video encoder, dimensionality projection MLP, and multi-head cross-attention. The parameters of the text encoder and augmented noun prototypes are frozen. After each iteration, the recognition accuracy of zero-sample combined actions is tested on the validation set. When the validation accuracy no longer improves for 5 consecutive iterations, an early stopping strategy is triggered, and training is stopped. The trained network weight file is saved, and the complete trained network weight file is finally output.

10. The classroom zero-shot combined action recognition method based on dual-attribute semantic cross-alignment according to claim 9, characterized in that, Step 4 specifically involves: dividing the collected test video into a video frame sequence according to the method in step 2.2, normalizing the video frame sequence, and then inputting it into the trained recognition model to obtain the combined action prediction score. The verb-noun combination corresponding to the maximum score is selected as the final recognition result, and the zero-sample combined action judgment of verb plus noun is completed. Finally, the final recognition result of classroom combined actions is output.

Images