A method, device, computer device and storage medium for classroom action recognition

Abstract

The present invention provides a classroom action recognition method, device, computer device and storage medium. The method includes: First, collect multi-modal data of human interaction actions. Map the multi-modal action data to the modality-associated space and the modality-specific space respectively by constructing a shared encoder and a differential encoder. In the modality-associated feature space, use the different modality data features corresponding to the same action to construct positive sample pairs, and use contrastive learning to reduce the semantic distance between the positive sample pairs. In the modality-specific feature space, use a graph convolutional neural network and a gated recurrent unit to extract visual skeleton data features and sensor perception data features respectively, calculate the probability distribution of each modality action recognition, and combine the KL divergence to reduce the difference of the probability distribution. Reduce the redundant information between the shared features and the differential features by constructing an orthogonal loss, and achieve the adaptive fusion between different modality features to perform interactive action recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal recognition, and particularly to a classroom action recognition method, device, computer device, and storage medium. Background Art

[0002] With the rapid development of computer technology and network technology, classroom teaching shows an intelligent and personalized development trend. Among them, multimodal action recognition, as an important part of classroom behavior analysis, can help identify the individual differences of students and provide technical support for personalized teaching. Multimodal action recognition collects action data through multi-channel sensory devices such as visual cameras and wearable sensors, and calculates, analyzes, and recognizes classroom multimodal teaching data, so that the computer can understand the behaviors and intentions between teaching subjects in the intelligent classroom.

[0003] Currently, the feature decoupling representation method applied to multimodal machine learning is mainly the decoupling of a multi-branch network architecture. The feature deconstruction of the multi-branch architecture mainly includes a shared feature encoding branch and a differential feature encoding branch. On the one hand, the feature decoupling of the multi-branch architecture maps different modal data to a modality-independent feature subspace through a shared encoder, fully mining the feature correlation between modalities; on the other hand, a differential encoder is used to mine the unique semantic information of different modalities themselves.

[0004] However, the above method mines the correlation information of each modality based on the Euclidean distance in the modality-related feature subspace. Using the Euclidean distance will ignore the semantic offset problem between different modalities, thus affecting the recognition accuracy of the model; moreover, in the modality-specific feature subspace, only using the cross-entropy loss function to train the recognition model makes the difference between the features of each modality not strong enough, resulting in low recognition accuracy of the model. Summary of the Invention

[0005] Based on the above defects existing in the prior art, the present invention provides a classroom action recognition method, device, computer device, and storage medium, which can solve the technical problem of low accuracy of action recognition in the prior art.

[0006] The present invention provides a classroom action recognition method, including:

[0007] Obtain the classroom action data to be recognized including visual skeleton data and sensor perception data;

[0008] Reduce the semantic distance between the visual skeleton data and the sensor perception data corresponding to the same classroom action to be recognized through contrastive learning, and obtain the shared features with semantic alignment;

[0009] Maximize the probability distribution of visual skeleton data and sensor perception data to expand the difference between visual skeleton data and sensor perception data, and obtain the difference features corresponding to the visual skeleton data and sensor perception data;

[0010] Adaptive weighted fusion of the concatenated shared features and difference features is performed by introducing the weights of Gumbel noise to obtain the fusion features; and the fusion features are input into the softmax function to output the recognition result of the action data to be recognized.

[0011] Further, the action data to be recognized in the classroom including visual skeleton data and sensor perception data includes a multi-modal action data set D composed of visual skeleton data d v and sensor perception data d t and the corresponding label

[0012]

[0013] where d1 to d n represent n actions, and n represents the number of action samples; respectively represent the visual skeleton data and sensor perception data of the i-th action; represents the label of the i-th action.

[0014] Further, the obtaining of the semantically aligned shared features includes:

[0015] Use a shared encoder to map the action data to be recognized containing different modalities to a modality-associated subspace to obtain shared features:

[0016]

[0017] where E * (·) is the shared encoder, are respectively the visual skeleton shared feature and sensor perception shared feature of the i-th action;

[0018] In the modality-associated subspace, take as a positive sample pair, and use the loss function of contrastive learning to narrow the semantic distance between positive sample pairs:

[0019]

[0020] where, represents and the distance between, τ represents the temperature coefficient, and n represents the number of sample pairs;

[0021] When the loss function of contrastive learning During the convergence process, a compact representation of the action data is learned by gradually reducing the spatial distance between positive sample pairs and gradually increasing the spatial distance between negative sample pairs, so as to obtain the semantic information of the aligned shared features.

[0022] Furthermore, the differential features include:

[0023] Construct a graph convolutional neural network and a gated recurrent unit differential encoder, and use the graph convolutional neural network and the gated recurrent unit to extract the visual skeleton data features and the sensor perception data features respectively:

[0024]

[0025] where GCN(·) and GRU(·) are the graph convolutional neural network and the gated recurrent unit respectively, are the visual skeleton data features and the sensor perception data features of the i-th action respectively;

[0026] Use the softmax function to obtain the probability distributions of the visual skeleton data features and the sensor perception data features:

[0027]

[0028] where, are the probability distributions of the visual skeleton data features and the sensor perception data features of the i-th action respectively;

[0029] Obtain the KL divergence based on the probability distributions of the visual skeleton data features and the sensor perception data features:

[0030]

[0031] where, is the KL divergence, and the meaning of this formula is to maximize the KL divergence to maximize the probability distributions of different modalities;

[0032] Construct a differential feature loss function based on the multi-class cross-entropy loss function and the KL divergence:

[0033]

[0034] where, represents the differential feature loss function, represents the cross-entropy loss function of the visual skeleton data features, represents the cross-entropy loss function of the sensor perception data features;

[0035] In the differential feature loss function during the convergence process, and Reduce the improvement of the representation ability of the skeleton action feature and enhance the perception ability of the sensor action data; D KL Increase the presence of differential information between the skeleton feature and the sensor action feature; when the loss function reaches the optimal value, the skeleton action feature and the sensor action feature not only have the representation ability but also contain the differential information unique to their respective modalities: the visual skeleton data differential feature f i v and the sensor perception data differential feature f i t .

[0036] Furthermore, the differential feature loss function further includes:

[0037]

[0038] where j ∈ {v, t}.

[0039] Furthermore, the obtaining of the fused feature includes:

[0040] Concatenate the shared feature using concatenation fusion:

[0041]

[0042] where Concat(·) is the feature concatenation operation, and f i c is the concatenated feature;

[0043] Initialize the weights of the concatenated feature f i c , the visual skeleton data differential feature f i v , and the sensor perception data differential feature f i t ;

[0044] Add Gumbel noise to the weights to obtain a new weight vector α′:

[0045]

[0046] where j ∈ {c, v, t}, ∈ ∼ U(0, 1);

[0047] Based on the weight vector α′, calculate the weighted fused feature f′:

[0048]

[0049] where ⊙ is the vector inner product operation.

[0050] Further, the recognition result of the action data to be recognized includes:

[0051] Taking the fused feature f′ as the input of the Softmax layer and calculating the conditional probability p of each action i :

[0052]

[0053] Selecting the category with the maximum conditional probability p i as the action type c(i) of the action to be recognized:

[0054] c(i) = argmax p i , i = 1, 2, …, n.

[0055] A classroom action recognition method device includes:

[0056] A data acquisition module, configured to acquire classroom action data to be recognized including visual skeleton data and sensor perception data;

[0057] An action recognition module, configured to reduce the semantic distance between the visual skeleton data and the sensor perception data corresponding to the same classroom action to be recognized through contrastive learning to obtain a semantically aligned shared feature; expand the difference between the visual skeleton data and the sensor perception data by maximizing the probability distributions of the visual skeleton data and the sensor perception data to obtain the difference features corresponding to the visual skeleton data and the sensor perception data; configured to adaptively weight and fuse the concatenated shared feature and difference feature by introducing the weights of Gumbel noise to obtain a fused feature; and input the fused feature into the softmax function to output the recognition result of the action data to be recognized

[0058] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned classroom action recognition method is implemented.

[0059] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned classroom action recognition method is implemented.

[0060] The present invention provides a classroom action recognition method, device, computer device, and storage medium. Compared with the prior art, the beneficial effects are as follows:

[0061] The present invention reduces the semantic distance between shared features through contrastive learning. The obtained semantically aligned shared features avoid the semantic shift problem between different modalities. Moreover, by maximizing the probability distribution between modalities to mine the unique semantic information of each modality, more distinctive difference features unique to each modality data are obtained. Based on the combination of the semantically aligned shared features and the more distinctive difference features, more accurate information and more reliable feature representations can be provided for tasks such as action recognition, thereby obtaining more accurate action recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention.

[0063] In the drawings:

[0064] Figure 1 is the flowchart of the classroom action recognition method provided in this specification;

[0065] Figure 2 is the framework diagram of the action recognition model provided in this specification;

[0066] Figure 3 is the schematic diagram of contrastive learning provided in this specification;

[0067] Figure 4 is the process of obtaining difference features provided in this specification;

[0068] Figure 5 is the schematic diagram of the computer device provided in this specification DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. However, it should be understood that the protection scope of the present invention is not limited by the specific embodiments.

[0070] Embodiment

[0071] Multi-modal action recognition mainly uses data from multiple sensory channels such as vision and touch for human pose recognition, and is one of the important research directions in the fields of intelligent education, human-computer interaction, etc. However, existing multi-modal classroom action recognition methods often focus on the correlation information between different modality data and ignore the unique semantic features of each modality data, which affects the recognition effect of the model. Therefore, to solve the above problems, this specification proposes as Figure 1 、 Figure 2The multi-modal action recognition model based on feature disentangling representation learning (Feature Disentangling Representation Learning For Multimodal Action Recognition, FDRL-MAR) is shown as follows. First, FDRL-MAR uses a shared encoder to map the action data of each modality to a modality-associated space. In the modality-associated feature space, FDRL-MAR constructs positive sample pairs from the data features of different modalities of the same action, and uses contrastive learning to narrow the semantic distance between positive sample pairs, realizing the mining of shared features and semantic alignment between the data of each modality. Second, FDRL-MAR maps the action data to a modality-specific space through a difference encoder. In the modality-specific feature space, a graph convolutional neural network (GCN) and a gated recurrent unit (GRU) are used to extract the skeleton data features and wearable sensor data features respectively, and the probability distribution of the recognition results of each modality is calculated through a Softmax layer. By combining the KL divergence, the difference of the probability distributions is reduced, thus completing the extraction of differential features. Then, FDRL-MAR constructs an orthogonal loss to eliminate the redundant information between the shared features and the differential features, and creates an adaptive feature fusion module to achieve adaptive fusion between different features with the help of Gumbel-softmax, improving the action recognition accuracy. It specifically includes the following steps:

[0072] Step 1: Construct a multi-modal action recognition model

[0073] According to the device type for collecting action data, the classroom action recognition method can be divided into a vision-based classroom action recognition method and a touch-based classroom action recognition method.

[0074] The vision-based classroom action recognition method can extract rich semantic information. For example, Luo et al. used the human motion information, scene context, and human interaction information in action video data to improve the action recognition effect. However, common RGB and video data contain a large amount of personal privacy information, such as face information, which is likely to cause the leakage of sensitive information, thus infringing on the privacy rights of users. Therefore, skeleton data has gradually become the main data type of visual data, which can not only protect the privacy and security of users, but also eliminate complex background information, facilitating the analysis and mining of the motion features of the limbs during the action recognition process. For example, Liu et al. constructed a Transformer-based skeleton data modeling framework, used a partition-aggregation temporal transformation module to capture the long-range dependencies between actions, and combined a topology-aware spatial Transformer to mine the spatial association information of actions.

[0075] The haptic-based classroom action recognition method uses wearable sensors, such as accelerometers, gyroscopes, and bending sensors, to collect human body posture and motion information. For example, Han et al. achieved human action recognition based on wearable sensors through a heterogeneous convolutional neural network. Common wearable sensor integration devices include smart bracelets, data gloves, smartphones, etc. Yuan et al. designed a data glove with double-arm rings to collect fine motion information of human actions and adopted a multi-granularity data feature fusion method to improve the action recognition accuracy. However, single-modal action data cannot comprehensively describe human body posture and motion information. For example, skeleton data focuses on human body spatial posture information, while wearable sensors focus on the spatial motion information of actions.

[0076] Multi-modal machine learning can simultaneously utilize information from multiple perceptual modalities to improve the performance of machine learning tasks, such as action recognition, intelligent question answering, sentiment analysis, etc. Multi-modal machine learning applied to action recognition has the characteristics of data heterogeneity, semantic consistency, and feature complementarity. Due to the differences in the storage structure and sampling frequency of multi-modal data, there is a semantic gap between the data, making it difficult to directly compare and fuse the data. Therefore, Islam et al. mapped different modal action data to the same feature subspace and enhanced the multi-modal action recognition performance through feature fusion. At the same time, the information described by different modal data of the same action is semantically consistent or mutually related. This means that different modal data can jointly express the same semantic concept or information, enabling better understanding and fusion of multi-modal data. Therefore, Liu et al. used the method of multi-modal knowledge distillation to make the modal information of the student network imitate the modal information of the teacher and combined the distillation loss to ensure the representational consistency between different modalities.

[0077] Currently, multimodal machine learning methods for action recognition can be roughly divided into two categories: homogeneous multimodal action recognition and heterogeneous multimodal action recognition. Homogeneous multimodal action recognition primarily refers to data derived from different perspectives of the same modality. For example, Xiao et al. used a Transformer to extract RGB information, optical flow information, and grayscale image information from action video data, and combined this with a category score fusion mechanism to achieve homogeneous multimodal action recognition. However, homogeneous multimodal action recognition often suffers from modality bias and fails to capture diverse action feature information. For example, the RGB, optical flow, and grayscale image information from video data often focuses on changes in the spatial posture of the human body and fails to capture spatial motion information. Heterogeneous multimodal action recognition leverages data from different modalities (such as RGB, skeleton, and wearable sensor data) to fully exploit the intermodal correlation and complementary information, thereby improving action recognition accuracy. Franco et al. used skeleton data and RGB data to describe the spatial posture information and spatiotemporal dependency information of the human body, respectively, to identify dangerous behaviors or prevent dangerous situations (such as fall detection). At the same time, Ni et al. proposed the Vision2Sensor knowledge distillation framework, which guides wearable sensor data to learn the semantic information of RGB data through semantic loss, thereby enhancing the representation ability of motion features. Since RGB data contains a large amount of personal privacy information (such as facial information), it is easy to cause sensitive information to be leaked, thereby infringing on the user's privacy rights. Ouyang et al. used contrastive learning to extract the associated and complementary semantic information of skeleton data and wearable sensor data to improve motion recognition performance. Existing multimodal classroom motion recognition methods often focus on the associated and complementary information between each modality data, ignoring its own unique semantic information. Therefore, this manual adopts the feature decoupling method to decompose multimodal motion data into shared features and differential features. Shared features are used to mine the associated and complementary information between different modalities, and differential features are used to analyze the unique semantic features of the original data itself. Therefore, this manual follows the idea of multimodal data complementarity, takes skeleton data and sensor data as research objects, and studies and analyzes motion recognition algorithms.

[0078] Step 1.1: Define the problem and related concepts

[0079] Definition 1: Multimodal Action Recognition

[0080] Given a set of visual data d v and tactile data d t Multimodal action dataset composed of The corresponding label is Through the multimodal action recognition model f(d i ; θ), its parameter θ is used to adjust the model recognition effect and introduce a loss function suitable for multimodal data. to measure the difference between the predicted Y = {y1, y2, …, y n} of the model and the actual action labels . The ultimate goal is to find the optimal parameter θ * through an optimization problem to achieve accurate classification and recognition of actions.

[0081] Definition 2: Multimodal Feature Decoupling Representation Learning

[0082] Feature decoupling representation learning is a technique for processing data features, aiming to improve the interpretability, generalization ability of data representation or reduce the redundancy between features. Early feature decoupling representation methods were mainly generative feature decoupling, that is, using generative networks such as autoencoders and generative adversarial networks to decouple feature vectors strongly related to the task from the original data. Aiming at the distribution difference problem existing in the original action data, Qian et al. used variational autoencoders to decouple the original action data into domain-independent and domain-specific semantic information, so as to learn a more generalizable action feature representation and improve the accuracy of the action recognition model. At the same time, Shahroudy et al. proposed an autoencoder network based on Shared-Special feature decomposition, which fully mined the complementary information of different modal action data and obtained higher recognition performance. Currently, the feature decoupling representation methods applied to multimodal machine learning are mainly multi-branch network architecture decoupling [36-38]. The multi-branch architecture feature decoupling mainly includes a shared feature encoding branch and a differential feature encoding branch. On the one hand, the multi-branch architecture feature decoupling maps different modal data to a modality-independent feature subspace through a shared encoder, fully mining the feature correlation between modalities; on the other hand, a differential encoder is used to mine the unique semantic information of different modalities themselves. However, in the modality-related feature subspace, the Euclidean distance between different modal features is often minimized to mine the correlation information of each modality, ignoring the semantic offset problem between different modalities, thus affecting the recognition accuracy of the model. Therefore, this specification uses contrastive learning methods to mine the correlation information between different modalities and achieve semantic alignment between the data features of each modality. At the same time, in the modality-specific feature subspace, only using the cross-entropy loss function to train the recognition model, the obtained feature vectors often cannot guarantee the difference between the features of each modality. Therefore, this specification uses the KL divergence that maximizes the probability distribution between modalities to mine the unique semantic information of different modalities themselves.

[0083] Through the shared encoder E * (·) and the differential encoder E(·) = {E v (·), E t(·)} Map the multi-modal action data to the modality-associated feature subspace and the modality-specific feature subspace respectively. The modality-associated feature space is used to mine the correlation between different modality features, and the specific modality feature space is used to retain the unique feature information of each modality.

[0084]

[0085] Definition 3: Shared Feature

[0086] Shared Feature Refers to the correlation characteristics between different modality features of the same action, such as the semantic consistency information between different modality features of the same action;

[0087] Definition 4: Specific Feature

[0088] Differential Feature Refers to the unique action information of each modality data. For example, the skeleton data has human body spatial pose information, while the wearable sensor focuses on the spatial motion information of the action.

[0089] Step 1.2: Multi-modal Feature Decoupling Representation Learning

[0090] Multi-modal action recognition often uses visual data (skeleton data) and tactile data (wearable sensor data) to describe the pose and motion information of human actions. Since different modality data of the same action are correlated, that is, the change in the pose space of the skeleton data during the movement of the same action will cause changes in the relevant wearable sensor motion information (such as acceleration, direction angle); at the same time, each modality data contains special semantic information by itself. For example, the skeleton data focuses on the spatial information of the human body pose, while the wearable sensor data focuses on the motion information of the action. Therefore, this specification decouples the features of the multi-modal action data to fully mine the correlation information between each modality and its own unique semantic information.

[0091] Assume a given set of multi-modal action data set composed of visual data d v and tactile data d t where n represents the number of action samples. This specification uses a shared encoder Ε * * (·) and a differential encoder Ε(·) = {Ε v (·), Ε t (·)} to map the multi-modal action data to the modality-associated feature space and the specific modality feature space respectively, as shown in formula (1):

[0092]

[0093] where For skeleton data and wearable sensor data, Ε * (·) is a shared encoder, Ε(·) = {Ε v (·), Ε t (·)} is a differential encoder, are the skeleton shared feature and the wearable sensor shared feature respectively, are the skeleton differential feature and the wearable sensor differential feature respectively.

[0094] Step 2: Consistency representation learning:

[0095] Common multi-modal correlation feature extraction methods include transfer learning, multi-modal joint learning, etc. Transfer learning can achieve knowledge transfer between modalities, but information between different modalities cannot be shared. Multi-modal joint learning promotes information interaction between modalities through a common inference or learning task, often ignoring the problem of semantic shift between multi-modal data. Therefore, this specification uses a multi-modal contrast learning model to mine correlation features, eliminating the semantic shift problem between different modalities while extracting different-modal correlation information. As Figure 3 shown, contrast learning can aggregate action features of different modalities for the same action, increasing the distance between the feature of each modality for different actions, that is:

[0096]

[0097] where i≠j, s(·) represents the feature distance, and its calculation formula is as shown in the formula:

[0098]

[0099] Therefore, in the modal correlation subspace, this specification uses the same-action different-modal data as positive sample pairs and uses the contrast loss function to mine correlation information, as shown in the formula:

[0100]

[0101] where i≠j, τ represents the temperature coefficient.

[0102] Step 3: Discriminative representation learning

[0103] In the field of action recognition, skeleton data is usually represented by the coordinates of joints or skeletons, which describe the pose space information of the human body. Wearable sensors consist of measurement values of sensors such as accelerometers, gyroscopes, and magnetometers, and are used to record information such as human movement, direction, and speed. As Figure 4As shown, for the characteristics of different modal data information, such as the skeleton data focusing on the spatial pose information of the human body, while the wearable sensors focusing on the motion timing information of the human body, the graph convolutional neural network and the gated recurrent unit are used to extract the skeleton data features and the wearable sensor data features respectively, that is:

[0104]

[0105] Among them, GCN(·) and GRU(·) are the graph convolutional neural network and the gated recurrent unit respectively.

[0106] The action data features obtained according to different differential encoders Calculate the probability distribution of each modality through the softmax layer, that is:

[0107]

[0108] Introduce the KL divergence to maximize the probability distributions of different modalities, so as to realize the mining of the differential information of different modal data features, that is

[0109]

[0110] Among them, and represent the probability distributions of vision and touch respectively.

[0111] Finally, based on the multi-class cross-entropy loss function, combined with the KL divergence, mine the differential information between different modalities, so as to enhance the semantic information of different modal features and improve the action recognition accuracy.

[0112]

[0113] Among them is the cross-entropy loss function.

[0114] In addition, in order to eliminate the redundant information between the shared features and the differential features, this specification ensures the independence between the feature decouplings through orthogonal constraints, and the calculation formula is as follows:

[0115]

[0116] where j ∈ {v, t}.

[0117] Step 4, Adaptive Feature Fusion

[0118] To make full use of the correlation information between multi-modal features and their own unique semantic information, it is necessary to effectively fuse the shared features and differential features obtained by feature decoupling. Common feature fusion methods include pooling fusion, concatenation fusion, and weighted fusion. Pooling fusion performs pooling operations (such as average pooling or max pooling) on the features of each modality, and then splices the features of each modality. Pooling fusion can reduce the dimension of features, extract the core information of each modality, and reduce the computational complexity, but some detailed information may be lost. Concatenation fusion can connect the features of different modalities in sequence to form a longer feature vector. Concatenation fusion retains all the information of each modality without information loss, so it can provide a more comprehensive feature representation in some cases. At the same time, the contribution values of different types of feature vectors to action recognition vary greatly. The method of manually assigning weights not only brings artificial interference problems but also makes the model unable to adapt to the dynamic changes of data. Therefore, this specification uses the method of concatenation fusion to retain the semantic information of the shared feature space, combines the method of weighted fusion, and dynamically adjusts the weights according to the importance of each modality through Gumbel-softmax, so as to achieve more flexible feature fusion.

[0119] First, splice the shared features using concatenation fusion, retaining the skeleton shared features and the wearable sensor shared features of the semantic information, as shown in the formula.

[0120]

[0121] Among them, Concat(·) is the feature splicing operation.

[0122] Then, initialize the weights of each type of modality feature and add Gumbel noise to obtain a new weight vector α′, and the calculation formula is as follows:

[0123]

[0124] Among them ∈~U(0,1), and τ is the temperature coefficient.

[0125] Finally, based on the weight vector α′, calculate the weighted feature vector f′, and the calculation formula is as follows:

[0126]

[0127] Among them, ⊙ is the vector inner product operation.

[0128] Step 5, Multimodal Action Recognition

[0129] Multimodal action recognition is a typical multi-classification problem. First, the feature vector f′ obtained through the adaptive feature fusion model is used as the input of the Softmax layer, and the conditional probability p of each action (the number of action types is n) is calculated i , then the conditional probability p i is calculated by the formula:

[0130]

[0131] Finally, the max() function is used to obtain the action classification result, that is, the category with the largest conditional probability in the Softmax layer is selected as the action type c(i), which is expressed by the formula:

[0132] c(i) = argmax p i , i = 1, 2, …, n (16)

[0133] Step 6, Target Optimization

[0134] First, the multimodal feature decoupling model decouples the multimodal features into shared features and differential features through feature decoupling. In the shared feature space, contrastive learning is used to mine the correlation between the shared features of different modalities. In the differential feature space, the KL divergence is used to measure the soft label probability distribution of the recognition results of different modalities, and combined with the cross-entropy loss function, the differential features are maximally feature learned. Then, in order to eliminate the redundancy between the shared features and the differential features, the orthogonal loss is used to eliminate the redundant information between each modality. Finally, the cross-entropy loss function is combined to complete the recognition of multimodal action data. Therefore, the final training loss function of the model consists of contrastive loss, differential loss, orthogonal loss, and cross-entropy loss function, as shown below:

[0135]

[0136] where λ is the control coefficient.

[0137] This specification presents a multi-modal action recognition method based on feature decoupling representation learning, which can effectively explore the correlation between different modalities and retain the unique semantic information of each modality. Specifically, FDRL-MAR decouples multi-modal action data into shared features and differential features through a shared encoder and a differential encoder. In the shared feature subspace, contrastive learning is used to explore the correlation between different modalities of the same action; in the differential feature subspace, KL divergence is used to measure the spatial distance between different modalities, retaining the unique semantic information of each modality. Finally, an adaptive feature fusion module is constructed to assign weights to the shared features and differential features, enabling the model to adaptively improve the action recognition effect according to the characteristics of the data itself. Although FDRL-MAR has achieved good recognition results, in real life, the distribution of action data is not uniformly distributed. Therefore, facing real-world scenarios, the problem of mining long-tailed distribution data is the focus of future research. The main contributions of this specification are as follows:

[0138] (1) Apply the idea of feature decoupling to decouple multi-modal action data into shared features and differential features, explore the correlation information between shared features through contrastive learning methods, and use KL divergence to reduce the probability distribution between differential features.

[0139] (2) Construct an adaptive feature fusion module, automatically assign weights to differential features and shared features with the help of Gumbel-Softmax, enhance the semantic information strongly related to associated actions, and thus achieve deep fusion between features.

[0140] For the specific limitations of the action recognition device, reference can be made to the limitations on action recognition in the above text, which will not be elaborated here. Each module in the above action recognition device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0141] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above action recognition method.

[0142] This specification also provides Figure 5 the structural schematic diagram of the computer device shown in Figure 5 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, there may also be other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above action recognition method.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0144] The above embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A classroom action recognition method, characterized in that: include: Obtaining classroom action data to be recognized, including visual skeleton data and sensor perception data; By contrastive learning, the semantic distance between the visual skeleton data and sensor perception data corresponding to the same classroom action to be recognized is narrowed, and semantically aligned shared features are obtained; By maximizing the probability distribution of visual skeleton data and sensor perception data, the difference between visual skeleton data and sensor perception data is expanded to obtain the difference features corresponding to the visual skeleton data and sensor perception data; By introducing the weight of Gumbel noise, the shared features and difference features of the serial splicing are adaptively weighted and fused to obtain the fused features; and the fused features are input into the softmax function to output the recognition results of the action data to be recognized.

2. A classroom action recognition method according to claim 1, characterized in that: The method of obtaining classroom action data to be identified including visual skeleton data and sensor perception data includes the visual skeleton data d v and sensor perception data d t The multimodal action dataset D and the corresponding labels Among them, d1~d n Represents n actions, where n represents the number of action samples; Represent the visual skeleton data and sensor perception data of the i-th action respectively; Indicates the label of the i-th action.

3. A classroom action recognition method according to claim 2, characterized in that: The obtaining of semantically aligned shared features includes: Use a shared encoder to map the action data to be recognized containing different modalities into the modality association subspace to obtain shared features: Among them, E * (·) is a shared encoder, are the visual skeleton shared features and sensor perception shared features of the i-th action respectively; In the modal correlation subspace, As a positive sample pair, using the loss function of contrastive learning Narrow the semantic distance between positive sample pairs: in, express and The distance between them, τ represents the temperature coefficient, and n represents the number of sample pairs; When the loss function of contrastive learning During the convergence process, a compact representation of the action data is learned by gradually reducing the spatial distance between positive sample pairs and gradually increasing the spatial distance between negative sample pairs, so as to obtain the semantic information of the aligned shared features.

4. A classroom action recognition method according to claim 3, characterized in that: The difference features include: Construct a graph convolutional neural network and a gated recurrent unit difference encoder, and use the graph convolutional neural network and the gated recurrent unit to extract visual skeleton data features and sensor perception data features respectively: Among them, GCN(·) and GRU(·) are graph convolutional neural networks and gated recurrent units, respectively. are the visual skeleton data features and sensor perception data features of the i-th action respectively; Use the softmax function to obtain the probability distribution of visual skeleton data features and sensor perception data features: in, are the probability distribution of visual skeleton data features and sensor perception data features of the i-th action respectively; The KL divergence is obtained based on the probability distribution of visual skeleton data features and sensor perception data features: in, is the KL divergence, which means that the probability distribution of different modes is maximized by maximizing the KL divergence; Based on the multi-classification cross entropy loss function and KL divergence, the difference feature loss function is constructed: in, represents the difference feature loss function, represents the cross entropy loss function of visual skeleton data features, Represents the cross entropy loss function of sensor perception data features; In the difference feature loss function During the convergence process, and The ability to represent skeleton motion features is improved, and the perception ability of sensor motion data is enhanced; D KL Adding information indicating that there is difference between skeleton features and sensor motion features; when the loss function reaches the optimal value, skeleton motion features and sensor motion features not only have representation capabilities, but also contain the difference information unique to each modality: visual skeleton data difference feature f i v , sensor perception data difference characteristics f i t .

5. A classroom action recognition method according to claim 4, characterized in that: The difference feature loss function also includes: Where j∈{v,t}.

6. A classroom action recognition method according to claim 4, characterized in that: The obtaining of fusion features includes: Shared features Use concatenated fusion for stitching: Among them, Concat(·) is the feature concatenation operation, f i c It is a splicing feature; Initialize the splicing feature f i c , visual skeleton data difference feature f i v , sensor perception data difference characteristics f i t Weight In weight Add Gumbel noise to Get the new weight vector α′: Among them, j∈{c,v,t},∈~U(0,1); Based on the weight vector α′, the weighted fusion feature f′ is calculated: Among them, ⊙ is the vector inner product operation.

7. A classroom action recognition method according to claim 6, characterized in that: The outputting of the recognition result of the action data to be recognized includes: The fused feature f′ is used as the input of the Softmax layer to calculate the conditional probability p of each action i : Choose the maximum conditional probability p i The category is the action type c(i) of the action to be identified: c(i)=argmaxp i ,i=1,2,…,n。 8. A classroom action recognition method and device, characterized in that: Used to implement the classroom action recognition method according to any one of claims 1 to 6, the classroom action recognition method system includes: A data acquisition module is used to acquire classroom action data to be identified, including visual skeleton data and sensor perception data; The action recognition module is used to narrow the semantic distance between the visual skeleton data and the sensor perception data corresponding to the same classroom action to be identified through comparative learning, so as to obtain semantically aligned shared features; to expand the difference between the visual skeleton data and the sensor perception data by maximizing the probability distribution of the visual skeleton data and the sensor perception data, so as to obtain the difference features corresponding to the visual skeleton data and the sensor perception data; to perform adaptive weighted fusion of the serially spliced shared features and difference features by introducing the weight of Gumbel noise, so as to obtain fused features; and to input the fused features into the softmax function to output the recognition results of the action data to be identified.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a data processing program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images