A cross-domain small sample self-action recognition method based on multi-modal frequency domain distillation

Abstract

The application discloses a cross-domain small sample self-action recognition method based on a multi-modal frequency domain distillation, and relates to the technical field of computer vision. The method comprises the following steps: acquiring an RGB frame sequence of a hand action video of a target domain, wherein the hand action video of the target domain refers to a hand action video under a hand use scene; pre-processing the RGB frame sequence to obtain a pre-processed RGB frame sequence; inputting the pre-processed RGB frame sequence into a trained single-modal student network to classify the hand action contained in the RGB frame sequence, and obtaining a classification score, wherein the trained single-modal student network is obtained by training a multi-modal teacher network and a single-modal student network by using a distillation training mechanism, and by simultaneously combining a frequency domain feature enhancement and a boundary enhancement distillation; and normalizing the classification score to obtain an action category prediction result. The application can improve the distillation efficiency and the generalization performance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a cross-domain few-sample self-action recognition method based on multimodal frequency domain distillation. Background Technology

[0002] In recent years, with the rapid development of wearable devices and computer vision technology, first-person perspective video analysis has become an important research direction in artificial intelligence. Among them, egocentric action recognition (EAR) aims to identify the actions, interactions, or intentions of an actor from videos captured by head-mounted or wearable devices, and is widely used in scenarios such as augmented reality, autonomous driving, human-computer interaction, and home services.

[0003] Existing EAR methods largely rely on large-scale labeled datasets (such as Kinetics, EPIC-Kitchens, etc.) and typically extract features based on Convolutional Neural Networks (CNNs) or Transformers. Although they perform well in a single data domain, they are susceptible to differences in lighting, viewpoint, and background when transferring data across domains, leading to a decline in recognition performance. At the same time, the high cost of acquiring large-scale labeled data limits the application of these models in small sample or unsupervised scenarios.

[0004] To enhance model robustness, researchers have attempted to improve action understanding capabilities through multimodal fusion (RGB frames, optical flow, hand pose, etc.). However, such methods are computationally complex, have high device requirements, and are difficult to deploy in real-time or embedded environments. Therefore, how to fully extract knowledge from multimodal models and achieve efficient cross-domain few-shot learning using only single-modal input has become a key issue in this field. Currently, while research combining knowledge distillation and cross-domain adaptation has made some progress, problems such as insufficient distillation efficiency and generalization performance still exist. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a cross-domain small sample self-action recognition method based on multimodal frequency domain distillation.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] This invention provides a cross-domain few-sample self-action recognition method based on multimodal frequency domain distillation, comprising:

[0008] Obtain the RGB frame sequence of the hand action video in the target domain, wherein the hand action video in the target domain refers to the hand action video in a hand use scenario;

[0009] The RGB frame sequence is preprocessed to obtain a preprocessed RGB frame sequence;

[0010] The preprocessed RGB frame sequence is input into the trained unimodal student network to classify the hand movements contained in the RGB frame sequence and obtain a classification score. The trained unimodal student network is trained using a distillation training mechanism of multimodal teacher network and unimodal student network, combined with frequency domain feature enhancement and boundary enhancement distillation.

[0011] The classification scores are normalized to obtain the action category prediction results.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] This invention constructs a distillation training mechanism for a multimodal teacher network and a unimodal student network. Using this distillation training mechanism, and combining frequency domain feature enhancement and boundary enhancement distillation methods, the unimodal student network is trained through distillation. This achieves effective transfer of cross-domain features and improved small-sample recognition performance, thereby improving distillation efficiency and generalization performance.

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the cross-domain small sample self-action recognition method based on multimodal frequency domain distillation provided in this embodiment of the invention.

[0016] Figure 2 This is a flowchart illustrating the training method for a single-modal student network provided in an embodiment of the present invention.

[0017] Figure 3 This is an example diagram of a hand joint posture;

[0018] Figure 4 This is another exemplary diagram of hand joint posture.

[0019] Specific implementation method

[0020] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0021] Figure 1 This is a flowchart illustrating a cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0022] S101. Obtain the RGB frame sequence of the hand action video in the target domain, where the hand action video in the target domain refers to the hand action video in a hand usage scenario.

[0023] Here, "hand usage scenario" refers to any scenario in which hands are used for operation, such as a kitchen scenario or a daily assembly scenario.

[0024] S102. Preprocess the RGB frame sequence to obtain the preprocessed RGB frame sequence.

[0025] Here, preprocessing of the RGB frame sequence may include: resizing the RGB frame sequence, normalizing the RGB frame sequence, etc., in order to make the processed RGB frame sequence meet the requirements of the trained single-modal student network for input data.

[0026] S103. Input the preprocessed RGB frame sequence into the trained unimodal student network to classify the hand movements contained in the RGB frame sequence and obtain the classification score. The trained unimodal student network is trained by using the distillation training mechanism of the multimodal teacher network and the unimodal student network, combined with frequency domain feature enhancement and boundary enhancement distillation.

[0027] Here, the trained unimodal student network includes a trained target RGB encoder and a trained classifier, and the output of the trained target RGB encoder is connected to the input of the trained classifier. Specifically, the preprocessed RGB frame sequence is input into the trained target RGB encoder, which outputs RGB encoded features. These RGB encoded features are then input into the trained classifier, which outputs a classification score.

[0028] S104. Normalize the classification scores to obtain the action category prediction results.

[0029] Here, the classification score (logits) output by the classifier is processed by the Softmax function to generate the corresponding probability distribution, thereby obtaining the final action category prediction result.

[0030] In some embodiments, such as Figure 2 As shown, the training method for a trained unimodal student network includes steps S1 to S6:

[0031] S1. Using the unlabeled target domain dataset and the labeled source domain dataset, jointly train the initial multimodal teacher network to obtain the trained multimodal teacher network. The target domain dataset contains multiple first samples. Each first sample is a multimodal dataset consisting of an RGB frame sequence, optical flow features, and hand joint pose map of a hand action video in a first hand use scenario. The source domain dataset contains multiple second samples. Each second sample is a multimodal dataset consisting of an RGB frame sequence, optical flow features, and hand joint pose map of a hand action video in a second hand use scenario. The first hand use scenario and the second hand use scenario are different scenarios.

[0032] Here, as Figure 3 and Figure 4 As shown, a hand joint pose diagram refers to a hand pose diagram composed of joint points, commonly known as a hand pose. Figure 3 and Figure 4 As shown, different colored dots represent the joints on different fingers.

[0033] For example, the first hand usage scenario could be a kitchen scenario, and the second hand usage scenario could be a scenario of repairing items. It should be noted that the first hand usage scenario and the second hand usage scenario merely refer to different hand usage scenarios. This invention does not specifically limit what kind of scenario the first hand usage scenario and the second hand usage scenario are.

[0034] S2. Construct an initial unimodal student network using the trained multimodal teacher network. The initial unimodal student network includes an initial target RGB encoder and a multilayer perceptron. The output of the initial target RGB encoder is connected to the input of the multilayer perceptron. The initial target RGB encoder is the trained RGB encoder in the trained multimodal teacher network.

[0035] S3. Using the target domain dataset and the trained multimodal teacher network, the initial unimodal student network is distilled and trained. During each training process, wavelet transform is performed on the multimodal features generated by the trained multimodal teacher network to perform frequency domain decomposition and obtain multi-scale frequency components. The multi-scale frequency components are adjusted based on the learnable scaling factor, and then inverse wavelet transform is performed to obtain global frequency domain enhancement features. The multimodal features generated by the trained multimodal teacher network are processed in the spatial domain to obtain local spatial domain features. Based on the global frequency domain enhancement features, local spatial domain features, and multimodal features generated by the unimodal student network, the relationship difference between the multimodal features of the trained multimodal teacher network and the unimodal student network is calculated. Based on the classification uncertainty of the trained multimodal teacher network, a sample weighting factor is generated to calculate the boundary enhancement distillation loss. The parameters of the target RGB encoder in the unimodal student network are updated by backpropagation. After completing the distillation training of the initial unimodal student network, the trained target RGB encoder is obtained.

[0036] S4. Construct a new initial unimodal student network using the trained target RGB encoder and the initial classifier.

[0037] Here, the new initial unimodal student network includes a trained target RGB encoder and an initial classifier, and the output of the trained target RGB encoder is connected to the input of the initial classifier. It should be noted that in this invention, the encoder architecture, decoder architecture, and classifier structure are the same in both the unimodal student network and the multimodal teacher network. For example, the encoders in both the unimodal student network and the multimodal teacher network can be encoders using the ViT architecture, where ViT is short for Vision Transformer. The classifiers in both the unimodal student network and the multimodal teacher network are lightweight feature classifiers using a multilayer perceptron architecture.

[0038] S5. Obtain the labels of the target domain dataset, and construct the support set and query set respectively using some second samples in the labeled target domain dataset.

[0039] It should be noted that each first sample in the target domain dataset has a label, but the labels of each first sample in the target domain dataset are neither obtained nor used before step S5. When it is necessary to construct the support set and query set, the labels of a subset of the first samples in the target domain dataset are obtained, thus obtaining... The first labeled sample will A portion of the labeled first samples is used as the support set, and the other portion is used as the query set. It is a positive integer greater than 1.

[0040] S6. Use the support set and query set to train the classifier in the new initial unimodal student network, and after completing the training of the classifier, obtain the trained unimodal student network.

[0041] This invention first constructs a multimodal teacher network comprising image frames, optical flow, and pose heatmaps, and performs joint pre-training on unlabeled samples in the source and target domains. Then, using a proposed frequency-aware distillation method, the multimodal features of the multimodal teacher network are mapped to the frequency domain. Wavelet transform is used to achieve multi-scale feature decomposition and reconstruction, guiding the unimodal student network to learn cross-modal semantic features. Furthermore, a boundary enhancement distillation method is employed, introducing a weighting mechanism based on classification uncertainty during the distillation process to strengthen the unimodal student network's discriminative ability on decision boundary samples. Finally, a few-shot feature classifier is used to optimize and classify features from a limited set of labeled samples. This invention can significantly improve the performance of cross-domain few-shot action recognition by fully utilizing multimodal knowledge for transfer learning and feature distillation using only unimodal input. It effectively solves the problems of cross-domain generalization and multimodal feature transfer in few-shot video recognition, and has high application value.

[0042] In some embodiments, step S1 is implemented through steps S11 to S14:

[0043] S11. Construct an initial multimodal teacher network, which includes an RGB branch, an optical flow branch, and a pose branch. Each branch contains an encoder, a decoder, and a classifier. The output of the encoder in each branch is connected to the input of the decoder and the classifier, respectively.

[0044] To distinguish them, this invention refers to the encoder, decoder, and classifier in the RGB branch as RGB encoder, RGB decoder, and RGB classifier, respectively; the encoder, decoder, and classifier in the optical flow branch as optical flow encoder, optical flow decoder, and optical flow classifier, respectively; and the encoder, decoder, and classifier in the pose branch as pose encoder, pose decoder, and pose classifier, respectively. Each branch is used to extract video feature information under different modalities. Specifically, the RGB branch is responsible for capturing the appearance features, texture changes, and scene background of the video; the optical flow branch is responsible for extracting the temporal dynamic information and directional motion patterns during the action process; and the pose branch is responsible for extracting human key points and limb movement structure features.

[0045] S12. When training the initial multimodal teacher network for the dth time, some samples are selected from the target domain dataset and the source domain dataset as training samples for the dth time. After masking each training sample for the dth time, they are input into the multimodal teacher network obtained from the (d-1)th training. The RGB branch, optical flow branch, and pose branch of the multimodal teacher network obtained from the (d-1)th training respectively extract features, reconstruct features, and perform classification prediction on the RGB frame sequence, optical flow features, and hand joint pose map of each training sample after masking. d is a positive integer greater than 1. When d is 1, it means that the first training is performed. When d is 1, the multimodal teacher network obtained from the (d-1)th training is the initial multimodal teacher network.

[0046] Here, the Tube Masking operation can be used to perform masking, and the masking ratio during masking is... For example, using the Tube Masking operation on the RGB frame sequence of each training sample The mask ratio is After masking, the RGB frame sequence of the training sample after masking can be obtained. .

[0047] Specifically, the RGB encoder in the RGB branch processes the input mask to produce the RGB frame sequence of each training sample. Feature extraction is performed to obtain the feature representation of the RGB modes. and represent the features of the RGB mode. The inputs are given to the RGB decoder and RGB classifier respectively. The RGB decoder then uses the feature representations of the RGB modes. Reconstruct the input to obtain the reconstructed RGB frame sequence. Furthermore, the RGB classifier is based on the feature representation of the RGB modes. Obtain the classification score for the RGB modality. The processing of optical flow features for each training sample after input masking by the optical flow branch, and the processing of hand joint pose maps for each training sample after input masking by the pose branch, are the same as those by the RGB branch, and will not be repeated here. The difference is that the optical flow encoder outputs the feature representation of the optical flow mode, the pose encoder outputs the feature representation of the pose mode, the optical flow decoder outputs the reconstructed optical flow features, the pose decoder outputs the reconstructed hand joint pose map, and the optical flow classifier outputs the classification score of the optical flow mode, and the pose classifier outputs the classification score of the pose mode.

[0048] S13. Based on the data generated by the multimodal teacher network obtained in the (d-1)th training iteration when performing feature extraction, feature reconstruction, and classification prediction on the training samples of the dth iteration, as well as the labels of the labeled training samples in the dth training iteration, calculate the pre-training loss of the dth iteration. Perform backpropagation based on the pre-training loss of the dth iteration to update the parameters of the RGB encoder, optical flow encoder, and pose encoder in the multimodal teacher network obtained in the (d-1)th training iteration, and obtain the multimodal teacher network obtained in the dth training iteration.

[0049] Specifically, the pre-training loss for the d-th iteration. The expression is: ,in, This is the source domain data reconstruction loss for the d-th iteration. It is the target domain data reconstruction loss of the dth iteration. It is the data classification loss for the d-th iteration. It is a balancing hyperparameter.

[0050] This is the sum of the reconstruction losses for the RGB modes, optical flow modes, and pose modes of the samples from the source domain dataset in the d-th training sample. It should be noted that the reconstruction loss for the RGB modes of the samples from the source domain dataset in the d-th training sample is calculated by inputting both the masked RGB sequence and the reconstructed RGB frame sequence of the samples from the source domain dataset in the d-th training sample into the Mean Squared Error (MSE) loss function. The calculation methods for the reconstruction losses for the optical flow modes and pose modes of the samples from the source domain dataset in the d-th training sample are the same as those for the RGB mode reconstruction loss. Similarly, It is the sum of the reconstruction losses of the RGB modes, optical flow modes, and pose modes of the samples from the target domain dataset in the d-th training sample. and The calculation principle is the same, so it will not be repeated here. This is the sum of the classification losses for the RGB modes, optical flow modes, and pose modes of the samples from the source domain dataset in the d-th training sample. It should be noted that the classification loss for the RGB modes of the samples from the source domain dataset in the d-th training sample is calculated by inputting the labels of the samples from the source domain dataset in the d-th training sample and the classification scores of the RGB modes after processing by the Softmax function into the cross-entropy loss function. The classification losses for the optical flow modes and pose modes of the samples from the source domain dataset in the d-th training sample are calculated using the same method as the classification loss for the RGB modes, and will not be repeated here.

[0051] S14. Set d=d+1 and return to S12 above to continue execution until the obtained pre-training loss is less than the preset loss or the number of iterations reaches the preset number. Then, end the training of the multimodal teacher network and obtain the trained multimodal teacher network.

[0052] In some embodiments, step S3 is implemented through steps S31 to S36:

[0053] S31. During the c-th training iteration of the initial unimodal student network, obtain from the target domain dataset... The first sample is used as the training sample for this training, and the training sample is input into the pre-trained multimodal teacher network to generate the first multimodal feature and the multimodal classification score of each training sample. At the same time, the RGB frame sequence in the training sample is input into the unimodal student network obtained from the (c-1)th training to generate the second multimodal feature. c is a positive integer greater than 1. When c is 1, it means that the first training is performed. When c is 1, the unimodal student network obtained from the (c-1)th training is the initial unimodal student network.

[0054] Here, after inputting the training samples into the pre-trained multimodal teacher network, the RGB encoder in the pre-trained multimodal teacher network generates RGB encoded features for each training sample, the optical flow encoder generates optical flow encoded features for each training sample, and the pose encoder generates pose encoded features for each training sample. The total RGB encoded features formed by the RGB encoded features of all training samples are called the first RGB encoded features, the total features formed by the optical flow encoded features of all training samples are called the first optical flow encoded features, and the total features formed by the pose encoded features of all training samples are called the first pose encoded features. The multimodal features formed by the first RGB encoded features, the first optical flow encoded features, and the first pose encoded features are called the aforementioned first multimodal features. Accordingly, after the RGB encoding features of each training sample are input into the RGB classifier in the trained multimodal teacher network, the RGB classifier outputs the classification score of the RGB mode for each training sample; after the optical flow encoding features of each training sample are input into the optical flow classifier in the trained multimodal teacher network, the optical flow classifier outputs the classification score of the optical flow mode for each training sample; after the pose encoding features of each training sample are input into the pose classifier in the trained multimodal teacher network, the pose classifier outputs the classification score of the pose mode for each training sample.

[0055] Similarly, after inputting the current training samples into the single-modal student network obtained from the (c-1)th training iteration, the RGB encoder in the (c-1)th training iteration generates RGB encoded features for each current training sample. These RGB encoded features are then input into the subsequent multilayer perceptron. The multilayer perceptron processes these RGB encoded features, projecting optical flow encoded features and pose encoded features for each current training sample. The total features formed by the RGB encoded features of all current training samples generated by the (c-1)th training iteration are called the second RGB encoded features. The total features formed by the projected optical flow encoded features of all current training samples are called the second optical flow encoded features. The total features formed by the projected pose encoded features of all current training samples are called the second pose encoded features. The multimodal features formed by the second RGB encoded features, the second optical flow encoded features, and the second pose encoded features are called the aforementioned second multimodal features.

[0056] It should be noted that the multilayer perceptron used in this invention includes a linear layer, a GELU activation function, and a normalized layer. Furthermore, this multilayer perceptron is an existing multilayer perceptron, so it will not be described in detail here.

[0057] S32. Perform a two-dimensional Haar wavelet transform on the first multimodal feature to obtain multi-scale frequency components. Adjust the multi-scale frequency components based on the learnable scaling factor and then perform an inverse wavelet transform to obtain global frequency domain enhancement features. At the same time, perform spatial domain processing on the first multimodal feature to obtain local spatial domain enhancement features. Determine the multimodal frequency domain reconstruction features based on the global frequency domain enhancement features and the local spatial domain enhancement features.

[0058] This invention employs four sets of convolutional filters to implement a two-dimensional Haar wavelet transform. These four sets of convolutional filters are filter banks for the Haar wavelet transform, also known as wavelet bases. Each set of convolutional filters consists of a filter and a convolutional layer with a kernel size of 2×2 and a stride of 2. The processing principle of each set of convolutional filters for the input features is as follows: first, the input features are filtered using the filter, and then the filtered features are convolved using the convolutional layer to obtain the frequency components of the input features at one scale. Specifically, the filter in the first set of convolutional filters is called a low-pass filter. The filter in the second group of convolutional filters is called a high-pass filter. The filter in the third group of convolutional filters is called a high-pass filter. The filter in the fourth group of convolutional filters is called a high-pass filter. ,in, Used to capture low-frequency semantic information globally; , , Used to capture high-frequency detail information in the horizontal, vertical, and diagonal directions respectively.

[0059] Specifically, the expressions for these four filters are as follows:

[0060] .

[0061] Specifically, step S32 is implemented through steps S321 to S325:

[0062] S321. Perform two-dimensional Haar wavelet transform on the first RGB coding feature, the first optical flow coding feature, and the first attitude coding feature respectively to obtain the four frequency band components of each of the first RGB coding feature, the first optical flow coding feature, and the first attitude coding feature.

[0063] For example, the process of performing two-dimensional Haar wavelet transform on the first RGB encoded feature, the first optical flow encoded feature, or the first attitude encoded feature using four sets of convolutional filters to obtain four frequency band components is expressed by the following formula:

[0064] ;

[0065] in, This represents the first RGB coding feature, the first optical flow coding feature, or the first attitude coding feature. represent The multi-scale frequency components are composed of four frequency band components. This represents a convolution operation in a convolutional layer with a kernel size of 2×2 and a stride of 2. , , and They represent Four different frequency band components.

[0066] S322. Four learnable scaling factors are used to adaptively adjust the four frequency band components respectively to obtain the scaled four frequency band components.

[0067] For example, the scaled multi-scale frequency components are composed of the scaled four frequency band components of the first RGB encoding feature, the first optical flow encoding feature, or the first attitude encoding feature. The expression is as follows:

[0068] ;

[0069] in, represent The first frequency band component, represent The second frequency band component, represent The third frequency band component, represent The fourth frequency band component, , , and This represents four learnable scaling factors.

[0070] S323. Perform inverse wavelet transform on the four frequency band components of the first RGB coding feature, the first optical flow coding feature, and the first attitude coding feature after scaling, to obtain the globally frequency domain enhanced RGB coding feature, the globally frequency domain enhanced optical flow coding feature, and the globally frequency domain enhanced attitude coding feature.

[0071] For example, the expressions for globally frequency domain enhanced RGB coding features, globally frequency domain enhanced optical flow coding features, or globally frequency domain enhanced attitude coding features are as follows:

[0072] ;

[0073] in, This represents globally frequency domain enhanced RGB coding features, globally frequency domain enhanced optical flow coding features, or globally frequency domain enhanced attitude coding features. This represents the transpose convolution operation, used to implement the inverse transform. represent , represent , represent , represent .

[0074] Here, the scaled multi-scale frequency components, which are composed of the scaled four frequency band components of the first RGB coding feature, the first optical flow coding feature, or the first attitude coding feature, can be reconstructed back into the spatial domain by inverse wavelet transform.

[0075] S324. After performing convolution processing on the first RGB coding feature, the first optical flow coding feature, and the first pose coding feature, an adaptive adjustment is performed using a learnable scaling factor to obtain locally spatially enhanced RGB coding features, locally spatially enhanced optical flow coding features, and locally spatially enhanced pose coding features.

[0076] For example, the expressions for locally spatially enhanced RGB coding features, locally spatially enhanced optical flow coding features, or locally spatially enhanced pose coding features are as follows:

[0077] ;

[0078] in, These represent RGB coding features with local spatial enhancement, optical flow coding features with local spatial enhancement, or attitude coding features with local spatial enhancement. These are the learnable parameters of a convolutional layer with a kernel size of 1×1. This refers to the convolution operation of a convolutional layer with a kernel size of 1×1. It is a learnable scaling factor.

[0079] S325. Based on the globally frequency domain enhanced RGB coding features, the locally spatial domain enhanced RGB coding features, the globally frequency domain enhanced optical flow coding features, the locally spatial domain enhanced optical flow coding features, the globally frequency domain enhanced attitude coding features, and the locally spatial domain enhanced attitude coding features, multimodal frequency domain reconstruction features are obtained; wherein, the multimodal frequency domain reconstruction features include: frequency domain reconstructed RGB coding features, frequency domain reconstructed optical flow coding features, and frequency domain reconstructed attitude coding features.

[0080] Specifically, the RGB coding features enhanced in the global frequency domain are added to the RGB coding features enhanced in the local spatial domain to obtain the RGB coding features after frequency domain reconstruction; the optical flow coding features enhanced in the global frequency domain are added to the optical flow coding features enhanced in the local spatial domain to obtain the optical flow coding features after frequency domain reconstruction; and the attitude coding features enhanced in the global frequency domain are added to the attitude coding features enhanced in the local spatial domain to obtain the attitude coding features after frequency domain reconstruction.

[0081] For example, the expressions for the frequency-domain reconstructed RGB coding features, frequency-domain reconstructed optical flow coding features, or frequency-domain reconstructed attitude coding features are as follows:

[0082] ;

[0083] in, This represents the RGB coding features, optical flow coding features, or attitude coding features after frequency domain reconstruction.

[0084] During the distillation process, student features are aligned with teacher frequency domain features, and the frequency-aware processing method described above is introduced. This enables the student network to not only mimic the teacher network's output in the spatial domain, but also to learn the energy distribution pattern of the teacher network in the frequency domain, thereby significantly improving the student network's representation ability and cross-modal learning efficiency.

[0085] S33. Based on the multimodal frequency domain reconstruction features and the second multimodal features, determine the multimodal relationship difference score for each training sample in this test, and based on the multimodal classification score for each training sample in this test, determine the multimodal classification uncertainty score for each training sample in this test.

[0086] Here, the multimodal relationship difference score refers to the relationship difference score of RGB mode, the relationship difference score of optical flow mode, and the relationship difference score of attitude mode; the multimodal classification uncertainty score refers to the classification uncertainty score of RGB mode, the classification uncertainty score of optical flow mode, and the classification uncertainty score of attitude mode.

[0087] Specifically, step S33 is implemented through S331~S333:

[0088] S331. Calculate the cosine similarity between the RGB coding features, optical flow coding features, and attitude coding features of each pair of training samples in the frequency domain reconstructed RGB coding features, frequency domain reconstructed optical flow coding features, and frequency domain reconstructed attitude coding features, respectively, to obtain the first RGB relationship matrix, the first optical flow relationship matrix, and the first attitude relationship matrix. Also, calculate the cosine similarity between the RGB coding features, optical flow coding features, and attitude coding features of each pair of training samples in the second RGB coding features, the second optical flow coding features, and the second attitude coding features, respectively, to obtain the second RGB relationship matrix, the second optical flow relationship matrix, and the second attitude relationship matrix.

[0089] Because the training samples this time are Let there be a first sample, and assume that the dimension of each first sample input into the trained multimodal teacher network is... Then the RGB coding features, optical flow coding features, and attitude coding features reconstructed in the frequency domain are all... OK The feature matrix of the column; because the input to the single-modal student network is The RGB frame sequence of each first sample in the first sample, therefore assuming the dimension of the RGB frame sequence of each first sample input to the unimodal student network is... Then the second RGB encoding feature, the second optical flow encoding feature, and the second pose encoding feature are all OK The feature matrices of the columns. The first RGB relation matrix, the first optical flow relation matrix, the first pose relation matrix, as well as the second RGB relation matrix, the second optical flow relation matrix, and the second pose relation matrix are all similarity matrices.

[0090] Specifically, the first RGB relationship matrix refers to OK The second RGB relation matrix is ​​a matrix formed by the similarity between every two rows of features in the frequency domain reconstructed RGB encoded features of the column. OK The matrix is ​​formed by the similarity between every two rows of features in the second RGB encoded features of the column; the first optical flow relation matrix, first pose relation matrix, second optical flow relation matrix, and second pose relation matrix are similarly constructed and will not be repeated here. Therefore, the first RGB relation matrix, first optical flow relation matrix, first pose relation matrix, second RGB relation matrix, second optical flow relation matrix, and second pose relation matrix are all... OK A matrix of columns, and the first column in each relation matrix. Line number The elements of the column represent the current number. The training sample and the current training sample Cosine similarity of corresponding features between training samples and The values ​​are all from 1 to .

[0091] S332. Based on the first RGB relation matrix, the second RGB relation matrix, the first optical flow relation matrix, the second optical flow relation matrix, the first attitude relation matrix, and the second attitude relation matrix, the mean square error is calculated to obtain the relationship difference score of RGB mode, optical flow mode, and attitude mode for each training sample.

[0092] For example, the first time this time The expression for the difference score of the relationship between RGB modes, optical flow modes, or pose modes of a training sample is as follows:

[0093] ;

[0094] in, The value ranges from 1 to , This indicates the first The relationship difference scores of RGB mode, optical flow mode, or pose mode of each training sample, where, when This indicates the first When scoring the relationship difference between the RGB modes of each training sample, Represents the first RGB relation matrix. Line number Column elements, Represents the second RGB relation matrix. Line number The elements of the column; when This indicates the first When scoring the difference in optical flow modes among training samples... Represents the first optical flow relation matrix. Line number Column elements, The second optical flow relation matrix represents the first... Line number The elements of the column; when This indicates the first When scoring the relationship differences in pose modalities among training samples, Represents the first attitude relation matrix. Line number Column elements, Represents the second attitude relation matrix. Line number The elements of the column. When the current [number]th [element]... The higher the difference score of the relationship between the RGB mode, optical flow mode, or pose mode of a training sample, the worse the consistency between the trained multimodal teacher network and the single-modal student network obtained from the (c-1)th training in the representation of the training sample in the RGB mode, optical flow mode, or pose mode.

[0095] S333. Normalize the classification scores of RGB mode, optical flow mode and attitude mode for each training sample, and calculate the information entropy of RGB mode, optical flow mode and attitude mode for each training sample based on the normalization results, and use the information entropy as the classification uncertainty score.

[0096] For example, the first time this time The expression for the classification uncertainty score of the RGB mode, optical flow mode, or pose mode of a training sample is as follows:

[0097] ;

[0098] ;

[0099] in, This indicates the first The classification uncertainty score of the RGB mode, optical flow mode, or attitude mode of each training sample. This indicates the first The normalized classification score of the RGB mode, optical flow mode, or pose mode of each training sample. This indicates the first The information entropy of the RGB mode, optical flow mode, or attitude mode of each training sample. This represents the softmax function used for normalization. This represents the logarithmic function. The higher the classification uncertainty score of a training sample in RGB mode, optical flow mode, or pose mode, the more uncertain the trained multimodal teacher network is in classifying that training sample in RGB mode, optical flow mode, or pose mode.

[0100] S34. Using the multimodal classification uncertainty score of each training sample in this iteration, the multimodal relationship difference score of each training sample in this iteration is weighted to obtain the multimodal boundary enhancement distillation loss for the cth iteration.

[0101] For example, the expression for the multimodal boundary-enhanced distillation loss at the c-th time is as follows:

[0102] ;

[0103] ;

[0104] in, This represents the multimodal boundary-enhanced distillation loss at the c-th time. This represents the RGB modal distillation loss in the c-th iteration. This represents the optical flow mode distillation loss at the c-th order. This represents the attitude mode distillation loss at the c-th iteration. This represents the RGB mode distillation loss of the c-th iteration, or the optical flow mode distillation loss of the c-th iteration, or the attitude mode distillation loss of the c-th iteration.

[0105] S35. Based on the multimodal boundary enhancement distillation loss of the cth training iteration, the gradient is calculated using the backpropagation algorithm to update the single-modal student network obtained in the (c-1)th training iteration, thus obtaining the single-modal student network obtained in the cth training iteration.

[0106] S36. After setting c=c+1, return to S31 above and continue execution until the cutoff condition is met to end the distillation training of the initial single-modal student network and obtain the trained target RGB encoder.

[0107] For example, when the calculated multimodal boundary enhancement distillation loss is less than the preset loss value, or when the training count reaches the preset number, it indicates that the cutoff condition has been met. In this case, the RGB encoder obtained from the last training is used as the trained target RGB encoder.

[0108] In some embodiments, step S6 is implemented through steps S61 to S65:

[0109] S61. During the e-th training iteration of the new initial unimodal student network, data is obtained from the support set. The single-modal student network generated from the first sample input during the (e-1)th training iteration is used to generate... The classification score of each first sample in the first sample; e is a positive integer greater than 1. When e is 1, it indicates that the first training is performed, and when e is 1, the unimodal student network obtained from the (e-1)th training is the new initial unimodal student network. It is less than or equal to Positive integers.

[0110] S62, will The classification score of each first sample in the first sample is normalized, and then... The normalized classification score of the first sample and The label of the first sample is input into the cross-entropy loss function to calculate the classification loss for the e-th iteration.

[0111] S63. Perform backpropagation based on the classification loss of the eth iteration to update the parameters of the classifier in the unimodal student network obtained from the (e-1)th training iteration, and obtain the unimodal student network obtained from the eth training iteration.

[0112] S64. Retrieve from the query set The single-modal student network generated from the first sample input during the e-th training iteration is used to generate... The classification score of each first sample in the first sample will be The classification score of each first sample in the first sample is normalized to obtain... The prediction result of the action category of each first sample in the first sample, according to The action category prediction results for the first sample and The label of the first sample is used to determine the classification accuracy of the unimodal student network obtained from the e-th training. It is less than or equal to Positive integers.

[0113] S65. When the classification accuracy of the unimodal student network obtained in the e-th training session does not reach the preset accuracy, and the classification loss does not reach the preset classification loss threshold or the number of training sessions does not reach the set number of training sessions threshold, let e = e + 1 and return to S61 above to continue execution until the classification accuracy reaches the preset accuracy, the classification loss reaches the preset classification loss threshold, or the number of training sessions reaches the set number of training sessions threshold, then end the training and obtain the trained unimodal student network.

[0114] Through the above training process, the unimodal student network can learn the knowledge representation of the pre-trained multimodal teacher network in the multimodal space when only RGB frame sequences are input, thus realizing the action recognition task under cross-domain and small sample conditions.

[0115] As described above, this invention constructs a distillation framework of "multimodal teacher-unimodal student" and integrates unlabeled data from the source and target domains for joint pre-training. Compared with existing cross-domain action recognition methods, it solves the problems of incomplete target domain modalities and scarce annotations, thus improving the effectiveness of cross-domain knowledge transfer. Furthermore, this invention innovatively introduces wavelet transform for frequency domain decomposition and adaptive weighting, enabling the model to simultaneously capture and transmit global semantic information and local detailed features, thereby generating more discriminative frequency domain enhanced features. In addition, by combining the differences in teacher-student relationship features with the uncertainty of teacher classification through a boundary-enhanced distillation method, the training process adaptively focuses on difficult samples, significantly optimizing distillation efficiency and ultimately improving the model's generalization ability and recognition accuracy in small-sample scenarios in the target domain.

[0116] Compared with the prior art, the present invention has the following beneficial effects:

[0117] 1) This invention achieves efficient multimodal to unimodal knowledge transfer, effectively addressing the challenges of scarce target domain data and incomplete modalities. By constructing a "multimodal teacher-unimodal student" distillation architecture and fusing labeled source domain data with unlabeled target domain data for joint pre-training, this invention successfully transfers rich multimodal information (RGB frames, optical flow, hand pose) from the source domain to a student network using only RGB frame sequence input.

[0118] 2) This invention introduces a frequency domain awareness mechanism, enhancing feature discriminative power and cross-modal representation capabilities. By performing a two-dimensional Haar wavelet transform on the multimodal features of the teacher network, this invention decomposes them into multiple frequency sub-bands and introduces a learnable scaling factor to adaptively weight different frequency components. This mechanism enables the student network to simultaneously learn global semantics (low frequency) and local details (high frequency), achieving more comprehensive and discriminative feature transfer in knowledge distillation, significantly improving the representation quality of the student network in the target domain.

[0119] 3) This invention proposes a boundary-enhanced distillation strategy, which improves the model's ability to discriminate difficult samples and its robustness across domains. This invention constructs a relationship matrix between teacher and student features and combines this with the classification uncertainty output by the teacher model to weight the differences in samples. This mechanism enables the training process to adaptively focus on uncertain samples near the classification boundary, optimizing distillation efficiency and the model's discrimination boundary, effectively enhancing the model's generalization ability and recognition accuracy in cross-domain, small-sample scenarios.

[0120] To verify the effectiveness of the method proposed in this invention, simulation experiments were conducted and compared with various existing action recognition models to further illustrate the advantages of this invention in recognition performance, generalization effect, and computational efficiency in cross-domain and low-sample scenarios.

[0121] 1. Simulation experimental conditions:

[0122] Operating system: Ubuntu 22.04, Python 3.9

[0123] Experimental platform: PyTorch-2.0.1

[0124] Graphics card: NVIDIA RTX 3090

[0125] Memory: 48GB

[0126] 2. Simulation Experiment Content

[0127] Simulation Experiment 1: Performance Evaluation of Cross-Domain Small Sample Action Recognition. It should be noted that all experiments below were conducted under a unified experimental environment. The dataset used is a representative cross-domain video action recognition dataset, and all benchmark methods and the method proposed in this invention under the same sample partitioning conditions were trained and tested.

[0128] (1) First-person video large dataset (called Ego4D): used as source domain dataset for pre-training of multimodal teacher networks and cross-domain knowledge distillation;

[0129] (2) Kitchen Scene First-Person View Action Recognition Dataset (referred to as EPIC): This is a kitchen scene action recognition dataset containing a large number of first-person view operation behaviors;

[0130] (3) Daily assembly operation action recognition dataset (referred to as MECCANO): It is a daily assembly scene action dataset, which includes complex object operations and lighting changes;

[0131] (4) Wearable device perspective human interaction and motion control dataset (referred to as WEAR): It is a wearable device perspective dataset, which mainly includes human interaction and motion control tasks.

[0132] The experiment adopted the standard 5-way 1-shot and 5-way 5-shot settings, that is, 5 categories were randomly selected for each task, and 1 or 5 labeled samples were used as the support set in each category, and the remaining samples were used as the query set to test the performance.

[0133] Table 1

[0134]

[0135] Table 1 compares the performance (in percentage points) of the baseline method and the method proposed in this invention on different datasets, where "1-shot" represents 5-way 1-shot and "5-shot" represents 5-way 5-shot. As shown in Table 1, the experimental results compare the performance of the baseline method and the method proposed in this invention in different cross-domain few-shot action recognition tasks. The experimental results show that the method proposed in this invention significantly outperforms the baseline method in both 1-shot and 5-shot few-shot settings. Especially in the first-person view kitchen scene action recognition task, the method proposed in this invention improves performance by 12.34% and 2.87% compared to the baseline method in the 1-shot and 5-shot settings, respectively; in the daily assembly scene action recognition task, it improves performance by 7.10% and 8.31%, respectively; and in the wearable device view action recognition task, it improves performance by 6.46% and 2.79%, respectively.

[0136] Based on the overall results of different test scenarios, the method of this invention achieved an average performance improvement of approximately 6.17% and 5.77% under 1-shot and 5-shot settings, respectively, which verifies that the multimodal frequency domain distillation strategy proposed in this invention can effectively enhance the action recognition capability and generalization performance under cross-domain and small sample conditions.

[0137] Simulation Experiment 2: Ablation Experiment of the Main Methods of the Small-Sample Cross-Domain Action Recognition Method Based on Multimodal Distillation. All experiments were conducted under the same experimental configuration and data conditions. To analyze the role of each core method in this invention, ablation comparison experiments were performed by progressively removing different methods.

[0138] Table 2

[0139]

[0140] Table 2 shows the experimental results of the main ablation experiments of the few-shot cross-domain action recognition method based on multimodal distillation. As shown in Table 2, the experimental results provide a performance comparison of the proposed method and its three variants on two cross-domain few-shot tasks: 1-shot and 5-shot. The experimental results are as follows: (1) Removing frequency domain perception leads to a significant performance decrease; the accuracy of 1-shot decreases by 5.22%, and the accuracy of 5-shot decreases by 1.63%, indicating that frequency domain enhancement distillation can effectively improve the multi-scale modeling ability of teacher feature representation. (2) Removing edge enhancement also causes a loss of stability performance; compared with the complete model, 1-shot and 5-shot decrease by 2.66% and 1.09% respectively, verifying the important role of the uncertainty relation distillation mechanism in boundary sample modeling. (3) Removing the few-shot feature classifier also results in performance regression; 1-shot and 5-shot decrease by 0.92% and 0.52% respectively, indicating that the feature classifier optimized for few shots can further improve the cross-domain generalization ability of the model.

[0141] Simulation Experiment 3: The Impact of Different Teacher Modal Combinations on Distillation Performance. All experiments below were conducted under the same experimental configuration and data conditions. To analyze the effect of different modal combinations in a multimodal teacher network on the distillation performance of cross-domain few-shot action recognition, this experiment tested the performance of different modal combinations in 1-shot and 5-shot tasks.

[0142] Table 3

[0143]

[0144] Table 3 shows the results of the influence of different teacher modality combinations on distillation performance. As shown in Table 3, the experimental results provide a comparison of the performance of the proposed method and its different teacher modality combinations. The experimental results are as follows: (1) Adding the optical flow modality significantly improves performance, with improvements of approximately 4.1% and 2.26% for 1-shot and 5-shot, respectively, indicating that optical flow can effectively capture the dynamic features of actions and provide important motion information for the distillation process. (2) Adding the posture modality also brings stable gains, with improvements of approximately 3.7% and 1.11% for 1-shot and 5-shot, respectively, enhancing the teacher network's spatial understanding of actions. (3) Trimodal fusion achieves the highest performance. Compared with single-modality or dual-modality, the trimodal combination achieves the best accuracy in both 1-shot and 5-shot, indicating that posture features and motion information are complementary and can provide richer spatiotemporal knowledge for the distillation process, thereby improving the performance of cross-domain small sample action recognition.

[0145] Simulation Experiment 4: Evaluation of the computational complexity and inference efficiency of a small-sample cross-domain action recognition method based on multimodal distillation. It should be noted that all the following experiments were conducted in the same experimental environment. To comprehensively evaluate the performance of the proposed method in terms of model performance and computational efficiency, the differences between different methods in terms of parameter quantity, computational complexity, and inference speed were compared.

[0146] Table 4

[0147]

[0148] Table 4 compares the computational complexity of small-sample cross-domain action recognition methods based on multimodal distillation. GFLOPs stands for Giga Floating Point Operations Per Second, referring to the number of floating-point operations that can be performed per second, measured in billions. FPS stands for Frames Per Second, representing the frame rate, and is an important indicator for measuring the performance of deep learning models. As shown in Table 4, the method of this invention maintains leading recognition accuracy while having only 23.04M parameters, reducing GFLOPs to 56.33, achieving an inference speed of 71.91 FPS, and a single-frame inference time of only 13.91 ms, significantly outperforming the baseline method.

[0149] In summary, the simulation experiments above show that:

[0150] 1. This invention significantly outperforms existing cross-modal distillation algorithms in all three target domains, with stable improvement and strong generalization ability;

[0151] 2. Each core method component in this invention contributes positively to performance improvement, and the synergistic effect of the three is optimal;

[0152] 3. Multimodal fusion distillation can significantly enhance the effective transfer of teachers' knowledge;

[0153] 4. The model has a lightweight structure and fast inference speed, making it suitable for embedded devices and real-time recognition tasks.

[0154] Therefore, this invention combines the advantages of accuracy, robustness, and real-time performance in the field of cross-domain few-sample self-perspective action recognition.

[0155] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0156] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0157] In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0158] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation, characterized in that, include: Obtain the RGB frame sequence of the hand action video in the target domain, wherein the hand action video in the target domain refers to the hand action video in a hand use scenario; The RGB frame sequence is preprocessed to obtain a preprocessed RGB frame sequence; The preprocessed RGB frame sequence is input into the trained unimodal student network to classify the hand movements contained in the RGB frame sequence and obtain a classification score. The trained unimodal student network is trained using a distillation training mechanism of multimodal teacher network and unimodal student network, combined with frequency domain feature enhancement and boundary enhancement distillation. The classification scores are normalized to obtain the action category prediction results; The training method for the pre-trained unimodal student network includes: S1. Using the unlabeled target domain dataset and the labeled source domain dataset, jointly train the initial multimodal teacher network to obtain the trained multimodal teacher network; the target domain dataset contains multiple first samples, each first sample being a multimodal data set consisting of an RGB frame sequence, optical flow features, and hand joint pose map of a hand action video in a first hand use scenario; the source domain dataset contains multiple second samples, each second sample being a multimodal data set consisting of an RGB frame sequence, optical flow features, and hand joint pose map of a hand action video in a second hand use scenario; the first hand use scenario and the second hand use scenario are different scenarios; S2. Construct an initial unimodal student network using the trained multimodal teacher network. The initial unimodal student network includes an initial target RGB encoder and a multilayer perceptron. The output of the initial target RGB encoder is connected to the input of the multilayer perceptron. The initial target RGB encoder is the trained RGB encoder in the trained multimodal teacher network. S3. Using the target domain dataset and the trained multimodal teacher network, the initial unimodal student network is distilled and trained. During each training process, wavelet transform is performed on the multimodal features generated by the trained multimodal teacher network to perform frequency domain decomposition to obtain multi-scale frequency components. The multi-scale frequency components are adjusted based on the learnable scaling factor, and then inverse wavelet transform is performed to obtain global frequency domain enhancement features. The multimodal features generated by the trained multimodal teacher network are processed in the spatial domain to obtain local spatial domain features. Based on the global frequency domain enhancement features, local spatial domain features, and multimodal features generated by the unimodal student network, the relationship difference between the multimodal features of the trained multimodal teacher network and the unimodal student network is calculated. Based on the classification uncertainty of the trained multimodal teacher network, a sample weighting factor is generated to calculate the boundary enhancement distillation loss. The parameters of the target RGB encoder in the unimodal student network are updated by backpropagation. After completing the distillation training of the initial unimodal student network, the trained target RGB encoder is obtained. S4. Construct a new initial unimodal student network using the trained target RGB encoder and the initial classifier; S5. Obtain the labels of the target domain dataset, and construct the support set and query set respectively using a portion of the second samples in the labeled target domain dataset; S6. Use the support set and the query set to train the classifier in the new initial unimodal student network, and after completing the training of the classifier, obtain the trained unimodal student network.

2. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 1, characterized in that, The trained unimodal student network includes: a trained target RGB encoder and a trained classifier; the step of inputting the preprocessed RGB frame sequence into the trained unimodal student network to classify the hand movements contained in the RGB frame sequence and obtain a classification score includes: The preprocessed RGB frame sequence is input into the trained target RGB encoder, which outputs RGB encoded features. The RGB encoded features are input into the trained classifier, and the classification score is output.

3. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 1, characterized in that, S3 includes: S31. During the c-th training iteration of the initial unimodal student network, obtain from the target domain dataset... The first sample is used as the training sample for this training iteration. The training sample is then input into the trained multimodal teacher network to generate the first multimodal feature and the multimodal classification score for each training sample. Simultaneously, the RGB frame sequence from the training sample is input into the unimodal student network obtained from the (c-1)th training iteration to generate the second multimodal feature. c is a positive integer greater than 1. When c is 1, it indicates that the first training iteration is being performed. Furthermore, when c is 1, the unimodal student network obtained from the (c-1)th training iteration is the initial unimodal student network. S32. Perform a two-dimensional Haar wavelet transform on the first multimodal feature to obtain multi-scale frequency components. Adjust the multi-scale frequency components based on the learnable scaling factor and then perform an inverse wavelet transform to obtain global frequency domain enhancement features. At the same time, perform spatial domain processing on the first multimodal feature to obtain local spatial domain enhancement features. Determine the multimodal frequency domain reconstruction features based on the global frequency domain enhancement features and the local spatial domain enhancement features. S33. Based on the multimodal frequency domain reconstruction features and the second multimodal features, determine the multimodal relationship difference score for each training sample in this exercise, and based on the multimodal classification score for each training sample in this exercise, determine the multimodal classification uncertainty score for each training sample in this exercise. S34. Using the multimodal classification uncertainty score of each training sample in this iteration, the multimodal relationship difference score of each training sample in this iteration is weighted to obtain the multimodal boundary enhancement distillation loss for the cth iteration. S35. Based on the multimodal boundary enhancement distillation loss of the cth training iteration, the gradient is calculated using the backpropagation algorithm to update the single-modal student network obtained in the (c-1)th training iteration, thus obtaining the single-modal student network obtained in the cth training iteration. S36. After setting c=c+1, return to S31 above and continue execution until the cutoff condition is met to end the distillation training of the initial single-modal student network and obtain the trained target RGB encoder.

4. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 3, characterized in that, The first multimodal feature includes: a first RGB encoding feature composed of the RGB encoding features of the current training samples, a first optical flow encoding feature composed of the optical flow encoding features of the current training samples, and a first pose encoding feature composed of the pose encoding features of the current training samples; S32 includes: S321. Perform two-dimensional Haar wavelet transform on the first RGB coding feature, the first optical flow coding feature, and the first attitude coding feature respectively to obtain four frequency band components of each of the first RGB coding feature, the first optical flow coding feature, and the first attitude coding feature. S322. Adaptively adjust the four frequency band components using four learnable scaling factors to obtain the scaled four frequency band components. S323. Perform inverse wavelet transform on the four scaled frequency band components of the first RGB coding feature, the first optical flow coding feature, and the first attitude coding feature respectively to obtain globally frequency domain enhanced RGB coding feature, globally frequency domain enhanced optical flow coding feature, and globally frequency domain enhanced attitude coding feature. S324. The first RGB coding feature, the first optical flow coding feature, and the first pose coding feature are respectively convolved and then adaptively adjusted using a learnable scaling factor to obtain locally spatially enhanced RGB coding features, locally spatially enhanced optical flow coding features, and locally spatially enhanced pose coding features. S325. Based on the globally frequency-domain enhanced RGB coding features, the locally spatial-domain enhanced RGB coding features, the globally frequency-domain enhanced optical flow coding features, the locally spatial-domain enhanced optical flow coding features, the globally frequency-domain enhanced attitude coding features, and the locally spatial-domain enhanced attitude coding features, the multimodal frequency-domain reconstruction features are obtained; wherein, the multimodal frequency-domain reconstruction features include: frequency-domain reconstructed RGB coding features, frequency-domain reconstructed optical flow coding features, and frequency-domain reconstructed attitude coding features.

5. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 4, characterized in that, Specifically, S325 includes: The RGB coding features enhanced in the global frequency domain are added to the RGB coding features enhanced in the local spatial domain to obtain the RGB coding features after frequency domain reconstruction. The optical flow coding features enhanced in the global frequency domain are added to the optical flow coding features enhanced in the local spatial domain to obtain the optical flow coding features after frequency domain reconstruction. The attitude coding features enhanced in the global frequency domain are added to the attitude coding features enhanced in the local spatial domain to obtain the attitude coding features reconstructed in the frequency domain.

6. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 4, characterized in that, The second multimodal feature includes: a second RGB encoding feature composed of the RGB encoding features of the RGB frame sequences in the current training sample, a second optical flow encoding feature composed of the optical flow encoding features of the current training sample, and a second pose encoding feature composed of the pose encoding features of the current training sample; the second optical flow encoding feature is formed by projecting the RGB encoding features of the RGB frame sequences in the current training sample onto the multilayer perceptron; the second pose encoding feature is formed by projecting the RGB encoding features of the RGB frame sequences in the current training sample onto the multilayer perceptron; the multimodal relationship difference score refers to the relationship difference score between the RGB mode, the optical flow mode, and the pose mode; the multimodal classification uncertainty score refers to the classification uncertainty score between the RGB mode, the optical flow mode, and the pose mode; the multimodal classification score refers to the classification score between the RGB mode, the optical flow mode, and the pose mode; S33 includes: S331. Calculate the cosine similarity between the RGB coding features, optical flow coding features, and attitude coding features of each pair of training samples in the frequency domain reconstructed RGB coding features, frequency domain reconstructed optical flow coding features, and frequency domain reconstructed attitude coding features, respectively, to obtain a first RGB relationship matrix, a first optical flow relationship matrix, and a first attitude relationship matrix. Also, calculate the cosine similarity between the RGB coding features, optical flow coding features, and attitude coding features of each pair of training samples in the second RGB coding features, second optical flow coding features, and second attitude coding features, respectively, to obtain a second RGB relationship matrix, a second optical flow relationship matrix, and a second attitude relationship matrix. S332. Calculate the mean square error based on the first RGB relationship matrix, the second RGB relationship matrix, the first optical flow relationship matrix, the second optical flow relationship matrix, the first attitude relationship matrix, and the second attitude relationship matrix to obtain the relationship difference score of RGB mode, optical flow mode, and attitude mode for each training sample in this exercise. S333. Normalize the classification scores of RGB mode, optical flow mode and attitude mode for each training sample, and calculate the information entropy of RGB mode, optical flow mode and attitude mode for each training sample based on the normalization results, and use the information entropy as the classification uncertainty score.

7. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 6, characterized in that, This time The expression for the difference score of the relationship between RGB modes, optical flow modes, or pose modes of a training sample is as follows: ； in, The value ranges from 1 to , This indicates the first The relationship difference score between RGB modes, optical flow modes, or pose modes of each training sample. This represents the first RGB relation matrix, the first optical flow relation matrix, or the first attitude relation matrix. Line 1 Column elements, This represents the second RGB relation matrix, the second optical flow relation matrix, or the second attitude relation matrix. Line 1 The elements of the column.

8. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 3, characterized in that, The expression for the multimodal boundary-enhanced distillation loss at the c-th time is as follows: ； ； in, This represents the multimodal boundary-enhanced distillation loss at the c-th time. This represents the RGB modal distillation loss in the c-th iteration. This represents the optical flow mode distillation loss at the c-th order. This represents the attitude mode distillation loss at the c-th iteration. This represents the RGB mode distillation loss, optical flow mode distillation loss, or attitude mode distillation loss at the c-th time. The value ranges from 1 to , Representing the first training samples, This indicates the first The classification uncertainty score of the RGB mode, optical flow mode, or attitude mode of each training sample. This indicates the first The difference score of the relationship between the RGB modes, optical flow modes, or pose modes of each training sample.

9. The cross-domain small-sample self-action recognition method based on multimodal frequency domain distillation according to claim 1, characterized in that, The multimodal teacher network includes an RGB branch, an optical flow branch, and a pose branch. Each branch contains an encoder, a decoder, and a classifier. The output of the encoder in each branch is connected to the input of the decoder and the classifier, respectively.

Images