A multimodal fusion dance motion recognition method and device with small samples

Abstract

The present invention discloses a multimodal fusion method and device for small-sample dance motion recognition, proposes a method for multimodal data fusion based on video preview guidance, applies the feature learning method of multimodal data to the small-sample dance motion recognition task, and alleviates the recognition difficulty problem caused by too little data in the small sample field by the intersection and fusion of different modal data. Compared with traditional small-sample motion recognition methods, this method uses multimodal information to effectively solve the problem that RGB single-modal data is not robust enough in video motion recognition; in addition, the method of multimodal data fusion guided by video preview strengthens the semantic extraction ability of the model in each modal space. Compared with traditional small-sample motion recognition methods, the method proposed by the present invention has stronger recognition performance, makes more full use of video information features, and is more suitable for small-sample dance motion recognition tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of small sample motion recognition, and specifically relates to a multimodal fusion small sample dance motion recognition method and device. Background Art

[0002] Small-sample action recognition is an information technology that uses computer algorithms to analyze, identify, and classify the actions contained in a small amount of action video sequence data. It is a key research area in computer vision. Small-sample action recognition combines the principles of small-sample learning and action recognition. Its goal is to use a small number of samples to learn a network model that can effectively recognize human actions. Small-sample action recognition is a relatively new and cutting-edge research field that can effectively reduce the cost of collecting high-quality action data. It has great practical application value, especially in areas where large datasets are difficult to collect.

[0003] Small-sample dance movement recognition presents a new task and challenge for motion recognition technology in a specific field. Compared to traditional small-sample motion recognition, dance movement recognition faces more specific challenges, such as the variable lighting conditions of the stage environment and frequent occlusion caused by human movement. These difficulties make dance movement recognition more challenging and the trained model less robust, especially when the training sample size is small. Using small-sample technology to quickly fit dance movements in this unique stage setting is a novel and challenging task.

[0004] The essence of small-sample dance motion recognition is to leverage small-sample motion recognition techniques to address the problem of dance motion recognition in a stage setting. Existing small-sample motion recognition methods can be broadly categorized as model-fine-tuning-based, data-augmentation-based, and transfer-learning-based. Model-fine-tuning-based small-sample motion recognition solutions typically pre-train a model on large-scale data and then fine-tune the parameters of the top layers of the model on a target small-sample dataset to obtain a fine-tuned model suitable for the target task. Data-augmentation-based small-sample motion recognition solutions attempt to improve sample diversity and model performance through data augmentation when data is limited. Transfer-learning-based small-sample motion recognition methods attempt to transfer knowledge learned from models in similar domains to the small-sample learning domain through transfer learning. Transfer-learning-based solutions offer fewer restrictions than other solutions and only require a certain correlation between the source and target domains. Furthermore, various current studies have demonstrated the superior performance of these solutions, making them a hot topic of research in the field of small-sample motion recognition. In the invention, a transfer-learning-based solution is also used as the basic architecture of the model.

[0005] While recent research and exploration of small-sample action recognition has yielded significant progress, the overall trend remains towards smaller datasets and faster learning, with little attention paid to recognizing specific actions in complex environments like stage performance. This has led to a focus on broad, macro-level approaches rather than specialized, detailed ones. Furthermore, due to the relatively recent development of small-sample action recognition, most methods are still borrowing from small-sample image recognition methods. When considering improving recognition performance, greater emphasis has been placed on increasing the accuracy of small-sample action recognition in common scenarios, with less attention paid to enhancing the accuracy of small-sample action recognition in complex, real-world settings. These issues hinder the application of existing methods in complex, real-world environments. For example, they often struggle to perform well in stage performance, characterized by variable lighting conditions and occlusion caused by human movement. Therefore, addressing the challenges of action recognition caused by complex lighting and occlusion, and improving the recognition performance of difficult actions in specific environments, has become a question of both practical and academic significance. Summary of the Invention

[0006] In response to the above-mentioned deficiencies in the prior art, the present invention provides a multimodal fusion small sample dance motion recognition method and device. The invention uses a method based on video preview to guide multimodal data fusion, allowing the model to focus more on information in the video sequence stream that is beneficial to motion recognition.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] According to a first aspect of the present specification, a multimodal fusion dance motion recognition method with a small sample size is provided, comprising the following steps:

[0009] S1, using a multimodal action recognition dataset, pre-trains feature extractors of different modalities, and uses feature extractors of different modalities to extract frame-by-frame feature information corresponding to different modalities;

[0010] S2, using the frame-by-frame feature information of different modalities in step S1, training a multimodal action feature fuser based on video preview guidance;

[0011] S3, based on the feature extractor and the multimodal action feature fusion, builds a multimodal action feature extractor;

[0012] S4, using a multimodal action recognition dataset, trains a small-sample dance action recognition model based on a multimodal action feature extractor;

[0013] S5, using the trained small sample dance movement recognition model to perform small sample dance movement recognition.

[0014] Furthermore, the step S1 is specifically as follows:

[0015] S11, using a multimodal action recognition dataset, extracting a predetermined number of video frames from different modalities, performing feature extraction on the video frames using a ResNet50 network, and obtaining frame-by-frame feature information of different modalities;

[0016] S12, performing global feature extraction on the frame-by-frame feature information of different modalities in step S11 to obtain global coding features of different modalities;

[0017] S13, using the linear layer and the Softmax layer, classifies the global encoding features of different modalities to obtain the video action prediction results;

[0018] S14, using a cross entropy loss function to calculate the loss between the video action prediction result and the true label to train the action recognition model of each modality, and fixing the parameters of the action recognition model after training each modality, and selecting the feature extraction module of the corresponding action recognition model as the feature extractor corresponding to each modality;

[0019] S15 , using the feature extractors of each modality in step S14 to perform feature extraction on the predetermined number of video frames in step S11 , to obtain frame-by-frame feature information of different modalities.

[0020] Furthermore, the step S2 is specifically as follows:

[0021] S21, using the order of the video frames as the position code required by the Transformer network, and using the Transformer network to extract temporal features from the frame-by-frame feature information of different modalities in step S1 to obtain spatiotemporal semantic features of the videos of different modalities, where the different modalities include RGB modalities and non-RGB modalities, and the non-RGB modalities include skeleton heat map modalities, depth modalities, and infrared modalities;

[0022] S22, by performing pooling processing on the spatiotemporal semantic features of the RGB modal video, the RGB modal single-modal action features are obtained;

[0023] S23, add modality embedding and time sequence embedding to the non-RGB modality video spatiotemporal semantic features and splice them together to obtain multimodal spatiotemporal semantic features without RGB;

[0024] S24, using the multimodal Transformer network to fuse multimodal spatiotemporal semantic features without RGB, to obtain preliminary multimodal fused spatiotemporal semantic features without RGB;

[0025] S25, using the Transformer network based on video preview, the spatiotemporal semantic features of the RGB modality used as guidance and the preliminary multimodal fusion features without RGB are fused to obtain the multimodal fusion spatiotemporal semantic features;

[0026] S26, pooling the multimodal fused spatiotemporal semantic features to obtain multimodal action features;

[0027] S27, using the linear layer and the Softmax layer, classifies the RGB mode single-modal action features and multi-modal action features to obtain the video action prediction results;

[0028] S28, using the cross entropy loss function, calculates the loss between the video action prediction results and the true label to train the multimodal action feature fusion.

[0029] Furthermore, the step S4 is specifically as follows:

[0030] S41, using a multimodal action feature extractor to extract features from different modalities of the support set and different modalities of the query set, respectively, to obtain RGB modal unimodal action features and multimodal action features of the corresponding support set and query set, wherein the support set and query set are obtained from a multimodal action recognition dataset;

[0031] S42, concatenating the RGB mode single-modal action features and multi-modal action features in the support set and the query set to obtain the corresponding multi-modal action fusion features of the support set and the query set;

[0032] S43, mapping the multimodal action fusion features of the support set and the query set in step S42 to a lower-dimensional feature space through a linear layer to obtain low-dimensional multimodal action fusion features of the support set and the query set;

[0033] S44, using a similarity function to calculate the similarity of the low-dimensional multimodal action fusion features of the support set and the query set in step S43, to obtain a similarity score for each video in the query set and the support set;

[0034] S45, selecting the similarity score of each video in the query set and the support set as the action prediction result of the query set video;

[0035] S46 uses the cross entropy loss function to calculate the loss between the action prediction results and the true labels of the query set videos, fine-tune the multimodal action feature extractor, and build a small sample dance action recognition model.

[0036] According to a second aspect of this specification, a multimodal fusion small sample dance movement recognition device is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the multimodal fusion small sample dance movement recognition method as described in the first aspect.

[0037] The present invention has the following beneficial effects: The present invention utilizes multimodal information to alleviate the recognition difficulties caused by insufficient data in small sample areas. Compared to traditional small sample motion recognition methods, this method utilizes multimodal information to effectively address the issue of insufficient robustness of RGB single-modal data in video motion recognition. Furthermore, the use of video preview to guide multimodal data fusion enhances the model's semantic extraction capabilities in each modal space. Compared to traditional small sample motion recognition methods, the proposed method has stronger recognition performance, more fully utilizes video information features, is more suitable for small sample dance motion recognition tasks, and improves the accuracy of small sample dance motion recognition in complex scenarios. Furthermore, the present invention addresses the issue of how to complement the learning of different modalities during multimodal fusion. A multimodal fusion method based on video preview guidance is proposed. This method first learns key video features by previewing the overall information of the video. Then, the model uses these key video features to determine the importance of different modalities. Based on the importance of each modality, the model selectively incorporates supplementary information from each modality to supplement and improve the feature information, thereby improving performance. This method promotes the interaction of multimodal information in videos, making it easier for the model to extract motion features from videos, thereby improving the performance of small-sample dance movement recognition models. This method proposes using video preview to guide multimodal data fusion, applying feature learning from multimodal data to the small-sample dance movement recognition task. By integrating and fusing data from different modalities, this invention addresses the existing issues of insufficient utilization of video information features and poor dance movement recognition performance in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flowchart of a multimodal fusion dance motion recognition method using a small sample size is provided as an exemplary embodiment;

[0040] Figure 2 A network structure diagram of a multimodal action feature extractor provided by an exemplary embodiment;

[0041] Figure 3 A network structure diagram of a small sample dance movement recognition model provided by an exemplary embodiment;

[0042] Figure 4A structural diagram of a multimodal fusion small sample dance motion recognition device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0043] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0044] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0045] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0046] To address the inadequate utilization of video information features and poor performance in action recognition in complex scenarios in existing technologies, this paper proposes a small-sample dance action recognition method based on multimodal data fusion guided by video preview. This method, in addition to the conventional small-sample action recognition method trained on single-modal clips, introduces multiple modalities (depth modality, skeleton heat map module, infrared modality) as RGB supplementary information. A clever fusion method is needed to aggregate the information between different modalities. First, the overall information of the video is previewed to learn key video features. The importance of different modalities is then determined based on these key video features. Supplementary information from other modalities is then incorporated to supplement and improve the feature information, thereby improving performance. In stage scenes, the RGB modality exhibits relatively simple recognition, which is believed to reflect its ability to more easily extract the key information required for small-sample action recognition in these scenarios. Therefore, RGB is used as the video preview modality. This method does not assume that only RGB can be used as the preview modality; on the contrary, in some dimly lit scenes, it is believed that using the infrared modality as the preview modality may provide better performance.

[0047] Stage and theater scenes often experience strong lighting due to performance requirements. More specifically, in some settings, such as magic shows and solo opera recitals, a single spotlight may be used to track the performer. In these situations, RGB modality alone can easily blur the scene due to rapidly changing lighting, leading to poor performance. Using modalities like skeleton and optical flow alone is inadequate for situations where the human body is sparse and movement is not obvious. Therefore, a skeleton heatmap module is introduced to enhance the robustness of action recognition. Furthermore, given that action recognition essentially focuses on human movement, and that human motion changes significantly in depth images, highlighting key points in the image, the depth modality is introduced to enhance the salience of key information for action recognition. The primary requirement for a network architecture that leverages multimodal information is to devise an appropriate modal fusion method. It is believed that different modalities have varying degrees of importance in different situations. Therefore, a modal embedding is added to the network to encourage the model to focus more on the impact of modal differences. In a stage setting, RGB mode is the easiest to identify among all modal information and is considered the modality most likely to extract key points from a video. Therefore, a method for guiding multimodal data fusion based on video preview is proposed. This method uses the feature information extracted from the RGB mode to guide the fusion of other modal information. By leveraging prior knowledge of RGB feature information, the model can focus more on information in the video sequence stream that is beneficial for action recognition.

[0048] like Figure 1 As shown, in one embodiment, a multimodal fusion small sample dance movement recognition method is provided, comprising the following steps:

[0049] S1, using a multimodal action recognition dataset, pre-trains feature extractors of different modalities, and uses feature extractors of different modalities to extract frame-by-frame feature information corresponding to different modalities;

[0050] In one embodiment, step S1 is specifically as follows:

[0051] S11, using a multimodal action recognition dataset, extracting a predetermined number of video frames from different modalities, performing feature extraction on the video frames using a ResNet50 network, and obtaining frame-by-frame feature information of different modalities;

[0052] In one embodiment, a method for constructing a multimodal action recognition dataset is as follows: obtain a public dataset containing RGB mode and infrared mode, use an algorithm to generate a corresponding skeleton heat map mode and depth mode, use the action label of the original public dataset as the action label of multimodal action recognition, and thus construct a multimodal action recognition dataset. Among them, the method for generating the skeleton heat map mode is as follows. Use the Alphapose network to extract the 2D human skeleton coordinates corresponding to the RGB mode, and generate a skeleton heat map through the 2D human skeleton coordinates and Gaussian distribution constraints. Skeleton heat map generation formula S kij It can be expressed as:

[0053] S kij =exp(-[(ix k ) 2 +(jy k ) 2 ] / (3*σ 2 ))*p k

[0054]

[0055] Among them, i and j represent the x-axis and y-axis coordinate values ​​of the current calculated pixel point, k represents the kth node, and x k 、y k Represents the x-axis and y-axis coordinate values ​​of the k-th node, p k represents the coordinate confidence of the kth node, S kij It represents the thermal intensity value of the pixel with coordinates i and j under the influence of the kth node after Gaussian distribution constraint, S ij Represents the final thermal intensity value of the pixel point with coordinate values ​​​​i and j. Then, according to the practical meaning of the skeleton coordinates, the skeleton nodes of the adjacent body parts are connected to obtain the thermal map of the human skeleton in a certain frame of the RGB image. The skeleton thermal map mode of all frames is extracted in sequence to obtain the skeleton thermal map mode of the video. The skeleton thermal map mode of all videos is extracted to obtain the skeleton thermal map mode of the dataset. The method for generating the depth mode is as follows. In sequence, the Monodepth2 network is used to extract the depth map of the video frames of the RGB mode video in sequence to obtain the depth mode of the video. The depth mode of all videos is extracted to obtain the depth mode of the dataset.

[0056] In a preferred embodiment, step S11 is specifically as follows:

[0057] For input videos of different modalities, n video frames are uniformly extracted sequentially. For videos with fewer than n frames, zero padding is used to bring the video length up to n frames. Feature extraction is then performed on these n frames using a ResNet50 network to obtain frame-by-frame feature information for each modality. In this model, n is set to 16.

[0058] S12, performing global feature extraction on the frame-by-frame feature information of different modalities in step S11 to obtain global coding features of different modalities;

[0059] In one embodiment, step S12 is specifically as follows:

[0060] Perform a mean operation on the frame-by-frame feature information of the videos of different modes obtained in step S11 to obtain the global encoding features of the videos of different modes. Taking the RGB mode as an example, this step is expressed by the formula:

[0061]

[0062] Among them, v t rgb represents the feature information of the tth frame extracted by the ResNet50 network in S11, O rgb Indicates the global encoding features of the video. The same method is used to obtain the global encoding features of the video for the skeleton heat map modality, depth modality, and infrared modality.

[0063] S13, using the linear layer and the Softmax layer, classifies the global encoding features of different modalities to obtain the video action prediction results;

[0064] In one embodiment, step S13 is specifically as follows:

[0065] The linear layer is used to change the data dimension of the global encoding features of different modalities, and the softmax layer is used to output the video action prediction results of the video. Taking the RGB mode as an example, the classification layer is expressed by the formula:

[0066]

[0067] Among them, f represents the linear layer, O rgb Indicates the global encoding features of the video, θ rgb Indicates the parameters of the linear layer processing RGB mode, This represents the action prediction results for RGB video. Using a linear layer, the input feature dimension is resized to match the number of categories in the dataset. This dimension is used to represent the likelihood of different categories. The Softmax function amplifies the difference in likelihood between categories, and the maximum probability output is used as the predicted category. Skeleton heatmap, depth, and infrared modalities are processed in the same manner, differing only in the input.

[0068] S14, using a cross entropy loss function to calculate the loss between the video action prediction result and the true label to train the action recognition model of each modality, and fixing the parameters of the action recognition model after training each modality, and selecting the feature extraction module of the corresponding action recognition model as the feature extractor corresponding to each modality;

[0069] In one embodiment, step S14 is specifically as follows:

[0070] Using the cross entropy loss function as the loss function, we calculate the loss between the action prediction results and the true labels of each modality video, and train the model's action recognition capabilities for each modality action recognition model. Taking the RGB modality as an example, this part can be expressed as:

[0071]

[0072]

[0073] Among them, y represents the true label of the video, Represents the video action prediction result of RGB mode, l rgb Represents the loss corresponding to the rgb mode, CrossEntropyLoss(.) represents the cross entropy loss function. (i) represents the true probability of the i-th classified action (if the action label is i, the probability is 1, otherwise it is 0); represents the predicted probability of the i-th action classification in the RGB modality. The same method is used to train the corresponding action recognition models for the skeleton heatmap modality, depth modality, and infrared modality. After training, the parameters of the trained action recognition models for each modality are fixed, and the feature extraction module of the corresponding action recognition model is selected as the feature extractor for each modality.

[0074] S15 , using the feature extractors of each modality in step S14 to perform feature extraction on the predetermined number of video frames in step S11 , to obtain frame-by-frame feature information of different modalities.

[0075] In one embodiment, step S15 is specifically as follows:

[0076] Use the feature extractors of each modality in step S14 to extract features from the predetermined number of video frames in step S11 to obtain frame-by-frame feature information of different modalities. Taking the RGB modality as an example, this process can be expressed as follows:

[0077] V rgb =g(M rgb ;p rgb )

[0078] Among them, M rgb represents a predetermined number of video frames of RGB modality input, g(.) represents the RGB modality feature extractor, ρ rgb Represents the parameters corresponding to the RGB modal feature extractor, V rgb Represents the frame-by-frame feature information of the RGB modality video. For the skeleton heat map modality, depth modality, and infrared modality, the same steps are used to extract the frame-by-frame feature information of the corresponding modality, that is, the corresponding V depth 、V skeleton 、V infrared .

[0079] S2, using the frame-by-frame feature information of different modalities in step S1, training a multimodal action feature fusion device guided by video preview;

[0080] In one embodiment, step S2 is specifically as follows:

[0081] S21, using the order of video frames as the position encoding required by the Transformer network, and using the Transformer network to extract temporal features from the frame-by-frame feature information of different modalities in step S1 to obtain spatiotemporal semantic features of videos of different modalities, where the different modalities include RGB modalities and non-RGB modalities, and the non-RGB modalities include skeleton heat map modalities, depth modalities, and infrared modalities;

[0082] In one embodiment, step S21 is specifically as follows:

[0083] Extract the spatiotemporal semantic features of the video from the frame-by-frame feature information of different modalities in step S1. Video is composed of a series of image sequences with a sequential order. However, the model cannot clearly capture this information, so temporal encoding is added to the model in the hope of improving the model's ability to extract action temporal features. Taking the RGB modality as an example, the order of video frames is used as the position encoding, and the encoding of n frames of video is set to 0, 1...n-1 respectively. The Transformer network is used to extract the spatiotemporal semantic features of the RGB modality video from the frame-by-frame feature information obtained in step S1. The formula of the Transformer network is expressed as follows:

[0084]

[0085] Among them, V rgb is the frame-by-frame feature information of RGB mode, t rgb is the corresponding temporal position code, represents the Transformer network, τ rgb is the parameter of the Transformer network, H rgb For the skeleton heat map modality, depth modality, and infrared modality, the same steps are used to extract the video spatiotemporal semantic features of the corresponding modality, that is, the corresponding video spatiotemporal semantic features H are obtained. skeleton 、 H depth 、H infrared .

[0086] S22, by performing pooling processing on the spatiotemporal semantic features of the RGB modal video, the RGB modal single-modal action features are obtained;

[0087] In one embodiment, step S22 is specifically as follows:

[0088] Perform mean pooling on the spatiotemporal semantic features of the RGB mode video obtained by S21 to obtain the RGB mode single-mode action features. The formula is:

[0089] O single =meanpool(H rgb )

[0090] Among them, meanpool represents the mean pooling operation, and the dimension of mean pooling is the time series dimension, H rgb is the spatiotemporal semantic feature of the video in RGB mode, O single Represents unimodal action features in RGB mode.

[0091] S23, add modality embedding and time sequence embedding to the non-RGB modality video spatiotemporal semantic features and splice them together to obtain multimodal spatiotemporal semantic features without RGB;

[0092] In one embodiment, step S23 is specifically as follows:

[0093] To help improve the fusion performance of multimodal fusion, modal coding and temporal coding are added to the spatiotemporal semantic information of skeleton heat maps, depth maps, and infrared maps. Temporal coding is the position coding of each frame. Modal coding means that different modalities will be encoded differently, such as setting the encoding of depth modality to 1, the encoding of infrared modality to 2, and the encoding of skeleton modality to 3. The temporal embedding and modal embedding corresponding to temporal coding and modal coding are learned through linear layers and converted into the same dimension as the video features. Then, the spatiotemporal semantic features, modal embedding, and temporal embedding of non-RGB modal videos are spliced. Expressed in formula:

[0094] X skeleton =H skeleton +M skeleton +T skeleton

[0095] X depth =H depth +M depth +T depth

[0096] X infrared =H infrared +M infrared +T infrared

[0097] X triplet =concatenate(X skeleton , X depth , X infrared )

[0098] Among them, H skeleton 、H depth 、H infrared is the spatiotemporal semantic feature of the video corresponding to the skeleton, depth, and infrared modes, M rgb 、M depth 、M infrared represents the added modality embedding of the corresponding modality, T skeleton 、T depth 、T infrared represents the added temporal embedding of the corresponding modality, X skeleton 、 X depth 、X infrared The corresponding modal splicing modality embedding and temporal sequence embedding are the spatiotemporal semantic features of the video. triplet Indicates that the multimodality does not contain RGB spatiotemporal semantic features.

[0099] S24, using the multimodal Transformer network to fuse multimodal spatiotemporal semantic features without RGB, to obtain preliminary multimodal fused spatiotemporal semantic features without RGB;

[0100] In one embodiment, step S24 is specifically as follows:

[0101] The multimodal Transformer network is used to fuse multimodal spatiotemporal semantic features without RGB, which can be expressed as follows:

[0102]

[0103] Among them, X triplet Indicates that the multimodality does not contain RGB spatiotemporal semantic features, τ triplet represents the parameters of the multimodal Transformer network, H triplet Indicates that the preliminary multimodal does not contain RGB fusion spatiotemporal semantic features, Represents the Transformer network.

[0104] S25, using the Transformer network based on video preview, the spatiotemporal semantic features of the RGB modality used as guidance and the preliminary multimodal fusion features without RGB are fused to obtain the multimodal fusion spatiotemporal semantic features;

[0105] In one embodiment, step S25 is specifically as follows:

[0106] For the multimodal fusion spatiotemporal semantic features obtained in step S25, the spatiotemporal semantic features of the video extracted from the RGB modality are used as a video preview guide to extract multimodal spatiotemporal semantic information. The difference between the Transformer network based on video preview and the general Transformer network is that it uses the spatiotemporal semantic information extracted from the RGB modality as the Q of the self-attention module and uses the trimodal spatiotemporal semantic information as the K and V of the self-attention module, so that when the network extracts multimodal features, it can explore the correlation between the RGB modality and other modalities. Expressed in formula:

[0107]

[0108] in, represents the Transformer network based on video preview, τ multi represents the model parameters of the Transformer network based on video preview, H multi Represents multimodal fusion of spatiotemporal semantic features.

[0109] S26, pooling the multimodal fused spatiotemporal semantic features to obtain multimodal action features;

[0110] In one embodiment, step S26 is specifically as follows:

[0111] Perform mean pooling on the multimodal fusion spatiotemporal semantic features obtained by S25 to obtain multimodal action features. The formula is:

[0112] O multi =meanpool(H multi )

[0113] Among them, meanpool(.) represents the mean pooling operation, and the dimension of mean pooling is the time series dimension, H multi Represents multimodal fusion spatiotemporal semantic features, O multi Represents multimodal action features.

[0114] S27, using the linear layer and the Softmax layer, classifies the RGB mode single-modal action features and multi-modal action features to obtain the video action prediction results;

[0115] In one embodiment, step S27 is specifically as follows: using a linear layer to change the data dimension of the RGB single-modal motion features and multi-modal motion features, and using a softmax layer to predict the video motion results. Taking the RGB mode as an example, the classification layer is expressed by the formula:

[0116]

[0117]

[0118] Among them, f represents the linear layer, O single Represents the unimodal action features of rgb and O multi Represents multimodal action features, θ single represents the parameters of the unimodal action feature of RGB using a linear layer, θ multi Indicates the parameters of using linear layer to process multimodal action features, Softmax indicates the Softmax layer, Represents the video action prediction result of RGB single mode, Represents multimodal video action prediction results.

[0119] S28, using the cross entropy loss function, calculates the loss between the video action prediction results and the true label to train the multimodal action feature fusion.

[0120] In one embodiment, step S28 is specifically as follows:

[0121] The cross entropy loss function is used to calculate the loss of the RGB single-modal video action prediction results, and the multi-modal action feature fusion function is trained to recognize the RGB mode action. The cross entropy loss function is used to calculate the loss of the multi-modal video action prediction results, and the multi-modal action feature fusion function is trained to recognize the action. The formula is:

[0122]

[0123]

[0124] Among them, y represents the true label of the video, Represents the video action prediction result of RGB single mode, Represents the multimodal video action prediction result, L rgb Represents the loss result of RGB single modality in the multimodal action feature fusion, L multi Represents the multimodal loss result in the multimodal action feature fusion.

[0125] like Figure 2 As shown, S3, the feature extractor and the multimodal action feature fusion device are spliced ​​to construct a multimodal action feature extractor;

[0126] like Figure 3 As shown in S4, a small sample dance action recognition model is trained based on a multimodal action feature extractor using a multimodal action recognition dataset;

[0127] In one embodiment, step S4 is specifically as follows:

[0128] S41, using a multimodal action feature extractor to extract features from different modalities of the support set and different modalities of the query set, respectively, to obtain RGB modal unimodal action features and multimodal action features of the corresponding support set and query set, wherein the support set and query set are obtained from a multimodal action recognition dataset;

[0129] In one embodiment, S41 specifically includes:

[0130] For each training session, unimodal and multimodal action features are obtained for the support and query sets. Few-shot task training utilizes a meta-training approach. Unlike traditional action recognition methods, this approach extracts k videos from each of the n categories in the dataset as the support set. A video from each of these n categories is randomly selected as the query set. The training then searches for the video in the support set that is most similar to the query set, and uses its category as the predicted category for the query set.

[0131] S42, concatenating the RGB mode single-modal action features and multi-modal action features in the support set and the query set to obtain the corresponding multi-modal action fusion features of the support set and the query set;

[0132] In one embodiment, step S42 can be expressed as:

[0133]

[0134]

[0135] in, represents the unimodal action feature of the RGB mode of the i-th video in the support set, represents the multimodal action features of the i-th video, Represents the unimodal action feature of the RGB mode of the query set, represents the multimodal action features of the query set, concatenate(.) represents the concatenation operation, support(i) represents the multimodal action fusion features of the i-th video in the support set, and query represents the multimodal action fusion features of the query set.

[0136] S43, mapping the multimodal action fusion features of the support set and the query set in step S42 to a lower-dimensional feature space through a linear layer to obtain low-dimensional multimodal action fusion features of the support set and the query set;

[0137] In one embodiment, step S43 can be expressed as:

[0138] support_low(i)=f(support(i),θ low )

[0139] query_low=f(query, θ low )

[0140] Among them, support(i) represents the multimodal action fusion feature of the i-th video in the support set, query represents the multimodal action fusion feature of the query set, f(.) represents the linear layer, θ low represents the parameters corresponding to the linear layer, support_low(i) represents the low-dimensional multimodal action fusion features of the i-th video in the support set, and query_low represents the low-dimensional multimodal action fusion features of the query set.

[0141] S44, using a similarity function to calculate the similarity of the low-dimensional multimodal action fusion features of the support set and the query set in step S43, to obtain a similarity score for each video in the query set and the support set;

[0142] In one embodiment, S44 can be expressed as:

[0143] Score i =cos(support(i),query),i∈(1,n*k)

[0144] Among them, query represents the query set, cos represents the cos function, Score i represents the similarity score between the i-th video in the support set and the query, and n*k represents the total number of videos formed by extracting k videos from each of the n categories in the dataset;

[0145] S45, selecting the similarity score of each video in the query set and the support set as the action prediction result of the query set video;

[0146] In a specific embodiment, S45 can be expressed as:

[0147]

[0148] in, is the action prediction result of the query set video, {Score0, Score1, ..., Score n*k} represents the similarity score of each video in the query set and the support set.

[0149] S46 uses the cross entropy loss function to calculate the loss between the action prediction results and the true labels of the query set videos, fine-tune the multimodal action feature extractor, and build a small sample dance action recognition model.

[0150] In a specific example, S46 is specifically:

[0151] The cross entropy loss function is used for the action prediction results of the real label and the query set video. The process is expressed by the formula:

[0152]

[0153] in, is the action prediction result of the query set video, Y represents the true label, and the true label represents the video corresponding to the query set in the support set, L Few_shot CrossEntropyLoss(.) represents the loss between the true labels and the action predictions for the query video. During training, since the model's feature extraction capabilities had already been roughly trained in the previous multimodal feature trainer, only a small learning rate was set to fine-tune the model.

[0154] S5, using the trained small sample dance movement recognition model to perform small sample dance movement recognition.

[0155] The method used in this embodiment alleviates the recognition difficulties caused by insufficient data in small sample sizes. Compared to traditional small sample motion recognition methods, this method leverages multimodal information to effectively address the lack of robustness of RGB single-modal data in video motion recognition. Furthermore, the use of video preview to guide multimodal data fusion enhances the model's semantic extraction capabilities across modal spaces. Compared to traditional small sample motion recognition methods, the proposed method offers stronger recognition performance, more comprehensive utilization of video information features, and is more suitable for small sample dance motion recognition tasks, improving the accuracy of small sample dance motion recognition in complex scenarios. Furthermore, the present invention addresses the issue of how to complement the learning of different modalities during multimodal fusion by proposing a multimodal fusion method guided by video preview. This method first learns key video features by previewing the overall video information. The model then uses these key video features to determine the importance of different modalities. Based on the importance of each modality, the model selectively incorporates supplementary information from each modality to complement and improve the feature information, thereby improving performance. This method promotes the interaction of multimodal information in videos, making it easier for the model to extract motion features from videos, thereby improving the performance of small-sample dance movement recognition models. This method proposes using video preview to guide multimodal data fusion, applying feature learning methods from multimodal data to small-sample dance movement recognition tasks. By integrating and fusing data from different modalities, this invention addresses the existing issues of insufficient utilization of video information features and poor dance movement recognition performance in complex scenarios.

[0156] Corresponding to the aforementioned embodiment of the multimodal fusion small sample dance movement recognition method, the present invention also provides an embodiment of the multimodal fusion small sample dance movement recognition device.

[0157] See also Figure 4 An embodiment of the present invention provides a multimodal fusion small sample dance movement recognition device, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the multimodal fusion small sample dance movement recognition method of the above embodiment.

[0158] The embodiment of the multimodal fusion small sample dance movement recognition device of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown in the figure, it is a hardware structure diagram of any device with data processing capability where the multimodal fusion small sample dance movement recognition device of the present invention is located. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0159] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0160] Since the device embodiments generally correspond to the method embodiments, reference will be made to the description of the method embodiments for relevant details. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present invention. Persons of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0161] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for multimodal fusion dance movement recognition using a small sample size in the above embodiment is implemented.

[0162] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0163] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0164] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0165] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit the one or more embodiments of this specification. As used in one or more embodiments of this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0166] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the term "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0167] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A multimodal fusion dance motion recognition method with a small sample size, characterized by: The following steps are involved: S1, using a multimodal action recognition dataset, pre-trains feature extractors of different modalities, and uses feature extractors of different modalities to extract frame-by-frame feature information corresponding to different modalities; S2, using the frame-by-frame feature information of different modalities in step S1, training a multimodal action feature fuser based on video preview guidance; step S2 specifically comprises: S21, using the order of video frames as the position code required by the Transformer network, and using the Transformer network to extract temporal features from the frame-by-frame feature information of different modalities in step S1 to obtain spatiotemporal semantic features of videos of different modalities, where the different modalities include RGB modalities and non-RGB modalities, and the non-RGB modalities include skeleton heat map modalities, depth modalities, and infrared modalities; S22, by performing pooling processing on the spatiotemporal semantic features of the RGB modal video, the RGB modal single-modal action features are obtained; S23, add modality embedding and time sequence embedding to the non-RGB modality video spatiotemporal semantic features and splice them together to obtain multimodal spatiotemporal semantic features without RGB; S24, using the multimodal Transformer network to fuse multimodal spatiotemporal semantic features without RGB, to obtain preliminary multimodal fused spatiotemporal semantic features without RGB; S25, using the Transformer network based on video preview, the spatiotemporal semantic features of the RGB modality used as guidance and the preliminary multimodal fusion features without RGB are fused to obtain the multimodal fusion spatiotemporal semantic features; S26, pooling the multimodal fused spatiotemporal semantic features to obtain multimodal action features; S27, using the linear layer and the Softmax layer, classifies the RGB mode single-modal action features and multi-modal action features to obtain the video action prediction results; S28, using the cross entropy loss function, calculates the loss between the video action prediction results and the true label to train the multimodal action feature fusion; S3, based on the feature extractor and the multimodal action feature fusion, builds a multimodal action feature extractor; S4, using a multimodal action recognition dataset, trains a small-sample dance action recognition model based on a multimodal action feature extractor; S5, using the trained small sample dance movement recognition model to perform small sample dance movement recognition.

2. The method according to claim 1, characterized in that The step S1 is specifically as follows: S11, using a multimodal action recognition dataset, extracting a predetermined number of video frames from different modalities, performing feature extraction on the video frames using a ResNet50 network, and obtaining frame-by-frame feature information of different modalities; S12, performing global feature extraction on the frame-by-frame feature information of different modalities in step S11 to obtain global coding features of different modalities; S13, using the linear layer and the Softmax layer, classifies the global encoding features of different modalities to obtain the video action prediction results; S14, using a cross entropy loss function to calculate the loss between the video action prediction result and the true label to train the action recognition model of each modality, and fixing the parameters of the action recognition model after training each modality, and selecting the feature extraction module of the corresponding action recognition model as the feature extractor corresponding to each modality; S15 , using the feature extractors of each modality in step S14 to perform feature extraction on the predetermined number of video frames in step S11 , to obtain frame-by-frame feature information of different modalities.

3. The method according to claim 1, characterized in that The step S4 is specifically as follows: S41, using a multimodal action feature extractor to extract features from different modalities of the support set and different modalities of the query set, respectively, to obtain RGB modal unimodal action features and multimodal action features of the corresponding support set and query set, wherein the support set and query set are obtained from a multimodal action recognition dataset; S42, concatenating the RGB mode single-modal action features and multi-modal action features in the support set and the query set to obtain the corresponding multi-modal action fusion features of the support set and the query set; S43, mapping the multimodal action fusion features of the support set and the query set in step S42 to a lower-dimensional feature space through a linear layer to obtain low-dimensional multimodal action fusion features of the support set and the query set; S44, using a similarity function to calculate the similarity of the low-dimensional multimodal action fusion features of the support set and the query set in step S43, to obtain a similarity score for each video in the query set and the support set; S45, selecting the similarity score of each video in the query set and the support set as the action prediction result of the query set video; S46 uses the cross entropy loss function to calculate the loss between the action prediction results and the true labels of the query set videos, fine-tune the multimodal action feature extractor, and build a small sample dance action recognition model.

4. A multimodal fusion dance movement recognition device with a small sample size, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the multimodal fusion small sample dance movement recognition method according to any one of claims 1 to 3.

Images