A TBW-based multi-modal human motion recognition method

Abstract

The application relates to a TBW-based multi-modal human action recognition method, which comprises the following steps: acquiring action video data, extracting RGB frames and audio signals; calculating an optical flow image and acquiring a spectrum image; using a TBW mechanism to perform interval sampling on the RGB frames, the optical flow image and the spectrum image; inputting the sampling results into a multi-modal model to perform feature extraction and feature fusion; inputting the fused features into a classifier to obtain action classification results corresponding to the TBW window. The application has the beneficial effects that the application simultaneously uses RGB frames, optical flows and audio as inputs, enhances the available information when the network recognizes human actions, and can effectively improve the accuracy of behavior recognition; and the application can process modal data with different sampling rates and can recognize actions with relatively large duration and speed changes by adding the TBW mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition, and more specifically, to a multimodal human action recognition method based on TBW. Background Art

[0002] In recent years, with the rapid development of computer and multimedia technologies and the increasing popularity of smartphones, surveillance cameras, and other video cameras, the amount of video data generated has exploded, creating a need for automated video retrieval and recognition. However, due to the unique structure of video data, the recorded information is difficult for computers to recognize and understand. Human motion is a crucial component of video content, and human motion recognition involves identifying the human body's movements and interactions with the environment and other individuals.

[0003] In traditional deep learning research, most strategies are based on convolutional neural networks or two-stream networks, employing fixed-interval frame sampling. These approaches have demonstrated significant performance in previous studies for human action recognition in videos. However, they still have limitations, particularly in leveraging multimodal information and robustly recognizing actions at varying frame rates or speeds. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the existing technology and propose a multimodal human action recognition method based on TBW.

[0005] In a first aspect, a multimodal human action recognition method based on TBW is provided, comprising:

[0006] S1. Acquire action video data, and extract RGB frames and audio signals based on the action video data;

[0007] S2. Calculate an optical flow image based on the RGB frame; and obtain a spectrum image based on the audio signal;

[0008] S3. Perform interval sampling on the RGB frame, optical flow image, and spectrum image using a TBW mechanism;

[0009] S4: Input the sampling results of S3 into the multimodal model for feature extraction and feature fusion;

[0010] S5. Input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

[0011] Preferably, S3 includes:

[0012] S301, divide the RGB frame sequence into K subsequences of equal length in time order, and randomly select a frame from each subsequence ; is the index number of the RGB frame sequence;

[0013] S302, from the modal spectrum image sequence and optical flow image sequences Select and ; Among them, the index number of the spectrum sequence and the index number of the optical flow map sequence The formula is:

[0014]

[0015]

[0016] in, and is the sequence sampling frequency of the two modes, is the sampling frequency of the RGB frame sequence, is half the length of the TBW window; the symbol Indicates rounding up to an integer.

[0017] As a preference, the TBW window length Independent of the subsequence length of the RGB frame sequence.

[0018] Preferably, S4 includes:

[0019] S401, extracting features of RGB frame, optical flow image and spectrum image through Transformer encoder, ResNet network and BN-Inception network respectively;

[0020] S402, through the cross attention mechanism and KAN network, the complementary features of the three modalities are fused, and the redundant features are eliminated to obtain the fused features; the formula corresponding to S4 is:

[0021]

[0022] in, It is the feature extraction module of the multimodal model. is the time domain pooling module, is a multimodal feature fusion module, is the fused multimodal feature vector.

[0023] Preferably, in S401, the ResNet network structure is:

[0024] The initial conv1 convolutional layer has a kernel size of 7x7 and a sliding stride of 2. This is followed by four residual units, where the output of each unit is the sum of the output of the convolutional layer within the unit and the residual connection of the initial input.

[0025] The first residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 64; the second residual unit contains 8 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 128; the third residual unit contains 12 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 256; the fourth residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 512.

[0026] Preferably, in S402, the mathematical expression of the KAN network is:

[0027]

[0028] in, and Both are learnable activation functions optimized by backpropagation, and the function It is the basic unit for introducing nonlinear transformation into KAN network. is a multivariable function The input dimension of is a multivariable function No. input variables.

[0029] Preferably, the method further includes: S6, training the multimodal model through relative norm alignment loss function and cross entropy loss function.

[0030] Preferably, in S6, the formula of the relative norm alignment loss function is:

[0031]

[0032] in, 、 and They are the feature vectors extracted by the three branches of the multimodal model from the RGB frame, optical flow image, and spectrum image; Indicates finding the L2 norm of the vector; represents the mathematical expectation of the eigenvector x, estimated by averaging all eigenvectors in each batch.

[0033] In a second aspect, a TBW-based multimodal human action recognition system is provided, which is configured to execute any of the TBW-based multimodal human action recognition methods described in the first aspect, including:

[0034] An acquisition module, configured to acquire motion video data and extract RGB frames and audio signals based on the motion video data;

[0035] A calculation module, configured to calculate an optical flow image based on the RGB frame; and obtain a spectrum image based on the audio signal;

[0036] A sampling module, configured to perform interval sampling on the RGB frame, the optical flow image, and the spectrum image using a TBW mechanism;

[0037] The feature extraction module is used to input the sampling results of the sampling module into the multimodal model for feature extraction and feature fusion;

[0038] The input module is used to input the fused features into the classifier to obtain the action classification results of the corresponding TBW window.

[0039] According to a third aspect, an electronic device is provided, comprising:

[0040] at least one processor; and,

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the TBW-based multimodal human motion recognition method as described in any one of the first aspects.

[0043] The beneficial effects of the present invention are:

[0044] 1. This invention uses RGB frames, optical flow, and audio as inputs simultaneously, which enhances the available information for the network to recognize human actions and can effectively improve the accuracy of action recognition.

[0045] 2. The present invention uses audio modalities to better distinguish actions that are similar in behavior but different in sound. For example, collision sounds can be used to help distinguish the material of moving objects, and water sounds can be used to help distinguish different types of swimming.

[0046] 3. By adding the TBW mechanism, the present invention can process modal data with different sampling rates and identify actions with large changes in duration and speed.

[0047] 4. By adding the RNA loss function, the present invention can adaptively balance the weights between different modalities, making full use of the complementary information in multiple modalities, thereby improving the accuracy and robustness of action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a multimodal human action recognition method based on TBW provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the structure of a TBW-based multimodal human action recognition model provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the following examples. The following examples are provided only to facilitate understanding of the present invention. It should be noted that, without departing from the principles of the present invention, it is possible for a person skilled in the art to make various modifications to the present invention, and such improvements and modifications fall within the scope of the claims of the present invention.

[0052] Example 1:

[0053] like Figure 1 As shown, Example 1 of the present application provides a multimodal human action recognition method based on TBW, including:

[0054] S1. Acquire action video data, and extract RGB frames and audio signals based on the action video data.

[0055] Specifically, you can use a program like OpenCV to read video data and split it into a sequence of RGB frames and audio data. The Lucas-Kanade algorithm is used to calculate an optical flow image from the RGB frames, and the audio data is converted into a spectrum image using a short-time Fourier transform (STFT).

[0056] S2. Calculate an optical flow image based on the RGB frame; and obtain a spectrum image based on the audio signal.

[0057] S3. Perform interval sampling on the RGB frame, optical flow image, and spectrum image using a TBW mechanism.

[0058] S3 includes:

[0059] S301, divide the RGB frame sequence into K subsequences of equal length in time order, and randomly select a frame from each subsequence ; is the index number of the RGB frame sequence;

[0060] S302, from the modal spectrum image sequence and optical flow image sequences Select and ; wherein the index number of the spectrum sequence and the index number of the optical flow sequence The formula is:

[0061]

[0062]

[0063] wherein, and are the sequence sampling frequencies of the two modalities, is the sampling frequency of the RGB frame sequence, is half the length of the TBW window; the symbol represents the upward integer.

[0064] It should be noted that the TBW window length is independent of the subsequence length of the RGB frame sequence. The duration of the subsequence does not affect the value of b. By relaxing the above constraint, the size of b can be flexibly set according to the duration of the action in the actual application scenario, improving the practicability of the model.

[0065] S4, input the sampling result of S3 into a multi-modal model as shown in Figure 2 for feature extraction and feature fusion.

[0066] In S4, the multi-modal model is divided into three branches, which respectively extract the features of the three modalities through the Transformer encoder, the ResNet network and the BN-Inception network.

[0067] S4 includes:

[0068] S401, extract the features of the RGB frame, the optical flow image and the spectrum image through the Transformer encoder, the ResNet network and the BN-Inception network respectively.

[0069] In S401, the ResNet network structure is:

[0070] The initial conv1 convolution layer has a convolution kernel size of 7x7 and a sliding step of 2; followed by 4 residual units, and the output of each unit is the output of the convolution layer in the unit plus the residual connection of the initial input.

[0071] The first residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 64; the second residual unit contains 8 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 128; the third residual unit contains 12 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 256; the fourth residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 512.

[0072] S402: Fuse the complementary features of the three modalities through the cross-attention mechanism and the KAN network, eliminate redundant features, and obtain the fused features.

[0073] In S402, the KAN network specifically inputs multimodal fusion features, outputs logits whose dimensions are determined by the number of actions to be classified, and then converts the logits into a probability distribution vector of the action to be classified through a Softmax layer. The mathematical expression of the KAN network is:

[0074]

[0075] in, and Both are learnable activation functions optimized by backpropagation, and the function It is the basic unit for introducing nonlinear transformation into KAN network. is a multivariable function The input dimension of is a multivariable function No. input variables.

[0076] According to the Kolmogorov-Arnold representation theorem, every multivariate continuous function can be represented as a two-parameter addition of a single-variable continuous function. The activation function of the KAN network is designed based on this theorem, introducing a favorable nonlinear transformation when converting feature dimensions to logits. This results in higher efficiency and better interpretability than traditional MLP layers.

[0077] The formula corresponding to S4 is:

[0078]

[0079] in, It is the feature extraction module of the multimodal model. is the time domain pooling module, is a multimodal feature fusion module, is the fused multimodal feature vector.

[0080] S5, input the fused features into a classifier to obtain an action classification result corresponding to the TBW window. In addition, the results of all TBW classifications can be averaged to obtain an action classification result of the video.

[0081] Embodiment 2

[0082] Based on embodiment 1, embodiment 2 of the present application provides a more specific TBW-based multi-modal human action recognition method, which includes:

[0083] S1, obtain action video data, and extract RGB frames and audio signals according to the action video data.

[0084] S2, calculate an optical flow image according to the RGB frames, and obtain a spectrum image according to the audio signals.

[0085] S3, use a TBW mechanism to perform interval sampling on the RGB frames, the optical flow image and the spectrum image.

[0086] S4, input the sampling results of S3 into a multi-modal model to perform feature extraction and feature fusion.

[0087] S5, input the fused features into a classifier to obtain an action classification result corresponding to the TBW window.

[0088] S6, train the multi-modal model through a relative norm alignment loss function and a cross-entropy loss function, adaptively adjust the weight of each modal feature, and ensure that the features extracted by the three branches of the multi-modal model are relatively balanced.

[0089] In S6, the formula of the relative norm alignment loss function is:

[0090]

[0091] wherein, , and are feature vectors extracted from RGB frames, optical flow images and spectrum images by the three branches of the multi-modal model respectively; denotes L2 norm of a vector; denotes mathematical expectation of a feature vector x, which is estimated by averaging all feature vectors in each batch.

[0092] It should be noted that the same or similar parts in this embodiment as in embodiment 1 can be mutually referenced, and will not be described herein again.

[0093] Embodiment 3

[0094] On the basis of embodiment 1, the application embodiment 3 provides a multi-modal human action recognition system based on TBW, comprising:

[0095] An acquisition module is configured to acquire action video data, extract RGB frames and audio signals according to the action video data;

[0096] A calculation module is configured to calculate an optical flow image according to the RGB frames, and acquire a spectrum image according to the audio signals;

[0097] A sampling module is configured to perform interval sampling on the RGB frames, the optical flow image and the spectrum image using a TBW mechanism;

[0098] A feature extraction module is configured to input the sampling result of the sampling module into a multi-modal model to perform feature extraction and feature fusion;

[0099] An input module is configured to input the fused features into a classifier to obtain an action classification result corresponding to a TBW window.

[0100] Specifically, the system provided in the present embodiment corresponds to the method provided in embodiment 1, and thus, the same or similar parts in the present embodiment and embodiment 1 can be mutually referred to, and will not be described herein again.

[0101] To sum up, the present application enhances the feature extraction and fusion capability through TBW interval sampling and cross-attention-based feature fusion, effectively improves the accuracy of human action recognition, and meets the demand for human action recognition scenarios in life and production scenarios.

Claims

1. A multimodal human action recognition method based on TBW, characterized in that: include: S1. Acquire action video data, and extract RGB frames and audio signals based on the action video data; S2. Calculate an optical flow image based on the RGB frame; Acquire a spectrum image according to the audio signal; S3, performing interval sampling on the RGB frame, optical flow image and spectrum image using the TBW mechanism; S3 includes: S301, divide the RGB frame sequence into K subsequences of equal length in time order, and randomly select a frame from each subsequence ; is the index number of the RGB frame sequence; S302, from the modal spectrum image sequence and optical flow image sequences Select and ; Among them, the index number of the spectrum sequence and the index number of the optical flow map sequence The formula is: in, and is the sequence sampling frequency of the two modes, is the sampling frequency of the RGB frame sequence, is half the length of the TBW window; the symbol Indicates rounding up to an integer; TBW window length Independent of the subsequence length of the RGB frame sequence; S4: Input the sampling results of S3 into the multimodal model for feature extraction and feature fusion; S5. Input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

2. The multimodal human motion recognition method based on TBW according to claim 1, characterized in that S4 include: S401, extracting features of RGB frame, optical flow image and spectrum image through Transformer encoder, ResNet network and BN-Inception network respectively; S402, through the cross attention mechanism and KAN network, the complementary features of the three modalities are fused, and the redundant features are eliminated to obtain the fused features; the formula corresponding to S4 is: in, It is the feature extraction module of the multimodal model. is the time domain pooling module, is a multimodal feature fusion module, is the fused multimodal feature vector.

3. The multimodal human motion recognition method based on TBW according to claim 2, characterized in that: In S401, the ResNet network structure is: The initial conv1 convolutional layer has a kernel size of 7x7 and a sliding stride of 2. This is followed by four residual units, where the output of each unit is the sum of the output of the convolutional layer within the unit and the residual connection of the initial input. The first residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 64; the second residual unit contains 8 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 128; the third residual unit contains 12 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 256; the fourth residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 512.

4. The multimodal human motion recognition method based on TBW according to claim 2 or 3, characterized in that: In S402, the mathematical expression of the KAN network is: in, and Both are learnable activation functions optimized by backpropagation, and the function It is the basic unit for introducing nonlinear transformation into KAN network. is a multivariable function The input dimension of is a multivariable function No. input variables.

5. The multimodal human motion recognition method based on TBW according to claim 4, characterized in that: Also includes: S6. Train the multimodal model using relative norm alignment loss function and cross entropy loss function.

6. The multimodal human motion recognition method based on TBW according to claim 5, characterized in that: In S6, the formula of the relative norm alignment loss function is: in, 、 and They are the feature vectors extracted from the RGB frame, optical flow image, and spectrum image by the three branches of the multimodal model; they represent the L2 norm of the vector; represents the mathematical expectation of the eigenvector x, estimated by averaging all eigenvectors in each batch.

7. A multimodal human action recognition system based on TBW, characterized in that: The method for performing the TBW-based multimodal human action recognition method according to any one of claims 1 to 6 comprises: An acquisition module, configured to acquire motion video data and extract RGB frames and audio signals based on the motion video data; A calculation module, configured to calculate an optical flow image based on the RGB frame; and obtain a spectrum image based on the audio signal; A sampling module, configured to perform interval sampling on the RGB frame, the optical flow image, and the spectrum image using a TBW mechanism; The feature extraction module is used to input the sampling results of the sampling module into the multimodal model for feature extraction and feature fusion; The input module is used to input the fused features into the classifier to obtain the action classification results of the corresponding TBW window.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the TBW-based multimodal human motion recognition method as described in any one of claims 1 to 6.

Images