Multi-modal human body action recognition method based on TBW

Abstract

The invention relates to a TBW-based multi-modal human body action recognition method, which comprises the following steps: acquiring action video data, and extracting RGB frames and audio signals; calculating an optical flow image and obtaining a frequency spectrum image; performing interval sampling on the RGB frame, the optical flow image and the frequency spectrum image by using a TBW mechanism; inputting a sampling result into a multi-modal model, and performing feature extraction and feature fusion; and inputting the fused features into a classifier to obtain an action classification result corresponding to the TBW window. The method has the advantages that RGB frames, optical streams and audios are used as input at the same time, available information generated when a network recognizes human body actions is enhanced, and the behavior recognition accuracy can be effectively improved; in addition, by adding a TBW mechanism, modal data with different sampling rates can be processed, and actions with large lasting time and speed changes can be recognized.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition, and more specifically, to a multimodal human action recognition method based on TBW. Background Art

[0002] In recent years, with the booming development of computer and multimedia technology, the gradual popularization of photography and video equipment such as smart phones and surveillance cameras, the scale of video data generated by people has exploded, and the demand for automatic retrieval and recognition of videos has emerged. However, due to the particularity of the video data structure, the information recorded in it is difficult for computers to recognize and understand. Human body movements are important information contained in video content. Human action recognition refers to the recognition of human body movements and interactions with the environment and other individuals.

[0003] In traditional deep learning research, most strategies are based on convolutional neural networks or two-stream networks, and use a frame sampling mechanism with a fixed time interval. These methods for dealing with the problem of human action recognition in videos have shown significant performance in previous studies. However, there are still some limitations, especially in the use of multimodal information and the robust recognition of actions at different frame rates or speed changes. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and propose a multimodal human action recognition method based on TBW.

[0005] In a first aspect, a multimodal human action recognition method based on TBW is provided, comprising:

[0006] S1, obtaining action video data, and extracting RGB frames and audio signals according to the action video data;

[0007] S2. Calculate an optical flow image according to the RGB frame; and obtain a spectrum image according to the audio signal;

[0008] S3, performing interval sampling on the RGB frame, the optical flow image and the spectrum image using a TBW mechanism;

[0009] S4, input the sampling results of S3 into the multimodal model for feature extraction and feature fusion;

[0010] S5. Input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

[0011] Preferably, S3 includes:

[0012] S301, divide the RGB frame sequence into K subsequences of equal duration in time order, and randomly select a frame M from each subsequence.rgb,i ; i is the index number of the RGB frame sequence;

[0013] S302, from the modal spectrum image sequence M audio and optical flow image sequence M flow Select M audio,j and M flow,k ; The formulas for the index number j of the spectrum map sequence and the index number k of the optical flow map sequence are:

[0014]

[0015] Among them, r audio and r flow is the sampling frequency of the two modes, r rgb is the sampling frequency of the RGB frame sequence, b is half the length of the TBW window; the symbol Indicates rounding up to an integer.

[0016] Preferably, the TBW window length 2b is independent of the subsequence length of the RGB frame sequence.

[0017] Preferably, S4 includes:

[0018] S401, extracting features of RGB frame, optical flow image and spectrum image through Transformer encoder, ResNet network and BN-Inception network respectively;

[0019] S402, through the cross attention mechanism and KAN network, the complementary features of the three modalities are fused, and the redundant features are eliminated to obtain the fused features; the formula corresponding to S4 is:

[0020] y=h(G(f tbw (M rgb,i ,M audio,j ,M flow,k )))

[0021] Among them, f tbw is the feature extraction module of the multimodal model, G is the time domain pooling module, h is the multimodal feature fusion module, and y is the fused multimodal feature vector.

[0022] Preferably, in S401, the ResNet network structure is:

[0023] The initial conv1 convolution layer has a kernel size of 7x7 and a sliding stride of 2. This is followed by 4 residual units, where the output of each unit is the output of the convolution layer within the unit superimposed with the residual connection of the initial input.

[0024] The first residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 64; the second residual unit contains 8 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 128; the third residual unit contains 12 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 256; the fourth residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 512.

[0025] Preferably, in S402, the mathematical expression of the KAN network is:

[0026]

[0027] Among them, Φ q and φ p,q All of them are learnable activation functions optimized by back propagation. Function f() is the basic unit for introducing nonlinear transformation in KAN network. n is the input dimension of multivariate function f(). p is the pth input variable of the multivariable function f().

[0028] Preferably, the method further includes: S6, training the multimodal model through relative norm alignment loss function and cross entropy loss function.

[0029] Preferably, in S6, the formula of the relative norm alignment loss function is:

[0030]

[0031] Among them, F rgb 、F flow and F audio are the feature vectors extracted from RGB frames, optical flow images and spectrum images by the three branches of the multimodal model; ‖‖ represents the L2 norm of the vector; E(x) represents the mathematical expectation of the feature vector x, which is estimated by taking the mean of all feature vectors in each batch.

[0032] In a second aspect, a TBW-based multimodal human motion recognition system is provided, which is used to execute any TBW-based multimodal human motion recognition method described in the first aspect, including:

[0033] An acquisition module, used for acquiring action video data, and extracting RGB frames and audio signals according to the action video data;

[0034] A calculation module, used to calculate an optical flow image according to the RGB frame; and obtain a spectrum image according to the audio signal;

[0035] A sampling module, used for performing interval sampling on the RGB frame, the optical flow image and the spectrum image using a TBW mechanism;

[0036] The feature extraction module is used to input the sampling results of the sampling module into the multimodal model to perform feature extraction and feature fusion;

[0037] The input module is used to input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

[0038] According to a third aspect, an electronic device is provided, the electronic device comprising:

[0039] at least one processor; and,

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the TBW-based multimodal human motion recognition method as described in any one of claims 1-8.

[0042] The beneficial effects of the present invention are:

[0043] 1. The present invention uses RGB frames, optical flow, and audio as inputs at the same time, which enhances the available information when the network recognizes human actions and can effectively improve the accuracy of behavior recognition.

[0044] 2. The present invention uses audio modality to better distinguish actions with similar behaviors but different sounds, such as using collision sounds to assist in distinguishing the materials of moving objects, and using water sounds to assist in distinguishing different types of swimming.

[0045] 3. By adding the TBW mechanism, the present invention is able to process modal data with different sampling rates and identify actions with relatively large changes in duration and speed.

[0046] 4. By adding the RNA loss function, the present invention can adaptively balance the weights between different modalities, so that the complementary information in multiple modalities can be fully utilized, thereby improving the accuracy and robustness of action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a TBW-based multimodal human action recognition method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the structure of a TBW-based multimodal human action recognition model provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0051] Embodiment 1:

[0052] like Figure 1 As shown, Embodiment 1 of the present application provides a multimodal human motion recognition method based on TBW, including:

[0053] S1. Acquire action video data, and extract RGB frames and audio signals according to the action video data.

[0054] Specifically, the video data can be read by using programs such as OpenCV and split into RGB frame sequences and audio data. The optical flow image is calculated based on the RGB frame using the Lucas-Kanade algorithm, and the audio data is converted into spectrum image data using the short-time Fourier transform (STFT).

[0055] S2. Calculate an optical flow image according to the RGB frame; and obtain a spectrum image according to the audio signal.

[0056] S3. Perform interval sampling on the RGB frame, the optical flow image and the spectrum image using a TBW mechanism.

[0057] S3 includes:

[0058] S301, divide the RGB frame sequence into K subsequences of equal duration in time order, and randomly select a frame M from each subsequence. rgb,i ; i is the index number of the RGB frame sequence;

[0059] S302, from the modal spectrum image sequence M audio and optical flow image sequence M flow Select M audio,j and M flow,k ; The formulas for the index number j of the spectrum map sequence and the index number k of the optical flow map sequence are:

[0060]

[0061] Among them, r audio and r flow is the sampling frequency of the two modes, rrgb is the sampling frequency of the RGB frame sequence, b is half the length of the TBW window; the symbol Indicates rounding up to an integer.

[0062] It should be noted that the TBW window length 2b is independent of the subsequence length of the RGB frame sequence. The duration of the subsequence does not affect the value of b. By relaxing the above constraints, the value of b can be flexibly set according to the duration of the action in the actual application scenario, thereby improving the practicality of the model.

[0063] S4, input the sampling result of S3 into Figure 2 The multimodal model shown performs feature extraction and feature fusion.

[0064] In S4, the multimodal model is divided into three branches, which extract features corresponding to the three modalities through Transformer encoder, ResNet network, and BN-Inception network respectively.

[0065] S4 includes:

[0066] S401, extract features of RGB frame, optical flow image and spectrum image through Transformer encoder, ResNet network and BN-Inception network respectively.

[0067] In S401, the ResNet network structure is:

[0068] The initial conv1 convolution layer has a kernel size of 7x7 and a sliding stride of 2. This is followed by 4 residual units, where the output of each unit is the output of the convolution layer within the unit superimposed with the residual connection of the initial input.

[0069] The first residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 64; the second residual unit contains 8 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 128; the third residual unit contains 12 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 256; the fourth residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 512.

[0070] S402. The complementary features of the three modalities are fused through the cross-attention mechanism and the KAN network, and the redundant features are eliminated to obtain the fused features.

[0071] In S402, the KAN network specifically inputs the multimodal fusion features, outputs logits whose dimensions are determined by the number of actions to be classified, and then converts the logits into the probability distribution vector of the actions to be classified through the Softmax layer. The mathematical expression of the KAN network is:

[0072]

[0073] Among them, Φ q and φ p,q All of them are learnable activation functions optimized by back propagation. Function f() is the basic unit for introducing nonlinear transformation in KAN network. n is the input dimension of multivariate function f(). p is the pth input variable of the multivariable function f().

[0074] According to the Kolmogorov-Arnold representation theorem, every multivariate continuous function can be represented as a superposition of two-parameter addition of a single-variable continuous function. The activation function of the KAN network is designed based on the Kolmogorov-Arnold representation theorem, which introduces a good nonlinear transformation when converting feature dimensions to logits. Compared with the traditional MLP layer, it has higher efficiency and better interpretability.

[0075] The formula corresponding to S4 is:

[0076] y=h(G(f tbw (M rgb,i ,M audio,j ,M flow,k )))

[0077] Among them, f tbw is the feature extraction module of the multimodal model, G is the time domain pooling module, h is the multimodal feature fusion module, and y is the fused multimodal feature vector. S5. The fused features are input into the classifier to obtain the action classification result of the corresponding TBW window. In addition, the results of all TBW classifications can be averaged to obtain the action classification result of the video.

[0078] Embodiment 2:

[0079] Based on Example 1, Example 2 of the present application provides a more specific TBW-based multimodal human motion recognition method, including:

[0080] S1. Acquire action video data, and extract RGB frames and audio signals according to the action video data.

[0081] S2. Calculate an optical flow image according to the RGB frame; and obtain a spectrum image according to the audio signal.

[0082] S3. Perform interval sampling on the RGB frame, the optical flow image and the spectrum image using a TBW mechanism.

[0083] S4: Input the sampling results of S3 into the multimodal model for feature extraction and feature fusion.

[0084] S5. Input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

[0085] S6. The multimodal model is trained through the relative norm alignment loss function and the cross entropy loss function, and the weight of each modal feature is adaptively adjusted to ensure that the features extracted by the three branches of the multimodal model are relatively balanced.

[0086] In S6, the formula of the relative norm alignment loss function is:

[0087]

[0088] Among them, F rgb 、F flow and F audio are the feature vectors extracted from RGB frames, optical flow images and spectrum images by the three branches of the multimodal model; ‖‖ represents the L2 norm of the vector; E(x) represents the mathematical expectation of the feature vector x, which is estimated by taking the mean of all feature vectors in each batch.

[0089] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0090] Embodiment 3:

[0091] On the basis of Example 1, Example 3 of the present application provides a TBW-based multimodal human motion recognition system, including:

[0092] An acquisition module, used for acquiring action video data, and extracting RGB frames and audio signals according to the action video data;

[0093] A calculation module, used to calculate an optical flow image according to the RGB frame; and obtain a spectrum image according to the audio signal;

[0094] A sampling module, used for performing interval sampling on the RGB frame, the optical flow image and the spectrum image using a TBW mechanism;

[0095] The feature extraction module is used to input the sampling results of the sampling module into the multimodal model to perform feature extraction and feature fusion;

[0096] The input module is used to input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

[0097] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiment 1. Therefore, the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be repeated in this application.

[0098] In summary, this application enhances the feature extraction and fusion capabilities through TBW interval sampling and cross-attention-based feature fusion, which can effectively improve the accuracy of human motion recognition and meet the needs of human motion recognition scenarios in life and production scenarios.

Claims

1. A multimodal human action recognition method based on TBW, characterized in that: include: S1, obtaining action video data, and extracting RGB frames and audio signals according to the action video data; S2. Calculate an optical flow image according to the RGB frame; and obtain a spectrum image according to the audio signal; S3, performing interval sampling on the RGB frame, the optical flow image and the spectrum image using a TBW mechanism; S4, input the sampling results of S3 into the multimodal model for feature extraction and feature fusion; S5. Input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

2. The multimodal human motion recognition method based on TBW according to claim 1, characterized in that S3 include: S301, divide the RGB frame sequence into K subsequences of equal duration in time order, and randomly select a frame M from each subsequence. rgb,i ; i is the index number of the RGB frame sequence; S302, from the modal spectrum image sequence M audio and optical flow image sequence M flow Select M audio,j and M flow,k ; The formulas for the index number j of the spectrum map sequence and the index number k of the optical flow map sequence are: Among them, r audio and r flow is the sequence sampling frequency of the two modes, r rgb is the sampling frequency of the RGB frame sequence, b is half the length of the TBW window; the symbol Indicates rounding up to an integer.

3. The multimodal human motion recognition method based on TBW according to claim 2, characterized in that: The TBW window length 2b is independent of the subsequence length of the RGB frame sequence.

4. The multimodal human motion recognition method based on TBW according to claim 3, characterized in that S4 include: S401, extracting features of RGB frame, optical flow image and spectrum image through Transformer encoder, ResNet network and BN-Inception network respectively; S402, through the cross attention mechanism and KAN network, the complementary features of the three modalities are fused, and the redundant features are eliminated to obtain the fused features; the formula corresponding to S4 is: y=h(G(f tbw (M rgb,i ,M audio,j ,M flow,k ))) Among them, f tbw is the feature extraction module of the multimodal model, G is the time domain pooling module, h is the multimodal feature fusion module, and y is the fused multimodal feature vector.

5. The multimodal human motion recognition method based on TBW according to claim 4, characterized in that: In S401, the ResNet network structure is: The initial conv1 convolution layer has a kernel size of 7x7 and a sliding stride of 2. This is followed by 4 residual units, where the output of each unit is the output of the convolution layer within the unit superimposed with the residual connection of the initial input. The first residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 64; the second residual unit contains 8 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 128; the third residual unit contains 12 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 256; the fourth residual unit contains 6 convolutional layers, whose convolution kernel size is 3x3, the sliding stride is 1, and the number of channels is 512.

6. The TBW-based multimodal human motion recognition method according to claim 4 or 5, characterized in that: In S402, the mathematical expression of the KAN network is: Among them, Φ q and φ p,q All of them are learnable activation functions optimized by back propagation. Function f() is the basic unit for introducing nonlinear transformation in KAN network. n is the input dimension of multivariate function f(). p is the pth input variable of the multivariable function f().

7. The multimodal human motion recognition method based on TBW according to claim 6, characterized in that: Also includes: S6. Train the multimodal model using relative norm alignment loss function and cross entropy loss function.

8. The multimodal human motion recognition method based on TBW according to claim 7, characterized in that: In S6, the formula of the relative norm alignment loss function is: Among them, F rgb 、F flow and F audio are the feature vectors extracted from RGB frames, optical flow images and spectrum images by the three branches of the multimodal model; ‖‖ represents the L2 norm of the vector; E(x) represents the mathematical expectation of the feature vector x, which is estimated by taking the mean of all feature vectors in each batch.

9. A multimodal human action recognition system based on TBW, characterized in that: The method for performing the TBW-based multimodal human motion recognition method according to any one of claims 1 to 8 comprises: An acquisition module, used for acquiring action video data, and extracting RGB frames and audio signals according to the action video data; A calculation module, used to calculate an optical flow image according to the RGB frame; and obtain a spectrum image according to the audio signal; A sampling module, used for performing interval sampling on the RGB frame, the optical flow image and the spectrum image using a TBW mechanism; The feature extraction module is used to input the sampling results of the sampling module into the multimodal model to perform feature extraction and feature fusion; The input module is used to input the fused features into the classifier to obtain the action classification result of the corresponding TBW window.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the TBW-based multimodal human motion recognition method as described in any one of claims 1-8.

Images