PSC-TNet video action recognition method based on fusion of spatial features and frame difference information

Abstract

This invention discloses a PSC-TNet video action recognition method based on the fusion of spatial features and frame difference information. The method involves preprocessing video frames and then inputting them into a PSC-TNet video action recognition model based on the fusion of spatial features and frame difference information. The PSC-TNet model includes three branches based on different frame intervals and an action classification module. The three branches are a motion branch, an appearance branch, and an action detail branch, each containing spatial information, motion information, and motion detail information. The original video is input into the motion branch, appearance branch, and action detail branch respectively, outputting their respective action features. All recognized features are then concatenated and input into the action classification module to obtain the action classification result. This invention can extract information from the temporal features of video images to adjust the spatial features of the video images, obtaining spatial features with enhanced weights for key spatial components, thereby achieving more accurate video image action recognition.

Description

Technical Field

[0001] This invention relates to a method for action recognition, specifically to an image action recognition method that associates temporal and spatial information. Background Technology

[0002] Video image action recognition is an important research area in computer vision, aiming to enable computers to automatically understand human actions in video content. This technology involves extracting action features from video sequences and using machine learning or deep learning models to analyze these features to identify ongoing actions or activities.

[0003] Motion recognition is crucial in many practical applications, such as video surveillance, human-computer interaction, sports analytics, healthcare, and the entertainment industry. In security monitoring, motion recognition can help detect abnormal behavior or potential security threats. In sporting events, it can be used to analyze athlete performance and technique. In healthcare, motion recognition can monitor patient recovery progress and daily activities to provide better care, among other things.

[0004] With the development of deep learning technology, it has achieved great success in fields such as image classification, object detection, and speech recognition. Deep learning can automatically learn high-level feature representations from data, avoiding the complexity and limitations of manually designed features. Deep learning has significantly improved the accuracy and efficiency of human action recognition. Typically based on models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Graph Neural Networks (GNNs), it can learn complex spatiotemporal features from video data. The addition of attention mechanisms further improves action recognition performance. There are two main methods for CNNs: one is based on dual-stream networks, which divides the video into RGB frames and optical flow frames, extracts spatial and temporal features using CNNs respectively, and then fuses the two features for classification; the other is based on three-dimensional convolutional neural networks (3D CNNs), which can directly perform 3D convolution on video blocks, simultaneously learning spatial and temporal features, and then uses fully connected layers or pooling layers for classification. Both methods have their own advantages and disadvantages. For example, dual-stream networks can capture action details better but require additional optical flow computation; 3D CNNs can better model the overall action but require more parameters and computational resources. Summary of the Invention

[0005] To address the problems of insufficient spatiotemporal information capture and easy interference from redundant information in deep learning-based image action recognition methods, this invention proposes a PSC-TNet video action recognition method based on the fusion of spatial features and frame difference information for video image processing.

[0006] The network framework of this invention adopts a three-branch structure of appearance branch, motion branch, and motion detail branch to extract spatial information, motion information, and action detail information. This invention designs two new plug-and-play modules: a temporal feature extraction module (TtS) and a spatial feature enhancement module (PSC). The TtS module aims to obtain weight scores based on the differential features at certain frame intervals, more effectively locating the motion difference information between two frames to enhance the spatial location features of the character's actions, thus achieving secondary enhancement of spatial features based on video temporal features. The PSC module, unlike traditional global pooling of channel dimensions which leads to feature loss, employs a filtering method. By compressing the channel dimensions while maintaining high resolution in the orthogonal spatial dimensions, and then multiplying the two, spatial feature weights are obtained without global pooling of the channels. Furthermore, the PSC spatial feature enhancement module employs a cross-multiplication strategy, processing features through convolutional kernels of different scales. Specifically, the spatial feature weights derived from 1×1 convolution kernels are used to adjust features processed by 3×3 convolution kernels, and the spatial feature weights derived from 3×3 convolution kernels are used to adjust features processed by 1×1 convolution kernels, achieving cross-spatial information aggregation. The PSC-T module proposed in this invention is a fusion of the PSC spatial feature enhancement module and the TtS temporal feature extraction module. New spatial feature weights are output by adding the spatial features extracted by the TtS feature extraction module to the new weights obtained from the sum of the cross-multiplication weights in the PSC module, achieving good results in action recognition and classification tasks.

[0007] The technical solution of the present invention is as follows:

[0008] The method of this invention preprocesses the input video frames, and then inputs the preprocessed video as the original video into the PSC-TNet video action recognition model based on fused spatial features and frame difference information to extract action recognition results. The PSC-TNet video action recognition model includes three branches based on different frame intervals and an action classification module. The three branches are the motion branch, the appearance branch, and the action detail branch. The original video is input into the motion branch, the appearance branch, and the action detail branch respectively to output their respective action features. Then, all the recognition features are concatenated and input into the action classification module to obtain the action classification results.

[0009] The input video is an image video to be identified.

[0010] The input video is sampled at higher frequencies, lower frequencies, and medium frequencies to obtain higher frequency video frames, lower frequency video frames, and medium frequency video frames, which together form the original video.

[0011] The intermediate frequency is a frequency that falls between higher and lower frequencies.

[0012] In the PSC-TNet video action recognition model, high-frequency video frames from the original video are used as feature tensors containing motion information and passed into the motion branch. Low-frequency video frames from the original video are used as feature tensors containing spatial information and passed into the appearance branch. Medium-frequency video frames from the original video are used as feature tensors containing action detail information and passed into the action detail branch. All three branches contain residual layers, and each residual layer includes a residual module. The features output from the motion branch and the action detail branch are passed to the appearance branch for channel-wise concatenation and feature fusion. The residual layer of the appearance branch is then processed by convolution. The three branches output recognition features containing spatial information, motion information, and motion detail information, respectively. Finally, the recognition features output from the three branches are concatenated and fused on each channel of the appearance branch and then input into the action classification module to obtain the final action recognition result.

[0013] The classifier uses a fully connected layer.

[0014] In the PSC-TNet video action recognition model, both the motion branch and the appearance branch include a convolutional block connected in sequence and four consecutive residual layers containing a spatial feature enhancement module (PSC). The action detail branch includes a temporal feature extraction module (TtS), a convolutional block, and four consecutive spatiotemporal feature fusion modules connected in sequence. Each spatiotemporal feature fusion module includes a TtS module connected in parallel and a residual layer containing a PSC module. The inputs of the spatiotemporal feature fusion modules are respectively input to the TtS module and the residual layers to obtain their respective results. The results are then added together and fused as the output of the spatiotemporal feature fusion module. In the spatiotemporal feature fusion module, the spatial feature weights generated in the last residual layer are added to the output of the temporal feature extraction module to generate new spatial feature weights to adjust the spatial features.

[0015] The four residual layers of the motion branch, appearance branch, and motion detail branch contain three, four, six, and three residual modules respectively, in the order of propagation. The convolutional blocks of the motion branch and the motion detail branch, as well as the convolutional block of the appearance branch, are concatenated on the channel of the appearance branch and then input into the first residual layer of the appearance branch. The nth residual layer of the motion branch and the motion detail branch, as well as the nth residual layer of the appearance branch, are concatenated on the channel of the appearance branch and then input into the (n+1)th residual layer of the appearance branch.

[0016] The residual layers in the motion branch, appearance branch, and motion detail branch each include multiple residual modules connected in sequence.

[0017] The residual module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a PSC-T attention mechanism module, and a fourth convolutional layer. The first, second, third, and PSC-T attention mechanism modules are connected in sequence. The inputs of the residual module are input to the first and fourth convolutional layers, respectively. The outputs of the PSC-T attention mechanism module and the fourth convolutional layer are added together to obtain the output of the residual module. The first and second convolutional layers are mainly composed of convolution operations, batch normalization operations, and activation operations connected in sequence. The third convolutional layer is mainly composed of convolution operations and batch normalization operations connected in sequence. The fourth convolutional layer is composed of a single convolution operation.

[0018] Therefore, the input features of the residual module are fed into multiple convolutional layers and the PSC-T attention mechanism module to obtain the output features of the residual module.

[0019] The PSC-T attention mechanism module of the residual module in the residual layer of the motion branch, appearance branch and action detail branch adopts the spatial feature enhancement module PSC.

[0020] More specifically, each motion branch includes four residual layers connected in sequence: a first convolutional block and a first residual layer to a fourth residual layer. The appearance branch includes four residual layers connected in sequence: a second convolutional block and a fifth residual layer to an eighth residual layer. Each residual layer in both the motion and appearance branches contains a spatial feature enhancement module.

[0021] The motion detail branch consists of four residual layers: a third convolutional block and four residual layers (ninth to twelfth). Each residual layer in the motion detail branch contains a temporal feature extraction module between itself and before the third convolutional block. Each residual layer in the motion detail branch contains a spatiotemporal feature fusion module. The four residual layers in the motion branch, appearance branch, and motion detail branch have 3, 4, 6, and 3 residual modules, respectively. The input to the motion detail branch is processed by the outputs of the first, second, and third convolutional layers in the residual modules, and then combined with the output of the temporal feature extraction module to form spatiotemporal feature fusion. The inputs to the fusion module are: the outputs of the first and third convolutional blocks are concatenated and fused along the channel dimension of the second convolutional block; the outputs of the first and ninth residual layers are concatenated and fused along the channel dimension of the fifth residual layer; the outputs of the second and sixth residual layers are concatenated and fused along the channel dimension of the tenth residual layer; the outputs of the third and seventh residual layers are concatenated and fused along the channel dimension of the eleventh residual layer; and the outputs of the fourth and eighth residual layers are concatenated and fused along the channel dimension of the twelfth residual layer.

[0022] The first, second, and third convolutional blocks have the same structure, each including a convolutional layer, a batch normalization layer, an activation layer, and a max pooling layer connected in sequence.

[0023] The time feature extraction module includes sequential video frame time feature extraction operations and several fifth convolutional layers. The video frame time feature extraction operation is to perform a difference operation between each two adjacent frames, and then input the difference result into a fifth convolutional layer for feature extraction to obtain differential motion features. All differential motion features are spliced ​​and fused together as the output of the time feature extraction module.

[0024] The fifth convolutional layer is mainly composed of convolution operations.

[0025] The number of fifth convolutional layers is equal to the number of medium-length video frames. The first video frame of the medium-length video frame is used as the input of the first convolutional layer. Each subsequent frame is subtracted from the previous video frame and then input into its corresponding convolutional layer for feature extraction to obtain differential motion features. Finally, the differential motion features obtained from all video frames are concatenated and fused as the output of the temporal feature extraction module.

[0026] The spatial feature enhancement module (PSC) includes two spatial feature branches, an addition operation, multiple sigmoid activation functions, and a pointwise multiplication operation. The PSC is input into the two spatial feature branches, which are a high receptive field branch and a low receptive field branch, respectively. The convolutional layer in the high receptive field branch has a larger convolutional size than that in the low receptive field branch. The convolutional layer in the high receptive field branch uses 3×3 convolutions for feature processing, while the convolutional layer in the low receptive field branch uses 1×1 convolutions for feature processing.

[0027] Each spatial feature branch has the same topology, including a sixth convolutional layer, a seventh convolutional layer, a global pooling layer, a first reconstruction layer, a second reconstruction layer, and a softmax activation function. The input of the spatial feature enhancement module PSC is used as the input of the spatial feature branch and is fed into the sixth and seventh convolutional layers respectively. The output of the sixth convolutional layer is passed through the global pooling layer, the first reconstruction layer, and the softmax activation function to obtain the AA feature. The output of the seventh convolutional layer is passed through the second reconstruction layer to obtain the BB feature. The AA feature and the BB feature are used as the output of the spatial feature branch.

[0028] The AA features output from the high receptive field branch and the BB features output from the low receptive field branch are processed by matrix multiplication and then input into the first sigmoid activation function. The AA features output from the low receptive field branch and the BB features output from the high receptive field branch are processed by matrix multiplication and then input into the second sigmoid activation function. The outputs of the first and second sigmoid activation functions are added together by their respective pixels and then input into the third sigmoid activation function. The output of the third sigmoid activation function and the input of the spatial feature enhancement module PSC are multiplied point by point and then output as the output of the spatial feature enhancement module PSC.

[0029] The spatial feature enhancement module PSC obtains spatial feature weights through global max pooling and matrix multiplication. Then, the weights generated by the low-scale receptive field branch are used to adjust the features of the high-scale branch, and the weights generated by the high-scale receptive field branch are used to adjust the features of the low-scale branch. Finally, the weights are added together to form the final spatial feature weights, which are then multiplied by the input features.

[0030] The action classification module specifically adopts a linear classification structure, which includes a global average pooling layer, a Dropout layer and a classifier connected in sequence. The recognition features containing spatial information, motion information and motion detail information are used as the input of the global average pooling layer, and the classifier outputs the action recognition result.

[0031] The computer program thereunder is an instruction that implements the method of any one of claims 1 to 9.

[0032] The method of this invention mainly includes the following steps: First, the input video image is preprocessed with video frames to extract three branches based on different frame intervals. High-frequency video frames are passed into the motion branch as feature tensors containing motion information, low-frequency video frames are passed into the appearance branch as feature tensors containing background information, and medium-frequency video frames are used to extract frame features to obtain action detail features, which are then passed into the action detail branch. After each residual module of the three branches, the feature tensors are concatenated on the appearance branch to perform feature fusion and convolution processing. Finally, the feature tensor results from the three branches after different processing are input into a linear classification layer to obtain the final action recognition result.

[0033] This invention can extract information from the temporal features of video images to adjust the spatial features of video images, thereby obtaining spatial features that enhance the weight of key spatial components, and thus achieving more accurate video image action recognition.

[0034] The beneficial effects of this invention are:

[0035] This invention proposes a PSC-TNet video action recognition method based on the fusion of spatial features and frame difference information. This invention designs two new plug-and-play modules, namely the temporal feature extraction module TtS and the spatial feature enhancement module PSC.

[0036] The temporal feature extraction module TtS aims to obtain weight scores based on the differential features at certain frame intervals, thereby more effectively locating the motion difference information between two frames. This enhances the spatial features of the character's actions, achieving secondary enhancement of spatial features based on the temporal features of the video.

[0037] The Spatial Feature Enhancement (PSC) module differs from traditional global pooling of the channel dimension, which leads to feature loss. PSC employs a filtering method, compressing the channel dimension while maintaining high resolution in the orthogonal spatial dimension, and then multiplying the two to obtain spatial feature weights without global pooling of the channels. Furthermore, the PSC module uses a cross-multiplication strategy, processing features through convolutional kernels of different scales. Specifically, spatial feature weights derived from 1×1 convolution kernels are used to adjust features processed by 3×3 convolution kernels, and vice versa, achieving cross-spatial information aggregation.

[0038] The PSC-T module proposed in this invention is a fusion of the PSC spatial feature enhancement module and the TtS temporal feature extraction module. The spatial features extracted by the TtS feature extraction module are added to the new weights obtained by adding the weights after cross-multiplication in the PSC module, and a new spatial feature weight is output. Attached Figure Description

[0039] Figure 1 This is a network framework diagram of the method of the present invention.

[0040] Figure 2 This is a schematic diagram of the structure of the temporal feature extraction module TtS and the spatial feature enhancement module PSC used in this invention.

[0041] Figure 3 This is a schematic diagram of the spatial feature enhancement module PSC used in this invention.

[0042] Figure 4 This is a schematic diagram of the residual module in the network of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] The embodiments of the present invention and their implementation process and circumstances are as follows:

[0045] The dataset used in this invention is the HMDB51 dataset, which contains 51 action categories and a total of 6849 videos. Each action contains at least 51 videos with a resolution of 320*240. The dataset comes from YouTube, Google Video, etc., and is 2GB in size.

[0046] 1) After preprocessing the input video images into video frames, low-frequency, high-frequency, and mid-frequency video frames are obtained. Specifically, the original input video images are normalized to obtain video image frames. In this video image frame sequence, video frames are taken at larger intervals to obtain a low-frequency video frame sequence, which serves as the input for the appearance branch; video frames are taken at smaller intervals to obtain a high-frequency video frame sequence, which serves as the input for the motion branch; and video frames are taken at medium intervals to obtain a mid-frequency video frame sequence. The size of the original input video image is normalized to a 64×112×112 3-channel RGB image, which is used as the input size for the neural network. The 3-channel RGB image is then normalized, mapping integers from 0 to 255 to floating-point numbers from 0 to 1.

[0047] 2) Input low-frequency video frames into the appearance branch, high-frequency video frames into the motion branch, and mid-frequency video frames into the motion detail branch. Then, concatenate and fused the feature tensors containing spatial information output from each layer in the appearance branch and the feature tensors containing motion information output from the corresponding layer in the motion branch in the channel dimension to obtain the final feature tensor.

[0048] In this embodiment, the input sizes of the appearance branch and the motion branch are 4×112×112 and 32×112×112, respectively, so as to extract spatial information and motion information features.

[0049] Figure 1 The diagram shows the overall framework of PSC-TNet. The motion branch (rightmost branch) consists of a first convolutional block and the first to fourth residual layers connected in sequence. A spatial feature enhancement module (PSC) is added to the residual module of each residual layer. The appearance branch (centermost branch) consists of a second convolutional block and the fifth to eighth residual layers connected in sequence. A spatial feature enhancement module (PSC) is added to the residual module of each residual layer. The motion detail branch (leftmost branch) consists of a temporal feature extraction module (TtS), a third convolutional block, and the ninth to twelfth residual layers connected in sequence. A temporal feature extraction module (TtS) is added before each residual layer. A spatiotemporal feature fusion module (PSC-T) is added to the residual module of each residual layer.

[0050] The time feature extraction module (TtS) is as follows: Figure 2As shown, it includes sequential video frame temporal feature extraction operations and several fifth convolutional layers. The video frame temporal feature extraction operation involves subtracting between each pair of adjacent frames, and then inputting the subtraction result into a fifth convolutional layer for feature extraction to obtain differential motion features. All differential motion features are then concatenated and fused as the output of the temporal feature extraction module. Each fifth convolutional layer is composed of convolution operations.

[0051] The number of fifth convolutional layers is equal to the number of medium-length video frames. The first video frame of the medium-length video frame is used as the input of the first convolutional layer. Each subsequent frame is subtracted from the previous video frame and then input into its corresponding convolutional layer for feature extraction to obtain differential motion features. Finally, the differential motion features obtained from all video frames are concatenated and fused as the output of the temporal feature extraction module.

[0052] The video frame temporal feature extraction operation consists of a convolutional layer with equal input and output channels, used to obtain weight scores. The specific formula is as follows:

[0053]

[0054] Score t =sigmoid(f t )

[0055] v=Concat(Score0,Score1,……,Score T )

[0056] Among them, f t Score is the motion information feature tensor obtained by subtracting frame T from frame T-1. t To obtain the spatial weight scores, Conv() is the spatial convolution function, Concat() is the stacking function, and x t Let v be the t-th input feature tensor, and v be the output feature tensor, i.e., the motion detail features.

[0057] The Spatial Feature Enhancement Module (PSC) is as follows: Figure 3 As shown, it is divided into two branches with different scales: the left branch is a 3×3 scale convolution, and the right branch is a 1×1 scale convolution. These are processed using global average max pooling and the Softmax function, respectively, as shown in the following formulas:

[0058] x l1 =Softmax reshape (GlobalPooling(Conv 3×3 (x)))

[0059] x l2 =reshape(Conv 3×3(x))

[0060] x r1 =Softmax reshape (GlobalPooling(Conv 1×1 (x)))

[0061] x r2 =reshape(Conv 1×1 (x))

[0062] Where x is the initial input feature, x l1 and x l2 For the feature branch with high receptive field, x r1 and x r2 This represents a feature branch with a low receptive field. To compensate for the limitations of a single scale and thus perform multi-scale spatial modeling, we next perform cross-spatial information aggregation, i.e., using x... r2 For x l1 Adjust the weights using x l2 For x r1 The weights are adjusted using the following formula:

[0063] x l =x l1 ×x r2

[0064] x r =x r1 ×x l2

[0065] x out =x l +x r

[0066] x = x × x out

[0067] Where, x l For the spatial weights of the low receptive field branch, x r The spatial weights of the high receptive field branches are summed to obtain the final x. out The spatial weights, multiplied by the initial x-features, yield the final output of the spatial feature enhancement module.

[0068] A spatiotemporal feature fusion module (PSC-T) is added to each residual module in the action detail branch. This module combines the spatial feature enhancement module (PSC) and the temporal feature extraction module (TtS). The spatiotemporal feature weights extracted by the temporal feature extraction module (TtS) are added at the final weight summation stage of the spatial feature module (PSC). The specific formula is as follows:

[0069] x weight =xl +x r +v

[0070] x = x × x weight

[0071] Where, x weight This is the final feature weight map of the spatiotemporal feature fusion module. Multiplying it by the initial feature x yields the output of the spatiotemporal feature fusion module.

[0072] In the motion detail branch, a spatiotemporal feature fusion module (PSC-T) was added to each residual module to enhance the performance of the staggered block and its sensitivity to space. The specific formula is as follows:

[0073] x1 = Relu(BN(Conv(x)))

[0074] x2 = Relu(BN(Conv(x1)))

[0075] x3 = BN(Conv(x2))

[0076]

[0077] The initial feature x passes through the first convolutional layer, the first batch of normalization layers, the first activation layer, the second convolutional layer, the second batch of normalization layers, the second activation layer, the third convolutional layer, and the third batch of normalization layers in sequence, and is input into the spatiotemporal feature fusion module (PSC-T) or the spatial feature enhancement module (PSC). That is, the middle is the motion detail branch, and the spatiotemporal feature fusion module (PSC-T) is inserted into the residual module of the motion detail branch. The fast and slow are the motion branch and the appearance branch, respectively, and the spatial feature enhancement module (PSC) is inserted in them.

[0078] Each residual layer comprises multiple sequentially connected residual modules. The number of residual modules in the four residual layers are 3, 4, 6, and 3, respectively. The first residual module in each residual layer undergoes a downsampling operation to reduce the dimensionality of the features. Each residual module includes a residual mapping and an identity mapping. The input to each residual module is denoted as the input feature tensor. After feature extraction via the residual mapping, the input feature tensor yields a first space-motion feature tensor. Simultaneously, after the identity mapping, the input feature tensor yields a second space-motion feature tensor. The output feature tensor of the residual module is obtained by adding the first and second space-motion feature tensors, as shown in the following formula:

[0079] H(x) = F(x) + G(x)

[0080] Where H() is the output function of the residual module; F() is the residual mapping function; G() is the identity mapping function; and x is the input feature tensor of the residual module.

[0081] The network structure diagram of the first residual module in each residual layer is as follows: Figure 4 As shown, in its identity mapping, it is determined whether the number of channels of the output feature tensor (i.e., the first feature tensor) of the residual mapping is the same as the number of channels of the input feature tensor. If they are the same, the input feature tensor is directly used as the output feature tensor of the identity mapping; otherwise, the feature tensor after pointwise convolution of the input feature tensor is used as the output feature tensor of the identity mapping, i.e., the second feature tensor. This can be set using the following formula:

[0082]

[0083] Where G() is the identity mapping function; It is a convolution function with a kernel size of 1×1×1 and an output channel of C.

[0084] The feature fusion strategy involves concatenating and fusing the outputs of the first and third convolutional blocks along the channel dimension of the output of the second convolutional block to obtain the first spatial-motion feature tensor. The specific formula is as follows:

[0085] T(x) = Concat(DownSample) fast (x fast ),DownSample middle (x middle ),x slow )

[0086] Where T(x) is the output function of the stacking result; Concat is the concatenation function, which performs splicing and fusion along the channel dimension; DownSample fast DownSample is the upsampling function on the motion branch. middle For the upsampling function on the motion detail branch, trilinear interpolation is used to preserve the stackable dimension size.

[0087] Similarly, the above is the first spatial-motion feature tensor. The outputs of the first residual layer and the ninth residual layer are concatenated and fused along the channel dimension of the fifth residual layer to obtain the second spatial-motion feature tensor. The second spatial-motion feature tensor, the output of the second residual layer, and the output of the sixth residual layer are concatenated, fused, and convolved along the channel dimension to obtain the third spatial-motion feature tensor. The third spatial-motion feature tensor, the output of the third residual layer, and the output of the seventh residual layer are concatenated, fused, and convolved along the channel dimension to obtain the fourth spatial-motion feature tensor. The fourth spatial-motion feature tensor, the output of the fourth residual layer, and the output of the eighth residual layer are concatenated, fused, and convolved along the channel dimension to obtain the final feature tensor.

[0088] After feature extraction through the three branches, the outputs from the three branches are added and fused to obtain an output feature tensor containing texture, temporal, and spatial information, which is then processed using the following formula:

[0089] B(X)=B fast (x)+B middle (x)+B slow (x)

[0090] Where B() is the fusion output function; B fast (x) represents the final output of the motion branch; B slow (x) is the final output of the appearance branch; B middle (x) is the final output of the motion detail branch; x is the input feature tensor.

[0091] 3) The final feature tensor is input into the action classification module to obtain the action recognition result. The action classification module includes a globally average pooling layer, a Dropout layer, and a classifier connected in sequence. The feature tensor containing spatial and motion information is used as the input to the globally average pooling layer, and the classifier outputs the action recognition result. In this embodiment, the classifier maps probability values ​​to 51 categories, and the category with the highest probability value is taken as the action category of the video image, processed using the following formula:

[0092] O(x)=Max(Linear(Dropout(GlobalAvgPool(x))))

[0093] Where O() is the network output function; Linear() is the fully connected function; Dropout() is the regularization function that randomly drops neurons with a given probability; Max() is the maximum value function; GlobalAvgPool() is the global average pooling function; and x is the input feature tensor.

[0094] This invention also provides a storage medium storing a computer program that, when executed by a processor, implements a PSC-TNet video action recognition method based on fused spatial features and frame difference information. The computer program consists of instructions corresponding to the implementation of the PSC-TNet video action recognition method based on fused spatial features and frame difference information.

[0095] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

[0096] In summary, a novel spatial feature enhancement module (PSC) and a temporal feature extraction module (TtS) are proposed, along with a three-branch structural innovation for the slowfast skeleton network. This results in a completely new action recognition network structure that achieves a certain improvement over the slowfast skeleton network on HMDB51.

Claims

1. A PSC-TNet video action recognition method based on fusion of spatial features and frame difference information, characterized in that, The method includes the following steps: sampling the input video at high, low, and medium frequencies to obtain video frames of the corresponding frequencies and construct the original video; inputting the original video into a PSC-TNet video action recognition model based on fused spatial features and frame difference information to extract action recognition results; the PSC-TNet video action recognition model includes three branches based on different frame intervals and an action classification module, namely a motion branch, an appearance branch, and an action detail branch; wherein, high-frequency video frames are used as feature tensors containing motion information input to the motion branch, low-frequency video frames are used as feature tensors containing spatial information input to the appearance branch, and medium-frequency video frames are used as feature tensors containing action detail information input to the action detail branch; the motion branch, appearance branch, and action detail branch all contain residual layers, and the features output from the motion branch and action detail branch are passed to the appearance branch and fused by channel splicing, and then input into the residual layer of the appearance branch; finally, the recognition features output from the three branches are spliced ​​and fused on the channel and input into the action classification module to obtain the final action classification result.

2. The PSC-TNet video action recognition method based on fused spatial features and frame difference information according to claim 1, characterized in that: In the PSC-TNet video action recognition model, both the motion branch and the appearance branch include a convolutional block connected in sequence and four consecutive residual layers containing a spatial feature enhancement module (PSC). The action detail branch includes a temporal feature extraction module (TtS), a convolutional block, and four consecutive spatiotemporal feature fusion modules connected in sequence. Each spatiotemporal feature fusion module includes a TtS module connected in parallel and a residual layer containing a PSC module. The inputs of the spatiotemporal feature fusion modules are respectively input to the TtS module and the residual layers to obtain their respective results. The results of each component are added together and fused to serve as the output of the spatiotemporal feature fusion module. The four residual layers of the motion branch, appearance branch, and action detail branch contain three, four, six, and three residual modules respectively, in the order of transmission. The convolutional blocks of the motion branch and the action detail branch, as well as the convolutional block of the appearance branch, are concatenated on the channel of the appearance branch and then input into the first residual layer of the appearance branch. The nth residual layer of the motion branch and the action detail branch, as well as the nth residual layer of the appearance branch, are concatenated on the channel of the appearance branch and then input into the (n+1)th residual layer of the appearance branch.

3. The PSC-TNet video action recognition method based on fused spatial features and frame difference information according to claim 2, characterized in that: The residual module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a PSC-T attention mechanism module, and a fourth convolutional layer. The first, second, third, and PSC-T attention mechanism modules are connected in sequence. The inputs of the residual module are input to the first and fourth convolutional layers, respectively. The outputs of the PSC-T attention mechanism module and the fourth convolutional layer are added together to obtain the output of the residual module. The first and second convolutional layers are mainly composed of convolution operations, batch normalization operations, and activation operations connected in sequence. The third convolutional layer is mainly composed of convolution operations and batch normalization operations connected in sequence. The fourth convolutional layer is composed of a single convolution operation.

4. The PSC-TNet video action recognition method based on fused spatial features and frame difference information according to claim 3, characterized in that: The PSC-T attention mechanism module of the residual module in the residual layer of the motion branch, appearance branch and action detail branch adopts the spatial feature enhancement module PSC.

5. The PSC-TNet video action recognition method based on fused spatial features and frame difference information according to claim 2, characterized in that: The temporal feature extraction module includes sequential video frame temporal feature extraction operations and several fifth convolutional layers. The video frame temporal feature extraction operation involves subtracting between each pair of adjacent frames, and then inputting the subtraction result into a fifth convolutional layer for feature extraction to obtain differential motion features. All differential motion features are then concatenated and fused as the output of the temporal feature extraction module. The fifth convolutional layers are mainly composed of convolution operations.

6. The PSC-TNet video action recognition method based on fused spatial features and frame difference information according to claim 2, characterized in that: The Spatial Feature Enhancement Module (PSC) includes two spatial feature branches, an addition operation, multiple sigmoid activation functions, and a pointwise multiplication operation. The PSC is input into the two spatial feature branches, which are a high receptive field branch and a low receptive field branch, respectively. The convolutional layer size in the high receptive field branch is larger than that in the low receptive field branch. Each spatial feature branch has the same topology, including a sixth convolutional layer, a seventh convolutional layer, a global pooling layer, a first reconstruction layer, a second reconstruction layer, and a softmax activation function. The input of the PSC is used as the input to the spatial feature branches and is fed into the sixth and seventh convolutional layers, respectively. The output of the sixth convolutional layer is sequentially processed through the global pooling layer, the first reconstruction layer, and the softmax activation function to obtain the AA feature. The output of the seventh convolutional layer is passed through the second reconstruction layer to obtain BB features, AA features, and BB features as the output of the spatial feature branch. The AA features output by the high receptive field branch and the BB features output by the low receptive field branch are processed by matrix multiplication and then input into the first sigmoid activation function. The AA features output by the low receptive field branch and the BB features output by the high receptive field branch are processed by matrix multiplication and then input into the second sigmoid activation function. The outputs of the first and second sigmoid activation functions are added together by their respective pixels and then input into the third sigmoid activation function. The output of the third sigmoid activation function and the input of the spatial feature enhancement module PSC are multiplied point by point and then output as the output of the spatial feature enhancement module PSC.

7. The PSC-TNet video action recognition method based on fused spatial features and frame difference information according to claim 1, characterized in that: The action classification module includes a global average pooling layer, a Dropout layer, and a classifier connected in sequence. Recognition features containing spatial information, motion information, and motion detail information are used as input to the global average pooling layer, and the classifier outputs the action recognition result.

8. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method described in any one of claims 1 to 7.

Images