A human action recognition method and system based on a double-flow self-attention mechanism

Abstract

The application belongs to the technical field of computer vision, and relates to a human action recognition method and system based on a double-flow self-attention mechanism, which comprises the following steps: collecting video data of human actions; extracting multiple modal features of the video data through a basic network of a double-flow self-attention mechanism model, and fusing the multiple modal features with a time flow; inputting the fused space-time features into a space-time memory network of the double-flow self-attention mechanism model, performing space-time feature interaction of different scales, and generating a preliminary action classification result; and estimating the action boundary and action center offset position of the preliminary action classification result through an action prediction head, and generating a final human action recognition result. The application can effectively extract human action video features, can extract human features from two levels of a macro instance level and a micro fine-grained level, and thus can reduce the interference of an action video background environment on an action recognition result.

Description

TECHNICAL FIELD

[0001] The application relates to a human action recognition method and system based on a double-flow self-attention mechanism, and belongs to the technical field of computer vision. BACKGROUND

[0002] As an important research direction in the field of computer vision, human action recognition has a profound impact on people's daily life. At present, the human action recognition method based on deep learning has a wide application prospect in human-computer interaction, safety monitoring, health evaluation, sports analysis and the like. The human action recognition method needs to use video data samples collected in daily life, and the data samples usually have multi-modal information such as RBG image data, optical flow information and audio information. When human action recognition is performed in an actual complex environment, not only the above-mentioned complex multi-modal data needs to be processed, but also the features of the human body and the background area in the video need to be accurately distinguished. In addition, the action recognition method needs a certain robustness to the changes of the light and the shooting angle of the data background; and the calculation amount required to improve the accuracy of human action recognition in a complex environment will increase greatly. In actual production application, the action recognition model algorithm usually has a real-time requirement, and the calculation amount is too large to affect the practicability of the recognition model. SUMMARY

[0003] In view of the above problems, the purpose of the present application is to provide a human action recognition method and system based on a double-flow self-attention mechanism, which balances the precision and speed.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a human action recognition method based on a double-flow self-attention mechanism, comprising the following steps: collecting video data of human action; extracting a plurality of modal features of the video data through a basic network of a double-flow self-attention mechanism model, and fusing the modal features with a time flow; inputting the fused spatio-temporal features into a spatio-temporal memory network of the double-flow self-attention mechanism model to interact the spatio-temporal features of different scales and generate a preliminary action classification result; and estimating the action boundary and the action center offset position of the preliminary action classification result through an action prediction head to generate a final human action recognition result.

[0005] Further, in the spatio-temporal memory network, the fused spatio-temporal features are subjected to local context feature enhancement through a plurality of convolution blocks, a part of the features output by the convolution blocks is encoded through a cross-channel interaction encoder, the encoded data is combined with another part of the features output by the convolution blocks, the combined features are subjected to feature dimension reduction, the features subjected to the dimension reduction are input into a plurality of spatio-temporal memory convolution blocks for feature encoding, and the preliminary action classification result is output after the feature encoding.

[0006] Further, the feature dimension reduction is realized through a double-flow Manhattan self-attention layer.

[0007] Further, the double-flow Manhattan self-attention layer includes an instance-level feature extraction flow, a window-level feature extraction flow, and an original path branch; the instance-level feature extraction flow distinguishes action and non-action features by expanding the feature distance of video features; the window-level feature extraction flow uses a larger receptive field than the original video data to understand high-dimensional semantic information features of the double-flow self-attention mechanism model; the original path branch performs identity mapping on the original features, and performs feature transmission of the deep network through the residual connection method.

[0008] Further, in the instance-level feature extraction flow, part of the data is input into a full connection layer, and the other part of the data is input into a Manhattan self-attention layer; the data after the full connection layer is multiplied by the data after the Manhattan self-attention layer to generate an output result; the window-level feature extraction flow is divided into three branches; the first branch is added to the second branch after k-group convolution; the data stream after addition is multiplied by the third branch after convolution to generate an output result; and the output results of the instance-level feature extraction flow and the window-level feature extraction flow are added to the output result of the original path branch to generate an output result of the double-flow Manhattan self-attention layer.

[0009] Further, the basic network performs feature extraction on the video data through RetNet x , y in two dimensions, performs local feature enhancement on the video data in two dimensions, and the expression of the features extracted from the video data is:

[0010]

[0011]

[0012] wherein, is a Manhattan self-attention mechanism, X is an input feature, QKV the matrix is three matrices used for calculating weights of the self-attention mechanism, Q is a query vector matrix, K is a key vector matrix, V is a value vector matrix, K T is K a transpose matrix of the matrix, D is a causal mask and an exponential decay matrix, D the matrix is a real matrix and an input feature X have the same dimension, store the relative distance in the one-dimensional sequence of the matrix, and provide explicit temporal prior information. D 2d is a two-dimensional D matrix, n andm are two dimensions of this matrix, are QKV exponential decay factors of the matrix.

[0013] Further, the base network is used to x , y sequence modeling of two-dimensional features, time sequence modeling of image data action sequence through Transformers layer with reservation mechanism, and the calculation formula of self-attention stage in the Transformers layer is as follows:

[0014] .

[0015] wherein, and are Q query vector matrix and K key vector matrix decomposed according to the direction of the axis, y and are Q query vector matrix and K key vector matrix decomposed according to the direction of the axis. B is the batch size, L is the length after the original spatial decay matrix is tiled into a one-dimensional tensor, C is the number of matrix channels, W and H are the width and height of the decay matrix after decomposition. is spatial self-attention decomposed according to H, y is spatial self-attention decomposed according to W, is spatial decay matrix decomposed according to H, is spatial decay matrix decomposed according to W. is Manhattan self-attention on the original features. On this basis, the basic topology structure of the feature extraction part is composed, and the expression is as follows: , is local feature enhancement, which is specifically performed by using a depth separable convolution to enhance the local features. Further, the action prediction head includes an action start boundary branch, an action end boundary branch and an intermediate offset branch, the action start boundary branch is used to predict the response strength of the start boundary of each action branch; the action end boundary branch is used to predict the response strength of the end boundary of each action branch; the intermediate offset branch takes a certain instance as a reference standard, and the start point or the end point of the two adjacent local time sets before and after the certain instance as the response strength, calculates the response expectation value through a local window to obtain the boundary prediction value of each instance.

[0016] Further, the overall loss function of the double-flow self-attention mechanism model is as follows:

[0017]

[0018]

[0019] wherein,​​ is a loss function, is a number of positive samples, l is a length of a spatial feature pyramid layer, t is a reading feature time length, is an L1 norm; is a classification label; is an IoU distance between a time dimension of a prediction sample and Ground Truth, is a classification loss function, is a regression loss, is a number of negative samples.

[0020] The application also discloses a human action recognition system based on a double-flow self-attention mechanism, comprising: a data acquisition module, which is used for acquiring video data of human actions; a basic network module, which is used for extracting multiple modal features of the video data through a basic network of a double-flow self-attention mechanism model and fusing the multiple modal features with a time flow; a space-time memory network module, which is used for inputting the fused space-time features into a space-time memory network of the double-flow self-attention mechanism model, performing space-time feature interaction of different scales, and generating a preliminary action classification result; and an action recognition module, which is used for estimating an action boundary and an action center offset position of the preliminary action classification result through an action prediction head and generating a final human action recognition result.

[0021] The technical scheme of the application has at least the following technical effects or advantages:

[0022] 1. The double-flow self-attention structure is adopted to build a lightweight network for effectively extracting human action video features.

[0023] 2. The application can extract human features from two levels of macro instance level and micro fine-grained level, thereby reducing the interference of action video background environment on the action recognition result.

[0024] 3. The application has high human action recognition accuracy in a complex environment, has a certain balance between model inference speed and precision, and has good practical application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a whole flowchart of a double-flow self-attention mechanism network in an embodiment of the application;

[0026] Figure 2 is a structural schematic diagram of a space-time memory network in an embodiment of the application;

[0027] Figure 3 is an action recognition schematic diagram of an action prediction head in an embodiment of the application. DETAILED DESCRIPTION

[0028] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be described in detail through specific embodiments. It should be understood, however, that the specific embodiments are provided only for better understanding of the present application, and should not be understood as limiting the present application. In the description of the present application, it should be understood that the terms used are only for the purpose of description, and should not be understood as indicating or implying relative importance.

[0029] In order to solve the problems in the prior art that the features of the human body and the background area in the video cannot be accurately distinguished, and the amount of calculation required to improve the accuracy of human action recognition in complex environments is increased, the present application builds a network structure based on a dual-flow self-attention mechanism, learns the spatio-temporal features of video data, and proposes a human action recognition method and system based on a dual-flow self-attention mechanism, which realizes a balance between precision and speed. The present application collects human action data, constructs, and trains a dual-flow self-attention network model; uses the spatio-temporal features extracted by the dual-flow self-attention network model; and uses the extracted spatio-temporal features for action classification and time positioning. The present application uses a dual-flow self-attention structure to build a lightweight network for effectively extracting features of human action videos. At the same time, the present method can extract human features from two levels of macro instance level and micro fine-grained level, thereby reducing the interference of the action video background environment on the action recognition result. The present application has high accuracy in human action recognition in complex environments, and has a certain balance between model inference speed and precision, and has practical application prospects. The present application will be described in detail below in conjunction with the drawings through embodiments.

[0030] Embodiment one

[0031] The present embodiment discloses a human action recognition method based on a dual-flow self-attention mechanism, as shown in Figure 1 The present embodiment discloses a human action recognition method based on a dual-flow self-attention mechanism, as shown in

[0032] S1 collects video data of human actions.

[0033] Video data of human actions is collected, and a data set is made. The data set can be selected as UCF101, HMDB51, Kinetics, etc. These data sets contain multiple action categories and rich video samples. Human action videos are obtained through channels such as camera video acquisition equipment, video website download, and common data set collection.

[0034] Before the data is input into the dual-flow self-attention mechanism model, the video data needs to be preprocessed, and the preprocessing process includes: first, video frame extraction is performed to decompose the video into continuous frames, and usually a certain number of frames per second are selected, and in the embodiment, 25 frames are preferably selected, but the number of frames can also be determined according to actual needs. Then, the extracted image data is enhanced: the image data enhancement method for each frame includes rotation, scaling, cropping, color transformation and the like, and the image data enhancement is performed to improve the robustness of the dual-flow self-attention mechanism model. Finally, label processing is performed, the action categories are counted and recorded, the action labels are made, and it is ensured that each video frame or sequence has a corresponding action label.

[0035] S2 extracts multiple modal features of the video data through the base network of the dual-flow self-attention mechanism model, and fuses them with the time flow.

[0036] The dual-flow self-attention mechanism network includes a base network and a space-time memory network.

[0037] The base network is used to process 2D image data, and the existing Retention layer is subjected to 1D to 2D operation. The base network performs feature extraction on the video data through RetNet x , y Two-dimensional feature extraction is performed on the video data to enhance the local features in two dimensions. The performances of 1D and 2D are different, and the 1D language task is one-dimensional unidirectional data, but the video data processed in the embodiment needs to be subjected to two-dimensional feature extraction. Therefore, the n-m part of the Retention layer needs to take the absolute value, which can be expressed as:

[0038]

[0039]

[0040]

[0041] wherein, is the output matrix corresponding to the n direction, is the query vector matrix corresponding to the n direction, is the transformer rotation position embedding angle, is the conjugate transpose, is the value vector matrix corresponding to the m direction, is the two-dimensional bidirectional retention mechanism, X is the input feature, T is the matrix transpose, D is the spatial decay matrix, V is the value vector matrix, n is the x dimension direction matrix column number, and m is the y dimension direction matrix row number, is the exponential decay factor.

[0042] The calculation process of the D matrix of the retention layer is also similar, and the absolute value needs to be taken in the x and y dimensions:

[0043]

[0044] wherein d is the matrix dimension, is the x coordinate of the token after decomposition in the n-dimensional direction, x is the x coordinate of the token after decomposition in the m-dimensional direction, is the y coordinate of the token after decomposition in the n-dimensional direction, y is the y coordinate of the token after decomposition in the m-dimensional direction, is the x coordinate of the token after decomposition in the n-dimensional direction, y is the x coordinate of the token after decomposition in the m-dimensional direction, is the y coordinate of the token after decomposition in the n-dimensional direction.

[0045] The expression of the feature extracted from the video data is:

[0046]

[0047]

[0048] wherein, is the Manhattan self-attention mechanism, X is the input feature, Q is the query vector matrix, K is the key vector matrix, T is the matrix transpose, D is the spatial decay matrix, d is the matrix dimension, V is the value vector matrix, n is the token position after flattening the matrix in the n-dimensional direction, m is the token position after flattening the matrix in the m direction, is the exponential decay factor.

[0049] The basic module follows the feature extraction form of the Transformer layer, and the image data action sequence is time-series modeled through the Transformers layer with a retention mechanism. The time decay brought by the retention mechanism is not possessed by the existing Transformers method, and the structure is used in the embodiment to time-series model the image data action sequence. The problem of too large token quantity in the self-attention stage, resulting in a huge increase in calculation amount, is solved, and the image is decomposed into x ,two axes, reducing the calculation consumption. The calculation formula of the self-attention stage is: y

[0050] .

[0051] wherein, is the query vector matrix after H direction decomposition, is the key vector matrix after H direction decomposition, B is the batch size, L is the length after matrix flattening, C is the channel number, W represents the width direction of the original feature matrix, H is the height direction of the original feature matrix, is the query vector matrix decomposed in W direction, is the key vector matrix decomposed in W direction, is the self-attention mechanism in H direction, is the self-attention mechanism in W direction, is the spatial decay matrix decomposed in H direction, is the spatial decay matrix decomposed in W direction, is the Manhattan self-attention mechanism. On this basis, the basic topology structure of the feature extraction part is composed, and its expression is , is the local feature enhancement, which performs local feature enhancement in two dimensions on the original feature.

[0052] The embodiment is for a time series action recognition task (TAD), and a video stream to be recognized wherein n represents the number of video frames, V i represents a video frame at a certain time. If the video data input into the network contains multi-modal features such as RBG channel image features, optical flow features, sound features, etc., the feature group input into the network is X i , is obtained by V i The original video data is extracted, and also the time series T corresponding. For the action label of the video frame K i , the starting frame of the action is recorded as s k , the end frame is e k , and the action category is c k , so there is an action label table . The basic network of the dual-stream self-attention mechanism is used for the feature extraction basic module. The feature pyramid is used to process actions of different time lengths.

[0053] S3 inputs the fused space-time features into the space-time memory network of the dual-stream self-attention mechanism model, interacts the space-time features of different scales, and generates a preliminary action classification result.

[0054] After the basic network, multiple down-sampling is performed through a maximum pooling step of 2, and a space-time memory network similar to a Transformer layer is built at the back end to enhance the space-time dimensions of the pyramid features of each scale. Through feature iterative down-sampling, different granularity and different time range feature interactions can be effectively perceived across scales.

[0055] In the space-time memory network, the fused space-time features are enhanced by local context features through several convolution blocks. Part of the features output by the convolution blocks are encoded by a cross-channel interaction encoder. The encoded data is combined with another part of the features output by the convolution blocks. The combined features are dimensionally reduced, and the reduced features are input into several space-time memory convolution blocks for feature encoding. After the feature encoding, a preliminary action classification result is output.

[0056] The enhanced context features are processed by a dual-flow self-attention mechanism. As shown below:

[0057]

[0058]

[0059]

[0060] wherein, is a dual-flow self-attention mechanism, is a video-level average feature, is a full convolution operation on the input feature X, is a local feature branch, is a convolution operation, is a convolution in a w window size, is a convolution in a w window size on the input feature x with a k expansion factor, and the k factor is used to measure large-granularity time information.

[0061] The biggest problem of the space-time feature enhancement mechanism is that it is easy to introduce complex space-time features, causing high computational burden. In this embodiment, the decomposition of the dual-flow Manhattan self-attention layer MaSA is used to ensure that the receptive fields between tokens are the same, thereby reducing the computational complexity. The dual-flow Manhattan self-attention layer is mainly used to realize feature dimension reduction. As shown in Figure 2 The dual-flow Manhattan self-attention layer is used to build a dual-flow self-attention feature pyramid, which includes an instance-level feature extraction flow, a window-level feature extraction flow, and an original path branch. The instance-level feature extraction flow distinguishes action and non-action features by expanding the feature distance of the video features. The window-level feature extraction flow uses a larger receptive field than the original video data to understand the high-dimensional semantic information features of the dual-flow self-attention mechanism model. The original path branch maps the original features identically and transmits the features of the deep network through the residual connection method. This network structure can improve the discrimination between action and non-action features, improve the understanding ability of the model for high-dimensional semantic information features, enhance the feature transmission ability of the deep network, and improve the learning effect.

[0062] Part of the data in the instance-level feature extraction stream enters the full connection layer, and the other part of the data enters the Manhattan self-attention layer. The data passing through the full connection layer is multiplied by the data passing through the Manhattan self-attention layer to generate an output result. The window-level feature extraction stream is divided into three branches. The first branch is added to the second branch after convolution and k-group convolution. The added data stream is multiplied by the third branch after convolution to generate an output result. The output results of the instance-level feature extraction stream and the window-level feature extraction stream are added to the output result of the original path branch to generate the output result of the double-flow Manhattan self-attention layer.

[0063] In order to cope with action features of different time lengths, a window scaling coefficient w is introduced in the window-level feature extraction stream to expand the distance between action and non-action and instance average features, and to distinguish action boundaries. The window-level feature extraction stream not only can introduce semantic feature information of different receptive fields to increase multi-scale feature recognition capability, but also can perform feature interaction on different time actions.

[0064] S4 estimates the action boundary and action center offset position of the preliminary action classification result through the action prediction head to generate a final human action recognition result.

[0065] The action prediction head includes an action start boundary branch, an action end boundary branch, and an intermediate offset branch. The action start boundary branch is used to predict the response strength of the start boundary of each action branch. The action end boundary branch is used to predict the response strength of the end boundary of each action branch. The intermediate offset branch takes a certain instance as a reference standard, takes the start point or end point of the two adjacent local time sets before and after the instance as the response strength, calculates the response expectation value through the local window, and obtains the boundary prediction value of each instance. For example, the action start point distance of the first instance is t d st , and is the start boundary prediction formula.

[0066]

[0067]

[0068]

[0069]

[0070] Similarly, the end boundary prediction formula is as follows:

[0071]

[0072] ​

[0073] where, is the probability of the start boundary of an action in a certain action set at a certain time, is the value of instance t in the action start set, is the value of instance t in the action offset set, b is the number of boundary prediction sets, is the expectation of instance t in the start boundary prediction, is the probability of the occurrence of an action in the bth prediction set, is the distance of an action relative to the start time, l is the feature scaling factor, t is the instance of the motion feature, is the distance of an action relative to the end time; B is the number of frames of the boundary prediction F s and F e are the response values at each time as the starting point and end point of an action, respectively, is the probability of the end boundary of an action in a certain action set at a certain time; is the distance of the end point of an action of the tth instance. t

[0074] The three action prediction heads are modeled using ordinary convolutional layers, sharing parameters across all feature pyramid scales to reduce the number of parameters.

[0075] The feature pyramid outputs time features for each instance t, and the action prediction head of the feature pyramid is used for action classification. The feature pyramid representation of each instance t is: .

[0076] The overall loss function of the dual-stream self-attention mechanism model is as follows:

[0077]

[0078] where, is the overall loss function, is the number of positive samples, l is the length of the spatial feature pyramid, t is the reading feature time length, is the one-norm; is the classification label; is the IoU distance between the prediction sample and the GT in the time dimension, is the classification loss function, is the regression loss function, is the number of negative samples.

[0079] ​In the embodiment, the hardware configuration of the dual-flow self-attention mechanism model is as follows: the CPU processor is AMD Ryzen 9 3950X 16-Core Processor, and the GPU is NVIDIA GeForce RTX 3090; the software configuration is as follows: the computer operating system is Ubuntu 24.04, the CUDA version is 11.2, the neural network framework used is Pytorch, and the version is 2.1.0. The initial learning rate is set to 10-4, the optimizer strategy is AdamW, the classification head is initialized as Gaussian distribution initialization (0, 0.1), the number of training iterations is 40, the Warmup strategy is used for the first 10 iterations, the window factor w in the training head dual-flow attention branch is set to 1.5 for scaling, and the parameters are appropriately adjusted.

[0080] The action recognition results on the THUMOS14 action recognition dataset are given below. The dataset is based on YouTube video collection, which contains sports categories in daily life. The dataset contains 13000 video clips, and the action categories are 20. Here, the average precision mAP (mean Average Precision) of different IoU is selected to measure the action recognition accuracy of different categories, and the inference time is selected to measure the model running efficiency. Table 1 shows different recognition results of the above dataset in different algorithm recognition models, and the measurement indicators are the average precision and the inference delay (ms).

[0081] As shown in Table 1, the embodiment achieves a balance in precision and speed in the THUMOS14 dataset action recognition task, which is better than other recognition models, proving the advantages of the method in the embodiment in recognition efficiency.

[0082] Table 1 shows different recognition results of the THUMOS14 dataset in different algorithm recognition models

[0083]

[0084] Embodiment Two

[0085] Based on the same inventive concept, the embodiment discloses a human action recognition system based on a dual-flow self-attention mechanism, which comprises:

[0086] A data acquisition module is configured to acquire video data of human actions.

[0087] A basic network module is configured to extract multiple modal features of the video data through a basic network of the dual-flow self-attention mechanism model, and fuse them with a time flow.

[0088] The spatio-temporal memory network module is configured to input the fused spatio-temporal features into a spatio-temporal memory network of a double-flow self-attention mechanism model, perform spatio-temporal feature interaction of different scales, and generate a preliminary action classification result.

[0089] The action recognition module is configured to estimate an action boundary and an action center offset position of the preliminary action classification result through an action prediction head, and generate a final human action recognition result.

[0090] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0091] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0092] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0094] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the above examples, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application. The above content is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A human action recognition method based on a two-stream self-attention mechanism, characterized in that, Includes the following steps: Collect video data of human movements; Multiple modal features of the video data are extracted using the base network of a dual-stream self-attention mechanism model and fused with the temporal stream. The fused spatiotemporal features are input into the spatiotemporal memory network of the two-stream self-attention mechanism model to perform spatiotemporal feature interaction at different scales and generate preliminary action classification results. The action prediction head is used to estimate the action boundaries and action center offset positions of the preliminary action classification results, and the final human action recognition results are generated. In the spatiotemporal memory network, the fused spatiotemporal features are enhanced with local contextual features through several convolutional blocks. Part of the features output by the convolutional blocks are encoded by a cross-channel interactive encoder. The encoded data is combined with the features output by another part of the convolutional blocks. The combined features are then subjected to feature dimensionality reduction. The dimensionality-reduced features are then input into several spatiotemporal memory convolutional blocks for feature encoding. After feature encoding, a preliminary action classification result is output. The feature dimensionality reduction is achieved through a dual-stream Manhattan self-attention layer; The dual-stream Manhattan self-attention layer includes an instance-level feature extraction stream, a window-level feature extraction stream, and an original path branch. The instance-level feature extraction stream distinguishes action and non-action features by expanding the feature distance of video features. The window-level feature extraction stream uses a larger receptive field than the original video data to understand the high-dimensional semantic information features of the dual-stream self-attention mechanism model. The original path branch performs an identity mapping on the original features and performs feature transfer of the deep network through the residual connection method. In the instance-level feature extraction stream, a portion of the data is input to the fully connected layer, and another portion enters the Manhattan self-attention layer. The data that has passed through the fully connected layer is multiplied with the data that has passed through the Manhattan self-attention layer to generate the output result. The window-level feature extraction stream is divided into three branches. The first branch undergoes convolution and is then added to the second branch, which has undergone k sets of convolutions. The data stream after addition is multiplied with the third branch that has undergone convolution to generate the output result. The output results of the instance-level feature extraction stream and the window-level feature extraction stream are then added to the output results of the original path branches to generate the output result of the dual-stream Manhattan self-attention layer.

2. The human action recognition method based on a two-stream self-attention mechanism as described in claim 1, characterized in that, The underlying network processes video data using RetNet. x , y Two-dimensional feature extraction involves performing local feature enhancement on the video data in two dimensions. The expression for the extracted features from the video data is as follows: in, This is the Manhattan self-attention mechanism, where X is the input feature matrix, Q is the query vector matrix, K is the key vector matrix, T is the matrix transpose, D is the exponential decay matrix, d is the matrix dimension, V is the value vector matrix, and n and m are the two dimensions of the matrix. It is an exponential decay factor.

3. The human action recognition method based on a two-stream self-attention mechanism as described in claim 2, characterized in that, The basic network x , y Sequence modeling is performed using two-dimensional features. Temporal modeling of the image data action sequence is then performed using Transformers layers with a preservation mechanism. The calculation formula for the self-attention stage in the Transformers layers is as follows: . in, It is the query vector matrix decomposed along the H direction. It is the key vector matrix decomposed along the H direction, B is the batch size, L is the length of the spatial decay matrix after tiling into a one-dimensional tensor, C is the number of matrix channels, and W and H are the two widths and heights of the exponential decay matrix. It is a query vector matrix decomposed along the W direction. It is a key vector matrix decomposed along the W direction. It is spatial self-attention in the H direction. It is spatial self-attention in the W direction. It is an exponentially decaying matrix decomposed along the H direction. It is an exponentially decaying matrix decomposed in reverse W. It is Manhattan self-attention, which forms the basic topological structure of the feature extraction part, and its expression is: , It is a local feature enhancement.

4. The human action recognition method based on a two-stream self-attention mechanism as described in claim 1, characterized in that, The action prediction head includes an action start boundary branch, an action end boundary branch, and an intermediate offset branch. The action start boundary branch is used to predict the response intensity of the start boundary of each action branch; the action end boundary branch is used to predict the response intensity of the end boundary of each action branch; the intermediate offset branch uses an instance as a reference standard, takes the start or end point of two adjacent local time sets before and after it as the response intensity, calculates the expected response value through a local window, and obtains the boundary prediction value for each instance.

5. The human action recognition method based on a two-stream self-attention mechanism as described in claim 1, characterized in that, The overall loss function of the two-stream self-attention mechanism model is as follows: in, It is the overall loss function. It is the number of positive samples. l It is the length of the spatial feature pyramid. t It is the length of time to read the features. It is to find a norm; These are category tags; It is the IoU distance in the time dimension between the predicted sample and the ground truth. It is classification loss. It is a regression loss. It represents the number of negative samples.

6. A human motion recognition system based on a two-stream self-attention mechanism, characterized in that, include: The data acquisition module is used to collect video data of human movements; The basic network module is used to extract multiple modal features of the video data through the basic network of the dual-stream self-attention mechanism model and fuse them with the temporal stream; The spatiotemporal memory network module is used to input the fused spatiotemporal features into the spatiotemporal memory network of the dual-stream self-attention mechanism model, perform spatiotemporal feature interaction at different scales, and generate preliminary action classification results. The action recognition module is used to estimate the action boundaries and action center offset positions of the preliminary action classification results through the action prediction head, and generate the final human action recognition results. In the spatiotemporal memory network, the fused spatiotemporal features are enhanced with local contextual features through several convolutional blocks. Part of the features output by the convolutional blocks are encoded by a cross-channel interactive encoder. The encoded data is combined with the features output by another part of the convolutional blocks. The combined features are then subjected to feature dimensionality reduction. The dimensionality-reduced features are then input into several spatiotemporal memory convolutional blocks for feature encoding. After feature encoding, a preliminary action classification result is output. The feature dimensionality reduction is achieved through a dual-stream Manhattan self-attention layer; The dual-stream Manhattan self-attention layer includes an instance-level feature extraction stream, a window-level feature extraction stream, and an original path branch. The instance-level feature extraction stream distinguishes action and non-action features by expanding the feature distance of video features. The window-level feature extraction stream uses a larger receptive field than the original video data to understand the high-dimensional semantic information features of the dual-stream self-attention mechanism model. The original path branch performs an identity mapping on the original features and performs feature transfer of the deep network through the residual connection method. In the instance-level feature extraction stream, a portion of the data is input to the fully connected layer, and another portion enters the Manhattan self-attention layer. The data that has passed through the fully connected layer is multiplied with the data that has passed through the Manhattan self-attention layer to generate the output result. The window-level feature extraction stream is divided into three branches. The first branch undergoes convolution and is then added to the second branch, which has undergone k sets of convolutions. The data stream after addition is multiplied with the third branch that has undergone convolution to generate the output result. The output results of the instance-level feature extraction stream and the window-level feature extraction stream are then added to the output results of the original path branches to generate the output result of the dual-stream Manhattan self-attention layer.

Images