Human action recognition method and system based on attention mechanism, and electronic device

Abstract

The application discloses a human action recognition method and system based on an attention mechanism and electronic equipment, and belongs to the technical field of human action recognition. The application introduces hourglass modules and coordinate attention modules, and constructs two light network modules, namely, an hourglass coordinate attention bottleneck module and an hourglass coordinate attention base module, on the basis, reduces the model parameter quantity and the operation complexity, and finally obtains long-range dependence and accurate position information of a feature map space direction. Thus, the technical problem that a high-resolution human action recognition network has large network parameter quantity and high operation complexity is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human action recognition, and more particularly, to a human action recognition method and system based on an attention mechanism and an electronic device. BACKGROUND

[0002] Human action recognition is the detection and positioning of human key points in a target, and as an important research topic in computer vision, it has a wide range of applications in pedestrian detection, human-computer interaction, and pedestrian re-identification. In the development of human action recognition research, the graph structure model algorithm has always been the mainstream way of traditional human action recognition algorithms. The graph structure model algorithm mainly consists of three parts, which are a human part detection model, a human graph model structure, and a corresponding pose reasoning algorithm. The algorithm first decomposes the human structure into different parts, then models the relationship between the parts using a graph structure model, and finally connects each part to form a complete human pose. However, in the case of complex background and highly flexible human poses, the detection accuracy and efficiency of the graph structure model will decrease sharply, making it difficult to achieve the level of practical application.

[0003] With the rise and development of deep neural networks, Toshev et al. first introduced deep neural networks into human action recognition algorithms in 2014, treating human action recognition as a human key point regression problem. The algorithm extracts features from the input image using a deep convolutional neural network, and uses a convolution kernel to model the relationship between key points. Because different convolution kernels learn different parameters, the detection performance of human key points is effectively improved. However, as the requirements for the performance of human action recognition networks continue to increase, the number of model parameters is constantly increasing, and the computational complexity is also rapidly increasing. Therefore, under the premise of maintaining the detection accuracy of human key points by the network model, how to reduce the parameter quantity and computational complexity of the network model during operation is an important problem faced by the improvement and optimization of current human action recognition network models.

[0004] The high-resolution network (HRNet) proposed in 2019 reconsiders the influence of resolution on the human motion recognition network model, which is different from the stacked hourglass network (Hourglass) which adopts a symmetrical structure of high-resolution downsampling to low-resolution and then upsampling from low-resolution to high-resolution. The HRNet model improves the Hourglass and proposes a human motion recognition network based on high resolution, which maintains high-resolution feature maps throughout the network model, to some extent, making up for the defect of spatial resolution loss caused by the upsampling process in Hourglass, and further verifying the importance of high-resolution feature maps in human motion recognition network model for human key point detection.

[0005] The invention application CN113158756A discloses a posture and behavior analysis module and analysis method based on HRNet deep learning, video stream acquisition is performed by a video stream acquisition module, human body detection is performed on images in the video stream by a human body detection module, and human body pictures meeting the requirements are cropped by a human body picture processing module. The HRNet model proposed in the invention application adopts a repeated multi-scale fusion method, and parallelly connects high-resolution to low-resolution subnets, uses low-resolution representations of the same depth and similar levels to improve high-resolution representations, so that the high-resolution representations are also sufficient for posture estimation, ensures that the predicted heat map is more accurate in space, and improves the accuracy, robustness and adaptability of the human key point detection algorithm.

[0006] That is, the HRNet model adopts a parallel subnet method, realizes sufficient fusion of multiple resolution feature maps, and enhances the feature information of the feature maps. However, since the HRNet model always maintains high-resolution feature maps, while improving the accuracy of predicting human key points, it also increases the parameter quantity and computational complexity of the network model. There is no corresponding solution to the problems of large network parameter quantity and high computational complexity of the high-resolution human motion recognition network. SUMMARY

[0007] In view of the deficiencies of the prior art, the present application provides a human motion recognition method and system based on an attention mechanism and an electronic device.

[0008] According to one aspect of the present application, a human motion recognition method based on an attention mechanism is provided, comprising:

[0009] The bottleneck module and the basic Block module in the existing HRNet model are redesigned to obtain a sandglass coordinate attention bottleneck module and a sandglass coordinate attention basic module.

[0010] The sandglass coordinate attention bottleneck module is used to replace the bottleneck module in the HRNet model, the sandglass coordinate attention base module is used to replace the base Block module in the HRNet model, the original architecture of the HRNet model is retained, and the SCANet model is obtained;

[0011] The target image of the human action to be recognized is input into the SCANet model, and a feature map with channel feature information and spatial direction feature information is output;

[0012] The feature map is subjected to vector distance calculation with a pre-trained visual relationship model, key node expression information is predicted, and finally, action classification is obtained; wherein,

[0013] The bottleneck module and the base Block module in the existing HRNet model are redesigned to obtain the operation of the sandglass coordinate attention bottleneck module and the sandglass coordinate attention base module, which specifically includes:

[0014] The sandglass module is used to replace the standard 3x3 convolution in the bottleneck module, and the scaling factor of the sandglass module is re-determined; wherein, the number of input channels of the sandglass module is equal to the number of output channels;

[0015] A coordinate attention module is added in the bottleneck module and the base Block module; wherein, the coordinate attention module is used to extract channel feature information and spatial direction feature information in the image, and the number of input channels of the coordinate attention module is equal to the number of output channels.

[0016] Optionally, the SCANet model inherits the original architecture of the HRNet model, and includes Stage1, Stage2, Stage3 and Stage4, a total of four stages; and the operation of inputting the target image of the human action to be recognized into the SCANet model to output a feature map with channel feature information and spatial direction feature information specifically includes:

[0017] The target image of the human action to be recognized is input into the SCANet model;

[0018] The SCANet model uses two standard 3x3 convolution kernels to pre-process the target image, so that the resolution of the target image becomes 1 / 4 of the original, and the number of channels changes from 3 to 64, obtaining a first feature map;

[0019] The SCANet model takes the first feature map as the input of Stage1, uses four sandglass coordinate attention bottleneck modules to extract features from the first feature map, and obtains a second feature map with channel feature information and spatial direction feature information;

[0020] The SCANet model takes the second feature map as an input of Stage 2, uses four hourglass coordinate attention base modules with different resolutions and different numbers of channels to extract features of the second feature map, and obtains a third feature map with channel feature information and spatial direction characteristic information;

[0021] The SCANet model takes the third feature map as an input of Stage 3, uses four hourglass coordinate attention base modules identical to those of Stage 2 to extract features of the third feature map, and obtains a fourth feature map with channel feature information and spatial direction characteristic information;

[0022] The SCANet model takes the fourth feature map as an input of Stage 3, uses four hourglass coordinate attention base modules identical to those of Stage 2 to extract features of the fourth feature map, and obtains a fifth feature map with channel feature information and spatial direction characteristic information and outputs the fifth feature map.

[0023] Optionally, the coordinate attention module extracts channel feature information and spatial direction feature information in the image by the following steps:

[0024] For the input feature map, the coordinate attention module uses an adaptive average pooling layer in the horizontal direction to extract spatial direction features of each feature channel, and uses an adaptive average pooling layer in the vertical direction to extract channel features of each feature channel;

[0025] The feature maps generated by the adaptive average pooling layers in the horizontal and vertical directions are spliced, and then a 1x1 convolution is used to generate an intermediate feature map with channel feature information and spatial direction feature information;

[0026] The intermediate feature map is divided into two temporary feature maps along the spatial direction, and a 1x1 convolution is used to convert the channel numbers of the two temporary feature maps, respectively, to obtain attention weights in the vertical and horizontal directions;

[0027] The attention weights in the vertical and horizontal directions are multiplied by the input feature map to obtain a feature map with channel feature information and spatial direction feature information.

[0028] Optionally, the scaling factor of the hourglass module is re-determined, and specifically includes:

[0029] Different scaling factors are respectively compared on the MPII dataset;

[0030] According to the result of the experimental comparison, the scaling factor of the hourglass module is determined as 1.

[0031] According to another aspect of the present application, there is provided a human action recognition system based on an attention mechanism, comprising:

[0032] a model design module, configured to redesign a bottleneck module and a basic Block module in an existing HRNet model to obtain a sandglass coordinate attention bottleneck module and a sandglass coordinate attention basic module;

[0033] a model determination module, configured to replace the bottleneck module in the HRNet model with the sandglass coordinate attention bottleneck module, replace the basic Block module in the HRNet model with the sandglass coordinate attention basic module, retain the original architecture of the HRNet model, and obtain a SCANet model;

[0034] a feature map acquisition module, configured to input a target image of a human action to be recognized into the SCANet model, and output a feature map with channel feature information and spatial direction feature information;

[0035] an action classification module, configured to perform vector distance calculation on the feature map and a pre-trained visual relationship model, predict key node expression information, and finally obtain action classification; wherein,

[0036] the model design module is specifically configured to:

[0037] the sandglass module is used to replace the standard 3x3 convolution in the bottleneck module, and the scaling factor of the sandglass module is re-determined; wherein the number of input channels of the sandglass module is equal to the number of output channels;

[0038] a coordinate attention module is added in the bottleneck module and the basic Block module; wherein the coordinate attention module is used to extract channel feature information and spatial direction feature information in an image, and the number of input channels of the coordinate attention module is equal to the number of output channels.

[0039] Optionally, the SCANet model inherits the original architecture of the HRNet model, and includes four stages, namely Stage1, Stage2, Stage3 and Stage4; and the feature map acquisition module is specifically configured to:

[0040] input the target image of the human action to be recognized into the SCANet model;

[0041] the SCANet model uses two standard 3x3 convolution kernels to pre-process the target image, so that the resolution of the target image becomes 1 / 4 of the original, and the number of channels changes from 3 to 64, to obtain a first feature map;

[0042] the SCANet model uses the first feature map as the input of Stage1, and uses four sandglass coordinate attention bottleneck modules to extract features from the first feature map, to obtain a second feature map with channel feature information and spatial direction feature information;

[0043] The SCANet model takes the second feature map as the input of Stage 2, uses four hourglass coordinate attention base modules with different resolutions and different numbers of channels to extract features from the second feature map, and obtains a third feature map with channel feature information and spatial direction characteristic information;

[0044] The SCANet model takes the third feature map as the input of Stage 3, uses the same four hourglass coordinate attention base modules as Stage 2 to extract features from the third feature map, and obtains a fourth feature map with channel feature information and spatial direction characteristic information;

[0045] The SCANet model takes the fourth feature map as the input of Stage 3, uses the same four hourglass coordinate attention base modules as Stage 2 to extract features from the fourth feature map, and obtains a fifth feature map with channel feature information and spatial direction characteristic information and outputs.

[0046] Optionally, the coordinate attention module extracts channel feature information and spatial direction feature information in the image by the following steps:

[0047] For the input feature map, the coordinate attention module uses an adaptive average pooling layer in the horizontal direction to extract spatial direction features for each feature channel, and uses an adaptive average pooling layer in the vertical direction to extract channel features for each feature channel;

[0048] The feature maps generated by the adaptive average pooling layers in the horizontal and vertical directions are spliced, and then a 1x1 convolution is used to generate an intermediate feature map with channel feature information and spatial direction feature information;

[0049] The intermediate feature map is divided into two temporary feature maps along the spatial direction, and a 1x1 convolution is used to convert the channel numbers of the two temporary feature maps, respectively, to obtain attention weights in the vertical and horizontal directions;

[0050] The attention weights in the vertical and horizontal directions are multiplied by the input feature map to obtain a feature map with channel feature information and spatial direction feature information.

[0051] Optionally, the scaling factor of the hourglass module is re-determined, and specifically includes:

[0052] Different scaling factors are respectively compared on the MPII dataset;

[0053] According to the results of the experimental comparison, the scaling factor of the hourglass module is determined to be 1.

[0054] According to still another aspect of the present application, there is provided a computer readable storage medium storing a computer program for performing the method according to any one of the preceding aspects of the present application.

[0055] According to still another aspect of the present application, there is provided an electronic device comprising: a processor; a memory for storing instructions executable by the processor; and the processor configured to read the executable instructions from the memory and execute the executable instructions to implement the method according to any one of the preceding aspects of the present application.

[0056] The sandglass module is first used to replace the standard 3x3 convolution in the original bottleneck module of the HRNet model, so as to ensure the performance of the network model and achieve the purpose of optimizing the network. Then, the coordinate attention module is introduced into the original bottleneck module and the basic Block module of the HRNet model, so as to ensure the acquisition of channel feature information of the feature map, strengthen the accurate positioning of the network model in the spatial direction feature information, and reduce the loss caused by the key point positioning. Thus, the sandglass module and the coordinate attention module are introduced, and on this basis, the sandglass coordinate attention bottleneck module and the sandglass coordinate attention basic module are constructed, so as to reduce the model parameter quantity and the operation complexity, and finally acquire the long-range dependence and accurate position information of the feature map in the spatial direction. Thus, the technical problems of large network parameter quantity and high operation complexity of the existing high-resolution human action recognition network are solved. BRIEF DESCRIPTION OF DRAWINGS

[0057] The exemplary embodiments of the present application can be more completely understood in reference to the following drawings:

[0058] Figure 1 is a flowchart of the human action recognition method based on the attention mechanism provided by an exemplary embodiment of the present application;

[0059] Figure 2 is a network structure diagram of the sandglass module provided by an embodiment of the present application;

[0060] Figure 3 is a network structure diagram of the coordinate attention module provided by an embodiment of the present application;

[0061] Figure 4 is a network structure diagram of the sandglass coordinate attention bottleneck module provided by an embodiment of the present application;

[0062] Figure 5 is a network structure diagram of the sandglass coordinate attention basic module provided by an embodiment of the present application;

[0063] Figure 6 is a network structure diagram of the SCANet model provided by an embodiment of the present application;

[0064] Figure 7 is a structure of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0065] Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that these described embodiments are merely exemplary of the present application and should not be considered limiting the scope of the present application. Therefore, the disclosure of these exemplary embodiments is intended to be illustrative, and not to be limiting of the scope of the present application.

[0066] It should be noted that the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments are not limiting of the scope of the application unless otherwise specifically stated.

[0067] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present application are only used to distinguish different steps, devices or modules and do not represent any specific technical meaning or indicate their logical order.

[0068] It should also be understood that "a plurality of" in the embodiments of the present application can mean two or more, and "at least one" can mean one, two or more.

[0069] It should also be understood that for any component, data or structure mentioned in the embodiments of the present application, unless specifically limited or given a contrary indication in the context, it can be understood as one or more in general.

[0070] In addition, the term "and / or" in the present application is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0071] It should also be understood that the description of the embodiments of the present application focuses on the differences between the embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0072] At the same time, it should be understood that, for the convenience of description, the size of each part shown in the drawings is not drawn according to the actual proportion relationship.

[0073] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting of the scope of the application or its applications or uses.

[0074] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0075] It should be noted that like reference numerals and characters refer to like items throughout the attached drawings and alternative embodiments thereof. Note, therefore, that no further discussion on the construction and operation of an item, once discussed in great detail in connection with one embodiment, is included in connection with another, functionally similar, embodiment.

[0076] Embodiments of the present application can be applied to electronic devices such as communication terminals, computer systems, servers, etc., which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known communication terminals, computing systems, environments, and / or configurations that can be suitable for use with electronic devices such as communication terminals, computer systems, servers, etc., include, but are not limited to, personal computers, server computers, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed cloud computing environments that include any of the above systems, and the like.

[0077] Electronic devices such as communication terminals, computer systems, servers, etc., can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, which perform particular tasks or implement particular abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media including storage devices.

[0078] The present application proposes a human action recognition method and system based on attention mechanism and electronic equipment. Figure 1 The flowchart of the human action recognition method based on attention mechanism provided by an exemplary embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the human action recognition method based on attention mechanism includes: Figure 1

[0079] Step S1: redesign the bottleneck module and the basic Block module in the existing HRNet model to obtain an hourglass coordinate attention bottleneck module and an hourglass coordinate attention basic module;

[0080] ​In some specific embodiments, the bottleneck module and the basic Block module in the existing HRNet model (high-resolution network model) are redesigned to obtain the operation of the sandglass coordinate attention bottleneck module and the sandglass coordinate attention basic module, specifically including: adopting a sandglass module to replace the standard 3x3 convolution in the bottleneck module, and re-determining the scaling factor of the sandglass module; wherein the number of channels of the input channel of the sandglass module is equal to the number of channels of the output channel; a coordinate attention module is added in the bottleneck module and the basic Block module; wherein the coordinate attention module is used to extract channel feature information and spatial direction feature information in the image, and the number of channels of the input channel of the coordinate attention module is equal to the number of channels of the output channel.

[0081] In the embodiments of the present application, the sandglass module: by flipping the inverted residual module (a common public module) and adding two 1x1 convolutions between the deep convolutions, it looks like a sandglass. This module adjusts the positions of the reduction layer and the expansion layer, avoids the defect that the compressed channels cannot retain all useful feature information when mapping high-dimensional features to low-dimensional features. At the same time, in order to retain more feature channels and spatial context information, a skip connection is established between high-dimensional features. The structure of the sandglass module is as shown in Figure 2 .

[0082] The coordinate attention (CoordAttention) module captures the precise position information and the field of interest in the spatial direction while obtaining the inter-channel feature information. The module realizes two functions of the coordinate information embedding module and the coordinate attention generation module in turn. The specific structure of the CoordAttention module is as shown in Figure 3 .

[0083] The application is to optimize the formation of the SCANet model on the HRNet (a commonly used public network model) model. Since the HRNet model is mainly composed of a bottleneck module (Bottleneck module) and a basic block module (Basic block module), the application redesigns the Bottleneck module and the Basic block module in the HRNet model to obtain a sandglass coordinate attention bottleneck module (SCAneck module) and a sandglass coordinate attention basic module (SCAblock module). The application first uses a sandglass module (Sandglass module) to replace the standard 3x3 convolution in the Bottleneck module, and reconsiders the scaling coefficients of the reduction layer and the expansion layer to ensure the feature extraction capability of the feature map; secondly, a coordinate attention module (CoordAttention module) is added in each module to obtain cross-channel feature information and accurate spatial position information, and finally a sandglass coordinate attention bottleneck (SCAneck) module and a sandglass coordinate attention basic (SCAblock) module are formed. The specific structure of the sandglass coordinate attention bottleneck module is shown in Figure 4 , and the specific structure of the sandglass coordinate attention basic module is shown in Figure 5 .

[0084] By introducing the Sandglass module and the CoordAttention module, the parameter quantity and the operation complexity of the network model are greatly reduced under the premise of ensuring the performance of the network model. When the input channel and the output channel number of the convolution layer are the same, the system access consumption is the smallest and the model speed is the fastest, so when designing the basic module of the SCANet model, the original residual architecture of the basic module in the HRNet model is retained, and the channel number C in of the Sandglass module and the CoordAttention module in the basic module is out . Wherein, C in and C out represent the input and output channel numbers of the module respectively.

[0085] Specifically, the sandglass module (Sandglass module) adjusts the positions of the reduction layer and the expansion layer by flipping the inverted residual module and adding two 1x1 convolution layers between the depth convolution layers, avoiding the defect that all useful feature information cannot be preserved when high-dimensional features are mapped to low-dimensional features. At the same time, in order to preserve more feature channels and spatial context information, a skip connection is established between the high-dimensional features. When the input and output channel numbers are not the same, the skip connection is not added.

[0086] The CoordAttention module captures the precise position information and the field of interest in the spatial direction while obtaining the inter-channel feature information. The module sequentially implements a coordinate information embedding module and a coordinate attention generation module.

[0087] Optionally, the coordinate attention module extracts the channel feature information and the spatial direction feature information in the image by the following steps: for the input feature map, the coordinate attention module uses an adaptive average pooling layer in the horizontal direction to extract the spatial direction feature of each feature channel, and uses an adaptive average pooling layer in the vertical direction to extract the channel feature of each feature channel; the feature maps generated by the adaptive average pooling layers in the horizontal direction and the vertical direction are spliced, and then a 1x1 convolution is used to generate an intermediate feature map with channel feature information and spatial direction feature information; the intermediate feature map is divided into two temporary feature maps along the spatial direction, and a 1x1 convolution is used to convert the channel numbers of the two temporary feature maps respectively to obtain attention weights in the vertical direction and the horizontal direction; the attention weights in the vertical direction and the horizontal direction are multiplied by the input feature map to obtain a feature map with channel feature information and spatial direction feature information.

[0088] In the embodiment of the application, the coordinate attention module extracts the channel feature information and the spatial direction feature information in the image by the following steps:

[0089] 1) For the input feature map, the coordinate attention module uses an adaptive average pooling layer in the horizontal direction to extract the spatial direction feature of each feature channel, and uses an adaptive average pooling layer in the vertical direction to extract the channel feature of each feature channel;

[0090] 2) The generated feature maps are spliced, and then a 1x1 convolution is used to generate an intermediate feature map with channel feature information and spatial direction feature information simultaneously;

[0091] 3) The intermediate feature map is divided into two feature maps along the spatial direction, and a 1x1 convolution is used to convert the channel numbers respectively to obtain attention weights in the vertical direction and the horizontal direction;

[0092] 4) The attention weights in the vertical and horizontal spatial directions are multiplied by the input feature map to obtain a feature map with channel feature information and spatial direction feature information.

[0093] Specifically, three 1x1 convolutions are used in the CoordAttention module, and the parameter amount P CAThe calculation formula of the operation complexity Gc A are respectively:

[0094] P CA =C in ×C mid +2×C mid ×C out ;

[0095] G CA =(H+W)×C in ×C mid +H×C mid ×C out +W×C mid ×C out ;

[0096] Wherein, C in represents the input channel of the module; C out represents the output channel of the module; C mid is the compressed feature channel; T is a scaling coefficient; H represents the height of the feature map; W represents the width of the feature map.

[0097] Optionally, the scaling coefficient of the hourglass module is re-determined, and specifically includes: respectively performing experimental comparison on different scaling coefficients on an MPII data set; according to the result of the experimental comparison, the scaling coefficient of the hourglass module is determined as 1.

[0098] In the embodiment of the application, different scaling coefficients T need to be respectively compared on the MPII (Max Planck Institute for Informatics) data set, and the experimental results are shown in Table 1, and it is found through research that when the scaling coefficient T = 1, the detection performance of human key points is 0.8% more than that of T = 2, therefore, the scaling coefficient T of the hourglass module is re-determined as 1.

[0099] Table 1

[0100] Network model Parameter quantity / 10 7 ]] GFOLPs Total Hourglass module (T=1) 13.2 6.4 89.1 Hourglass module (T=2) 11.5 5.3 88.3

[0101] In the embodiment of the application, the parameter quantity of the network model can be directly calculated according to the structure of the network and the parameter quantity of each layer to calculate the total parameter quantity. For example, for a convolution layer, the parameter quantity is mainly determined by the size of the convolution kernel, the input and output channel number; for a fully connected layer, the parameter quantity is mainly determined by the number of input and output units.

[0102] The parameter quantity of the network model can also be determined by using the API provided by a deep learning framework such as PyTorch: for example, PyTorch provides a method sum(p.numel() for p in model.parameters()) to quickly calculate the total parameter quantity of the model. In addition, there are special libraries such as thop, torchstat, ptflops, etc., which can more specifically calculate the parameter quantity and computational quantity of the model.

[0103] In summary, the parameter quantity of the network model can be determined in various existing ways, and the selection of which method depends on the specific requirements and scenarios.

[0104] It should be particularly noted that the determination of the parameter quantity of the network model is not the improvement point of the present application, and any one of the existing methods can achieve the determination of the parameter quantity of the network model.

[0105] GFLOPs is the operation speed, and the unit is: 10 billion operations per second.

[0106] Total represents the verification score.

[0107] The verification standard uses OKS (Object Keypoint Similarity), and the AP50 is used in this experiment, which is the accuracy of detecting key points when OKS = 0.5, and the specific implementation method is as shown in the formula:

[0108]

[0109] OKS is the similarity of each detected key point, d i represents the Euclidean distance between the detected key point and the key point labeled in the MPII dataset; v i is the flag of the real key point; s is the target scale; k i is the relevant control attenuation constant of each key point; sk i represents the standard deviation of each key point. The similarity of each detected key point is in the range of [0, 1], when OKS = 1, it means perfect predicted key points; when OKS = 0, it means that the predicted value is too different from the true value.

[0110] Step S2: replace the bottleneck module in the HRNet model with the sandglass coordinate attention bottleneck module, replace the basic Block module in the HRNet model with the sandglass coordinate attention basic module, and keep the original architecture of the HRNet model to obtain a SCANet model (a lightweight sandglass coordinate attention network model);

[0111] Step S3: input the target image of the human body action to be recognized into the SCANet model, and output a feature map with channel feature information and spatial direction feature information;

[0112] Optionally, the SCANet model inherits the original architecture of the HRNet model, including Stage1, Stage2, Stage3 and Stage4, a total of 4 stages; and the operation of inputting the target image of the human body action to be recognized into the SCANet model and outputting a feature map with channel feature information and spatial direction feature information specifically includes: inputting the target image of the human body action to be recognized into the SCANet model; the SCANet model uses two standard 3x3 convolution kernels to preprocess the target image, so that the resolution of the target image becomes 1 / 4 of the original, and the channel number changes from 3 to 64, to obtain a first feature map; the SCANet model takes the first feature map as the input of Stage1, uses 4 hourglass coordinate attention bottleneck modules to extract features from the first feature map, and obtains a second feature map with channel feature information and spatial direction characteristic information; the SCANet model takes the second feature map as the input of Stage2, uses 4 hourglass coordinate attention base modules with different resolutions and different channel numbers to extract features from the second feature map, and obtains a third feature map with channel feature information and spatial direction characteristic information; the SCANet model takes the third feature map as the input of Stage3, uses the same 4 hourglass coordinate attention base modules as Stage2 to extract features from the third feature map, and obtains a fourth feature map with channel feature information and spatial direction characteristic information; the SCANet model takes the fourth feature map as the input of Stage3, uses the same 4 hourglass coordinate attention base modules as Stage2 to extract features from the fourth feature map, and obtains a fifth feature map with channel feature information and spatial direction characteristic information and outputs.

[0113] In the embodiment of the present application, as shown in Figure 6 The SCANet model inherits the original architecture of the HRNet model, including Stage1, Stage2, Stage3 and Stage4, a total of 4 stages. This is to make the resolution of the feature map take a gradual and progressive descending mode. If the down-sampling operation is taken greatly, the details of the human body posture in the feature map will be lost rapidly, and even if the feature information is learned from the blurred image, it is also difficult to improve the accuracy of predicting the key points by fusing the feature information with the feature information extracted from the high-resolution feature map of the upper layer. And in each stage, 1, 2, 3 and 4 parallel branches with different resolutions and channel numbers are used respectively, which also avoids the loss of spatial information of the feature map in the down-sampling process.

[0114] Further, the specific processing process of the SCANet model is as follows:

[0115] 1) In the preprocessing stage, the input image resolution is reduced to 1 / 4 by two standard 3x3 convolution kernels, and the channel number is also changed from 3 to 64;

[0116] 2) The preprocessed feature map is used as the input of Stage1, and four SCAneck modules (hourglass coordinate attention bottleneck modules) are used to extract features from the feature map;

[0117] 3) In the subsequent three stages, SCAblock modules (hourglass coordinate attention basic modules) with different resolutions (1 / 4, 1 / 8, 1 / 16, 1 / 32) and channel numbers (C, 2C, 3C, 4C) are used to extract features from the feature map in each stage, and the obtained features are weighted to obtain a feature map with channel feature information and spatial direction characteristic information.

[0118] The network architecture of the present application adopts C=32, and the resolution and channel number of the feature map are adjusted between each stage. The present application refers to existing experience data, and the resolution is reduced by half and the channel number of the feature map is doubled to compensate for the loss of spatial positioning caused by the reduction of resolution.

[0119] Step S4: Perform vector distance calculation on the feature map and the pre-trained visual relationship model to predict key node expression information, and finally obtain action classification.

[0120] In the embodiment of the present application, the above-mentioned feature map and the pre-trained visual relationship model can be used for vector distance calculation to predict key node expression information, and finally obtain action classification. Specifically, the feature map contains the number and position relationship of key nodes, which are changed into feature vectors after passing through each layer of the model. The pre-trained model vector (containing multiple vectors, determined according to the number of action categories) performs a vector distance calculation (which can be Mahalanobis distance or cosine distance) in the last fully connected layer. After normalization, a value less than 1 is obtained as the vector distance score. The higher the vector distance score, the closer the distance. According to the vector distance score, the key point expression information can be predicted, and then it is determined which category of action the key point expression information belongs to, and finally the action classification of the feature map is obtained.

[0121] The SCANet model of the present application can store action data with low vector distance score, which is used as model optimization data for the pre-trained model. Through adaptive optimization of the learning rate, the accuracy of subsequent action category detection is improved.

[0122] The present application calculates the parameter amount of the basic module of the HRNet model and the basic module of the SCANet model according to the following formula:

[0123] PBottleneck = C mid x (C m + C mid x 3 x 3 + C out ) ;

[0124] P Basicblock = C in x C mid x 3 x 3 + C mid x C out x 3 x 3 ;

[0125] P SCAneck = C in x C mid + P CA + P SA + C mid x C out ;

[0126] P SCAblock = P SA + P CA + P SA ;

[0127] wherein P Bottleneck is the parameter quantity of the Bottleneck module; P Basicblock is the parameter quantity of the Basicblock module; P SCAneck is the parameter quantity of the SCAneck module; P SCAblock is the parameter quantity of the SCAblock module; C in represents the input channel of the module; C out represents the output channel of the module; C mid is the compressed feature channel; P CA is the parameter quantity of the CoordAttention module; P SA is the parameter quantity of the Hourglass module.

[0128] Further, the ratio of the SCAneck module and the SCAblock module in reducing the parameter quantity during network training is calculated by the following formula:

[0129]

[0130] wherein r nrckl is the ratio of the first SCAneck module in reducing the parameter quantity compared with the Bottleneck module; r nrck2 is the ratio of the last three SCAneck modules in reducing the parameter quantity compared with the Bottleneck module; r block is the ratio of the SCAblock module in reducing the parameter quantity compared with the Basicblock module; P BottleneckP is the parameter amount of the Bottleneck module; P Basicblock P is the parameter amount of the Basicblock module; P SCAneck P is the parameter amount of the SCAneck module; P SCAblock C is the parameter amount of the SCAblock module; C in C represents the input channel of the module; C out C represents the output channel of the module; C mid T is the scaling coefficient.

[0131] Since four SCAneck modules are applied in Stage 1, the input channel C in of the first SCAneck module is 64 and the output channel C out is 256, so when the skip connection is performed, a 1x1 convolution is added to convert the channel number, so that the channel number changes from 64 to 256; the input channel and the output channel of the remaining three SCAneck modules are both 256, so when the skip connection is performed, only the feature map is added, and no channel number conversion is performed, so r nrck1 is different from the result of r nrck2 .

[0132] In the entire model, the SCAblock module is applied to Stage 2, Stage 3 and Stage 4, and there are 32 SCAblock modules with 32 channels, 32 SCAblock modules with 64 channels, 28 SCAblock modules with 128 channels and 12 SCAblock modules with 256 channels, and the input channel of the SCAblock module is equal to the output channel. Since the scaling coefficient T = 1 in the Sandglass module, no channel number conversion is performed in the SCAblock module, so in the entire network model, the parameter amount ratio reduced by the SCAblock module of different parallel branches can be expressed as r block .

[0133] To sum up, the present application firstly uses the hourglass module to replace the standard 3x3 convolution in the original bottleneck module of the HRNet model, ensures the performance of the network model, and achieves the purpose of optimizing the network. Then, the coordinate attention module is introduced into the original bottleneck module and the basic Block module of the HRNet model, which ensures the acquisition of channel feature information of the feature map, strengthens the accurate positioning of the network model in the spatial direction feature information, and reduces the loss caused by key point positioning. Therefore, the hourglass module and the coordinate attention module are introduced, and on this basis, the hourglass coordinate attention bottleneck module and the hourglass coordinate attention basic module are constructed, which reduces the model parameter quantity and the operation complexity, and finally obtains the long-range dependence and accurate position information of the feature map in the spatial direction. Therefore, the technical problems of large network parameter quantity and high operation complexity of the existing high-resolution human action recognition network are solved.

[0134] The present application also provides a human action recognition system based on an attention mechanism, comprising:

[0135] A model design module is used to redesign the bottleneck module and the basic Block module in the existing HRNet model to obtain the hourglass coordinate attention bottleneck module and the hourglass coordinate attention basic module.

[0136] A model determination module is used to replace the bottleneck module in the HRNet model with the hourglass coordinate attention bottleneck module, replace the basic Block module in the HRNet model with the hourglass coordinate attention basic module, retain the original architecture of the HRNet model, and obtain the SCANet model.

[0137] A feature map acquisition module is used to input the target image of the human action to be recognized into the SCANet model, and output the feature map with channel feature information and spatial direction feature information.

[0138] An action classification module is used to perform vector distance calculation on the feature map and the pre-trained visual relationship model, predict key node expression information, and finally obtain action classification.

[0139] The model design module is specifically used for:

[0140] The hourglass module is used to replace the standard 3x3 convolution in the bottleneck module, and the scaling factor of the hourglass module is re-determined; wherein the number of input channels of the hourglass module is equal to the number of output channels.

[0141] The coordinate attention module is added to the bottleneck module and the basic Block module; wherein the coordinate attention module is used to extract channel feature information and spatial direction feature information in the image, and the number of input channels of the coordinate attention module is equal to the number of output channels.

[0142] Optionally, the SCANet model inherits the original architecture of the HRNet model, including Stage1, Stage2, Stage3 and Stage4, a total of 4 stages; and a feature map acquisition module, specifically configured to:

[0143] sending a target image of a human action to be recognized into the SCANet model;

[0144] The SCANet model uses two standard 3x3 convolution kernels to preprocess the target image, so that the resolution of the target image becomes 1 / 4 of the original, and the number of channels changes from 3 to 64, obtaining a first feature map;

[0145] The SCANet model takes the first feature map as the input of Stage1, uses 4 hourglass coordinate attention bottleneck modules to extract features from the first feature map, and obtains a second feature map with channel feature information and spatial direction characteristic information;

[0146] The SCANet model takes the second feature map as the input of Stage2, uses 4 hourglass coordinate attention base modules with different resolutions and different numbers of channels to extract features from the second feature map, and obtains a third feature map with channel feature information and spatial direction characteristic information;

[0147] The SCANet model takes the third feature map as the input of Stage3, uses the same 4 hourglass coordinate attention base modules as Stage2 to extract features from the third feature map, and obtains a fourth feature map with channel feature information and spatial direction characteristic information;

[0148] The SCANet model takes the fourth feature map as the input of Stage3, uses the same 4 hourglass coordinate attention base modules as Stage2 to extract features from the fourth feature map, and obtains a fifth feature map with channel feature information and spatial direction characteristic information and outputs.

[0149] Optionally, the coordinate attention module extracts channel feature information and spatial direction feature information in the image by the following steps:

[0150] For the input feature map, the coordinate attention module uses a horizontal direction adaptive average pooling layer to extract spatial direction features for each feature channel, and uses a vertical direction adaptive average pooling layer to extract channel features for each feature channel;

[0151] The feature maps generated by the horizontal direction and vertical direction adaptive average pooling layers are spliced, and then a 1x1 convolution is used to generate an intermediate feature map with channel feature information and spatial direction feature information;

[0152] The intermediate feature map is divided into two temporary feature maps along the spatial direction, and the channel number conversion is performed on the two temporary feature maps by using 1*1 convolution respectively, so as to obtain the attention weight in the vertical direction and the horizontal direction.

[0153] The attention weight in the vertical direction and the horizontal direction is multiplied with the input feature map, so as to obtain the feature map with channel feature information and spatial direction feature information.

[0154] Optionally, the scaling factor of the hourglass module is re-determined, and specifically includes:

[0155] The different scaling factors are respectively compared on the MPII dataset;

[0156] According to the comparison result, the scaling factor of the hourglass module is determined as 1.

[0157] The human action recognition system based on the attention mechanism of the embodiment of the present application corresponds to the human action recognition method based on the attention mechanism of another embodiment of the present application, which will not be described here.

[0158] Figure 7 is the structure of an electronic device provided by an exemplary embodiment of the present application. As shown in Figure 7 The electronic device 70 includes one or more processors 71 and a memory 72.

[0159] The processor 71 can be a central processing unit (CPU) or other forms of processing units having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0160] The memory 72 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache memory, and the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor 71 can run the program instructions to implement the methods of the software programs of the various embodiments of the present application described above and / or other desired functions. In one example, the electronic device can further include an input device 73 and an output device 74, and these components are interconnected by a bus system and / or other forms of connection mechanism (not shown).

[0161] In addition, the input device 73 can further include, for example, a keyboard, a mouse, and the like.

[0162] The output device 74 can output various information to the outside. The output device 74 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0163] Of course, in order to simplify, Figure 7 Only some of the components of the electronic device related to the present application are shown in the drawing, and components such as a bus, an input / output interface, and the like are omitted. In addition to this, the electronic device can include any other appropriate components according to the specific application.

[0164] In addition to the above-mentioned method and device, an embodiment of the present application can also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0165] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

[0166] In addition, an embodiment of the present application can also be a computer readable storage medium, which stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0167] The computer readable storage medium can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0168] The above describes the basic principles of the present application in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the above specific details disclosed are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to be necessarily implemented with the above specific details.

[0169] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between each embodiment can be understood by referring to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be understood by referring to the part of the method embodiment.

[0170] The block diagrams of the devices, systems, apparatuses, systems involved in the present application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, systems, apparatuses, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, which mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0171] The method and system of the present application can be implemented in many ways. For example, the method and system of the present application can be implemented by software, hardware, firmware or any combination of software, hardware and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present application are not limited to the above specific description, unless otherwise specifically described. In addition, in some embodiments, the present application can also be implemented as programs recorded in recording media, which include machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers the recording media storing the programs for executing the method according to the present application.

[0172] It should also be noted that, in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in this field to make or use the present invention. Various modifications to these aspects will be very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but according to the widest scope consistent with the principles disclosed here and novel features.

[0173] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for human action recognition based on attention mechanism, characterized in that, The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The application relates to a human action recognition method based on a sandglass coordinate attention network (SCANet) model. The The SCANet model takes the fourth feature map as an input of Stage 3, extracts the fourth feature map by using the same four hourglass coordinate attention base modules as Stage 2, and obtains a fifth feature map with channel feature information and spatial direction characteristic information and outputs the fifth feature map.

2. The method of claim 1, wherein, The coordinate attention module extracts the channel feature information and the spatial direction feature information in the image by the following steps: For the input feature map, the coordinate attention module extracts the spatial direction feature of each feature channel by using the horizontal direction adaptive average pooling layer, and extracts the channel feature of each feature channel by using the vertical direction adaptive average pooling layer; The feature maps generated by the horizontal direction adaptive average pooling layer and the vertical direction adaptive average pooling layer are spliced, and then a 1x1 convolution is used to generate an intermediate feature map with channel feature information and spatial direction feature information; The intermediate feature map is divided into two temporary feature maps along the spatial direction, and a 1x1 convolution is used to convert the channel number of the two temporary feature maps respectively to obtain the attention weight of the vertical direction and the horizontal direction; The attention weight of the vertical direction and the horizontal direction is multiplied by the input feature map to obtain a feature map with channel feature information and spatial direction feature information.

3. The method of claim 1, wherein, The scaling factor of the hourglass module is specifically determined by: The parameter amount P of the sandglass module is determined by the following parameters SA : P SA = C in × 3 × 3 + C in × (C in / T) + (C in / T) × C out + C out × 3 × 3; The operation complexity G of the sandglass module is determined by the following parameters SA : G SA = H x W x (C in x 3 x 3 + C in x (Cin / T) + H x W x ((C in / T) x C out + C out x 3 x 3); where C in represents the input channel of the module; C out represents the output channel of the module; T is a scaling factor; H represents the height of the feature map; and W represents the width of the feature map. According to the parameter quantity P of the sandglass module SA and the operation complexity G SA , the scaling factor of the sandglass module is determined.

4. A human action recognition system based on an attention mechanism, characterized by, including: The model design module is used to redesign the bottleneck module and the basic Block module in the existing HRNet model to obtain the hourglass coordinate attention bottleneck module and the hourglass coordinate attention basic module; The model determination module is used to replace the bottleneck module in the HRNet model with the hourglass coordinate attention bottleneck module, replace the basic Block module in the HRNet model with the hourglass coordinate attention basic module, and retain the original architecture of the HRNet model to obtain the SCANet model; The feature map acquisition module is used to input the target image of the human body action to be recognized into the SCANet model, and output a feature map with channel feature information and spatial direction feature information; The action classification module is used to calculate the vector distance between the feature map and the pre-trained visual relationship model, predict the key node expression information, and finally obtain the action classification; wherein The model design module is specifically used to: replace the standard 3x3 convolution in the bottleneck module with the hourglass module, and re-determine the scaling factor of the hourglass module; wherein the channel number of the input channel of the hourglass module is equal to the channel number of the output channel; add a coordinate attention module in the bottleneck module and the basic Block module; wherein the coordinate attention module is used to extract the channel feature information and the spatial direction feature information in the image, and the channel number of the input channel of the coordinate attention module is equal to the channel number of the output channel; the SCANet model inherits the original architecture of the HRNet model, and includes Stage 1, Stage 2, Stage 3 and Stage 4, a total of four stages; and the feature map acquisition module is specifically used to: input the target image of the human body action to be recognized into the SCANet model; The SCANet model uses two standard 3x3 convolution kernels to preprocess the target image, so that the resolution of the target image becomes 1 / 4 of the original, and the number of channels changes from 3 to 64, obtaining a first feature map; The SCANet model takes the first feature map as the input of Stage1, uses four hourglass coordinate attention bottleneck modules to extract features from the first feature map, and obtains a second feature map with channel feature information and spatial direction characteristic information; The SCANet model takes the second feature map as the input of Stage2, uses four hourglass coordinate attention base modules with different resolutions and different numbers of channels to extract features from the second feature map, and obtains a third feature map with channel feature information and spatial direction characteristic information; The SCANet model takes the third feature map as the input of Stage3, uses the same four hourglass coordinate attention base modules as Stage2 to extract features from the third feature map, and obtains a fourth feature map with channel feature information and spatial direction characteristic information; The SCANet model takes the fourth feature map as the input of Stage3, uses the same four hourglass coordinate attention base modules as Stage2 to extract features from the fourth feature map, and obtains a fifth feature map with channel feature information and spatial direction characteristic information and outputs.

5. The system of claim 4, wherein, The coordinate attention module extracts channel feature information and spatial direction feature information in the image by the following steps: For the input feature map, the coordinate attention module uses a horizontal direction adaptive average pooling layer to extract spatial direction features for each feature channel, and uses a vertical direction adaptive average pooling layer to extract channel features for each feature channel; The feature maps generated by the horizontal direction and vertical direction adaptive average pooling layers are spliced, and then a 1x1 convolution is used to generate an intermediate feature map with channel feature information and spatial direction feature information; The intermediate feature map is divided into two temporary feature maps along the spatial direction, and a 1x1 convolution is used to convert the number of channels of the two temporary feature maps, respectively, to obtain attention weights in the vertical direction and the horizontal direction; The attention weights in the vertical direction and the horizontal direction are multiplied by the input feature map to obtain a feature map with channel feature information and spatial direction feature information.

6. The system of claim 4, wherein, The scaling factor of the hourglass module is re-determined, specifically including: The parameter amount P of the sandglass module is determined by the following parameters SA : P SA = C in ×3×3+C in ×(C in / T)+(C in / T)×C out +C out ×3×3; The operation complexity G of the sandglass module is determined by the following parameters SA : G SA = H x W x (C in x 3 x 3 + C in x (C in / T)) + H x W x ((C in / T) x C out + C out + C out x 3 x 3); where C in represents the input channel of the module; C out represents the output channel of the module; T is a scaling factor; H represents the height of the feature map; and W represents the width of the feature map. According to the parameter quantity P of the sandglass module SA and the operation complexity G SA , the scaling factor of the sandglass module is determined.

7. A computer-readable storage medium, the storage medium storing a computer program, the computer program being used to execute the method of any one of claims 1-3.

8. An electronic device, comprising: The electronic device includes: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any one of claims 1-3.

Images