Human action recognition network and electronic device based on self-attention mechanism

Abstract

The present invention discloses a human motion recognition network and electronic device based on a self-attention mechanism. The network comprises a linear processing module, a first-level self-attention processing unit, a second-level self-attention processing unit, an average pooling processing unit, and a classification unit, which are sequentially connected in series. After a human skeleton data sequence is input into the linear processing module, a high-dimensional skeleton data sequence is obtained. After passing through the first-level self-attention processing unit, a first-level skeleton joint feature sequence is output. This is then divided into two branches in parallel, which are respectively input into the second-level self-attention processing unit for processing. A second-level skeleton joint feature sequence and a second-level body part feature sequence are output in parallel. These are then respectively input into the average pooling processing unit to obtain result feature data. After identification by the classification unit, the recognition result is output. The network adopts a self-attention processing mechanism, which can effectively integrate human joints and body parts, and accurately recognize human skeleton motion with high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image intelligent recognition, and in particular to a human motion recognition network and electronic device based on a self-attention mechanism. Background Art

[0002] Human action recognition has a wide range of applications, such as human-computer interaction, intelligent video surveillance, and robotics. The Transformer (TFormer) is a model that utilizes a self-attention mechanism to accelerate model training. For human action recognition based on human skeleton data, the self-attention mechanism in the Transformer naturally captures the correlations between joints in spatial and temporal dimensions, without requiring the mandatory connectivity constraints between joints, as in graph convolutional networks (GCNs).

[0003] However, there are currently few methods that use Transformer for 3D human action recognition, and most of them use a hybrid model of GCN and Transformer or a multi-task learning framework.

[0004] How to use the self-attention mechanism to learn the spatiotemporal relationship and representative motion features of human joints to improve the accuracy of human motion recognition remains an unresolved problem. Summary of the Invention

[0005] The main technical problem solved by the present invention is to provide a human motion recognition network and electronic device based on the self-attention mechanism, so as to solve the problem in the prior art of lacking the use of the attention mechanism to effectively recognize human motion.

[0006] In order to solve the above technical problems, the present invention provides a human action recognition network based on the self-attention mechanism, wherein the network comprises a linear processing module, a first-level self-attention processing unit, a second-level self-attention processing unit, an average pooling processing unit and a classification unit which are cascaded in sequence; a human skeleton data sequence X is converted into a human action recognition network; a human action recognition network based on the self-attention mechanism is constructed ... in After inputting the linear processing module, a high-dimensional skeleton data sequence X1 is obtained, and then after passing through the first-level self-attention processing unit, the first-level skeleton joint feature sequence X2 is output; the first-level skeleton joint feature sequence X2 is divided into two branches in parallel, and input into the second-level self-attention processing unit for processing, and the second-level skeleton joint feature sequence is output in parallel. and the second-level body part feature sequence The second-level skeleton joint feature sequence and the second-level body part feature sequence Input them into the average pooling processing unit respectively to obtain the resulting feature data X out , and then the recognition result is output after recognition by the classification unit.

[0007] Preferably, the first-level self-attention processing unit includes a cascaded first basic spatial self-attention module and a first local-global temporal self-attention module.

[0008] Preferably, the second-level self-attention processing unit includes a first processing branch and a second processing branch, wherein the first processing branch includes a cascaded key node selection module, a second basic spatial self-attention module, a first node and part cross-processing module and a second local-global temporal self-attention module, and the second processing branch includes a cascaded global body part encoding module, a third basic spatial self-attention module, a second node and part cross-processing module and a third local-global temporal self-attention module; and the second basic spatial self-attention module also has an output connected to the second node and part cross-processing module, and the third basic spatial self-attention module also has an output connected to the first node and part cross-processing module.

[0009] Preferably, the first basic spatial self-attention module includes H1 basic self-attention sub-modules. For the basic self-attention sub-module of each head, the first input data entering the input end of the sub-module must first be position-encoded by the first position encoder; the calculation results of the basic self-attention sub-modules of H1 heads are all input into the first linear connector, connected in series in feature dimensions and then linearly transformed, and then output to an adder for adding the first input data for the first time; the result of the first addition is then input into the first FFN calculator for linear transformation, and the result calculated by the first FFN calculator is added to the first input data for a second time, and the result of the second addition is used as the output result of the first basic spatial self-attention module; the second and third basic spatial self-attention modules have the same structure as the first basic spatial self-attention module.

[0010] Preferably, the first local-global temporal self-attention module includes H2 temporal self-attention sub-modules. For each temporal self-attention sub-module, the second input data entering the input end of the sub-module must first be position-encoded by a second position encoder; the calculation results of the H2 temporal self-attention sub-modules are all input into the second linear connector, and the feature dimensions are connected in series and then linearly transformed; the output end of the second linear connector is further added to the second input data through an adder, and the result of the addition is used as the output result of the first local-global temporal self-attention module; the second and third local-global temporal self-attention modules have the same structure as the first local-global temporal self-attention module.

[0011] Preferably, the key node selection module adaptively samples the key joint point subset in each frame data of the first-level skeleton joint point feature sequence X2 output by the first-level self-attention processing unit, discards non-key joint points, and outputs the key joint point sequence This includes using a trainable projection vector And sigmoid function, C2 represents the dimension of the joint points in the first-level skeleton joint point feature sequence X2, and predicts the joint point information score in each frame of the first-level skeleton joint point feature sequence X2 N3 and T3 represent the number of joint points and the number of frames in each frame corresponding to the first-level skeleton joint feature sequence X2, respectively, then:

[0012] S=sigmoid(X2W P / ||W P ||),

[0013] Among them, for the first-level skeleton joint feature sequence X2, each element S ij , represents the information score of the i-th joint point in the j-th frame in the sequence. Sort the scores of all joint points in each frame and obtain the top K joint points with the largest scores: idx = sort(S,K), idx is the index of the selected joint point with the largest score, and further obtain the key joint point sequence:

[0014] Preferably, the body part encoding module divides all the joints into P body parts based on the physical structure of the human skeleton and prior knowledge for the first-level skeleton joint feature sequence X2 output by the first-level self-attention processing unit; concatenates the features of the joints belonging to the same body part, and then converts all the body parts into a global body feature sequence through a shared linear layer.

[0015] Preferably, the first node and part cross-processing module is used for cross-attention processing from body parts to key joints, including H3 parallel processing body part to joint cross-processing submodules, a third linear connector for converging and connecting these body part to joint cross-processing submodules, and a second FFN calculator;

[0016] Each body part to joint point cross processing submodule has two data input terminals, which input the key joint point sequence respectively. and body global feature sequences The key joint sequence After linear transformation, three branch data Q are output respectively J ,K J ,V J, body global feature sequence After linear transformation, three branch data Q are output respectively P ,K P ,V P , and then calculate:

[0017]

[0018] Wherein, d3=C3 / H3, C3 is the feature dimension of the input data entering the body part to joint point cross processing submodule, and the attention matrix A JP ∈R K×P For Q J and K P Model the correlation between body parts and aggregate body part features to key joints;

[0019] The third linear connector is used to connect the output results of the H3 head body parts to the joint point cross processing submodule, connect them in series along the feature dimension, and then transform them linearly. Then, the output results are added to the key joint point sequence of the input through the adder. The result of the second FFN calculator is then added and input into the second FFN calculator for linear transformation. The result of the second FFN calculator is once again compared with the key joint point sequence. The sum is added, and the result of the addition is used as the output result of the first node and part cross processing module.

[0020] Preferably, the second node and part cross processing module includes H4 heads of parallel processing joint point to body part cross processing submodules, a fourth linear connector for converging and connecting these joint point to body part cross processing submodules, and a third FFN calculator; each joint point to body part cross processing submodule has two data input terminals, which respectively input the body global feature sequence and key joint sequence The global feature sequence of the body After linear transformation, three branch data Q are output respectively P ,K P ,V P , key joint sequence After linear transformation, three branch data Q are output respectively J ,K J ,V J , and then calculate:

[0021]

[0022] Among them, d4=C4 / H4, C4 is the feature dimension of the input data entering the joint point to body part cross processing submodule, the attention matrix A PJ ∈RP×K For Q P and K J Model the correlation and aggregate the body part features onto the key joint points;

[0023] The fourth linear connector is used to connect the output results of the H4 head joints to the body part cross processing submodule, connect them in series along the feature dimension, and then transform them linearly. Then, the output results are added to the input body global feature sequence through the adder. The sum is then input into the third FFN calculator for linear transformation calculation. The result calculated by the third FFN calculator is once again compared with the global feature sequence of the body. The sum is added, and the result of the addition is used as the output result of the second node and part cross processing module.

[0024] The present invention also provides an electronic device comprising the aforementioned three-dimensional human motion recognition network based on spatiotemporal local and global self-attention.

[0025] The beneficial effects of the present invention are as follows: the present invention discloses a human motion recognition network and electronic device based on a self-attention mechanism. The network includes a linear processing module, a first-level self-attention processing unit, a second-level self-attention processing unit, an average pooling processing unit, and a classification unit, which are cascaded in sequence. After the human skeleton data sequence is input into the linear processing module, a high-dimensional skeleton data sequence is obtained. After passing through the first-level self-attention processing unit, a first-level skeleton joint feature sequence is output. The first-level skeleton joint feature sequence is then divided into two branches in parallel and input into the second-level self-attention processing unit for processing. The second-level skeleton joint feature sequence and the second-level body part feature sequence are output in parallel. The second-level skeleton joint feature sequence and the second-level body part feature sequence are then input into the average pooling processing unit to obtain result feature data. After identification by the classification unit, the recognition result is output. The network adopts a self-attention processing mechanism, which can effectively integrate human joints and body parts, and accurately recognize human skeleton movements with high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a block diagram of an embodiment of a human action recognition network based on a self-attention mechanism according to the present invention;

[0027] Figure 2 1 is a block diagram of the composition of the first basic spatial self-attention module in the human action recognition network based on the self-attention mechanism according to the present invention;

[0028] Figure 3 1. It is a block diagram of the composition of the first local-global temporal self-attention module in the human action recognition network based on the self-attention mechanism according to the present invention;

[0029] Figure 4Schematic diagram of body part division by a global body part encoding module in a human action recognition network based on a self-attention mechanism according to the present invention;

[0030] Figure 5 1 is a block diagram of the composition of the first node and the part cross processing module in the human action recognition network based on the self-attention mechanism according to the present invention;

[0031] Figure 6 This is a block diagram of the composition of the second node and the part cross-processing module in the human action recognition network based on the self-attention mechanism according to the present invention. DETAILED DESCRIPTION

[0032] To facilitate understanding of the present invention, the present invention will be described in more detail below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present invention.

[0033] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification are intended solely for the purpose of describing specific embodiments and are not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0034] Figure 1 The block diagram of an embodiment of a human action recognition network based on self-attention mechanism is shown. Figure 1 It includes a linear change module 1, a first-level self-attention processing unit 2, a second-level self-attention processing unit 3, an average pooling processing unit 4 and a classification unit 5, which are cascaded in sequence.

[0035] Furthermore, the human skeleton data sequence X in After inputting the linear processing module 1, a high-dimensional skeleton data sequence X1 is obtained, and then after passing through the first-level self-attention processing unit 2, the first-level skeleton joint feature sequence X2 is output; the first-level skeleton joint feature sequence X2 is divided into two branches in parallel, and respectively input into the second-level self-attention processing unit 3 for processing, and the second-level skeleton joint feature sequence is output in parallel. and the second-level body part feature sequence The second-level skeleton joint feature sequence and the second-level body part feature sequence Input them into the average pooling processing unit 4 respectively to obtain the result feature data X out, and then the classification unit 5 recognizes it and outputs the recognition result.

[0036] The above is Figure 1 The components of the embodiment shown in the figure and the functions of each component module or unit are summarized. Figure 1 The core parts of the human skeleton action recognition network shown are the first-level self-attention processing unit 2 and the second-level self-attention processing unit 3, wherein the first-level self-attention processing unit 2 is used to model all joints in the human skeleton data sequence, learn the spatial and temporal correlations between the joints, and fully extract motion features to provide effective and reliable feature representation for the subsequent processing. The second-level self-attention processing unit 3 is used to select key joints with discriminative information for each human skeleton action. At the same time, considering the global spatial structure information of the human body, the joints are divided into multiple body parts, such as 10 body parts, according to the human body topological structure. In addition, the second-level self-attention processing unit 3 also explicitly models the interaction between body parts and key joints through a cross-attention mechanism, or models the motion information of key joints and global body parts. In this way, the accuracy of human skeleton action recognition can be improved.

[0037] The following further Figure 1 The illustrated embodiment is described in detail.

[0038] Preferably, for the human skeleton data sequence X in , refers to a data sequence consisting of human skeleton data, which can reflect the continuous movement of the human skeleton over time. Therefore, the embodiment of the present invention directly models the temporal motion information of human joints over a long sequence.

[0039] Preferably, the human skeleton data sequence X in Further expressed as: N represents the number of joint points in a frame of human skeleton data (or a frame of human skeleton image), T1 represents the number of frames of human skeleton data, for example, 128 frames is the length of the sequence, C in The dimension of the three-dimensional coordinates (x, y, z) of the joint points in a frame of human skeleton data, which constitutes a set of human skeleton data sequences Where R represents a set of real numbers.

[0040] For example, the human skeleton data sequence X in It can be that N=25 joint points in a frame of human skeleton image, for the sequence with less than 300 frames, zero padding is performed to fill it to 300 frames, and then linear interpolation is used to sample it to 128 frames to obtain X in ∈R 25×128×3 .

[0041] Furthermore, the human skeleton data sequence X in After linear transformation by the linear processing module 1, the three-dimensional coordinates C of each joint point are in Mapping to a high-dimensional space C1, such as a 64-dimensional space, to obtain a high-dimensional skeleton data sequence Obviously, the dimension of the high-dimensional space C1=64 is significantly larger than that of the three-dimensional space C in =3 dimensions.

[0042] Preferably, for the linear change module 1, in order to facilitate the construction of a network with better learning characteristics, the linear change implemented by the module is: y = LeakyReLU (BN (Wx + b)), x is input, y is output, W is weight, b is bias value, BN (batch normalization) is batch normalization processing, and LeakyReLU () is an activation function. In the present invention, for the linear change module 1, the input is a human skeleton data sequence The output is a high-dimensional skeleton data sequence For the input human skeleton data sequence Each frame of human skeleton data in the linear transformation module 1 outputs a frame of high-dimensional skeleton data. Then, multiple frames of high-dimensional skeleton data form a high-dimensional skeleton data sequence.

[0043] Further, such as Figure 1 As shown, the first-level self-attention processing unit 2 includes a cascaded first basic spatial self-attention module 21 and a first local-global temporal self-attention module 22.

[0044] As mentioned above, the first-level self-attention processing unit 2 is used to model all joints in the human skeleton data sequence, learn the spatial and temporal correlations between the joints, and fully extract motion features to provide effective and reliable feature representation for the subsequent processing. A sub-unit formed by the cascade of the first basic spatial self-attention module 21 and the first local-global temporal self-attention module 22 can further be combined with these two modules as a sub-unit to perform a cascade of multiple sub-units, so in Figure 1 In the figure, ×L1 indicates that there are L1 subunits formed by combining the cascaded dual modules. L1≥1, preferably L1=6.

[0045] Preferably, when L1=6, it is equivalent to having 6 combined subunits composed of the first basic spatial self-attention module 21 and the first local-global temporal self-attention module 22 in cascade, then the first basic spatial self-attention module 21 and the first local-global temporal self-attention module 22 in these 6 combined subunits have similar structural compositions, and the difference is mainly reflected in the change in the dimension of the data they process. Specifically, the first-level self-attention processing unit 2 includes 6 cascaded combined subunits, and the dimensions corresponding to the data of these 6 combined subunits are 64 dimensions, 64 dimensions, 128 dimensions, 128 dimensions, 256 dimensions, and 256 dimensions, respectively. Therefore, the first-level self-attention processing unit 2 finally outputs the first-level skeleton joint feature sequence Where C2 represents the dimension of the joint points of each frame of data in the first-level skeleton joint point feature sequence, which is also a high-dimensional space. Preferably, C2 is 256 dimensions.

[0046] The following combination Figure 2 and Figure 3 , and the first basic spatial self-attention module 21 and the first local-global temporal self-attention module 22 are described in detail respectively.

[0047] Preferably, Figure 2 As shown, for the first basic spatial self-attention module 21, which includes H1 head basic self-attention sub-modules 212, these H1 head self-attention sub-modules 212 are in parallel and simultaneously process the first input data from the input end through the H1 head self-attention mechanism. It should be noted that each frame of data input to the first basic spatial self-attention module 21 is used as the first input data and is processed simultaneously and in parallel by the H1 head basic self-attention sub-modules 212. Therefore, the first input data here is simultaneously input to the H1 head basic self-attention sub-modules 212 for parallel processing.

[0048] Preferably, for each head's basic self-attention submodule 212, the first input data at the input end is first position-encoded by the first position encoder 211, denoted as PE. Because different types of joints in a frame of image data contain different semantic information, position encoding is used to provide a unique label for each joint.

[0049] Preferably, sine and cosine functions of different frequencies are used as encoding functions, namely:

[0050]

[0051]

[0052] Where x represents the position of the joint point in the first input data, i represents the dimension index of the position encoding vector, and C11 is the feature dimension of the first input data.

[0053] This can include spatial position coding and temporal position coding. Spatial position coding sequentially encodes the joint points in the same frame of image data. The same joint points in different frames of image data have the same code. Similarly, different frames of image data contain different temporal information. To encode this information, sine and cosine position coding can be used on the first input data. Joint points in the same frame of image data have the same temporal code, and the same joint points in different frames are encoded in temporal order.

[0054] exist Figure 2 It can be seen that the first input data is divided into three branches and enters the basic attention submodule 212 of H1 head, of which the two left branches are the first position encoder 211 (corresponding to Figure 2 The module marked Position Encoding) performs position encoding on the first input data and adds it to the input data X1, and then inputs it into the two linear transformers 2121 and 2122 in the basic self-attention submodule 212, wherein the first linear transformer 2121 outputs Q1 branch data, the second linear transformer 2122 outputs K1 branch data, and the third linear transformer 2123 directly performs linear transformation on the first input data and outputs V1 branch data. Here, Q1, K1, and V1 are identification symbols for distinguishing the three branch data in the basic self-attention submodule 212. The three linear transformers 2121, 2122, and 2123 correspond to Figure 2 Label the three modules of Linear.

[0055] Specifically, the relationship between the first input data X1 and the Q1 branch data of each frame is: Q1 = (X1 + PE) W q The relationship between the first input data X1 and the K1 branch data is K1 = (X1 + PE) W k : The relationship between the first input data X1 and the V1 branch data is: V1=X1W v , where W q 、W k 、W v It is the linear mapping matrix representing the above three linear transformers, and PE corresponds to the aforementioned position encoding.

[0056] Furthermore, in the basic self-attention submodule 212, the Q1 branch data and the K1 branch data are input to the first calculator 2124 (corresponding to Figure 2 The module marked MatMul&Scale) is calculated, that is:

[0057]

[0058] Wherein, h represents the serial number of the head corresponding to the basic self-attention submodule 212, and the value of h is selected in the range of 1 to H1, preferably H1=3, then h has three values of 1, 2, and 3. Figure 2 There are three parallel basic self-attention submodules 212 in Q1. h It represents the Q1 branch data in the basic self-attention submodule 212 with the serial number h, K1 h It represents the K1 branch data in the basic self-attention submodule 212 with the serial number h, and T represents the transposed matrix, because the K1 branch data is matrix data, (K1 h ) T It's K1 h The corresponding transposed matrix. In addition, d1=C 11 / H1, as above, C 11 is the feature dimension of the first input data, and H1 indicates that the first basic spatial self-attention module 21 includes a basic self-attention submodule 212 with H1 heads. Here, in order to prevent the inner product Q1 h (K1 h ) T Too large, so it is normalized by dividing by the square root of d1.

[0059] After the output of the first calculator 2124 is further input to the first softmax calculator 2125 (corresponding to Figure 2 The module marked with softmax) completes the calculation Softmax is an activation function used to normalize a numerical vector into a probability distribution vector.

[0060] The first softmax calculator 2125 outputs the attention matrix Right now: The attention matrix Each element in represents the correlation between two joint points, and the basic self-attention submodules 212 with different sequence numbers h correspond to Then the attention matrix Plus a global regularized attention matrix shared by all input high-dimensional skeleton data sequences It is a learnable matrix initialized to all 1s. The basic self-attention submodules 212 with different sequence numbers h learn their own Therefore, after the addition operation, the feature of each joint point is updated to the weighted sum of all joint points.

[0061] Furthermore, the basic self-attention submodule 212 further includes a second calculator 2126 (corresponding to Figure 2 The module marked MatMul) completes the calculation: V1h It represents the V1 branch data in the basic self-attention submodule 212 with serial number h.

[0062] The calculation result is further input to the first linear connector 213 (corresponding to Figure 2 The module marked Concat&Linear) can use the first linear connector 213 to concatenate the results of the basic self-attention submodule 212 of H1 heads along the feature dimension and then linearly transform them before output.

[0063] There is an adder after the first linear connector 213, which is used to add the result output by the first linear connector 213 and the first input data, and then enter the first FFN calculator 214 (corresponding to Figure 2 The first FFN calculator 214 performs a linear transformation on the first input data (labeled as the FFN module), and uses the LeakyReLU() function as the activation function. The result calculated by the first FFN calculator 214 is once again added to the first input data, and the result of the addition serves as the output of the first basic spatial self-attention module 21.

[0064] The two adders before and after the first FFN calculator 214 are respectively added to the first input data, which belongs to the residual connection. Adding the residual connection will make the established network training more stable, avoid the gradient disappearance with the increase of network depth, and accelerate the convergence of the network.

[0065] Further, such as Figure 3 As shown, for the first local-global time self-attention module 22, this module is Figure 2 The first basic spatial self-attention module 21 shown has many similarities in structural composition, but also has obvious differences, which are described in detail below.

[0066] Figure 3 It includes H2-head temporal self-attention submodules 222, and these H2-head temporal self-attention submodules 222 are in a parallel relationship, and simultaneously perform local-global temporal multi-head self-attention mechanism processing on the second input data X11 from the input end.

[0067] Preferably, for each head's temporal self-attention submodule 222, the second input data at the input end first passes through the second position encoder 221 (corresponding to Figure 3 The module marked Position Encoding) performs position encoding on the second input data. The implementation method of the position encoding is referenced to Figure 2 The description of the embodiments will not be repeated here.

[0068] It can be seen that the second input data is divided into three branches and enters the temporal self-attention submodule 222. The upper two branches are position-encoded by the second position encoder 221 through the adder and are respectively input into the two linear transformers 2221 and 2222 in the temporal self-attention submodule 222 (corresponding to Figure 3 The first linear transformer 2221 outputs Q2 branch data, and the second linear transformer 2222 outputs K2 branch data. The second input data is also input to the first dilated time convolution transformer 2223 (corresponding to Figure 3 The module marked as Dilated Temporal Conv) outputs the V2 branch data. Here, Q2, K2, and V2 are the identifiers that distinguish the three branch data in the temporal self-attention submodule 222. The functions of the two linear transformers 2221 and 2222 are the same as those mentioned above. Figure 2 The two linear converters 2121 and 2122 in the embodiment have the same linear relationship. The main difference lies in the specific parameter values after network training optimization, which will not be repeated here.

[0069] exist Figure 3 In the above example, the local and global temporal relationships of the joints can be explicitly utilized simultaneously. When the temporal self-attention submodule 222 is used to perform multi-head attention calculation, the first dilated temporal convolution transformer 2223 performs a dilated temporal convolution on the second input data to generate V2 branch data. This has the following beneficial effects: (1) it can explicitly learn the short-term temporal motion characteristics of the joints from adjacent frames; (2) it introduces a beneficial local bias to the attention mechanism. At the same time, the attention matrix generated by the multi-head attention mechanism of the temporal self-attention submodule 222 models the global temporal correlation.

[0070] Therefore, the first dilated temporal convolution transformer 2223 integrates dilated temporal convolution into the multi-head self-attention mechanism. For a specific joint point in a frame of image data, the short-term temporal motion features of the joint points or body parts from adjacent frames can also be incorporated. The short-term feature representations of all joint points are then further fused using the weights obtained through global self-attention. In this way, the resulting features integrate local temporal relationships and global temporal information.

[0071] The first linear transformer 2221 outputs Q2 branch data and the second linear transformer 2222 outputs K2 branch data, which are respectively represented as Q2, Among them, N2 and T2 correspond to the number of joint points of the second input data X11 in one frame and the number of frames of the second input data, respectively. 21 The first dilated time convolution transformer 2223 uses a convolution kernel size of Kt ×1, dilated time convolution with a dilation rate of q, to obtain V2 branch data, denoted as V2, specifically:

[0072]

[0073] Among them, TCN dilate () represents the dilated temporal convolution calculation, and X11 is the second input data entering the temporal self-attention submodule 222. Preferably, K t =7, the expansion rate q is 1 or 2.

[0074] Furthermore, in the temporal self-attention submodule 222, the third calculator 2224 (corresponding to Figure 3 The module marked MatMul&Scale) and the second softmax calculator 2225 (corresponding to Figure 3 Module labeled softmax), similar to the above Figure 2 The first calculator 2124 and the first softmax calculator 2125 have the same function, that is, the Q2 branch data and the K2 branch data are input to the third calculator 2224 for calculation, that is:

[0075]

[0076] Among them, h2 represents the serial number of the head corresponding to the time self-attention submodule 222, and the value of h2 is selected in the range of 1 to H2, preferably H2 = 2, then h2 has two values 1 and 2, that is, in Figure 3 There are two parallel temporal self-attention submodules 222 in Q2. h2 It represents the Q2 branch data in the temporal self-attention submodule 222 of the h2 head, K2 h2 It represents the K2 branch data in the time self-attention submodule 222 of the h2 head, and T represents the transposed matrix, because the K2 branch data is usually matrix data, (K2 h2 ) T It's K2 h2 The corresponding transposed matrix. In addition, d2=C 21 / H2,C 21 represents the feature dimension of the second input data of a frame, and H2 represents that the first local-global temporal self-attention module 22 includes a temporal self-attention submodule 222 with H2 heads. Here, in order to prevent the inner product Q2 h2 (K2 h2 ) T Too large, so it is normalized by dividing by the square root of d2.

[0077] After the output of the third calculator 2224 is further input into the second softmax calculator 2225, the calculation is completed. Get the global temporal self-attention matrix Right now Then used to fuse local features

[0078] The first correlation calculator 2226 (corresponding to Figure 3 The module marked with Attention Map is a T2×T2 matrix calculation, which is used to calculate the correlation corresponding to the second input data of any two frames in the input action sequence.

[0079] Furthermore, the fourth calculator 2227 (corresponding to Figure 3 The module marked MatMul) completes the calculation:

[0080] Attention(Q2 h2 ,K2 h2 ,V2 h2 )=A2 h2 V2 h2

[0081] Linear converter 2228 (corresponding to Figure 3 The module labeled Reshape & Linear) is used to further process the result output by the fourth calculator 2227. Perform a linear transformation to produce the output:

[0082] Attention(Q2 h2 ,K2 h2 ,V2 h2 )W2 h2 =A2 h2 V2 h2 W2 h2 .

[0083] Then for the temporal self-attention submodule 222, the output result is also:

[0084] Attention(Q2 h2 ,K2 h2 ,V2 h2 )W2 h2 =A2 h2 V2 h2 W2 h2 ,

[0085] The result is obtained by adding the result output by the first dilated time convolution transformer 2223. Adding them together, we have:

[0086] head(X11) h2 =Attention(Q2 h2 ,K2h2 ,V2 h2 )W2 h2 +V2 h2

[0087] for Figure 3 The first local-global temporal self-attention module 22 shown in FIG. 1 includes H2 temporal self-attention submodules 222 of parallel computation, which are all input to the second linear connector 223 (corresponding to Figure 3 The outputs of the temporal self-attention submodules 222 of all heads are concatenated in the feature dimension by the second linear connector 223 and connected through Performing linear transformation and using LeakyReLU as activation function, we get:

[0088] Concat[head(X11) 1 ,…,head(X11) H2 ]W0

[0089] The output end of the second linear connector 223 is further added to the second input data from the input end through an adder, and can be used as the second input data of another first local-global temporal self-attention module 22 of the same type in a subsequent cascade connection, that is:

[0090] X l+1 =Concat[head(X11) 1 ,…,head(X11) H2 ]W0+X l

[0091] Among them, X l Indicates the second input data of this level, X l+1 It represents the second input data of the first local-global temporal self-attention module 22 of the next level input with the same structure. Preferably, here l∈L1, there can be 6 levels of second input data.

[0092] above Figure 2 and Figure 3 The embodiments are respectively Figure 1 The first-level self-attention processing unit 2 includes a first basic spatial self-attention module 21 and a first local-global temporal self-attention module 22. As mentioned above, in the first-level self-attention processing unit 2, there can be multiple sub-units consisting of the first basic spatial self-attention module 21 and the first local-global temporal self-attention module 22, which are cascaded in series.

[0093] Combine Figure 1As shown, the second-level self-attention processing unit 3 includes a first processing branch and a second processing branch, wherein the first processing branch includes a cascaded key node selection module 31, a second basic spatial self-attention module 33, a first node and part cross-processing module 35 and a second local-global temporal self-attention module 37, and the second processing branch includes a cascaded global body part encoding module 32, a third basic spatial self-attention module 34, a second node and part cross-processing module 36 and a third local-global temporal self-attention module 38; and the second basic spatial self-attention module 33 also has an output connected to the second node and part cross-processing module 36, and the third basic spatial self-attention module 34 also has an output connected to the first node and part cross-processing module 35.

[0094] Further, combined Figure 1 As shown, the second-level self-attention processing unit 3 includes a key joint point selection module 31, which adaptively samples the key joint point subset in each frame data of the first-level skeleton joint point feature sequence X2 output by the first-level self-attention processing unit 2, discards non-key joint points, and outputs a key joint point sequence

[0095] To do this, a trainable projection vector is used And sigmoid function, C2 represents the dimension of the joint points in the first-level skeleton joint point feature sequence X2, and predicts the joint point information score in each frame of the first-level skeleton joint point feature sequence X2 Here N3 and T3 represent the number of joint points and the number of frames in each frame corresponding to the first-level skeleton joint feature sequence X2, respectively, then:

[0096] S=sigmoid(X2W P / ||W P ||)

[0097] Among them, for the first-level skeleton joint feature sequence X2, each element S ij , represents the information score of the i-th joint in the j-th frame in the sequence. The larger the score, the more information the joint contains. Sort the scores of all joints in each frame and get the top K joints with the largest scores:

[0098] idx=sort(S,K)

[0099] idx is the index of the selected joint point with the maximum score, and the key joint point sequence is further obtained:

[0100] Further, such as Figure 1As shown, the second-level self-attention processing unit 3 includes a global body part encoding module 32, which divides all joint points of the first-level skeleton joint feature sequence X2 output by the first-level self-attention processing unit 2 into P body parts based on the physical structure and prior knowledge of the human skeleton.

[0101] In order to encode the first-level skeleton joint feature sequence X2 into P body parts, we concatenate the features of the joints belonging to the same body part, and then convert all parts into a global body feature sequence through a shared linear layer

[0102] Specific as Figure 4 As shown, the joints are marked as: 1 base of spine, 2 middle of spine, 3 neck, 4 head, 5 left shoulder, 6 left elbow, 7 left wrist, 8 left hand, 9 right shoulder, 10 right elbow, 11 right wrist, 12 right hand, 13 left hip, 14 left knee, 15 left ankle, 16 left foot, 17 right hip, 18 right knee, 19 right ankle, 20 right foot, 21 spine, 22 left hand tip, 23 left thumb, 24 right hand tip, 25 right thumb.

[0103] These 25 nodes are further divided into 10 body parts, namely (1, 2, 21), (3, 4), (5, 6), (7, 8, 22, 23), (9, 10), (11, 12, 24, 25), (13, 14), (15, 16), (17, 18), and (19, 20). It can be seen that each partition contains at most 4 nodes. In order to pass a shared linear layer, body parts with less than 4 nodes are padded with 0.

[0104] The above key joint selection module 31 and the global body part encoding module 32 respectively extract the key joint features and body part features of the input first-level skeleton joint feature sequence X2, and then perform cross-attention mechanism processing in the next level of processing, which is conducive to more accurate and efficient recognition of motion movements.

[0105] Preferably, Figure 1 In the second-level self-attention processing unit 3, a second basic spatial self-attention module 33 and a third basic spatial self-attention module 34 are respectively included. The composition and function of these two modules are the same as those of the first basic spatial self-attention module 21 mentioned above, and will not be repeated here.

[0106] Furthermore, the second-level self-attention processing unit 3 includes a first node and part cross-processing module 35 for cross-attention processing from body parts to key joints. Figure 5As shown, the module includes multiple parallel processing body part to joint point cross processing sub-modules 351, a third linear connector 352 for converging and connecting these body part to joint point cross processing sub-modules 351, and a second FFN calculator 353.

[0107] Preferably, each body part to joint point cross processing submodule 351 has two data input terminals, which input the key joint point sequence and body global feature sequences The key joint sequence After linear transformation, three branch data Q are output respectively J ,K J ,V J (correspond Figure 5 Mark Q respectively J ,K J ,V J three modules), body global feature sequence After linear transformation, three branch data Q are output respectively P ,K P ,V P (correspond Figure 5 Mark Q respectively P ,K P ,V P The linear transformation method here is the same as that of the above Figure 2 The description of the linear converter in the embodiment will not be repeated here.

[0108] Then the fifth calculator 3511 (corresponding to Figure 5 The module labeled MatMul&Scale) is Q-scaling for the input J and K P Calculated as:

[0109]

[0110] Where d3 = C3 / H3. As mentioned above, C3 is the feature dimension of the input data entering the body part to joint point cross processing submodule 351, and H3 indicates that the first node and part cross processing module 35 includes H3 heads of the body part to joint point cross processing submodule 351. Preferably, H3 = 3. Here, in order to prevent the inner product Q J (K P ) T Too large, so normalize by dividing by the square root of d3.

[0111] Then pass through the third softmax calculator 3512 (corresponding to Figure 5 The module marked with softmax) is calculated as follows:

[0112]

[0113] The second correlation calculator 3513 (corresponding to Figure 5 The module labeled Attention Map) is a K×P matrix calculation, which is used to calculate the correlation between the key joints of the skeleton and the body parts of any frame in the input action sequence.

[0114] Then, the sixth calculator 3514 (corresponding to Figure 5 The module marked MatMul) calculates:

[0115]

[0116] Among them, the attention matrix A JP ∈R K×P It is used to model the correlation between the two branches and aggregate body part features onto key joints.

[0117] It can be seen that when the fifth calculator 3511 performs calculations, it is respectively Q after linear transformation J , and the global feature sequence from the body After linear transformation, K P Perform cross calculation; the sixth calculator 3514 further calculates the global feature sequence from the body After linear transformation, V P Perform cross calculations. As a result, body part features are aggregated to key joints.

[0118] Furthermore, the third linear connector 352 (corresponding to Figure 5 The module marked Concat & Linear is used to concatenate the output results of the H3 body parts to the joint point cross processing submodule 351 along the feature dimension and then linearly transform the output, and then add it to the input key joint point sequence through the adder. Add, and then input to the second FFN calculator 353 (corresponding to Figure 5 The module marked FFN) performs linear transformation calculation processing and uses LeakyReLU as the activation function. The result calculated by the second FFN calculator 353 is once again compared with the key joint point sequence The sum is added, and the result of the addition is used as the output result of the first node and part intersection processing module 35.

[0119] Combine Figure 1 As shown, the second-level self-attention processing unit 3 also includes a second node and part cross-processing module 36, which is used for cross-attention processing from key joints to body parts.

[0120] Specific as Figure 6 As shown, the module includes multiple parallel processing joint point to body part cross processing sub-modules 361, a fourth linear connector 362 for converging and connecting these joint point to body part cross processing sub-modules 361, and a third FFN calculator 363.

[0121] Preferably, each joint point to body part cross processing submodule 361 has two data input terminals, which input the body global feature sequence respectively. and key joint sequence The global feature sequence of the body After linear transformation, three branch data Q are output respectively P ,K P ,V P (correspond Figure 6 Mark Q respectively P ,K P ,V P three modules), key joint sequence After linear transformation, three branch data Q are output respectively J ,K J ,V J (correspond Figure 6 Mark Q respectively J ,K J ,V J The linear transformation method here is the same as that of the above Figure 2 The description of the linear converter in the embodiment will not be repeated here.

[0122] Then the seventh calculator 3611 (corresponding to Figure 6 The module labeled MatMul&Scale) is Q-scaling for the input P and K J Calculated as:

[0123]

[0124] Where d4 = C4 / H4. As mentioned above, C4 is the feature dimension of the input data entering the joint point to body part cross processing submodule 361, and H4 indicates that the second node and part cross processing module 36 includes H4 heads of the joint point to body part cross processing submodule 361. Preferably, H4 = 3. Here, in order to prevent the inner product Q J (K P ) T Too large, so normalize by dividing by the square root of d3.

[0125] Then pass through the fourth softmax calculator 3612 (corresponding to Figure 6The module marked with softmax) is calculated as follows:

[0126]

[0127] The third correlation calculator 3613 (corresponding to Figure 6 The module labeled Attention Map) is a P×K matrix calculation, which is used to calculate the correlation between the body parts and key joints of any frame skeleton in the input action sequence.

[0128] Then, the eighth calculator 3614 (corresponding to Figure 6 The module marked MatMul) calculates:

[0129]

[0130] Among them, the attention matrix A PJ ∈R P×K It is used to model the correlation between the two branches and aggregate the features of key joint points into body part features.

[0131] It can be seen that when the seventh calculator 3611 performs calculations, it is respectively Q after linear transformation P , and the sequence from key joint points After linear transformation, K J Perform cross calculation; the eighth calculator 3614 further calculates the sequence of key joint points After linear transformation, V J Perform cross calculations. Thus, the features of key joints are aggregated into body part features.

[0132] Furthermore, the fourth linear connector 362 (corresponding to Figure 6 The module marked Concat & Linear is used to convert the output of the H4 head joint points to the body part cross processing submodule 361, concatenate along the feature dimension and then linearly transform the output, and then add it to the input body global feature sequence through the adder Add, and then input to the third FFN calculator 363 (corresponding to Figure 6 The result of the third FFN calculator 363 is compared with the body global feature sequence again. The sum is added, and the result of the addition is used as the output result of the second node and part intersection processing module 36.

[0133] Preferably, combined Figure 1As shown, the data output from the first node and part cross-processing module 35 is further input into the second local-global temporal self-attention module 37 for processing, and the data output from the second node and part cross-processing module 36 is further input into the third local-global temporal self-attention module 38 for processing. Structurally, the second local-global temporal self-attention module 37 and the third local-global temporal self-attention module 38 are similar to the aforementioned first local-global temporal self-attention module 22, differing primarily in the different parameter values after training, which will not be further described here.

[0134] Similar to the first-level self-attention processing unit 2, the second basic spatial self-attention module 33, the third basic spatial self-attention module 34, the first node and part cross-processing module 35 and the second node and part cross-processing module 36, the second local-global time self-attention module 37 and the third local-global time self-attention module 38 in the second-level self-attention processing unit 3 can be combined to form a sub-unit, which is then cascaded to form the second-level self-attention processing unit 3. Therefore, in Figure 1 ×L2 indicates that there are L2 such sub-units in cascade. L2 ≥ 1, preferably L2 = 2. Note that the key joint point selection module 31 and the global body part encoding module 32 are only used once in the second-level self-attention processing unit 3 and do not need to be used in multiple levels of cascade.

[0135] Furthermore, the second-level self-attention processing unit 3 includes two outputs, which are the second-level skeleton joint feature sequence output and the second-level body part feature sequence Enter the average pooling processing unit 4 respectively and perform global average pooling layer (GAP) processing, namely: and The feature dimension after pooling is C5 represents the value of the feature dimension after pooling, and then it is spliced along the feature channel to finally get Among them, C out =2C5, the X out Output from the average pooling processing unit 4. Then input to the fully connected layer (FC) to convert X out Mapped to the number of action categories, and finally classified and output by softmax classifier through classification unit 5.

[0136] Based on the human action recognition network embodiment based on the self-attention mechanism constructed in the above embodiment, the network training is carried out by training data so that the loss of the cross entropy loss function is continuously reduced and finally stabilized. Among them, the PyTorch framework is used on the RTX 3090 GPU. The network model is trained for 80 rounds using the SGD optimizer with a momentum of 0.9 and a weight decay of 0.0005. The warm-up strategy is used for the first 5 rounds of training, and the learning rate increases from 0.002 to 0.01. The initial learning rate is set to 0.01 and decays 10 times in the 50th and 70th rounds. The trained network model is then tested for classification of the test set data.

[0137] To validate the role of the second-level self-attention processing unit, ablation experiments were conducted on its key components: the key joint selection module 31, the global body part encoding module 32, the first node and part intersection processing module 35, and the second node and part intersection processing module 36 (JP-CA). Using the second basic spatial self-attention module 33 and the third basic spatial self-attention module 34 as a foundation, the system was verified by gradually adding modules.

[0138] The experimental results are shown in the following table.

[0139]

[0140] Model A uses the key joint selection module 31 in the second-level self-attention processing unit of the network, improving the performance of the second basic spatial self-attention module 33 and the third basic spatial self-attention module 34 by 0.5%, that is, 88.3%-87.8%=0.5%. This shows that identifying key joint selection is beneficial. Then, Model B further introduces the global body part encoding module 32 into the second-level self-attention processing unit. This further improves the performance by 0.8%, that is, 89.1%-88.3%=0.8%. Finally, by further adding the first node and part cross-processing module 35 and the second node and part cross-processing module 36 to Model C, the accuracy is further improved by 0.4%, that is, 89.5%-89.1%=0.4%. This shows that the interaction between body parts and selected key nodes helps to distinguish action categories.

[0141] Furthermore, we explored different network level configurations for stages 1 and 2 by adjusting the number of first-level self-attention processing units (L1) and second-level self-attention processing units (L2). The total number of levels was fixed at 8. The results are shown in the table below.

[0142]

[0143] Comparing Models A, B, and C, using more than four levels in the first stage yields higher performance, with the best performance (89.5%) achieved when L1 = 6 and L2 = 2. Accuracy degraded when only one level was assigned to the second stage in Model D. These observations suggest that the first stage should fully model the relationships between all relevant nodes; otherwise, focusing on unreliable key joints and body parts may degrade performance.

[0144] The present invention also provides an electronic device comprising the aforementioned 3D human motion recognition network based on spatiotemporal local and global self-attention. Preferably, the electronic device comprises a computer, mobile phone, server, electronic device with an embedded processor, or other intelligent electronic terminal with program processing capabilities. The aforementioned recognition network embodiment can be stored as program code in the memory of such electronic device, and the aforementioned 3D human motion recognition function can then be implemented by executing this program code.

[0145] Based on the above embodiments, the present invention discloses a human motion recognition network and electronic device based on a self-attention mechanism. The network includes a linear processing module, a first-level self-attention processing unit, a second-level self-attention processing unit, an average pooling processing unit, and a classification unit, which are cascaded in sequence. After the human skeleton data sequence is input into the linear processing module, a high-dimensional skeleton data sequence is obtained. After passing through the first-level self-attention processing unit, a first-level skeleton joint feature sequence is output, which is then divided into two branches in parallel and input into the second-level self-attention processing unit for processing. The second-level skeleton joint feature sequence and the second-level body part feature sequence are output in parallel. They are then input into the average pooling processing unit to obtain result feature data. After being identified by the classification unit, the recognition result is output. The network adopts a self-attention processing mechanism, which can effectively integrate human joints and body parts, and accurately identify human skeleton movements with high precision.

[0146] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A human action recognition system based on self-attention mechanism, characterized in that: The system includes a linear processing module, a first-level self-attention processing unit, a second-level self-attention processing unit, an average pooling processing unit and a classification unit that are cascaded in sequence; Human skeleton data sequence After inputting the linear processing module, a high-dimensional skeleton data sequence is obtained Then, after the first-level self-attention processing unit, the first-level skeleton joint feature sequence is output ; The first-level skeleton joint feature sequence The parallel branch is divided into two branches, which are input into the second-level self-attention processing unit for processing, and the second-level skeleton joint feature sequence is output in parallel. and the second-level body part feature sequence The second-level self-attention processing unit includes a first processing branch and a second processing branch, wherein the first processing branch includes a cascaded key node selection module, a second basic spatial self-attention module, a first node and part cross-processing module, and a second local-global temporal self-attention module; the second processing branch includes a cascaded global body part encoding module, a third basic spatial self-attention module, a second node and part cross-processing module, and a third local-global temporal self-attention module; Furthermore, the second basic spatial self-attention module further outputs a connection to the second node and part cross-processing module, and the third basic spatial self-attention module further outputs a connection to the first node and part cross-processing module; The second-level skeleton joint feature sequence and the second-level body part feature sequence Input them into the average pooling processing unit respectively to obtain the resulting feature data , and then the recognition result is output after recognition by the classification unit.

2. The human action recognition system based on self-attention mechanism according to claim 1, characterized in that The first-level self-attention processing unit includes a cascaded first basic spatial self-attention module and a first local-global temporal self-attention module.

3. The human action recognition system based on self-attention mechanism according to claim 2, characterized in that The first basic spatial self-attention module includes H1 basic self-attention sub-modules. For each basic self-attention sub-module, the first input data entering the input end of the sub-module must first be position-encoded by a first position encoder; The calculation results of the basic self-attention submodules of H1 heads are all input to the first linear connector, concatenated in feature dimensions and then linearly transformed, and then output to an adder for adding the first input data for the first time; the result of the first addition is then input to the first FFN calculator for linear transformation, and the result calculated by the first FFN calculator is added to the first input data for the second time, and the result of the second addition is used as the output result of the first basic spatial self-attention module; The second and third basic spatial self-attention modules have the same structure as the first basic spatial self-attention module.

4. The human action recognition system based on self-attention mechanism according to claim 3 is characterized in that The first local-global temporal self-attention module includes H2 temporal self-attention sub-modules. For each temporal self-attention sub-module, the second input data entering the input end of the sub-module must first be position-encoded by the second position encoder; The calculation results of the temporal self-attention sub-modules of H2 heads are all input to the second linear connector, and are concatenated in feature dimensions and then linearly transformed. The output end of the second linear connector is further added to the second input data through an adder, and the addition result is used as the output result of the first local-global temporal self-attention module. The second and third local-global temporal self-attention modules have the same structure as the first local-global temporal self-attention module.

5. The human action recognition system based on self-attention mechanism according to claim 4 is characterized in that: The key node selection module selects the first-level skeleton joint feature sequence output by the first-level self-attention processing unit , adaptively sample the key joint point subset in each frame data in the sequence, discard non-key joint points, and output the key joint point sequence ; This includes using a trainable projection vector and the sigmoid function, Represents the first-level skeleton joint feature sequence The dimension of the joint points in the prediction of the first-level skeleton joint feature sequence The joint information score in each frame , and Represent the first-level skeleton joint feature sequence The corresponding number of joint points and frames in each frame are: ， Among them, for the first-level skeleton joint feature sequence , each element , represents the information score of the i-th joint point in the j-th frame in the sequence, sorts the scores of all joint points in each frame, and obtains the top K joint points with the largest scores: , idx is the index of the selected joint point with the maximum score, and the key joint point sequence is further obtained: .

6. The human action recognition system based on self-attention mechanism according to claim 5, characterized in that: The body part encoding module processes the first-level skeleton joint feature sequence output by the first-level self-attention processing unit Based on the physical structure of the human skeleton and prior knowledge, all joints are divided into P body parts; the features of joints belonging to the same body part are connected in series, and then all body parts are converted into a global body feature sequence through a shared linear layer. .

7. The human action recognition system based on self-attention mechanism according to claim 6, characterized in that: The first node and part cross-processing module is used for cross-attention processing from body parts to key joints, including H3 parallel processing body part to joint cross-processing sub-modules, a third linear connector for converging and connecting these body part to joint cross-processing sub-modules, and a second FFN calculator; Each body part to joint point cross processing submodule has two data input terminals, which input the key joint point sequence respectively. and body global feature sequences , where the key joint sequence After linear transformation, three branch data are output respectively , , , body global feature sequence After linear transformation, three branch data are output respectively , , , and then calculate: ， in, , is the feature dimension of the input data entering the body part to joint point cross processing submodule, the attention matrix Used for and Model the correlation between body parts and aggregate body part features to key joints; The third linear connector is used to connect the output results of the H3 head body parts to the joint point cross processing submodule, connect them in series along the feature dimension, and then transform them linearly. Then, the output results are added to the key joint point sequence of the input through the adder. The result of the second FFN calculator is then added and input into the second FFN calculator for linear transformation. The result of the second FFN calculator is once again compared with the key joint point sequence. The sum is added, and the result of the addition is used as the output result of the first node and part cross processing module.

8. The human action recognition system based on self-attention mechanism according to claim 6, characterized in that The second node and part cross processing module includes H4 heads of parallel processing joint point to body part cross processing submodules, a fourth linear connector for converging and connecting these joint point to body part cross processing submodules, and a third FFN calculator; Each joint point to body part cross processing submodule has two data input terminals, which input the body global feature sequence respectively. and key joint sequence , where the body global feature sequence After linear transformation, three branch data are output respectively , , , key joint sequence After linear transformation, three branch data are output respectively , , , and then calculate: ， in, , is the feature dimension of the input data entering the joint point to body part cross processing submodule, the attention matrix Used for and Model the correlation and aggregate the body part features onto the key joints; The fourth linear connector is used to connect the output results of the H4 head joints to the body part cross processing submodule, connect them in series along the feature dimension, and then transform them linearly. Then, the output results are added to the input body global feature sequence through the adder. The sum is then input into the third FFN calculator for linear transformation calculation. The result calculated by the third FFN calculator is once again compared with the global feature sequence of the body. The sum is added, and the result of the addition is used as the output result of the second node and part cross processing module.

9. An electronic device, characterized in that: A human action recognition system based on a self-attention mechanism comprising any one of claims 1-8.

Images