Human action recognition method based on human bone distribution, electronic device, and storage medium

Abstract

The application discloses a human action recognition method based on human skeleton distribution, an electronic device and a storage medium, and belongs to the technical field of human-computer cooperation and intelligent manufacturing. In order to solve the accuracy problem of complex action recognition, the application comprises the following steps: collecting human action data by wearing sensors at main joints of a human body, setting sensor wearing positions as nodes according to a human skeleton structure diagram, adding edges to multi-dimensional features collected by each node according to human morphology and constructing an adjacency list, mapping the multi-dimensional features collected by each node to a hidden layer, updating by using an attention mechanism, and obtaining a multi-dimensional feature sequence of each node after updating; constructing a sparse routing selection method of a time step dimension attention mechanism of time series data, dividing the multi-dimensional feature sequence of each node after updating into multiple windows, processing each window to obtain the most relevant neighbor window of each window, performing sparse attention calculation, and obtaining a human action recognition result based on human skeleton distribution.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human-computer collaboration and intelligent manufacturing, and particularly relates to a human action recognition method based on human skeleton distribution, an electronic device and a storage medium. BACKGROUND

[0002] In the past few years, deep learning has gradually replaced machine learning in the field of HAR, and deep learning has improved the performance and robustness of HAR, and has improved the efficiency and accuracy of action recognition.

[0003] The intention behind human daily behavior can be to complete a daily task, convey an idea, or engage in non-verbal communication as part of human interaction.

[0004] Research results of HAR problems are widely used in sports fitness, medical health, abnormal behavior detection, content-based video analysis, robot technology, visual monitoring, smart home, environmental intelligence, and many other fields. For example, computer vision, which includes pictures, videos, depth images, etc.; wearable sensors; and more generally, simple portable devices such as smartphones and smartwatches, which are already commercially available. Among them, commonly used sensors include Kinect sensors and wearable sensors such as accelerometers, gyroscopes, and stretch sensors.

[0005] In comparison, wearable sensors have small data volume, high portability, and are more widely used. Inertial sensors such as accelerometers, gyroscopes, and magnetometers are also widely used for human activity recognition. Wearable sensors have been attached to or worn on human skeletal joint parts for a long time to detect human activities and recognize and segment them. Soft wearable stretch sensors have become popular in recent years and are widely used in the fields of healthcare to human-computer interaction. The advantage of stretch sensors is that they have high body-fitting, little discomfort when worn, and are more life-like, so they can more easily obtain data. The sensor can be sewn into clothes or directly attached to the human skin to measure the degree of joint bending and straightening.

[0006] The current action recognition method has the following problems: low action recognition accuracy, large model size difficult to deploy to small terminals, and slow inference speed. Traditional human activity recognition models usually use fully connected attention mechanisms or complex convolutional networks, resulting in a large number of parameters. For example, the parameter quantity of a model based on Transformer generally exceeds 2000M, making it difficult to deploy on resource-constrained edge devices. Moreover, the inference delay of traditional methods on edge devices generally exceeds 100ms, which cannot meet the real-time requirements. SUMMARY

[0007] The problem to be solved by this invention is to improve the accuracy of complex action recognition, and to propose a human action recognition method, electronic device and storage medium based on the distribution of human skeleton.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for human action recognition based on human skeleton distribution includes the following steps:

[0010] S1. The sensor is worn on the main joints of the human body to collect human motion data. Then, the sensor wearing position is set as a node according to the human skeletal structure diagram. The multi-dimensional features collected by each node are added with edges according to human morphology and an adjacency list is constructed. Then, the multi-dimensional features collected by each node are mapped to a hidden layer and updated using an attention mechanism to obtain the updated multi-dimensional feature sequence of each node.

[0011] S2. Construct a sparse routing selection method based on the time step dimension attention mechanism of time series data. Divide the updated multi-dimensional feature sequence of each node obtained in step S1 into multiple windows, and process each window to obtain the k most relevant neighbor windows of each window.

[0012] S3. Perform sparse attention calculation on the k most relevant neighbor windows of each window obtained in step S2 to obtain human action recognition results based on the distribution of human skeleton.

[0013] Furthermore, the specific implementation method of step S1 includes the following steps:

[0014] S1.1. Place the sensor on the main joints of the human body, and then set the sensor placement according to the human skeletal structure diagram as nodes v1…,v i ...vn, where n is the total number of joints equipped with sensors, v i For the i-th node;

[0015] S1.2. Add edges to the nodes obtained in step S1.1 according to human morphology, and then construct the adjacency list E. n×n (e ij :(v i ,v j ), where e ij For the i-th node v i With the j-th node v j The indicator of whether they are connected or not; if connected, then e ij If it is 1, then e is not connected. ij =0;

[0016] S1.3. Map the multi-dimensional features of each node to a hidden layer, resulting in the following expression:

[0017] s ij = a([ω i v i || ω j v j ]) (1)

[0018] wherein ω i i and ω j j are the learnable weight matrix corresponding to the ith node v i and the jth node v j , [·||·] is the concatenation of the transformed features of the ith node v i and the jth node v j , a(·) is a mapping function, s ij is the high-dimensional feature mapping result of the concatenation of the ith node v i and the jth node v j ;

[0019] S1.4. Calculate the attention weight between nodes by using the attention mechanism, and the calculation expression is:

[0020]

[0021] wherein s iq is the high-dimensional feature mapping result of the concatenation of the ith node and the qth node, A ij is the attention weight between the ith node v i and the jth node v j ;

[0022] S1.5. Aggregate the features of the nodes by using the calculated A ij , to generate the updated multi-dimensional feature sequence h' i of the ith node, and the expression is:

[0023]

[0024] wherein σ(·) is an activation function, ω is the weight matrix of h j , h j is the feature of the jth node in N i , and N i is the number of all nodes adjacent to the ith node.

[0025] S1.6. Based on steps S1.1-S1.5, update the nodes for multiple rounds, which can be fixed update rounds, or set a threshold, and stop updating when the feature h of each node is less than the threshold, to obtain the updated multi-dimensional feature sequence of each node.

[0026] Further, the specific implementation method of step S2 comprises the following steps:

[0027] S2.1. Constructing the time window division:

[0028] Divide the updated multi-dimensional feature sequence of each node into multiple windows, and the expression of window division is:

[0029]

[0030] where m is the number of window division, w is the window length, and T is the time dimension length of the time series (usually the number of time steps); each window obtained after division is W p ;

[0031] S2.2. Regional routing:

[0032] S2.2.1. Calculate the regional representation for each window, and the expression is:

[0033] Q p = AvgPool(W p ω Q ) (5)

[0034] K p = AvgPool(W p ω K ) (6)

[0035] where Q p is the query, K p is the key, AvgPool(·) is the average pooling, are two learnable weight matrices of W p , d is the original dimension of W p , and d h is related to d as follows:

[0036]

[0037] S2.2.2. Construct the window correlation matrix β, and a(·) is the mapping function:

[0038]

[0039] where Q and K are the queries and keys of all windows, respectively, and ReLU is the activation function named ReLU;

[0040] β ij is the position element in the window correlation matrix β, representing the correlation strength between window W i and window W j , which is used to measure the importance of information interaction between windows;

[0041] S2.2.3. Dynamically select the k most relevant neighbor windows for each window: Dynamically select the k most relevant neighbor windows I for each window based on the window association matrix α. r The expression is:

[0042]

[0043] Among them, argtopk k (·) returns the set of indices of the k most relevant neighboring windows for each window.

[0044] Furthermore, the specific implementation method of step S3 is to perform standard attention Attn(Q,K,V) on the k most relevant neighbor windows of each window, with the expression as follows:

[0045]

[0046] Where Softmax(·) is the normalized attention weight, ensuring that the sum of the weights is 1. isI r The query, isI r The key, isI r The value;

[0047] The sparse attention mechanism is used to aggregate features from the k most relevant neighbor windows of each window. Finally, the output features are restored to their dimensionality through low-rank projection, as shown in the expression:

[0048]

[0049] Where H is the final output result, ω O It is a learnable weight matrix used to multiply with Attn(Q,K,V), and d h Mapping back to the original dimensions d, b O This is a bias term that enhances the model's expressive power.

[0050] An electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the human motion recognition method based on human skeletal distribution.

[0051] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned human motion recognition method based on human skeletal distribution.

[0052] The beneficial effects of this invention are:

[0053] The human action recognition method based on human skeleton distribution disclosed by the application significantly compresses the model volume by designing a sparse adaptive attention mechanism: a regional routing strategy is adopted to reduce the calculation complexity from O(n 2 ) to O(k·n), and low-rank projection and parameter sharing are used to reduce the model parameter quantity to 66% of that of ST-GCN. Experiments show that on the UCI HAR data set, the model volume is compressed from 381.86MB to 252.28MB.

[0054] The human action recognition method based on human skeleton distribution disclosed by the application improves the inference efficiency by optimizing the calculation process and hardware adaptation: an elastic calculation architecture is designed to dynamically adjust the window length (l=0.35√M) according to the available memory; a sparse attention mechanism is adopted to improve the inference speed. Through multi-modal feature fusion and dynamic weight distribution, the model performance is improved:

[0055] The human action recognition method based on human skeleton distribution disclosed by the application introduces an anatomical constraint attention mechanism and adopts a multi-round feature update strategy, and the F1-score of complex action recognition is improved to 95.17%; through a cascade architecture, the network achieves an accuracy of 95.67% on the self-built data set. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 A flowchart of the human action recognition method based on human skeleton distribution disclosed by the application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application, i.e., the described specific embodiments are only a part of the embodiments of the application, but not all the specific embodiments. The components of the specific embodiments of the application described and shown in the drawings herein can be arranged and designed in various different configurations, and the application can also have other embodiments.

[0058] Therefore, the detailed description of the specific embodiments of the application provided in the drawings below is not intended to limit the scope of the claimed application, but only represents selected specific embodiments of the application. Based on the specific embodiments of the application, all other specific embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0059] In order to further understand the inventive content, characteristics and effects of the application, the following specific embodiments are exemplified, and the drawings are Figure 1 The detailed description is as follows:

[0060] Example 1:

[0061] A method for human action recognition based on human skeleton distribution includes the following steps:

[0062] S1. The sensor is worn on the main joints of the human body to collect human motion data. Then, the sensor wearing position is set as a node according to the human skeletal structure diagram. The multi-dimensional features collected by each node are added with edges according to human morphology and an adjacency list is constructed. Then, the multi-dimensional features collected by each node are mapped to a hidden layer and updated using an attention mechanism to obtain the updated multi-dimensional feature sequence of each node.

[0063] Furthermore, the specific implementation method of step S1 includes the following steps:

[0064] S1.1. Place the sensor on the main joints of the human body, and then set the sensor placement according to the human skeletal structure diagram as nodes v1…,v i ...vn, where n is the total number of joints equipped with sensors, v i For the i-th node;

[0065] S1.2. Add edges to the nodes obtained in step S1.1 according to human morphology, and then construct the adjacency list E. n×n (e ij :(v i ,v j ), where e ij For the i-th node v i With the j-th node v j The indicator of whether they are connected or not; if connected, then e ij If it is 1, then e is not connected. ij =0;

[0066] S1.3. Map the multi-dimensional features of each node to a hidden layer, resulting in the following expression:

[0067] s ij =a([ω i v i ||ω j v j (1)

[0068] Where, ω i ω j They are the i-th node v i With the j-th node v j The corresponding learnable weight matrix, [·||·] represents the weights for the i-th node v. i With the j-th node v jtransformed features of the i-th node and the j-th node, a(·) is a mapping function, s ij is the mapping result of the concatenated high-dimensional features of the i-th node and the j-th node; i is the mapping result of the concatenated high-dimensional features of the i-th node and the j-th node; j is the mapping result of the concatenated high-dimensional features of the i-th node and the j-th node;

[0069] S1.4. Calculate the attention weight between nodes using the attention mechanism, and the calculation expression is:

[0070]

[0071] where s iq is the mapping result of the concatenated high-dimensional features of the i-th node and the j-th node, A ij is the attention weight between the i-th node and the j-th node; i is the attention weight between the i-th node and the j-th node; j

[0072] S1.5. Aggregate the features of the nodes using the calculated A ij to generate the updated multi-dimensional feature sequence h' i of the i-th node, and the expression is:

[0073]

[0074] where σ(·) is an activation function, ω is a weight matrix of h j , h j is the feature of the j-th node in N i , and N i is the number of all nodes adjacent to the i-th node;

[0075] S1.6. Based on steps S1.1-S1.5, update the nodes for multiple rounds, which can be fixed update rounds or set a threshold, and stop updating when the feature h of each node is less than the threshold, to obtain the updated multi-dimensional feature sequence of each node.

[0076] S2. Construct a sparse routing method of time series data time step dimension attention mechanism, divide the updated multi-dimensional feature sequence of each node obtained in step S1 into multiple windows, and process each window to obtain the most relevant k neighbor windows of each window.

[0077] Further, the specific implementation method of step S2 includes the following steps:

[0078] S2.1. Construct a time series window division:

[0079] Divide the updated multi-dimensional feature sequence of each node into multiple windows, and the window division expression is:

[0080] ​

[0081] where m is the number of window partition, w is the window length, and T is the time dimension length of time series; each partitioned window is W p ;

[0082] The window length can be dynamically adjusted or set as a fixed value.

[0083] S2.2. Regional routing:

[0084] S2.2.1. Calculate the regional representation for each window, the expression is:

[0085] Q p = AvgPool(W p ω Q ) (5)

[0086] K p = AvgPool(W p ω K ) (6)

[0087] where Q p is the query, K p is the key, AvgPool(·) is the average pooling, are two learnable weight matrices of W p , d is the original dimension of W p , and d h is related to d as follows.

[0088]

[0089] S2.2.2. Construct the window correlation matrix β, and a(·) is the mapping function:

[0090]

[0091] where Q and K are the query and key of all windows, and ReLU is the activation function named ReLU.

[0092] β ij is the position element in the window correlation matrix β, representing the correlation strength between window W i and window W j , used to measure the importance of information interaction between windows.

[0093] S2.2.3. Dynamically select the most relevant k neighbor windows for each window: according to the window correlation matrix α, dynamically select the most relevant k neighbor windows I r for each window, the expression is:

[0094]

[0095] where argtop k k(·) is the index set of the k most relevant neighbor windows for each window.

[0096] By dynamic routing, only the most informative window interactions are preserved, avoiding the high computational cost of full connection attention mechanism.

[0097] S3. Perform sparse attention calculation on the k most relevant neighbor windows of each window obtained in step S2 to obtain human action recognition results based on human skeleton distribution.

[0098] Further, the specific implementation method of step S3 is to perform standard attention Attn(Q, K, V) on the k most relevant neighbor windows of each window, expressed as:

[0099]

[0100] where Softmax(·) is the normalized attention weight, ensuring that the weight sum is 1, is the query of I r , is the key of I r , is the value of I r ;

[0101] By sparse attention mechanism, feature aggregation is performed on the k most relevant neighbor windows of each window, and finally the output feature is restored in dimension by low-rank projection, expressed as:

[0102]

[0103] where H is the final output result, ω O is a learnable weight matrix, used to multiply Attn(Q, K, V) to map d h back to the original dimension d, b O is a bias term to enhance the expression ability of the model.

[0104] The human action recognition method based on human skeleton distribution described in the embodiment is verified by experiments as follows:

[0105] 1. Benchmark dataset performance: Four benchmark datasets and four baseline networks are selected to compare the performance of the model.

[0106] CZU-MHAD contains 22 actions performed by 5 male subjects. Each subject repeats each action 8 times or more. The dataset contains more than 880 samples in total. Data from 10 wearable inertial sensors is used.

[0107] The self-built dataset consists of 15 subjects wearing smart clothes embedded with 24-way stretch sensors, which can perform 8 actions in total.

[0108] The daily and sports activity dataset contains motion sensor data for 19 daily and sports activities, each performed by 8 subjects for 5 minutes in their own style.

[0109] The ASL sensor dataglove dataset contains sensor data captured from a data glove designed to perform American Sign Language (ASL) gestures.

[0110] The comparison results are shown in Table 1:

[0111] Table 1

[0112]

[0113] 2. Ablation experiments were conducted to analyze the performance of the network, and two important parts of the network were ablated, namely the anatomical constraint attention mechanism and the time step dimension attention mechanism, and the experimental results are shown in Table 2:

[0114] Table 2

[0115]

[0116] 3. Complexity comparison analysis: The parameter quantity and flops of the method of embodiment 1 and the mainstream method are compared, and the comparison results are shown in Table 3:

[0117] Table 3

[0118] network parameters flops LSTM 0.35M 1102.99M CNN-LSTM 0.25M 761.18M Transformer 1.59M 5051.94M ST-GCN 0.13M 381.86M Example 1 0.13M 252.28M

[0119] Embodiment 2:

[0120] An electronic device, characterized in that it comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps of the human action recognition method based on human bone distribution in embodiment 1.

[0121] The computer device of the application can be a device comprising a processor and a memory, such as a single-chip microcomputer comprising a central processing unit. Moreover, the processor is used to execute the computer program stored in the memory to realize the steps of the human action recognition method based on human bone distribution.

[0122] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0123] The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0124] Embodiment 3:

[0125] A computer readable storage medium, having stored thereon a computer program, the computer program being executed by a processor to implement the human action recognition method based on human bone distribution according to embodiment 1.

[0126] The computer readable storage medium of the present application can be any form of storage medium readable by the processor of the computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc., and the computer readable storage medium has stored thereon a computer program, when the processor of the computer device reads and executes the computer program stored in the memory, the steps of the human action recognition method based on human bone distribution can be implemented.

[0127] The computer program includes computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium, etc. It should be noted that the computer-readable medium contains content that can be appropriately added or deleted according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunications signals.

[0128] Abbreviations and key terms are defined as follows:

[0129] HAR: human action recognition;

[0130] LSTM: long short-term memory network;

[0131] CNN-LSTM: convolutional long short-term memory network;

[0132] Transformer: self-attention transformer;

[0133] ST-GCN: spatio-temporal graph convolutional network;

[0134] NVIDIA 3070Ti: NVIDIA 3070Ti graphics card;

[0135] F1-score: F1-score;

[0136] CZU-MHAD: Changzhou University dataset;

[0137] ASL sensor dataglove datase: American sign language gesture dataset;

[0138] GAT: graph attention mechanism.

[0139] It has to be noted that the terms "first", "second", and the like in connection with an entity or action refer to this entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without further constraints, exclude the presence of additional elements of the process, method, article, or apparatus.

[0140] While the application has been described with reference to specific implementations thereof, it should be understood that various modifications and substitutions can be made by those skilled in the art without departing from the scope of the present application. In particular, any one of the features of the present application disclosed above can be utilized independently of any other and the scope of the application should not be limited by the specific embodiments disclosed herein, but should be given the widest coverage possible in its true scope.

Claims

1. A human action recognition method based on human bone distribution, characterized in that, Comprising the following steps: S1. Wear sensors at the main joints of the human body to collect human action data, then set the sensor wearing position as a node according to the human bone structure diagram, add edges to each node according to the human morphology, and construct an adjacency table, then map the multi-dimensional features collected by each node to a hidden layer, and update using an attention mechanism to obtain the updated multi-dimensional feature sequence of each node; S2. Construct a sparse routing method of time series data time step dimension attention mechanism, divide the updated multi-dimensional feature sequence of each node obtained in step S1 into multiple windows, process each window to obtain the most relevant neighbor window of each window; S3. For each window obtained in step S2, the most relevant Sparse attention calculation is performed on each neighbor window to obtain human action recognition results based on the distribution of human skeleton; The specific implementation method of step S3 is to perform standard attention on each window most relevant to the neighbor window , and the expression is: ； wherein, is a normalized attention weight, ensuring the weights sum up to 1, is a query of is a key of is a value of The most relevant window is selected using a sparse attention mechanism. Feature aggregation is performed using neighbor windows, and the final output features are restored to their dimensionality through low-rank projection, expressed as follows: ； where, is the final output result, is a learnable weight matrix that is multiplied with to map back to the original dimension , is a bias term that enhances the expressive power of the model.

2. The human action recognition method based on human skeleton distribution according to claim 1, characterized in that, The specific implementation method of step S1 comprises the following steps: S1.

1. Wear sensors at the main joints of the human body, and then set the sensor wearing positions as nodes according to the human bone structure diagram , is the total number of nodes wearing sensors, is the ith node; S1.

2. Add edges to the nodes obtained in step S1.1 according to human morphology, and then construct an adjacency list wherein, is the i-th node is the j-th node is a flag indicating whether the i-th node is connected to the j-th node or not, if connected, then is 1, if not connected is 0; S1.

3. Map the multi-dimensional features of each node to a hidden layer to obtain the expression: wherein, , is the i-th node and the j-th node corresponding learnable weight matrix, is the transformed feature of the i-th node and the j-th node , is the mapping function, is the high-dimensional feature mapping result of the i-th node and the j-th node after splicing. S1.

4. Calculate the attention weight between nodes using an attention mechanism, and the calculation expression is: ； wherein, is the high-dimensional feature mapping result of the i-th node after splicing the q-th node, is the high-dimensional feature mapping result of the i-th node is the attention weight between the i-th node and the j-th node . S1.

5. utilizing the calculated characteristics of the aggregated nodes, generating an updated multi-dimensional feature sequence of the i-th node , the expression is: ； wherein, is an activation function, is a weight matrix of, is the feature of the node in, is the number of all nodes adjacent to the i-th node; S1.

6. Based on steps S1.1-S1.5, multiple rounds of updating are performed on the nodes, and the number of updating rounds can be fixed or a threshold value can be set, and when the characteristics of each node are less than the threshold value, the updating is stopped, and the updated multi-dimensional feature sequence of each node is obtained. S1.

6. Based on steps S1.1-S1.5, multiple rounds of updating are performed on the nodes, and the number of updating rounds can be fixed or a threshold value can be set, and when the characteristics of each node are less than the threshold value, the updating is stopped, and the 3. The human action recognition method based on human skeleton distribution according to claim 2, characterized in that, The specific implementation method of step S2 comprises the following steps: S2.

1. Construct a time sequence window division: Divide the updated multi-dimensional feature sequence of each node into multiple windows, and the window division expression is: ； wherein, is the number of windows, is the window length, is the time dimension length of the time series; each window after partitioning is ; S2.

2. Regional routing selection: S2.2.

1. Calculate the regional representation for each window, and the expression is: ； ； wherein, is a query, is a key, is an average pooling, is two learnable weight matrices of is original dimension of is a relationship with as follows; ； S2.2.

2. Constructing the window correlation matrix , is the mapping function: ； wherein, are the query and key for all windows, respectively, is the activation function named ReLU; position element in the window association matrix represents the association strength between windows and windows and is used to measure the importance of information interaction between windows S2.2.

3. Dynamically selecting the most relevant neighbor window for each window: according to the window association matrix dynamically selecting the most relevant neighbor window for each window , the expression is: ； wherein, is a set of indices returning the most relevant neighbor windows for each window.

4. An electronic device, comprising: A computer device comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the human action recognition method based on human bone distribution according to any one of claims 1-3.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the human action recognition method based on human bone distribution according to any one of claims 1-3.

Images