Human skeleton action recognition method for electric power field operation

Abstract

The invention discloses a human skeleton action recognition method for electric power field operation. The method comprises the following steps: acquiring a video of an actual electric power field operation scene; preprocessing the video and then building a three-dimensional human body posture estimation network model; performing feature extraction in the three-dimensional human body posture estimation network model by using a self-attention mechanism to obtain three-dimensional skeleton joint point sequence data of a human body; and building a human skeleton action recognition network model to capture the motion information of the skeleton sequence and the correlation between time and space, and decoding through a full connection layer to obtain a final action recognition result. According to the method, the three-dimensional human body posture estimation technology and the human body skeleton action recognition technology are utilized, the defects that the robustness is poor, the background is complex and illumination changes when action recognition is directly carried out through video images are overcome, the accuracy of action recognition is improved, and therefore the accuracy of action risk judgment of field operation personnel is improved, and the working efficiency is improved. And the field operation efficiency and the safety management level are improved, and possible risks are reduced.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision, and in particular to a human skeleton motion recognition method for power field operations. Background Art

[0002] The safety management of power site operations is directly related to the stability of the entire industry and society. The guarantee of power safety depends largely on the management and construction links. Power site operations involve high-risk operations, which place extremely high demands on the safety of personnel. By implementing effective risk monitoring, the safety management level of power site operations can be significantly improved, potential safety hazards can be reduced, and the life safety of construction personnel and the overall safety performance of the construction site can be guaranteed. Action recognition based on human skeletons relies on human posture estimation technology. Its basic principle is to extract the spatial position information of human key points (such as head, shoulders, elbows, knees, etc.) from videos or images to construct a skeleton structure that reflects human posture. Skeleton data has the characteristics of high abstraction, low complexity and good robustness. It is not easily affected by factors such as background, scale, perspective, and lighting. Skeleton data is more in line with the actual physical meaning of human actions, can better characterize the process of human movement, improve the accuracy of action recognition, thereby improving the accuracy of action risk judgment for on-site workers, improving the efficiency and safety management level of on-site operations, and reducing possible risks. Summary of the invention

[0003] The present invention proposes three-dimensional human posture estimation technology and human skeleton motion recognition technology, which overcomes the defects of poor robustness, complex background and changing lighting of direct motion recognition through video images in the power field operation environment, improves the accuracy of motion recognition, thereby improving the accuracy of risk judgment of on-site workers' actions, improving the efficiency and safety management level of on-site operations, and reducing possible risks.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows: a method for recognizing human skeleton movements in power field operations comprises the following steps:

[0005] S1. Collect videos of actual power field operation scenes;

[0006] S2, preprocessing the video image data;

[0007] S3, building a 3D human posture estimation network model, and inputting the preprocessed data into the 3D human posture estimation network model to obtain 3D skeleton joint point sequence data of the human body;

[0008] S4, building a human skeleton action recognition network model, inputting the three-dimensional skeleton joint point sequence data into the human skeleton action recognition network model to extract features in spatial and temporal dimensions;

[0009] S5. Input the posture features after spatial and temporal dimension feature extraction into the fully connected layer for decoding, and output the final action prediction result.

[0010] In some embodiments, in S1, the initialization video image data of the workers in the actual power field operation scene is obtained, which includes interactive behaviors such as equipment operation, inspection, maintenance or installation, and also records different environmental settings on site, including weather, lighting conditions, and surrounding facility layouts;

[0011] Furthermore, in S2, the initialized video image data of the workers in the actual power field operation scene is subjected to size uniformity processing and data normalization processing;

[0012] Furthermore, in S3, a 3D human posture estimation network model is built. The network model is constructed by an encoder and a decoder. The encoder is mainly composed of a slice embedding module and multiple Transformer modules based on the self-attention mechanism. The encoder model has a simple structure, strong scalability, flexible training paradigm and excellent effect, which is suitable for power operation sites with complex environments. Each Transformer module consists of a multi-head self-attention layer (MHSA) and a feedforward network layer (FFN).

[0013]

[0014] Among them, F i represents the output of the i-th Transformer layer, represents the output of the multi-head self-attention layer of the i-th Transformer layer, F i+1 represents the output of the i+1th Transformer layer. The decoder module consists of two deconvolution blocks, each of which contains a deconvolution layer followed by batch normalization and ReLU.

[0015] K=Conv 1×1 (Deconv(Deconv(F out )))

[0016] Among them, F outRepresents the output after passing through the encoder module, and K represents the estimated heat map (one for each skeleton joint point). Furthermore, in S4, a human skeleton action recognition network model is built: a multi-scale spatiotemporal cross human skeleton action recognition model based on self-attention mechanism. The entire model designs a multi-scale spatiotemporal cross Transformer architecture, which effectively captures the temporal information and joint dependency of human features at different scales. This model also designs an effective branch interaction method to learn the consistency of time channels and spatial channels. The entire model framework consists of three key modules: spatiotemporal encoding module, multi-scale spatiotemporal cross Transformer module, and multi-scale spatiotemporal cross decoding module.

[0017] In the temporal-spatial encoding module, human features of different scales are encoded and embedded in the temporal and spatial dimensions. We adopt a dual-stream architecture of time and space to encode the temporal and spatial channels independently, and map the newly generated motion sequence into two independent embedding spaces (time channel E t and spatial channel E s ).

[0018] When performing temporal encoding, we use a separate linear layer to map the data of all joints into a D-dimensional space and learn the contextual information of each dimension on the timeline.

[0019]

[0020] in, Represents the coordinates of the nth joint at time t, weight and bias is for each joint n. The temporal channel successfully captures the characteristics of parameter changes over time, while the spatial channel deeply understands the potential dependencies between skeleton parameters at each time point, which enables accurate expression of the reasonable posture of the human body. In terms of spatial encoding, we also use a linear layer to expand the sequence length from T to T+K and map it to a D-dimensional space to obtain the feature representation E in the spatial dimension. s :

[0021]

[0022] in, Represents the coordinates of the nth joint at time t, weight and bias is for each time t. Subsequently, we use a multi-scale spatiotemporal cross-Transformer module to deeply capture the spatiotemporal information of human body dynamics at different scales, distinguish the motion information by time dimension and space dimension, and use the time transformer and space transformer to independently process the time series data and spatial relationship respectively. The spatiotemporal cross-attention mechanism is used to synchronously capture the consistency of time and space dimensions and strengthen the connection between time and space branches.

[0023] The Temporal Transformer adopts the "Scaled Dot-ProductAttention" mechanism of the original Transformer model to update the embedding of each joint by considering the information of the same joint in historical frames. This process involves the concepts of query, key, and value, where the representation of "value" is a sequence of all key indices. Using the dot product attention mechanism, we compare the query with all keys to evaluate the similarity between different joints. If a key is very close to the query (has a high attention weight), then the value corresponding to the key is considered to be highly relevant to the query. The position similarity is calculated as follows:

[0024]

[0025] M is the mask matrix.

[0026] The spatial transformer uses a similar architecture to the temporal transformer and implements a multi-head attention mechanism. In this mechanism, the query (Q), key (K), and value (V) of each head are linearly transformed through three independent learnable weight matrices. The difference from the temporal transformer is that the spatial transformer aims to explore spatial relationships, so there is no need to introduce a mask matrix M to prevent future information leakage in this scenario. The spatial transformer focuses on analyzing the interdependencies between human joints within the same time step t, with the goal of simulating the spatial connections between different joints in the same frame, covering the information of all human joints at that point in time.

[0027] We propose an innovative cross-attention interaction between spatial and temporal branches. In both spatial and temporal branches, an additional information (Info) token is introduced to facilitate the interaction between branches. This interaction includes not only the exchange of messages and patch tokens within the branch, but also the exchange of information across branches. In the spatial branch, the information token aggregates the information about the human joint dynamics within the branch, and this stage is called "information aggregation". The information token aggregates the information of all patch tokens in its branch, and then uses the self-attention mechanism to exchange information with the patch tokens of other branches. After receiving the information from another branch, the updated information token returns to its original branch with the information through the next cross-attention mechanism module, aiming to integrate the knowledge obtained from other branches into its own spatiotemporal representation, a process called "information feedback". The multi-head attention mechanism (MHC) is adopted in the cross-attention mechanism module, which allows the human body dynamic information to be projected from multiple perspectives, and layer normalization (LN) and residual connection are added after each module to enhance the model performance. The output of the cross-attention mechanism module is defined as follows:

[0028]

[0029]

[0030] is the information token, is a patch token, fs(·) and gs(·) represent the projection and back-projection functions for dimension alignment, respectively. s is the output of the cross-attention mechanism module of the spatial branch, while the temporal branch adopts the same processing flow. This interaction is achieved through a series of K cross-attention mechanism modules, each of which is dedicated to learning the consistency between the spatial and temporal branch information. After the interaction is completed, the fused features are used to update the input of the spatiotemporal Transformer.

[0031] In the multi-scale spatiotemporal cross decoding module, the outputs Y1, Y2, and Y3 from the spatiotemporal cross Transformer encoding modules at each scale are compared with the inputs at the corresponding scales. The purpose of fusing residual connections is to deal with the gradient vanishing or exploding problem that may be encountered as the model depth increases. After completing the residual connection, all results are summarized.

[0032]

[0033] Furthermore, in S5, the posture features extracted by the human skeleton action recognition network module are input into the fully connected layer for decoding to obtain the prediction result of the human skeleton action recognition.

[0034] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, a human skeleton motion recognition method for power field operations is implemented.

[0035] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the human skeleton motion recognition method for power field operations.

[0036] Compared with the prior art, the advantages of the present invention are as follows:

[0037] The present invention utilizes three-dimensional human posture estimation technology and human skeleton motion recognition technology to overcome the defects of poor robustness, complex background, and changing lighting in direct motion recognition through video images, thereby improving the accuracy of motion recognition, thereby improving the accuracy of risk judgment of on-site workers' actions, improving the efficiency and safety management level of on-site operations, and reducing possible risks.

[0038] Skeleton data is highly abstract, low in complexity, and robust, and is not easily affected by factors such as background, scale, perspective, and lighting. Skeleton data is more in line with the actual physical meaning of human motion and can better characterize the process of human motion. There is a strong correlation between the joints of the skeleton and its adjacent nodes, so rich body structure information can be found in the skeleton data in an intra-frame manner. The above advantages make the skeleton-based human motion recognition algorithm fast, robust, and scalable. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a step framework diagram of a method for recognizing human skeleton movements for electric power field operations in an embodiment of the present invention;

[0040] Figure 2 A model diagram of a coding module of a three-dimensional human posture estimation network model of a human skeleton action recognition method for power field operations in an embodiment of the present invention;

[0041] Figure 3 A model diagram of a decoding module of a 3D human posture estimation network model of a human skeleton action recognition method for power field operations in an embodiment of the present invention;

[0042] Figure 4 This is a diagram of a human skeleton motion recognition network model of a human skeleton motion recognition method for power field operations in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the description of the present invention, the meaning of "several" is more than one, the meaning of "many" is more than two, and the meaning of "greater than", "less than", "exceeding", etc. are understood to exclude the number itself, and the meaning of "above", "below", "within", etc. are understood to include the number itself. If there is a description of the first and the second, it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0044] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0045] Example 1

[0046] like Figure 1 As shown, the human skeleton action recognition method for power field operations includes the following steps:

[0047] S1. Collect videos of actual power field operation scenes;

[0048] In S1, the initialization video image data of the workers in the actual power field operation scene is obtained, which includes interactive behaviors such as equipment operation, inspection, maintenance or installation. Different environmental settings on site, such as weather, lighting conditions, and surrounding facility layout, are also recorded;

[0049] S2, preprocessing the video image data;

[0050] In S2, the initialization video image data of the workers in the actual power field operation scene is subjected to size consistency processing and data normalization processing;

[0051] S3, building a 3D human posture estimation network model, and inputting the preprocessed data into the 3D human posture estimation network model to obtain 3D skeleton joint point sequence data of the human body;

[0052] In S3, a 3D human posture estimation network model is built. The network model is built by an encoder and a decoder. The encoder is mainly composed of a slice embedding module and multiple Transformer modules based on the self-attention mechanism. The encoder model has a simple structure, strong scalability, flexible training paradigm and excellent results. Each Transformer module consists of a multi-head self-attention layer (MHSA) and a feedforward network layer (FFN).

[0053]

[0054] Among them, F i represents the output of the i-th Transformer layer, represents the output of the multi-head self-attention layer of the i-th Transformer layer, F i+1 represents the output of the i+1th Transformer layer. The decoder module consists of two deconvolution blocks, each of which contains a deconvolution layer followed by batch normalization and ReLU.

[0055] K=Conv 1×1 (Deconv(Deconv(F out )))

[0056] Among them, F out represents the output after passing through the encoder module, and K represents the estimated heat map (one for each skeleton joint point).

[0057] S4, building a human skeleton action recognition network model, inputting the three-dimensional skeleton joint point sequence data into the human skeleton action recognition network model to extract features in spatial and temporal dimensions;

[0058] In S4, a human skeleton action recognition network model is built: a multi-scale spatiotemporal cross human skeleton action recognition model based on the self-attention mechanism. The entire model designs a multi-scale spatiotemporal cross Transformer architecture, which effectively captures the temporal information and joint dependency of human features at different scales. This model also designs an effective branch interaction method to learn the consistency of time channels and space channels. The entire model framework consists of three key modules: time-space encoding module, multi-scale spatiotemporal cross Transformer module, and multi-scale spatiotemporal cross decoding module.

[0059] In the temporal-spatial encoding module, human features of different scales are encoded and embedded in the temporal and spatial dimensions. We adopt a dual-stream architecture of time and space to encode the temporal and spatial channels independently, and map the newly generated motion sequence into two independent embedding spaces (time channel E t and spatial channel E s).

[0060] When performing temporal encoding, we use a separate linear layer to map the data of all joints into a D-dimensional space and learn contextual information for each dimension on the timeline.

[0061]

[0062] in, Represents the coordinates of the nth joint at time t, weight and bias is for each joint n. The temporal channel successfully captures the characteristics of parameter changes over time, while the spatial channel deeply understands the potential dependencies between skeleton parameters at each time point, which enables accurate expression of the reasonable posture of the human body. In terms of spatial encoding, we also use a linear layer to expand the sequence length from T to T+K and map it to a D-dimensional space to obtain the feature representation E in the spatial dimension. s :

[0063]

[0064] in, Represents the coordinates of the nth joint at time t, weight and bias is for each time t. Subsequently, we use a multi-scale spatiotemporal cross-Transformer module to deeply capture the spatiotemporal information of human body dynamics at different scales, distinguish the motion information by time dimension and space dimension, and use the time transformer and space transformer to independently process the time series data and spatial relationship respectively. The spatiotemporal cross-attention mechanism is used to synchronously capture the consistency of time and space dimensions and strengthen the connection between time and space branches.

[0065] The Temporal Transformer adopts the "Scaled Dot-ProductAttention" mechanism of the original Transformer model to update the embedding of each joint by considering the information of the same joint in historical frames. This process involves the concepts of query, key, and value, where the representation of "value" is a sequence of all key indices. Using the dot product attention mechanism, we compare the query with all keys to evaluate the similarity between different joints. If a key is very close to the query (has a high attention weight), then the value corresponding to the key is considered to be highly relevant to the query. The position similarity is calculated as follows:

[0066]

[0067] M is the mask matrix.

[0068] The spatial transformer uses a similar architecture to the temporal transformer and implements a multi-head attention mechanism. In this mechanism, the query (Q), key (K), and value (V) of each head are linearly transformed through three independent learnable weight matrices. The difference from the temporal transformer is that the spatial transformer aims to explore spatial relationships, so there is no need to introduce a mask matrix M to prevent future information leakage in this scenario. The spatial transformer focuses on analyzing the interdependencies between human joints within the same time step t, with the goal of simulating the spatial connections between different joints in the same frame, covering the information of all human joints at that point in time.

[0069] We propose an innovative cross-attention interaction between spatial and temporal branches. In both spatial and temporal branches, an additional information (Info) token is introduced to facilitate the interaction between branches. This interaction includes not only the exchange of messages and patch tokens within the branch, but also the exchange of information across branches. In the spatial branch, the information token aggregates the information about the human joint dynamics within the branch, and this stage is called "information aggregation". The information token aggregates the information of all patch tokens in its branch, and then uses the self-attention mechanism to exchange information with the patch tokens of other branches. After receiving the information from another branch, the updated information token returns to its original branch with the information through the next cross-attention mechanism module, aiming to integrate the knowledge obtained from other branches into its own spatiotemporal representation, a process called "information feedback". The multi-head attention mechanism (MHC) is adopted in the cross-attention mechanism module, which allows the human body dynamic information to be projected from multiple perspectives, and layer normalization (LN) and residual connection are added after each module to enhance the model performance. The output of the cross-attention mechanism module is defined as follows:

[0070]

[0071]

[0072] is the information token, is a patch token, fs(·) and gs(·) represent the projection and back-projection functions for dimension alignment, respectively. sis the output of the cross-attention mechanism module of the spatial branch, while the temporal branch adopts the same processing flow. This interaction is achieved through a series of K cross-attention mechanism modules, each of which is dedicated to learning the consistency between the spatial and temporal branch information. After the interaction is completed, the fused features are used to update the input of the spatiotemporal Transformer.

[0073] In the multi-scale spatiotemporal cross decoding module, the outputs Y1, Y2, and Y3 from the spatiotemporal cross Transformer encoding modules at each scale are compared with the inputs at the corresponding scales. The purpose of fusing residual connections is to deal with the gradient vanishing or exploding problem that may be encountered as the model depth increases. After completing the residual connection, all results are summarized.

[0074]

[0075] S5. Input the posture features extracted by the human skeleton action recognition network module into the fully connected layer for decoding to obtain the prediction results of human skeleton action recognition.

[0076] Example 2

[0077] This embodiment provides a human skeleton motion recognition device for power field operations, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method according to Example 1.

[0078] Example 3

[0079] This embodiment provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1. These computer program instructions may also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process described in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A human skeleton motion recognition method for power field operations, characterized in that: The steps include: S1. Collect videos of actual power field operation scenes; S2, preprocessing the video image data; S3, building a 3D human posture estimation network model, and inputting the preprocessed data into the 3D human posture estimation network model to obtain 3D skeleton joint point sequence data of the human body; S4, building a human skeleton action recognition network model, inputting the three-dimensional skeleton joint point sequence data into the human skeleton action recognition network model to extract features in spatial and temporal dimensions; S5. Input the posture features after spatial and temporal dimension feature extraction into the fully connected layer for decoding, and output the final action prediction result.

2. The human skeleton motion recognition method for power field operations according to claim 1 is characterized in that: In S1, the initialization video image data of the workers in the actual power field operation scene is obtained, which includes equipment operation, inspection, maintenance or installation interaction behaviors, and also records different environmental adjustments on site, including weather, lighting conditions, and surrounding facility layout.

3. The human skeleton motion recognition method for power field operations according to claim 1 is characterized in that: In S2, the initialized video image data of the workers in the actual power field operation scene are subjected to size consistency processing and data normalization processing.

4. The human skeleton motion recognition method for power field operations according to claim 1 is characterized in that: In S3, a 3D human posture estimation network model is built. The network model is composed of an encoder and a decoder. The encoder is mainly composed of a slice embedding module and multiple Transformer modules based on the self-attention mechanism. The encoder model has a simple structure, strong scalability, flexible training paradigm and excellent effect. It is suitable for power operation sites with complex environments. Each Transformer module consists of a multi-head self-attention layer (MHSA) and a feedforward network layer (FFN). Among them, F i represents the output of the i-th Transformer layer, represents the output of the multi-head self-attention layer of the i-th Transformer layer, F i+1 represents the output of the i+1th Transformer layer. The decoder module consists of two deconvolution blocks, each of which contains a deconvolution layer followed by batch normalization and ReLU. K=Conv 1×1 (Deconv(Deconv(F out ))) Among them, F out represents the output after passing through the encoder module, and K represents the estimated heat map, one for each skeleton joint point.

5. The human skeleton motion recognition method for power field operations according to claim 1 is characterized in that: In S4, a human skeleton action recognition network model is built: a multi-scale spatiotemporal cross human skeleton action recognition model based on the self-attention mechanism. The entire model designs a multi-scale spatiotemporal cross Transformer architecture. The entire model framework consists of three key modules: time-space encoding module, multi-scale spatiotemporal cross Transformer module, and multi-scale spatiotemporal cross decoding module. In the time-space encoding module, human features of different scales are encoded and embedded in the time and space dimensions. The dual-stream architecture of time and space is adopted to encode the time and space channels independently, and the newly generated motion sequence is mapped into two independent embedding spaces, namely the time channel E and the spatial channel E. t and spatial channel E s , When performing temporal encoding, a separate linear layer is used to map the data of all joints into a D-dimensional space to learn the contextual information of each dimension on the timeline. in, Represents the coordinates of the nth joint at time t, weight and bias is for each joint n. The time channel successfully captures the characteristics of parameter changes over time, while the spatial channel deeply understands the potential dependencies between skeleton parameters at each time point, which enables accurate expression of the reasonable posture of the human body. In terms of spatial encoding, a linear layer is used to expand the sequence length from T to T+K and map it to a D-dimensional space to obtain the feature representation E in the spatial dimension. s : in, Represents the coordinates of the nth joint at time t, weight and bias For each time t, a multi-scale spatiotemporal cross-Transformer module is then used to deeply capture the spatiotemporal information of human body dynamics at different scales, distinguish the motion information by time dimension and space dimension, and use the time Transformer and space Transformer to independently process the time series data and spatial relationship respectively, and use the spatiotemporal cross-attention mechanism to synchronously capture the consistency in time and space dimensions, strengthen the connection between time and space branches, Temporal Transformer adopts the "Scaled Dot-Product Attention" mechanism of the original Transformer model, and updates the embedding of each joint by considering the information of the same joint in the historical frames. This process involves the concepts of query, key, and value, where the "value" is represented by the sequence of all key indexes. The dot product attention mechanism is used to compare the query with all keys to evaluate the similarity between different joints. If a key is very close to the query (has a higher attention weight), then the value corresponding to the key is considered to be highly correlated with the query. The position similarity is calculated as follows: M is the mask matrix, The spatial Transformer adopts a similar architecture to the temporal Transformer and implements a multi-head attention mechanism, in which the query (Q), key (K), and value (V) of each head are linearly transformed through three independent learnable weight matrices. The spatial Transformer focuses on analyzing the interdependence between human joints within the same time step t, with the aim of simulating the spatial connection between different joints in the same frame, covering the information of all human joints at that time point, and proposing an innovative cross-attention interaction method between spatial and temporal branches. In the spatial and temporal branches, an additional information (Info) token is introduced to promote the interaction between branches. In the spatial branch, the information token aggregates the branch. The information about the dynamics of human joints is collected in this stage. This stage is called "information aggregation". The information token aggregates all patch token information of its branch, and then uses the self-attention mechanism to exchange information with patch tokens of other branches. After receiving information from another branch, the updated information token returns to its original branch with this information through the next cross-attention mechanism module, aiming to integrate the knowledge obtained from other branches into its own spatiotemporal representation. This process is called "information feedback". The multi-head attention mechanism (MHC) is adopted in the cross-attention mechanism module, which allows the projection of human dynamic information from multiple angles, and layer normalization (LN) and residual connection are added after each module to enhance the model performance. The output of the cross-attention mechanism module is defined as follows: is the information token, is the patch token, f s (·) and gs(·) represent the projection and back-projection functions for dimension alignment, respectively. s is the output of the cross-attention mechanism module of the spatial branch, while the temporal branch adopts the same processing flow. This interaction is achieved through a series of K cross-attention mechanism modules, each of which is dedicated to learning the consistency between the spatial and temporal branch information. After the interaction is completed, the fused features are used to update the input of the spatiotemporal Transformer. In the multi-scale spatiotemporal cross decoding module, the outputs Y1, Y2, and Y3 from the spatiotemporal cross Transformer encoding modules at each scale are fused with the inputs Ds1, Ds2, and Ds3 at the corresponding scales through residual connections. This is to deal with the gradient vanishing or explosion problems that may occur as the model depth increases. After the residual connection is completed, all the results are summarized.

6. The human skeleton motion recognition method for power field operations according to claim 1 is characterized in that: In S5, the posture features extracted by the human skeleton action recognition network module are input into the fully connected layer for decoding to obtain the prediction result of human skeleton action recognition.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for human skeleton motion recognition for power field operations as described in any one of claims 1 to 6 above is implemented.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by the processor, the human skeleton motion recognition method for power field operations as described in any one of claims 1-6 is implemented.

Images