Method, device and equipment for human behavior recognition in power production based on skeleton

Abstract

The present invention discloses a method, device and equipment for identifying human behavior in power production based on skeletons, which relates to the field of power safety technology and can solve the technical problem that whether human behavior in power production is standardized requires manual monitoring, resulting in low efficiency and high labor costs. It includes: obtaining a video of human behavior in power production to be identified, generating skeleton sequence data according to the video of human behavior in power production, and the skeleton sequence data includes at least one group of actions; inputting the skeleton sequence data into a trained human behavior recognition model for power production to identify the target human behavior in power production, and the human behavior recognition model for power production includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target human behavior in power production.

Description

Technical Field

[0001] The present invention relates to the field of electric power safety technology, and in particular to a skeleton-based method, device and equipment for identifying human behavior in electric power production. Background Art

[0002] Safety in production is a very important part of the operation of the power system. The existing safety standard methods are mainly through the formulation of safety regulations and codes, video surveillance systems, safety risk management platforms, etc. However, the above methods rely on the production personnel's own safety awareness and real-time human monitoring, which has the defects of low efficiency and high labor costs.

[0003] Therefore, there is an urgent need for a method to efficiently and intelligently identify human behavior in power production in order to regulate the illegal operations or behaviors of power production personnel. Summary of the invention

[0004] In view of this, the present invention provides a skeleton-based method, device and equipment for identifying human behavior in power production, which can solve the technical problem that whether human behavior during power production is currently standardized requires manual monitoring, resulting in low efficiency and high labor costs.

[0005] According to a first aspect of the present invention, a skeleton-based method for identifying human behavior in power production is provided, the method comprising:

[0006] Acquire a power generation human behavior video to be identified, and generate skeleton sequence data according to the power generation human behavior video, wherein the skeleton sequence data includes at least one set of actions;

[0007] The skeleton sequence data is input into a trained power production human behavior recognition model for recognition to obtain target power production human behavior. The power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior.

[0008] According to a second aspect of the present invention, there is provided a skeleton-based power production human behavior recognition device, the device comprising:

[0009] An acquisition module, used for acquiring a power generation human behavior video to be identified, and generating skeleton sequence data according to the power generation human behavior video, wherein the skeleton sequence data includes at least one group of actions;

[0010] A prediction module is used to input the skeleton sequence data into a trained power production human behavior recognition model for recognition, so as to obtain a target power production human behavior. The power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior.

[0011] According to a third aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned skeleton-based power production human behavior recognition method is implemented.

[0012] According to the fourth aspect of the present application, a computer device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned skeleton-based power production human behavior recognition method when executing the program.

[0013] Through the above technical scheme, the present invention provides a skeleton-based power production human behavior recognition method, device and equipment, which obtains a power production human behavior video to be recognized, generates skeleton sequence data according to the power production human behavior video, and the skeleton sequence data includes at least one group of actions; the skeleton sequence data is input into a trained power production human behavior recognition model for recognition to obtain the target power production human behavior, and the power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior. Through the technical scheme of the present invention, compared with the static topological structure of the existing spatiotemporal graph convolutional network, The spatiotemporal graph convolutional network of the present application adopts a dynamic topological structure, which makes the model have stronger generalization ability. Compared with the existing spatiotemporal graph convolutional network strategy of sharing weights in different channels, the spatiotemporal graph convolutional network of the present application adopts a strategy of not sharing weights in different channels, which makes the model have stronger accuracy. The present application adopts a spatiotemporal transformer network, which uses the self-attention mechanism of the time dimension and the self-attention mechanism of the space dimension to assign appropriate weights to the corresponding skeleton point features according to specific behaviors, that is, in some behaviors, only specific parts of the skeleton points will have an impact on the behavior judgment, so the weights of other key points can be ignored or reduced, so that even in scenes with occlusion, the behavior can still be accurately identified, which improves the occlusion robustness of the model. The first feature obtained by the spatiotemporal graph convolutional network of the present application is combined with the second feature obtained by the spatiotemporal transformer network of the present application, which can more accurately identify the target power production human behavior.

[0014] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation on the local application. In the drawings:

[0016] Figure 1 A schematic diagram of a process of a skeleton-based power production human behavior recognition method provided by an embodiment of the present invention is shown;

[0017] Figure 2 A schematic flow chart of another skeleton-based method for identifying human behavior in power production provided by an embodiment of the present invention is shown;

[0018] Figure 3 A schematic diagram of the structure of a skeleton-based power production human behavior recognition device provided by an embodiment of the present invention is shown;

[0019] Figure 4 A schematic diagram showing the structure of another skeleton-based power production human behavior recognition device provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the structure of a human behavior recognition model for power production provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0022] This embodiment provides a skeleton-based method for identifying human behavior in power production. Figure 1 As shown, the method includes:

[0023] 101. Obtain a power generation human behavior video to be identified, and generate skeleton sequence data according to the power generation human behavior video, wherein the skeleton sequence data includes at least one set of actions.

[0024] It should be noted that the power production human behavior video to be identified is a real-time video stream. By intelligently identifying the real-time video stream, the target power production human behavior can be identified in real time. If the target power production human behavior involves illegal operations, an alarm will be automatically issued or the personnel performing power production will be reminded that their actions involve illegal operations. The entire process does not require manual video monitoring, which reduces labor costs and improves efficiency.

[0025] For the present embodiment, since the biggest feature of the YOLO (You Only Look Once) algorithm is that it runs very fast, it can be used to generate skeleton sequence data in real time. Specifically, the YOLO algorithm is used to generate the coordinate position and confidence of each joint point on different time dimension information according to the power production human behavior video (wherein, the number of joint points of the human body is fixed), and the coordinate position and confidence of each joint point are used as spatial dimension information, thereby obtaining the skeleton sequence data, that is, the skeleton sequence data includes the time dimension information of each joint point, and the spatial dimension information on different time dimension information. Among them, the method of generating the coordinate position and confidence of each joint point on different time dimension information according to the power production human behavior video using the YOLO algorithm is a prior art, which will not be repeated here.

[0026] The skeleton sequence data includes at least one group of actions, for example, the skeleton sequence data includes a group of actions of walking from a first point to a second point, and the corresponding target power production human behavior is walking, and also includes another group of actions of crossing from the second point to a third point, and the corresponding target power production human behavior is crossing. No matter how many groups of actions the skeleton sequence data includes, in step 102 of the embodiment, the skeleton sequence data is input into the trained power production human behavior recognition model for recognition, and the target power production human behavior corresponding to each group of actions can be obtained.

[0027] 102. Input the skeleton sequence data into a trained power production human behavior recognition model for recognition to obtain target power production human behavior. The power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior.

[0028] It should be noted that the spatiotemporal graph convolutional network of this embodiment is improved compared with the existing spatiotemporal graph convolutional networks (Spatial-Temporal Graph Convolutional Networks, ST-GCN). The existing spatiotemporal graph convolutional networks adopt a static topological structure and a strategy of sharing weights in different channels, wherein the static topological structure refers to the topology (connection relationship of nodes) is not dynamically adjusted during the reasoning process, but compared with the static topological structure, the spatiotemporal graph convolutional network of this embodiment adopts a dynamic topological structure, so that the model has a stronger generalization ability, compared with the strategy of sharing weights in different channels, the spatiotemporal graph convolutional network of this embodiment adopts a strategy of not sharing weights in different channels, so that the model has a stronger accuracy, the spatiotemporal graph convolutional network of this embodiment adopts a dynamic topological structure and a strategy of not sharing weights in different channels so that the adjacency matrix in the graph convolution process can change accordingly with different inputs during the reasoning process, and for different categories of inputs, the weight parameters are independent of each other between channels, which can enhance the model's ability to extract different input features.

[0029] In the actual operation scenarios of power stations, many illegal operations or behaviors will be lost due to occlusion or part of the skeleton sequence data will be lost during the posture recognition process. Whether it is the existing spatiotemporal graph convolutional network or the improved spatiotemporal graph convolutional network of this embodiment, the basic architecture method belongs to the graph convolutional network, which relies heavily on the integrity of the skeleton sequence data. In the case of occlusion, the accuracy of behavior recognition will be greatly reduced. Therefore, the spatiotemporal transformer network is introduced, which uses the self-attention mechanism of the time dimension and the self-attention mechanism of the space dimension to assign appropriate weights to the corresponding skeleton point features according to specific behaviors. That is, in some behaviors, only specific parts of the skeleton points will affect the behavior judgment, so the weights of other key points can be ignored or reduced, so that even in scenes with occlusion, the behavior can still be accurately identified, thereby improving the occlusion robustness of the model.

[0030] For this embodiment, before the skeleton sequence data is input into the trained power production human behavior recognition model for identification to obtain the target power production human behavior, a video data set from the actual production recording of the power station is obtained, and the video data set includes scenes where the actions are blocked. The video data set is processed using the YOLO algorithm to obtain skeleton sequence sample data, and the skeleton sequence sample data includes labels of multiple behavior categories, and each group of actions of the skeleton sequence sample data corresponds to a label of a behavior category, and the labels of the behavior categories include, for example, crossing a fence, operating a cabinet door, moving quickly, and making and receiving calls, etc. The skeleton sequence sample data is used to train the initial power production human behavior recognition model, wherein the training method is the same as the method of inputting the skeleton sequence data into the trained power production human behavior recognition model for identification to obtain the target power production human behavior, which will not be repeated here, and the loss function is calculated until the loss function is less than a preset value to obtain the trained power production human behavior recognition model.

[0031] The present invention provides a skeleton-based power production human behavior recognition method, device and equipment, which obtains a power production human behavior video to be recognized, generates skeleton sequence data according to the power production human behavior video, and the skeleton sequence data includes at least one group of actions; the skeleton sequence data is input into a trained power production human behavior recognition model for recognition to obtain the target power production human behavior, and the power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior. Through the technical solution of the present invention, compared with the static topological structure of the existing spatiotemporal graph convolutional network, the present application The spatial graph convolutional network adopts a dynamic topological structure, which makes the model have stronger generalization ability. Compared with the existing spatiotemporal graph convolutional network strategy of sharing weights in different channels, the spatiotemporal graph convolutional network of the present application adopts a strategy of not sharing weights in different channels, which makes the model have stronger accuracy. The present application adopts a spatiotemporal transformer network, which uses the self-attention mechanism of the time dimension and the self-attention mechanism of the spatial dimension to assign appropriate weights to the corresponding skeleton point features according to specific behaviors. That is, in some behaviors, only specific skeleton points will have an impact on the behavior judgment, so the weights of other key points can be ignored or reduced, so that even in scenes with occlusion, the behavior can still be accurately identified, which improves the occlusion robustness of the model. The first feature obtained by the spatiotemporal graph convolutional network of the present application is combined with the second feature obtained by the spatiotemporal transformer network of the present application, which can more accurately identify the target power production human behavior.

[0032] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another skeleton-based power production human behavior recognition method is provided, such as Figure 2 As shown, the method includes:

[0033] 201. Obtain a power generation human behavior video to be identified, and generate skeleton sequence data according to the power generation human behavior video, wherein the skeleton sequence data includes at least one set of actions.

[0034] For this embodiment, the specific implementation is the same as step 101 of the embodiment, and will not be repeated here.

[0035] 202. Input the skeleton sequence data into a trained power production human behavior recognition model for recognition to obtain target power production human behavior. The power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module. The spatiotemporal graph convolutional network includes a feature dimension upgrading module and an aggregation splicing module connected in sequence, and also includes a feature extraction module, a multi-layer perceptron, an aggregation module and the aggregation splicing module connected in sequence. The spatiotemporal transformer network includes a spatial transformer module and a splicing module connected in sequence, and also includes a temporal transformer module and the splicing module connected in sequence. The skeleton sequence data is input into the feature dimension upgrading module for feature dimension upgrading processing to obtain a first dimension upgraded feature.

[0036] Among them, the human behavior recognition model for power production is as follows Figure 5 shown.

[0037] For this embodiment, as an implementation method, the feature dimension-upgrading module performs a linear transformation on the bone sequence data, and the linear transformation does not change the feature information of the bone sequence data itself, wherein the data dimension of the bone sequence data is N times C, N is the number of joints, C is the number of channels, and the data dimension of the first dimension-upgrading feature is N times C′, N is the number of joints, and C′ is the number of channels after dimension-upgrading;

[0038] 203. Input the bone sequence data into the feature extraction module for feature extraction to obtain output features of the joint points, input the output features of the joint points into the multi-layer perceptron for feature dimension upgrading to obtain second dimension upgraded features, input the second dimension upgraded features into the aggregation module, and use the aggregation module to perform multiplication calculation of the second dimension upgraded features and the adjacency matrix of the same group of actions to obtain aggregated features.

[0039] For this embodiment, the step of inputting the skeleton sequence data into the feature extraction module for feature extraction to obtain output features of the joint points includes: inputting the skeleton sequence data into the feature extraction module, and determining, for each group of actions, the input features of the jth joint point of the group of actions in the skeleton sequence data; constructing an adjacency matrix for the group of actions, wherein the adjacency matrix is ​​the degree of connection between the jth joint point and the ith joint point in the group of actions, and i is not equal to j; multiplying the adjacency matrix with the weight matrix to obtain a first generalized weight matrix between the i-th joint point and the j-th joint point, constructing a first constraint condition corresponding to the second generalized weight matrix and the third generalized weight matrix under the same group of actions, and constructing a second constraint condition corresponding to the first generalized weight matrix and the first generalized weight matrix under different groups of actions; multiplying the first generalized weight matrix with the input features of the j-th joint point to obtain a second multiplication result, adding the second multiplication results corresponding to all joint points to obtain the output features of the i-th joint point of the group of actions, so as to obtain the output features of the joint points.

[0040] The data dimension of the skeleton sequence data is N times C. Taking the kth group of actions as an example, the input features of the jth joint point of the kth group of actions can be obtained from the skeleton sequence data. Construct the adjacency matrix of this set of actions Its reason Such an element composition represents the degree of connection between the jth joint point and the ith joint point of the kth group of actions. The weight matrix is ​​recorded as W. The first generalized weight matrix corresponding to the kth group of actions is

[0041] For this embodiment, the first generalized weight matrix is ​​multiplied by the input feature of the j-th joint point to obtain a second multiplication result The second multiplication results corresponding to all joint points are added together to obtain the output feature of the i-th joint point of the group of actions, so as to obtain the output feature of the joint point.

[0042] The output feature of the i-th joint point of the group of actions Thus, the output features of all relevant nodes can be obtained.

[0043] For this embodiment, the calculation of generalized weights is performed under the constraints of the first constraint and the second constraint, and these two constraints have the least restrictions on the calculation of generalized weights. The first constraint and the second constraint are used to implement a dynamic topological structure and a strategy for sharing weights in different channels, thereby improving the generalization ability of the model and achieving higher accuracy.

[0044] The first constraint condition corresponding to the second generalized weight matrix and the third generalized weight matrix under the same group of actions is constructed, and the second constraint condition corresponding to the first generalized weight matrix and the first generalized weight matrix under different groups of actions is constructed, including: constructing under the same group of actions, the second generalized weight matrix between the i-th joint point and the j1-th joint point is equal to the first preset coefficient matrix multiplied by the third generalized weight matrix between the i-th joint point and the j2-th joint point, j1 and j2 both belong to j, and j1 is not equal to j2; constructing under different groups of actions, the first generalized weight matrix between the i-th joint point and the j-th joint point is equal to the second preset coefficient matrix multiplied by the first generalized weight matrix between the i-th joint point and the j-th joint point.

[0045] Among them, under the same group of actions (taking the kth group of actions as an example), the second generalized weight matrix between the i-th joint point and the j1-th joint point is The third generalized weight matrix between the i-th joint point and the j2-th joint point is Then the first constraint is

[0046] Under different groups (taking the k1 group and k1 action as an example), the second constraint is

[0047] For this embodiment, the second dimensionality-upgraded feature is input into the aggregation module, and the aggregation module is used to perform multiplication calculation of the second dimensionality-upgraded feature and the adjacency matrix of the same group of actions to obtain a single aggregation feature corresponding to each group of actions, and the aggregation feature of all groups of actions can be obtained from the single aggregation feature.

[0048] 204. Use the aggregation and splicing module to multiply the first dimension-raising feature and the aggregation feature of the same channel to obtain a first multiplication result, and perform feature splicing on the first multiplication results of all the channels to obtain a first feature.

[0049] 205. Input the skeleton sequence data into the spatial transformer module to calculate the spatial mixed attention score, and calculate the spatial output feature according to the calculated spatial mixed attention score.

[0050] For this embodiment, the bone sequence data is input into the spatial transformer module to calculate the spatial mixed attention score, and the spatial output feature is calculated according to the calculated spatial mixed attention score, including: inputting the bone sequence data into the spatial transformer module, calculating a multi-channel first query matrix, a multi-channel first key matrix and a multi-channel first value matrix; determining a group of single first query matrices and single first key matrices for each channel according to the multi-channel first query matrix and the multi-channel first key matrix, calculating self-attention according to the single first query matrix and the single first key matrix of the same group, and obtaining a first self-attention score corresponding to each group; randomly arranging each group, calculating self-attention between the single first query matrix of one of two adjacent groups and the single first key matrix of the other group, and obtaining a second self-attention score corresponding to each adjacent two groups; for each group, calculating the first self-attention score corresponding to the group and the second self-attention score related to the group, and adding them to obtain a spatial mixed attention score, wherein the second self-attention score corresponding to each adjacent two groups is related to the two adjacent groups; calculating the spatial output feature according to the spatial mixed attention score and the multi-channel first value matrix.

[0051] Among them, the method of calculating the multi-channel first query matrix, the multi-channel first key matrix and the multi-channel first value matrix according to the skeleton sequence data is the same as the method of the prior art, which will not be repeated here. The multi-channel first query matrix refers to multiple channels, each channel corresponds to a single first query matrix. Similarly, the multi-channel first key matrix refers to multiple channels, each channel corresponds to a single first key matrix, and the multi-channel first value matrix refers to multiple channels, each channel corresponds to a single first value matrix. The single first query matrix and the single first key matrix of the same channel are taken as the same group. The method of calculating self-attention according to the single first query matrix and the single first key matrix of the same group is the same as the method of the prior art, which will not be repeated here, and the first self-attention score corresponding to each group is obtained. For this embodiment, the calculated spatial mixed attention score is not the self-attention in the prior art, but the groups are randomly arranged, and the self-attention is calculated between the single first query matrix of one of the two adjacent groups and the single first key matrix of the other group. For example, if the first group is adjacent to the second group, then the self-attention is calculated between the single first query matrix in the first group and the single first key matrix in the second group to obtain the second self-attention score corresponding to the first group and the second group (the self-attention can also be calculated between the single first key matrix in the first group and the single first query matrix in the second group, which is not limited here). Similarly, if the second group is adjacent to the third group, then the self-attention is calculated between the single first query matrix in the second group and the single first key matrix in the third group to obtain the second self-attention score corresponding to the second group and the third group. In this way, the corresponding second self-attention scores are calculated for each two adjacent groups.

[0052] Then, for each group, the first self-attention score corresponding to the group is calculated and added to the second self-attention score related to the group to obtain a spatial mixed attention score. For example, for the first group, the second self-attention score related to the first group is the second self-attention scores corresponding to the first group and the second group. Then the objects of addition calculation are the first self-attention score corresponding to the first group and the second self-attention scores corresponding to the first group and the second group, and the spatial mixed attention score corresponding to the first group is obtained; for the second group, the second self-attention score related to the second group is the second self-attention score corresponding to the first group and the second group and the second self-attention score corresponding to the second group and the third group. Then the objects of addition calculation are the first self-attention score corresponding to the second group, the second self-attention scores corresponding to the first group and the second group, and the second self-attention scores corresponding to the second group and the third group, and the spatial mixed attention score corresponding to the second group is obtained. The same is true for the other groups, which will not be repeated here.

[0053] Therefore, each group corresponds to a spatial mixed attention score. Since each group corresponds to a channel, each channel corresponds to a spatial mixed attention score. Then, the spatial output features are calculated based on the spatial mixed attention scores and the multi-channel first value matrix. That is, the spatial mixed attention scores of the same channel are calculated with a single first value matrix to obtain the spatial output features of the channel. The calculation method is the same as the prior art and will not be repeated here. Therefore, the spatial output features of all channels can be obtained.

[0054] 206. Input the skeleton sequence data into the temporal transformer module to calculate the temporal attention score, and calculate the temporal output feature according to the calculated temporal attention score.

[0055] For this embodiment, the skeleton sequence data is input into the time transformer module to calculate the time attention score, and the time output feature is calculated according to the calculated time attention score, including: inputting the skeleton sequence data into the time transformer module, calculating the multi-channel second query matrix, the multi-channel second key matrix and the multi-channel second value matrix; calculating the self-attention according to the multi-channel second query matrix and the multi-channel second key matrix to obtain the third self-attention score, and obtaining the time output feature according to the multi-channel second value matrix and the third self-attention score.

[0056] For this embodiment, for calculating self-attention according to the multi-channel second query matrix and the multi-channel second key matrix, obtaining the third self-attention score includes: calculating self-attention for a single second query matrix and a single second key matrix of the same channel to obtain the third self-attention score of the channel, thereby obtaining the third self-attention scores of all channels, and the calculation method is the same as that in the prior art, which will not be repeated here.

[0057] The temporal output feature is a temporal relationship matrix showing the correlation between different frames. This matrix is ​​used to adjust the final feature representation to reflect the temporal dynamics of the entire action sequence.

[0058] 207. Perform feature splicing processing on the spatial output feature and the temporal output feature according to the splicing module to obtain a second feature.

[0059] 208. Obtain the target power-producing human behavior according to the first feature and the second feature by the recognition module.

[0060] For this embodiment, the target power-producing human behavior is obtained according to the first feature and the second feature of the recognition module, including: for each group of actions, a first confidence score of each channel is obtained according to the first feature, and a second confidence score of each channel is obtained according to the second feature; a weighted sum of the first confidence score and the second confidence score of the same channel is performed to obtain a final confidence score of each channel; the final confidence score is input into the softmax layer to obtain the action probability corresponding to each channel, and the target channel corresponding to the maximum value of the action probability is determined, and the action corresponding to the target channel is used as the target power-producing human behavior corresponding to the group of actions.

[0061] Among them, the skeleton sequence data includes at least one group of actions, for example, the skeleton sequence data includes a group of actions of walking from a first point to a second point, and the corresponding target power production human behavior is walking, and also includes another group of actions of crossing from the second point to a third point, and the corresponding target power production human behavior is crossing.

[0062] For each group of actions, since the first feature includes the first confidence score for predicting that the skeleton sequence data belongs to each channel (one channel corresponds to a behavior category), and the second feature includes the second confidence score for predicting that the skeleton sequence data belongs to each channel (one channel corresponds to a behavior category), the first confidence score and the second confidence score of the same channel are weighted and summed to obtain the final confidence score of each channel (the formula is as follows), thereby combining the spatiotemporal graph convolutional network with the spatiotemporal transformer network to jointly identify the target power production human behavior of the skeleton sequence data. The common method is to input the final confidence score of each channel into the softmax layer of the recognition module to obtain the action probability corresponding to each channel, and select the channel with the largest action probability as the target channel. The action corresponding to the target channel has a behavior category, and this behavior category is the target power production human behavior corresponding to the group of actions.

[0063] S=α·M gcn (P)+β·M former (P)

[0064] Among them, S is the final confidence score, M gcn (P) is the first confidence score, M former (P) is the second confidence score, α and β are the optimal weight parameter matrices obtained after training the spatiotemporal graph convolutional network and the spatiotemporal transformer network.

[0065] The present invention provides a skeleton-based power production human behavior recognition method, device and equipment, which obtains a power production human behavior video to be recognized, generates skeleton sequence data according to the power production human behavior video, and the skeleton sequence data includes at least one group of actions; the skeleton sequence data is input into a trained power production human behavior recognition model for recognition to obtain the target power production human behavior, and the power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior. Through the technical solution of the present invention, compared with the static topological structure of the existing spatiotemporal graph convolutional network, the present application The spatial graph convolutional network adopts a dynamic topological structure, which makes the model have stronger generalization ability. Compared with the existing spatiotemporal graph convolutional network strategy of sharing weights in different channels, the spatiotemporal graph convolutional network of the present application adopts a strategy of not sharing weights in different channels, which makes the model have stronger accuracy. The present application adopts a spatiotemporal transformer network, which uses the self-attention mechanism of the time dimension and the self-attention mechanism of the spatial dimension to assign appropriate weights to the corresponding skeleton point features according to specific behaviors. That is, in some behaviors, only specific skeleton points will have an impact on the behavior judgment, so the weights of other key points can be ignored or reduced, so that even in scenes with occlusion, the behavior can still be accurately identified, which improves the occlusion robustness of the model. The first feature obtained by the spatiotemporal graph convolutional network of the present application is combined with the second feature obtained by the spatiotemporal transformer network of the present application, which can more accurately identify the target power production human behavior.

[0066] Further, as Figure 1 and Figure 2 The specific implementation of the method shown in the figure, the embodiment of the present invention provides a skeleton-based power production human behavior recognition device, such as Figure 3 As shown, the device includes: an acquisition module 31, a prediction module 32;

[0067] An acquisition module 31 is used to acquire a power generation human behavior video to be identified, and generate skeleton sequence data according to the power generation human behavior video, wherein the skeleton sequence data includes at least one group of actions;

[0068] The prediction module 32 is used to input the skeleton sequence data into a trained power production human behavior recognition model for recognition, so as to obtain target power production human behavior. The power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior.

[0069] Correspondingly, the spatiotemporal graph convolution network includes a feature dimension-increasing module and an aggregation and splicing module connected in sequence, and also includes a feature extraction module, a multi-layer perceptron, an aggregation module and the aggregation and splicing module connected in sequence. In order to input the bone sequence data into the spatiotemporal graph convolution network to obtain the first feature, the prediction module 32 may specifically include: a dimension-increasing unit 321, an aggregation unit 322, and a splicing unit 323;

[0070] The dimension-increasing unit 321 may be used to input the bone sequence data into the feature dimension-increasing module for feature dimension-increasing processing to obtain a first dimension-increasing feature;

[0071] The aggregation unit 322 may be used to input the skeleton sequence data into the feature extraction module for feature extraction to obtain output features of joint points, input the output features of the joint points into the multi-layer perceptron for feature dimension upscaling to obtain second dimension upscaling features, input the second dimension upscaling features into the aggregation module, and use the aggregation module to perform multiplication calculation of the second dimension upscaling features of the same group of actions with an adjacency matrix to obtain aggregated features;

[0072] The splicing unit 323 can be used to use the aggregation splicing module to multiply the first dimension-raising feature and the aggregation feature of the same channel to obtain a first multiplication result, and perform feature splicing on the first multiplication results of all the channels to obtain a first feature.

[0073] Correspondingly, the spatiotemporal transformer network includes a spatial transformer module and a splicing module connected in sequence, and also includes a temporal transformer module and the splicing module connected in sequence. In order to input the skeleton sequence data into the spatiotemporal transformer network to obtain the second feature, the prediction module 32 can also be used to input the skeleton sequence data into the spatial transformer module to calculate the spatial mixed attention score, and calculate the spatial output feature according to the calculated spatial mixed attention score; input the skeleton sequence data into the temporal transformer module to calculate the temporal attention score, and calculate the temporal output feature according to the calculated temporal attention score; perform feature splicing processing on the spatial output feature and the temporal output feature according to the splicing module to obtain the second feature.

[0074] Correspondingly, in order to input the skeleton sequence data into the feature extraction module for feature extraction and obtain the output features of the joint points, the aggregation unit 322 can also be specifically used to input the skeleton sequence data into the feature extraction module, and for each group of actions, determine the input features of the jth joint point of the group of actions in the skeleton sequence data; construct an adjacency matrix for the group of actions, wherein the adjacency matrix is ​​the degree of connection between the jth joint point and the i-th joint point in the group of actions, and i is not equal to j; multiply the adjacency matrix with the weight matrix to obtain a first generalized weight matrix between the i-th joint point and the j-th joint point, construct a first constraint condition corresponding to the second generalized weight matrix and the third generalized weight matrix under the same group of actions, and construct a second constraint condition corresponding to the first generalized weight matrix and the first generalized weight matrix under different groups of actions; multiply the first generalized weight matrix with the input features of the j-th joint point to obtain a second multiplication result, add the second multiplication results corresponding to all joint points, and obtain the output features of the i-th joint point of the group of actions to obtain the output features of the joint points.

[0075] Correspondingly, in order to construct the first constraint condition corresponding to the second generalized weight matrix and the third generalized weight matrix under the same group of actions, and to construct the second constraint condition corresponding to the first generalized weight matrix and the first generalized weight matrix under different groups of actions, the aggregation unit 322 can also be used to construct under the same group of actions, the second generalized weight matrix between the i-th joint point and the j1-th joint point is equal to the first preset coefficient matrix multiplied by the third generalized weight matrix between the i-th joint point and the j2-th joint point, j1 and j2 both belong to j, and j1 is not equal to j2; and to construct under different groups of actions, the first generalized weight matrix between the i-th joint point and the j-th joint point is equal to the second preset coefficient matrix multiplied by the first generalized weight matrix between the i-th joint point and the j-th joint point.

[0076] Correspondingly, in order to input the skeleton sequence data into the spatial transformer module for calculating the spatial mixed attention score, the spatial output feature is calculated according to the calculated spatial mixed attention score, and the prediction module 32 can also be used to input the skeleton sequence data into the spatial transformer module, calculate the multi-channel first query matrix, the multi-channel first key matrix and the multi-channel first value matrix; determine a group of single first query matrices and single first key matrices for each channel according to the multi-channel first query matrix and the multi-channel first key matrix, calculate self-attention according to the single first query matrix and the single first key matrix of the same group, and obtain the first self-attention score corresponding to each group; randomly arrange the groups, calculate the self-attention of the single first query matrix of one of the two adjacent groups with the single first key matrix of the other group, and obtain the second self-attention score corresponding to each two adjacent groups; for each A group, calculating a first self-attention score corresponding to the group and a second self-attention score related to the group for addition calculation to obtain a spatial mixed attention score, wherein the second self-attention score corresponding to each two adjacent groups is related to the two adjacent groups; calculating a spatial output feature according to the spatial mixed attention score and the multi-channel first value matrix; inputting the skeleton sequence data into the time transformer module to calculate the time attention score, and calculating the time output feature according to the calculated time attention score, including: inputting the skeleton sequence data into the time transformer module, calculating a multi-channel second query matrix, a multi-channel second key matrix and a multi-channel second value matrix; calculating self-attention according to the multi-channel second query matrix and the multi-channel second key matrix to obtain a third self-attention score, and obtaining the time output feature according to the multi-channel second value matrix and the third self-attention score.

[0077] It should be noted that for other corresponding descriptions of the functional units involved in the skeleton-based power production human behavior recognition device provided in this embodiment, reference can be made to Figure 1 to Figure 2 The corresponding description will not be repeated here.

[0078] Based on the above Figure 1 to Figure 2 The method shown in the embodiment, accordingly, also provides a storage medium, which can be volatile or non-volatile, and stores a computer program, which is executed by a processor to implement the above-mentioned Figure 1 to Figure 2 The skeleton-based power production human action recognition method shown.

[0079] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present invention.

[0080] Based on the above Figure 1 to Figure 2 The method shown and Figure 3 , Figure 4 In order to achieve the above-mentioned purpose, the present embodiment further provides a computer device, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 to Figure 2 The skeleton-based power production human action recognition method shown.

[0081] Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0082] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0083] The storage medium may also include an operating network communication module. The operating is a program that manages the hardware and software resources of the above-mentioned computer device, supporting the operation of the information processing program and other software and / or programs. The network communication module is used to communicate between the components inside the storage medium, and to communicate with other hardware and software in the information processing entity device.

[0084] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform, or by hardware.

[0085] The present invention provides a skeleton-based power production human behavior recognition method, device and equipment, which obtains a power production human behavior video to be recognized, generates skeleton sequence data according to the power production human behavior video, and the skeleton sequence data includes at least one group of actions; the skeleton sequence data is input into a trained power production human behavior recognition model for recognition to obtain the target power production human behavior, and the power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior. Through the technical solution of the present invention, compared with the static topological structure of the existing spatiotemporal graph convolutional network, the present application The spatial graph convolutional network adopts a dynamic topological structure, which makes the model have stronger generalization ability. Compared with the existing spatiotemporal graph convolutional network strategy of sharing weights in different channels, the spatiotemporal graph convolutional network of the present application adopts a strategy of not sharing weights in different channels, which makes the model have stronger accuracy. The present application adopts a spatiotemporal transformer network, which uses the self-attention mechanism of the time dimension and the self-attention mechanism of the spatial dimension to assign appropriate weights to the corresponding skeleton point features according to specific behaviors. That is, in some behaviors, only specific skeleton points will have an impact on the behavior judgment, so the weights of other key points can be ignored or reduced, so that even in scenes with occlusion, the behavior can still be accurately identified, which improves the occlusion robustness of the model. The first feature obtained by the spatiotemporal graph convolutional network of the present application is combined with the second feature obtained by the spatiotemporal transformer network of the present application, which can more accurately identify the target power production human behavior.

[0086] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present invention. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0087] The above serial numbers of the present invention are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of the present invention, but the present invention is not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A skeleton-based method for identifying human behavior in power production, characterized in that: The method comprises: Acquire a power generation human behavior video to be identified, and generate skeleton sequence data according to the power generation human behavior video, wherein the skeleton sequence data includes at least one set of actions; Input the skeleton sequence data into a trained power production human behavior recognition model for recognition, and obtain the target power production human behavior, wherein the power production human behavior recognition model includes a spatiotemporal graph convolutional network, a spatiotemporal transformer network, and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior; The spatiotemporal graph convolutional network includes a feature dimension-increasing module and an aggregation splicing module connected in sequence, and also includes a feature extraction module, a multi-layer perceptron, an aggregation module and the aggregation splicing module connected in sequence. The bone sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, including: Inputting the skeleton sequence data into the feature dimension upgrading module to perform feature dimension upgrading processing to obtain a first dimension upgrading feature; Inputting the skeleton sequence data into the feature extraction module for feature extraction to obtain output features of joint points, inputting the output features of the joint points into the multi-layer perceptron for feature dimension upscaling to obtain second dimension upscaling features, inputting the second dimension upscaling features into the aggregation module, and using the aggregation module to perform multiplication calculation of the second dimension upscaling features of the same group of actions with an adjacency matrix to obtain aggregated features; The aggregation and splicing module is used to multiply the first dimension-raising feature and the aggregation feature of the same channel to obtain a first multiplication result, and the first multiplication results of all the channels are spliced ​​to obtain a first feature; The step of inputting the skeleton sequence data into the feature extraction module for feature extraction to obtain output features of joint points includes: Input the skeleton sequence data into the feature extraction module, and for each group of actions, determine the input feature of the j-th joint point of the group of actions in the skeleton sequence data; Constructing an adjacency matrix of the group of actions, wherein the adjacency matrix is ​​the degree of connection between the j-th joint point and the i-th joint point in the group of actions, where i is not equal to j; The adjacency matrix is ​​multiplied by the weight matrix to obtain a first generalized weight matrix between the ith joint point and the jth joint point, a first constraint condition corresponding to the second generalized weight matrix and the third generalized weight matrix under the same group of actions is constructed, and a second constraint condition corresponding to the first generalized weight matrix and the first generalized weight matrix under different groups of actions is constructed; wherein: under the same group of actions, the second generalized weight matrix between the ith joint point and the j1th joint point is equal to the first preset coefficient matrix multiplied by the third generalized weight matrix between the ith joint point and the j2th joint point, j1 and j2 both belong to j, and j1 is not equal to j2; under different groups of actions, the first generalized weight matrix between the ith joint point and the jth joint point is equal to the second preset coefficient matrix multiplied by the first generalized weight matrix between the ith joint point and the jth joint point; The first generalized weight matrix is ​​multiplied by the input feature of the j-th joint point to obtain a second multiplication result, and the second multiplication results corresponding to all joint points are added to obtain the output feature of the i-th joint point of the group of actions to obtain the output feature of the joint point.

2. The method according to claim 1, characterized in that The spatiotemporal transformer network includes a spatial transformer module and a splicing module connected in sequence, and also includes a temporal transformer module and the splicing module connected in sequence, and the bone sequence data is input into the spatiotemporal transformer network to obtain a second feature, including: Inputting the skeleton sequence data into the spatial transformer module to calculate the spatial mixed attention score, and calculating the spatial output feature according to the calculated spatial mixed attention score; Inputting the skeleton sequence data into the temporal transformer module to calculate the temporal attention score, and calculating the temporal output feature according to the calculated temporal attention score; The spatial output feature and the temporal output feature are subjected to feature splicing processing according to the splicing module to obtain a second feature.

3. The method according to claim 2, characterized in that The step of inputting the skeleton sequence data into the spatial transformer module to calculate the spatial mixed attention score, and calculating the spatial output feature according to the calculated spatial mixed attention score, comprises: Input the skeleton sequence data into the spatial transformer module, and calculate a multi-channel first query matrix, a multi-channel first key matrix, and a multi-channel first value matrix; Determine a group of single first query matrices and single first key matrices for each channel according to the multi-channel first query matrix and the multi-channel first key matrix, calculate self-attention according to the single first query matrix and the single first key matrix of the same group, and obtain a first self-attention score corresponding to each group; Randomly arrange the groups, calculate the self-attention of the single first query matrix of one of the two adjacent groups and the single first key matrix of the other group, and obtain the second self-attention score corresponding to each of the two adjacent groups; For each group, the first self-attention score corresponding to the group is calculated and added to the second self-attention score related to the group to obtain a spatial mixed attention score, where the second self-attention score corresponding to each two adjacent groups is related to the two adjacent groups; Calculate spatial output features according to the spatial mixed attention score and the multi-channel first value matrix; The step of inputting the skeleton sequence data into the temporal transformer module to calculate the temporal attention score, and calculating the temporal output feature according to the calculated temporal attention score, comprises: Input the skeleton sequence data into the time transformer module, calculate the multi-channel second query matrix, the multi-channel second key matrix and the multi-channel second value matrix; Self-attention is calculated according to the multi-channel second query matrix and the multi-channel second key matrix to obtain a third self-attention score, and a temporal output feature is obtained according to the multi-channel second value matrix and the third self-attention score.

4. The method according to claim 1, characterized in that: The step of inputting the first feature and the second feature into the recognition module to obtain the target power-producing human behavior includes: For each group of actions, obtaining a first confidence score for each channel according to the first feature, and obtaining a second confidence score for each channel according to the second feature; Performing a weighted summation on the first confidence score and the second confidence score of the same channel to obtain a final confidence score of each channel; The final confidence score is input into the softmax layer to obtain the action probability corresponding to each channel, the target channel corresponding to the maximum value of the action probability is determined, and the action corresponding to the target channel is used as the target power production human behavior corresponding to the group of actions.

5. A skeleton-based power production human behavior recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a power generation human behavior video to be identified, and generating skeleton sequence data according to the power generation human behavior video, wherein the skeleton sequence data includes at least one group of actions; A prediction module, used for inputting the skeleton sequence data into a trained power production human behavior recognition model for recognition, to obtain a target power production human behavior, wherein the power production human behavior recognition model comprises a spatiotemporal graph convolutional network, a spatiotemporal transformer network and a recognition module, wherein the skeleton sequence data is input into the spatiotemporal graph convolutional network to obtain a first feature, the skeleton sequence data is input into the spatiotemporal transformer network to obtain a second feature, and the first feature and the second feature are input into the recognition module to obtain the target power production human behavior; The spatiotemporal graph convolutional network includes a feature dimension-increasing module and an aggregation splicing module connected in sequence, and also includes a feature extraction module, a multi-layer perceptron, an aggregation module and the aggregation splicing module connected in sequence; The prediction module is further used for: inputting the skeleton sequence data into the feature dimension-upgrading module for feature dimension-upgrading processing to obtain a first dimension-upgrading feature; inputting the skeleton sequence data into the feature extraction module for feature extraction to obtain output features of joint points, inputting the output features of the joint points into the multi-layer perceptron for feature dimension-upgrading processing to obtain a second dimension-upgrading feature, inputting the second dimension-upgrading feature into the aggregation module, using the aggregation module to perform multiplication calculation of the second dimension-upgrading feature of the same group of actions with the adjacency matrix to obtain an aggregated feature; using the aggregation splicing module to perform multiplication calculation of the first dimension-upgrading feature of the same channel with the aggregated feature to obtain a first multiplication result, and performing feature splicing on the first multiplication results of all the channels to obtain a first feature; The prediction module is further used to: input the skeleton sequence data into the feature extraction module, and determine, for each group of actions, the input feature of the jth joint point of the group of actions in the skeleton sequence data; Constructing an adjacency matrix of the group of actions, wherein the adjacency matrix is ​​the degree of connection between the j-th joint point and the i-th joint point in the group of actions, where i is not equal to j; The adjacency matrix is ​​multiplied by the weight matrix to obtain a first generalized weight matrix between the ith joint point and the jth joint point, a first constraint condition corresponding to the second generalized weight matrix and the third generalized weight matrix under the same group of actions is constructed, and a second constraint condition corresponding to the first generalized weight matrix and the first generalized weight matrix under different groups of actions is constructed; wherein: under the same group of actions, the second generalized weight matrix between the ith joint point and the j1th joint point is equal to the first preset coefficient matrix multiplied by the third generalized weight matrix between the ith joint point and the j2th joint point, j1 and j2 both belong to j, and j1 is not equal to j2; under different groups of actions, the first generalized weight matrix between the ith joint point and the jth joint point is equal to the second preset coefficient matrix multiplied by the first generalized weight matrix between the ith joint point and the jth joint point; The first generalized weight matrix is ​​multiplied by the input feature of the j-th joint point to obtain a second multiplication result, and the second multiplication results corresponding to all joint points are added to obtain the output feature of the i-th joint point of the group of actions to obtain the output feature of the joint point.

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the skeleton-based power production human behavior recognition method described in any one of claims 1 to 4 is implemented.

7. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the skeleton-based power production human behavior recognition method described in any one of claims 1 to 4 is implemented.

Images