Behavior classification method and apparatus, computer device, storage medium, and program product

Abstract

The present application relates to a behavior classification method and apparatus, a computer device, a storage medium, and a program product. The behavior classification method comprises: acquiring a power monitoring image, wherein the power monitoring image comprises a target person; according to the power monitoring image, acquiring key point feature data corresponding to the target person; inputting the key point feature data and at least one piece of behavior query data into a behavior classification network to obtain a behavior classification result corresponding to the target person and outputted by the behavior classification network, wherein the behavior classification network is obtained by training on the basis of a multi-head cross attention mechanism, the at least one piece of behavior query data is determined according to the behavior classification requirements, and the behavior query data correspond to different behavior categories.

Description

Behavior classification method, device, computer equipment, storage medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202310708684.X, filed on June 14, 2023, entitled “Behavior classification method, device, computer equipment, storage medium and program product,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of electric power technology, and in particular to a behavior classification method, apparatus, computer equipment, storage medium, and program product. Background Art

[0004] In the power industry, in order to ensure the safety of operators and power equipment, it is necessary to monitor the behavior of operators in real time.

[0005] The current method for real-time monitoring of the behavior of power workers is to obtain a target image including the worker's body through monitoring equipment such as surveillance cameras, first detect the key points of the human body in the target image, and then classify and identify the detected key point data to obtain a classification result, where the classification result is used to characterize the behavior category of the worker in the target image, such as making a phone call, falling to the ground, climbing, etc.

[0006] However, the above behavior classification methods have low accuracy.

[0007] Summary of the Invention

[0008] Based on this, it is necessary to provide a behavior classification method, device, computer equipment, storage medium and program product that can improve classification accuracy in response to the above technical problems.

[0009] In a first aspect, the present application provides a behavior classification method. The method comprises:

[0010] Acquire a power monitoring image, where the power monitoring image includes a target person portrait;

[0011] According to the power monitoring image, key point feature data corresponding to the target portrait is obtained;

[0012] The key point feature data and at least one behavior query data are input into the behavior classification network to obtain the behavior classification result corresponding to the target portrait output by the behavior classification network. The behavior classification network is trained based on a multi-head cross-attention mechanism. The at least one behavior query data is determined according to the behavior classification requirements, and each behavior query data corresponds to a different behavior category.

[0013] In one embodiment, the behavior classification network includes a multi-head self-attention network and a multi-head cross-attention prediction network connected to each other; key point feature data and at least one behavior query data are input into the behavior classification network, and the behavior classification result corresponding to the target person portrait output by the behavior classification network is obtained, including:

[0014] Each behavior query data is input into the multi-head self-attention network to obtain the self-attention query data corresponding to each behavior query data. The spatial embedding tensor of the multi-head self-attention network is obtained by position encoding according to the dimension of the key point feature data;

[0015] The behavior classification results are obtained based on the respective attention query data, key point feature data and multi-head cross-attention prediction network.

[0016] In one embodiment, the multi-head cross-attention prediction network includes a multi-head cross-attention network and a prediction network sequentially connected to a multi-head self-attention network; obtaining behavior classification results based on respective attention query data, key point feature data, and the multi-head cross-attention prediction network includes:

[0017] Input the respective attention query data and key point feature data into the multi-head cross attention network, and obtain the cross query data corresponding to the respective attention query data output by the multi-head cross attention network;

[0018] Each cross-query data is input into the prediction network to obtain the behavior classification result output by the prediction network.

[0019] In one embodiment, the prediction network includes a feedforward subnetwork and a pooling subnetwork connected in sequence; each cross-query data is input into the prediction network to obtain a behavior classification result output by the prediction network, including:

[0020] Input each cross-query data into the feed-forward sub-network to obtain the cross-enhanced data corresponding to each cross-query data output by the feed-forward sub-network;

[0021] Based on each cross-enhancement data and pooling sub-network, the behavior classification results are obtained.

[0022] In one embodiment, the prediction network further includes an affine transformation subnetwork connected between the feedforward subnetwork and the pooling subnetwork; obtaining a behavior classification result based on each cross-enhancement data and the pooling subnetwork includes:

[0023] Inputting each cross-enhancement data into the affine transformation sub-network to obtain at least one cross-transformation data corresponding to each cross-enhancement data output by the affine transformation sub-network;

[0024] Each cross-transformed data is input into the pooling sub-network to obtain the behavior classification result.

[0025] In one embodiment, obtaining key point feature data corresponding to a target person's portrait based on a power monitoring image includes:

[0026] The power monitoring image is input into the portrait key point feature extraction network to obtain the key point feature data output by the portrait key point feature extraction network. The portrait key point feature extraction network is trained based on a multi-resolution parallel mechanism.

[0027] In a second aspect, the present application further provides a behavior classification device. The device comprises:

[0028] An image acquisition module is used to acquire a power monitoring image, wherein the power monitoring image includes a target person's portrait;

[0029] A feature acquisition module is used to obtain key point feature data corresponding to the target portrait based on the power monitoring image;

[0030] The behavior classification module is used to input key point feature data and at least one behavior query data into the behavior classification network to obtain the behavior classification result corresponding to the target portrait output by the behavior classification network. The behavior classification network is trained based on the multi-head cross-attention mechanism. The at least one behavior query data is determined according to the behavior classification requirements, and each behavior query data corresponds to a different behavior category.

[0031] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0033] In a fifth aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0034] The above-mentioned behavior classification method, device, computer equipment, storage medium and program product obtain an electric power monitoring image, which includes a target portrait; obtain key point feature data corresponding to the target portrait based on the electric power monitoring image; input the key point feature data and at least one behavior query data into the behavior classification network, and obtain the behavior classification result corresponding to the target portrait output by the behavior classification network, the behavior classification network is trained based on a multi-head cross-attention mechanism, at least one behavior query data is determined according to the behavior classification requirements, and each behavior query data corresponds to a different behavior category; use the behavior classification network trained based on the multi-head cross-attention mechanism to perform multi-head cross-attention learning on the behavior query data determined according to the behavior classification requirements and the key point feature data corresponding to the target portrait, and use ... The force mechanism focuses on the important information part and ignores the characteristics of unimportant information when processing data, so that each behavior query data learns the spatial feature relationship between each key point in the key point feature data. At the same time, the behavior classification network can focus on the correlation between different key points in the target portrait based on the multi-head cross-attention mechanism to improve the classification accuracy; it avoids the behavior classification method of using support vector machines in traditional technology, which can only perform data analysis on the positional relationship of each key point in the key point feature data, and cannot cross-learn the spatial feature relationship corresponding to the key point feature data and the behavior category determined according to the behavior classification requirements, resulting in low classification accuracy; the embodiment of the present application uses the behavior classification network to perform multi-head cross-attention learning on the key point feature data and each behavior query data, so the behavior classification result obtained is highly accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0036] FIG1 is a diagram illustrating an application environment of a behavior classification method according to an embodiment;

[0037] FIG2 is a schematic diagram of a flow chart of a behavior classification method according to an embodiment;

[0038] FIG3 is a schematic diagram of the structure of a behavior classification network in one embodiment;

[0039] FIG4 is a schematic diagram of a process for obtaining behavior classification results in one embodiment;

[0040] FIG5 is a schematic diagram of the structure of a behavior classification network in another embodiment;

[0041] FIG6 is a schematic diagram of a process for obtaining behavior classification results in another embodiment;

[0042] FIG7 is a schematic diagram of a flow chart of a behavior classification method according to another embodiment;

[0043] FIG8 is a structural block diagram of a behavior classification device according to an embodiment;

[0044] FIG9 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] The behavior classification method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0047] Terminal 102 obtains an electric power monitoring image, which includes a target portrait; based on the electric power monitoring image, key point feature data corresponding to the target portrait is obtained; the key point feature data and at least one behavior query data are input into a behavior classification network to obtain a behavior classification result corresponding to the target portrait output by the behavior classification network. The behavior classification network is trained based on a multi-head cross-attention mechanism, and at least one behavior query data is determined based on the behavior classification requirements, and each behavior query data corresponds to a different behavior category.

[0048] In one embodiment, as shown in FIG2 , a behavior classification method is provided. The method is described by taking the application of the method to the terminal 102 in FIG1 as an example, including the following steps:

[0049] Step 202: Acquire a power monitoring image, where the power monitoring image includes a target person portrait.

[0050] Power monitoring images refer to images containing human figures captured by monitoring systems at power operation sites. By identifying the behavior type of target human figures in power monitoring images, it is determined whether workers at the power operation site are engaging in abnormal behavior. Behavior type identification is also known as human pose estimation. The target human figures in power monitoring images can be one or multiple.

[0051] For example, drones can be used for power inspections, and power monitoring images can be obtained using high-definition cameras, infrared thermal imaging and other technologies carried by the drones; power monitoring images of the power operation site can also be obtained in real time through visual inspection tools, such as wire-type indoor inspection robots and insulated high-definition inspection devices; power monitoring images can also be obtained in real time using monitoring equipment at the power operation site.

[0052] Step 204 : Acquire key point feature data corresponding to the target person's portrait based on the power monitoring image.

[0053] Among them, the key point feature data refers to the position information of each key point in the target portrait in the power monitoring image and the positional relationship between each key point. Exemplarily, the key point feature data is a 17-dimensional key point heat map, where one dimension corresponds to one key point, and the key points include left ear, right ear, left eye, right eye, nose, right shoulder, head, left shoulder, right hand head, right elbow, left hand head, left elbow, right waist, left waist, right knee, left knee, right foot head, and left foot head. In the field of human posture estimation technology, key point feature data can be referred to as a human skeleton feature map.

[0054] If the power monitoring image includes multiple target portraits, key point feature data corresponding to each target portrait is obtained separately. For example, if the power monitoring image includes target portrait 1 and target portrait 2, key point feature data corresponding to target portrait 1 and key point feature data corresponding to target portrait 2 are obtained separately.

[0055] Obtaining key point feature data corresponding to the target portrait refers to using a posture estimation algorithm to extract key point features from the power monitoring image to obtain corresponding key point feature data. In this embodiment, there is no specific limitation on the posture estimation algorithm used. For example, the OPENPOSE network can be used to perform key point detection on the power monitoring image to obtain key point feature data output by the OPENPOSE network. For another example, the CPM (Convolutional Pose Machines) network or LCR-Net (Localization-Classification-Regression Net) can be used to perform key point detection on the power monitoring image to obtain key point feature data.

[0056] Step 206: Input the key point feature data and at least one behavior query data into the behavior classification network to obtain a behavior classification result corresponding to the target person portrait output by the behavior classification network.

[0057] The behavior classification network is trained using a multi-head cross-attention mechanism. At least one behavior query data set is determined based on the behavior classification requirements, and each behavior query data set corresponds to a different behavior category. A behavior query data set can correspond to one behavior category or multiple different behavior categories.

[0058] The behavior classification method used in this embodiment can be regarded as a two-stage target detection network architecture: the first stage is the feature extraction stage, and the second stage is the target detection stage based on key point feature data.

[0059] The behavior query data can be viewed as a feature matrix describing the behavior category. For example, the operation process of the multi-head cross attention mechanism can be expressed as the following formula:

[0060] Among them, N h is the number of cross-attention heads in the multi-head cross-attention mechanism, Q i is the input query vector of the i-th cross attention head, which is transformed from the corresponding behavior query data through the query transformation matrix, K i is the key vector of the i-th cross attention head input, which is transformed from the key point feature data through the key transformation matrix, V i is the value vector of the i-th cross attention head input, which is transformed from the key point feature data through the value conversion matrix, C h is the dimension of each cross-attention head to achieve scaled dot-product attention, W O is the learnable transformation matrix.

[0061] After the behavior classification network cross-learns the key point feature data and at least one behavior query data, the cross-learned data is input into the normalization layer and the residual connection layer for processing, and then feedforward prediction is performed to obtain the predicted probability of the target portrait corresponding to each behavior query data. The behavior category corresponding to the behavior query data with the largest predicted probability is determined as the behavior classification result corresponding to the target portrait, that is, the behavior category corresponding to the target portrait is obtained.

[0062] Because human posture has a strong spatial correlation, the features at different positions in the power monitoring image have different importance to the behavior classification results. When classifying the behavior of the target portrait, it is necessary to suppress non-important background information; the spatial relationship between the key points in the human posture has different importance in different behavior categories; in this embodiment, the behavior classification network obtained by training based on the multi-head cross-attention mechanism allows the key point feature data and the behavior query data to be cross-learned, so that the behavior query data learns the spatial feature relationship between the key points in the key point feature data, thereby making good use of useful features and achieving more accurate behavior classification and recognition.

[0063] In the above-mentioned behavior classification method, an electric power monitoring image is obtained, which includes a target portrait; based on the electric power monitoring image, key point feature data corresponding to the target portrait is obtained; the key point feature data and at least one behavior query data are input into the behavior classification network, and a behavior classification result corresponding to the target portrait output by the behavior classification network is obtained, the behavior classification network is trained based on a multi-head cross-attention mechanism, at least one behavior query data is determined according to the behavior classification requirements, and each behavior query data corresponds to a different behavior category; using the behavior classification network trained based on the multi-head cross-attention mechanism, multi-head cross-attention learning is performed on the behavior query data determined according to the behavior classification requirements and the key point feature data corresponding to the target portrait, and the multi-head cross-attention mechanism is used to process the data. Focusing on important information and ignoring unimportant information allows each behavior query data to learn the spatial feature relationship between each key point in the key point feature data. At the same time, the behavior classification network can focus on the correlation between different key points in the target portrait based on the multi-head cross-attention mechanism to improve the classification accuracy; it avoids the behavior classification method of using support vector machines in traditional technology, which can only perform data analysis on the positional relationship of each key point in the key point feature data, and cannot cross-learn the spatial feature relationship corresponding to the key point feature data and the behavior category determined according to the behavior classification requirements, resulting in low classification accuracy. The embodiment of the present application uses a behavior classification network to perform multi-head cross-attention learning on the key point feature data and each behavior query data, so the behavior classification results obtained are highly accurate.

[0064] In one embodiment, based on the embodiment shown in Figure 2, the behavior classification network provided in this embodiment includes an interconnected multi-head self-attention network and a multi-head cross-attention prediction network, as shown in Figure 3. Referring to Figure 4, this embodiment describes how to input key point feature data and at least one behavior query data into the behavior classification network and output a behavior classification result corresponding to the target person portrait. As shown in Figure 4, this process includes steps 402 and 404.

[0065] In step 402, each behavior query data is input into a multi-head self-attention network to obtain the self-attention query data corresponding to each behavior query data. The spatial embedding tensor E of the multi-head self-attention network is obtained by position encoding according to the dimensions of the key point feature data.

[0066] Among them, in order to add the spatial position information of key point feature data to the multi-head self-attention network, the spatial embedding tensor E of the multi-head self-attention network is obtained by position encoding according to the dimension of the key feature data.

[0067] For example, in this embodiment, the spatial embedding tensor E can be calculated using the sine-cosine position encoding method. The position encoding (PE) process can be expressed as follows:

[0068] Where d is the total dimension of the spatial embedding tensor E (the same as the dimension of the key feature data), i is the position index, and j is the index of the encoding dimension.

[0069] Exemplarily, the multi-head self-attention network includes a multi-head self-attention sub-network and a normalization and residual connection sub-network, wherein the operation process of the multi-head self-attention sub-network can be expressed as the following formula:

[0070] Among them, N h is the number of self-attention heads in the multi-head self-attention network, Q i is the input query vector of the i-th self-attention head, which is transformed from the corresponding behavioral query data and the spatial embedding tensor E through the query transformation matrix, K i is the key vector of the i-th self-attention head input, which is transformed from the behavior query data and the spatial embedding tensor E through the key transformation matrix, V i is the value vector of the i-th self-attention head input, which is transformed from the behavior query data and the spatial embedding tensor E through the value conversion matrix, C h is the dimension of each self-attention head to achieve scaled dot-product attention, W O is the learnable transformation matrix.

[0071] The output of the multi-head self-attention sub-network and the input data of the multi-head sub-attention sub-network are jointly input into the normalization and residual connection sub-network to obtain the self-attention query data.

[0072] Step 404: Obtain behavior classification results based on the respective attention query data, key point feature data, and the multi-head cross-attention prediction network.

[0073] In this embodiment, position encoding is performed according to the dimension of the key point feature data to obtain the spatial embedding tensor E of the multi-head self-attention network, so that the position information of the key point feature data is added to the multi-head self-attention network, and then each behavior query data is subjected to multi-head self-attention learning through the multi-head self-attention network, so that the behavior classification network is linked to the position information of the key point feature data. In this way, the behavior classification network does not need to be designed according to the corresponding structural design based on the method of obtaining the key point feature data, that is, the behavior classification network provided by this embodiment is universal.

[0074] In one embodiment, based on the embodiment shown in FIG4 , the multi-head cross-attention prediction network in the behavior classification network provided in this embodiment includes a multi-head cross-attention network and a prediction network sequentially connected to a multi-head self-attention network, as shown in FIG5 . Referring to FIG6 , this embodiment relates to the process of obtaining behavior classification results based on respective attention query data, key point feature data, and the multi-head cross-attention prediction network. As shown in FIG6 , this process includes steps 602 and 604 .

[0075] In step 602, the respective attention query data and key point feature data are input into the multi-head cross attention network to obtain cross query data corresponding to the respective attention query data output by the multi-head cross attention network.

[0076] For the multi-head cross-attention network, the input Q vector comes from the self-attention query data, and the K vector and V vector come from the key point feature data.

[0077] Through the multi-head cross self-attention network, the similarity between each attention query data learning and key point feature data is obtained to obtain the corresponding cross query data.

[0078] Step 604: Input each cross-query data into the prediction network to obtain the behavior classification result output by the prediction network.

[0079] In one possible embodiment, the prediction network includes a feedforward subnetwork and a pooling subnetwork connected in sequence. In this embodiment, step 604 inputs each cross-query data into the prediction network to obtain the behavior classification result output by the prediction network, including:

[0080] In step A1, each cross-query data is input into the feed-forward sub-network to obtain cross-enhanced data corresponding to each cross-query data output by the feed-forward sub-network.

[0081] Step A2: Obtain the behavior classification result based on each cross-enhancement data and the pooling sub-network.

[0082] In a possible implementation, in step A2, each cross-enhancement data is directly input into the pooling sub-network to obtain a behavior classification result output by the pooling sub-network.

[0083] In another possible implementation, the prediction network further includes an affine transformation subnetwork connected between the feedforward subnetwork and the pooling subnetwork. In this possible implementation, step A2 obtains the behavior classification result based on each cross-enhancement data and the pooling subnetwork, including:

[0084] Step B1: input each cross-enhancement data into the affine transformation sub-network to obtain at least one cross-transformation data corresponding to each cross-enhancement data output by the affine transformation sub-network.

[0085] In step B2, each cross-transformed data is input into the pooling sub-network to obtain the behavior classification result.

[0086] Generally speaking, one behavior query data corresponds to one behavior category. In some extreme behavior classification scenarios, the number of behavior categories will be large, which means that if you want to parse more behavior categories, you need to increase the number of corresponding behavior query data, but this will bring heavy parameters and calculations. As the number of behavior categories increases, the behavior classification network will become very computationally intensive and may reduce the classification performance. Among them, the multi-head self-attention network and the multi-head cross-attention network will show a quadratic complexity O(n) as the sequence length of the input data increases. 2 ): O(n 2 )=4nh 2 d 2 +2n 2 hd;

[0087] Where n is the sequence length of the input data; d is the dimension of each head; and h is the number of heads.

[0088] In order to solve the problem that the computational complexity of the behavior classification network presents a quadratic relationship with the increase of behavior classification, the present embodiment adopts the method of group decoding to reduce the number of parameters and computational complexity. By adding an affine transformation layer between the feedforward subnetwork and the pooling subnetwork, a behavior query data corresponds to multiple behavior categories through affine transformation and pooling operations. In this way, there is a linear complexity between the behavior classification network and the behavior query data. In some application scenarios that need to be actually implemented, this embodiment has a great advantage (such as edge terminals). In terms of computational complexity, the computational complexity of the affine transformation subnetwork is equal to that of the fully connected layer, and the computational complexity of the pooling subnetwork is N×D multiplications (N is the number of behavior query data, and D is the dimension of the key point feature data). It can be seen that the computational complexity of the affine transformation subnetwork and the pooling subnetwork are both linearly related to the number N of input behavior query data.

[0089] For example, we first define a grouping factor g = K / N, where N is the number of behavior query data and K is the number of behavior categories to be queried. The actual behavior category L to be classified is the output of the affine transformation subnetwork. i : L i =(W k Q k ) j , n=i div g, j=i mod g;

[0090] Where: Query data corresponding to the nth behavior; is the nth learnable transformation matrix.

[0091] When g=1, the behavior classification network in this embodiment can also be called a full-decoding behavior classification network, and one behavior query data corresponds to one behavior category. When g≠1, the behavior classification network in this embodiment can also be called a group decoding behavior classification network, which means that each behavior query data corresponds to several behavior categories, and the behavior classification network in this embodiment randomly divides the behavior categories into several groups.

[0092] In this implementation, an efficient grouping factor g can be set according to different task requirements. Usually, g is greater than 1. The relationship between behavior query data and groups is established through the affine transformation subnetwork, and then pooling along the embedding dimension of the token is used to obtain the final classification result.

[0093] The behavior classification network provided in this embodiment achieves lightweightness while maintaining accuracy and stability. At the same time, the grouping method can be easily extended to any category and has strong universality.

[0094] In one embodiment, based on the embodiment shown in Figure 2, this embodiment relates to a process for obtaining key point feature data corresponding to a target person's portrait based on a power monitoring image. This process includes inputting the power monitoring image into a portrait key point feature extraction network and obtaining key point feature data output by the portrait key point feature extraction network. The portrait key point feature extraction network is trained using a multi-resolution parallel mechanism.

[0095] Exemplarily, the portrait keypoint feature extraction network includes a stem, a backbone, and a regression head. The backbone uses a multi-resolution branch parallel computation method and cross-resolution connections between different branches for feature exchange, while maintaining strong semantic and positional information, making the predicted heatmap more spatially accurate.

[0096] The base is used to reduce the input power monitoring image to 1 / 4 the size of the original image to obtain a reduced image, thereby reducing the subsequent computational effort. The reduced image is then placed in a main body divided into a stage layer and a transition layer to extract features. The stage layer in the main body mainly plays the role of image feature extraction and multi-feature fusion. The feature extraction module uses a residual connection structure, repeated 4 times, to more fully extract features. Multi-feature fusion is achieved by nearest neighbor upsampling and convolution downsampling. The transition layer in the main body adds a downsampling branch based on the original branch, performs convolution downsampling operations on the existing feature map, and further obtains a feature map with richer semantic information. Finally, the regression head fuses the various features obtained from the last stage layer to predict the key point heat map corresponding to the target portrait, that is, the key point feature data corresponding to a target portrait containing 17 key points.

[0097] The loss function of the portrait key point feature extraction network in this embodiment during training is as follows: Similar to other deep learning algorithms, the loss function is the mean squared error loss. Since the prediction is a heat map, if a single ground truth (GT) point in the dataset is directly used for calculation, the training will be difficult to converge due to the excessive number of negative samples. Therefore, it is necessary to expand the GT into a GT heat map using a two-dimensional Gaussian distribution. Due to the different difficulty levels of different key points, it is necessary to calculate the corresponding weights for different key points when calculating the total loss.

[0098] Exemplarily, the portrait key point feature extraction network can be constructed based on the HRNet network.

[0099] In one embodiment, referring to FIG7 , a behavior classification method is provided, including:

[0100] Step 702: Acquire a power monitoring image, where the power monitoring image includes a target person portrait.

[0101] In step 704, the power monitoring image is input into a portrait key point feature extraction network to obtain key point feature data output by the portrait key point feature extraction network. The portrait key point feature extraction network is trained based on a multi-resolution parallel mechanism.

[0102] In step 706, at least one behavior query data is input into the multi-head self-attention network to obtain self-attention query data corresponding to each behavior query data. The spatial embedding tensor of the multi-head self-attention network is obtained by positionally encoding the keypoint feature data according to its dimensions. The at least one behavior query data is determined based on behavior classification requirements, and each behavior query data corresponds to a different behavior category.

[0103] In step 708, the respective attention query data and key point feature data are input into the multi-head cross attention network to obtain cross query data corresponding to the respective attention query data output by the multi-head cross attention network.

[0104] In step 710 , each cross-query data is input into the feed-forward sub-network to obtain cross-enhanced data corresponding to each cross-query data output by the feed-forward sub-network.

[0105] In step 712, each cross-enhancement data is input into the affine transformation sub-network to obtain at least one cross-transformation data corresponding to each cross-enhancement data output by the affine transformation sub-network.

[0106] In step 714, each cross-transformed data is input into the pooling sub-network to obtain the behavior classification result corresponding to the target portrait.

[0107] The process of the above-mentioned behavior classification method can correspond to a two-stage network structure of a feature encoder and a classification decoder. The applicant provides a method for performing network training based on the two-stage network structure corresponding to the behavior classification method provided in this embodiment. An exemplary training process is as follows:

[0108] Overall, the two-stage network structure is trained using a pre-training and fine-tuning approach. The feature encoder uses Microsoft's COCO2017 public dataset as a training dataset. The COCO2017 dataset is a mainstream dataset for pose estimation, including a rich collection of images with single and multiple people, large, medium and small objects. Related research uses it as a training and test dataset. The classification decoder is trained using a private dataset. This private dataset focuses on substation power scenes and contains 12,510 sample images. These images include a variety of indoor and outdoor scenes, various working postures (climbing, leaping, making phone calls, falling, lifting), and different population distributions (single person, multiple people). This simulates various actual on-site scenarios. The number of samples in the private dataset is shown in Table 1.

[0109] Table 1. Number of samples in private datasets

[0110] The training process provided in this example uses an Intel Xeon Gold 6242R processor (754GB of memory), a single NVIDIA A100 GPU (80GB of video memory), the Ubuntu 18.04 operating system, and the PyTorch deep learning framework. During the pre-training phase, the feature encoder is trained first, and the classification decoder is not included in the training. The training input image size is 384 pixels × 288 pixels, the batch size is 24, and the Adam optimizer is used. The initial learning rate is set to 1e-3, the learning rate is adjusted using MultiStep, the drop points are set at 170 and 200, and the drop rate is set to 0.1. Training is performed for a total of 210 epochs. During the fine-tuning phase, the feature encoder is fixed, and only the classification decoder is trained. The learning rate is set to 1e-5, the learning rate is adjusted using cosine annealing, the batch size is 128, and the weight decay is 1e-8. Training is performed for a total of 30 epochs. All other hyperparameters remain the same as those in the pre-training phase.

[0111] During the training process, data augmentation can be used to create new training samples to expand the size of the existing dataset. This improves the accuracy of the feature encoder and classification decoder, effectively prevents overfitting, and allows the model to generalize better, making it more accurate and reliable in practical applications. For example, the data augmentation methods shown in Table 2 can be used.

[0112] Table 2 Data augmentation methods for training samples

[0113] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0114] Based on the same inventive concept, the present application also provides a behavior classification device for implementing the aforementioned behavior classification method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more behavior classification device embodiments provided below can be found in the above-mentioned limitations of the behavior classification method and will not be repeated here.

[0115] In one embodiment, as shown in FIG8 , a behavior classification device is provided, including: an image acquisition module 802 , a feature acquisition module 804 , and a behavior classification module 806 , wherein:

[0116] The image acquisition module 802 is configured to acquire a power monitoring image, wherein the power monitoring image includes a target person portrait.

[0117] The feature acquisition module 804 is configured to acquire key point feature data corresponding to the target portrait based on the power monitoring image.

[0118] The behavior classification module 806 is used to input the key point feature data and at least one behavior query data into the behavior classification network to obtain the behavior classification result corresponding to the target portrait output by the behavior classification network. The behavior classification network is trained based on a multi-head cross-attention mechanism. The at least one behavior query data is determined according to the behavior classification requirements, and each behavior query data corresponds to a different behavior category.

[0119] In one embodiment, the behavior classification network includes a multi-head self-attention network and a multi-head cross-attention prediction network that are interconnected; the behavior classification module 806 is used to input each of the behavior query data into the multi-head self-attention network to obtain the self-attention query data corresponding to each of the behavior query data, and the spatial embedding tensor of the multi-head self-attention network is obtained by position encoding according to the dimension of the key point feature data; and is used to obtain the behavior classification result based on each of the self-attention query data, the key point feature data and the multi-head cross-attention prediction network.

[0120] In one embodiment, the multi-head cross-attention prediction network includes a multi-head cross-attention network and a prediction network connected in sequence to the multi-head self-attention network; the behavior classification module 806 is used to input each of the self-attention query data and the key point feature data into the multi-head cross-attention network to obtain cross-query data corresponding to each of the self-attention query data output by the multi-head cross-attention network; and is used to input each of the cross-query data into the prediction network to obtain the behavior classification result output by the prediction network.

[0121] In one embodiment, the prediction network includes a feedforward subnetwork and a pooling subnetwork connected in sequence; the behavior classification module 806 is used to input each cross-query data into the feedforward subnetwork to obtain cross-enhancement data corresponding to each cross-query data output by the feedforward subnetwork; and is used to obtain the behavior classification result based on each cross-enhancement data and the pooling subnetwork.

[0122] In one embodiment, the prediction network also includes an affine transformation subnetwork connected between the feedforward subnetwork and the pooling subnetwork; the behavior classification module 806 is used to input each of the cross-enhancement data into the affine transformation subnetwork to obtain at least one cross-transformation data corresponding to each of the cross-enhancement data output by the affine transformation subnetwork; and is used to input each of the cross-transformation data into the pooling subnetwork to obtain the behavior classification result.

[0123] In one embodiment, the feature acquisition module 804 is used to input the power monitoring image into the portrait key point feature extraction network to obtain the key point feature data output by the portrait key point feature extraction network, and the portrait key point feature extraction network is trained based on a multi-resolution parallel mechanism.

[0124] Each module in the aforementioned behavior classification device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0125] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 9. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store power monitoring images. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a behavior classification method.

[0126] Those skilled in the art will understand that the structure shown in Figure 9 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0127] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0131] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0132] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A behavior classification method, It is characterized in that The method comprises: Acquire a power monitoring image, wherein the power monitoring image includes a target person portrait; According to the power monitoring image, acquiring key point feature data corresponding to the target portrait; The key point feature data and at least one behavior query data are input into a behavior classification network to obtain a behavior classification result corresponding to the target portrait output by the behavior classification network. The behavior classification network is trained based on a multi-head cross-attention mechanism. The at least one behavior query data is determined according to the behavior classification requirements, and each behavior query data corresponds to a different behavior category.

2. The method according to claim 1, It is characterized in that The behavior classification network includes a multi-head self-attention network and a multi-head cross-attention prediction network connected to each other; the key point feature data and at least one behavior query data are input into the behavior classification network to obtain a behavior classification result corresponding to the target portrait output by the behavior classification network, including: Inputting each of the behavior query data into the multi-head self-attention network to obtain the self-attention query data corresponding to each of the behavior query data, wherein the spatial embedding tensor of the multi-head self-attention network is obtained by position encoding according to the dimension of the key point feature data; The behavior classification result is obtained according to the self-attention query data, the key point feature data and the multi-head cross-attention prediction network.

3. The method according to claim 2, It is characterized in that The multi-head cross-attention prediction network includes a multi-head cross-attention network and a prediction network connected in sequence to the multi-head self-attention network; the behavior classification result is obtained according to each of the self-attention query data, the key point feature data and the multi-head cross-attention prediction network, including: Input each of the self-attention query data and the key point feature data into the multi-head cross attention network to obtain the cross query data corresponding to each of the self-attention query data output by the multi-head cross attention network; Each of the cross-query data is input into the prediction network to obtain the behavior classification result output by the prediction network.

4. The method according to claim 3, It is characterized in that The prediction network includes a feedforward subnetwork and a pooling subnetwork connected in sequence; the step of inputting each of the cross-query data into the prediction network to obtain the behavior classification result output by the prediction network includes: Inputting each cross query data into the feedforward sub-network to obtain cross enhanced data corresponding to each cross query data output by the feedforward sub-network; The behavior classification result is obtained according to each of the cross-enhancement data and the pooling sub-network.

5. The method according to claim 4, It is characterized in that The prediction network also includes an affine transformation subnetwork connected between the feedforward subnetwork and the pooling subnetwork; and obtaining the behavior classification result according to each of the cross-enhancement data and the pooling subnetwork includes: Input each of the cross-enhancement data into the affine transformation sub-network to obtain at least one cross-transformation data corresponding to each of the cross-enhancement data output by the affine transformation sub-network; Each of the cross-transformed data is input into the pooling subnetwork to obtain the behavior classification result.

6. The method according to claim 1, It is characterized in that The step of acquiring key point feature data corresponding to the target person portrait according to the power monitoring image includes: The power monitoring image is input into a portrait key point feature extraction network to obtain the key point feature data output by the portrait key point feature extraction network, wherein the portrait key point feature extraction network is trained based on a multi-resolution parallel mechanism.

7. A behavior classification device, It is characterized in that The device comprises: An image acquisition module, used to acquire a power monitoring image, wherein the power monitoring image includes a target person portrait; A feature acquisition module, used to acquire key point feature data corresponding to the target portrait according to the power monitoring image; The behavior classification module is used to input the key point feature data and at least one behavior query data into the behavior classification network to obtain the behavior classification result corresponding to the target portrait output by the behavior classification network, wherein the behavior classification network is trained based on a multi-head cross-attention mechanism, and the at least one behavior query data is determined according to the behavior classification requirements, and each behavior query data corresponds to a different behavior category.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images