Behavior category identification method and device, computer equipment and readable storage medium

Abstract

The invention relates to a behavior category identification method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring a to-be-detected image sequence in power grid production operation; and inputting the to-be-detected image sequence to a target behavior category recognition model, and obtaining a category recognition result output by the target behavior category recognition model, the category recognition result being used for determining a power grid operation behavior category of a target detection object in the to-be-detected image sequence, the target behavior category recognition model is obtained based on sample image sequence training in power grid production operation. By adopting the method, the accuracy of a category identification result can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a behavior category recognition method, device, computer equipment, and computer-readable storage medium. Background Art

[0002] In order to ensure the safety of operators, the power grid has formulated detailed operating specifications, especially for behaviors that pose significant potential risks to personnel safety, such as climbing and operating high-voltage equipment. However, in actual production operations, it is still difficult to effectively supervise personnel violations.

[0003] Existing intelligent application technologies based on target detection face many challenges, including insufficient detection accuracy in complex environments, limited ability to make fine judgments on human behavior, and poor detection results under occlusion and long distance conditions. This can easily lead to missed reports or false reports, making it impossible to accurately identify dangerous behaviors in power grid operations. These problems make it impossible to accurately monitor the behavior of operators in high-risk operating environments. Summary of the invention

[0004] Based on this, it is necessary to provide a behavior category identification method, device, computer equipment and computer-readable storage medium that can improve the accuracy of power grid operation behavior category identification in response to the above technical problems.

[0005] In a first aspect, the present application provides a behavior category identification method, the method comprising:

[0006] Acquire a sequence of images to be detected in power grid production operations;

[0007] The image sequence to be detected is input into a target behavior category recognition model, and a category recognition result output by the target behavior category recognition model is obtained, wherein the category recognition result is used to determine the power grid operation behavior category of the target detection object in the image sequence to be detected, wherein the target behavior category recognition model is trained based on a sample image sequence in power grid production operations.

[0008] In one embodiment, the target behavior category recognition model includes a first target model and a second target model; inputting the image sequence to be detected into the target behavior category recognition model, and obtaining the category recognition result output by the target behavior category recognition model, includes:

[0009] Inputting the image sequence to be detected into a first target model, and obtaining a position prediction result output by the first target model, wherein the position prediction result includes a target detection object of each image to be detected in the image sequence to be detected, and a predicted position of each key point of the target detection object;

[0010] For the same target detection object, the predicted positions of the key points of the target detection object in each of the images to be detected are input into the second target model, and the category recognition result output by the second target model is obtained.

[0011] In one embodiment, inputting the to-be-detected image sequence into a first target model and obtaining a position prediction result output by the first target model includes:

[0012] Extracting initial features of each image to be detected in the sequence of images to be detected to obtain initial feature maps;

[0013] Performing a feature fusion operation on each of the initial feature maps to obtain a target feature map;

[0014] According to the target feature map, the target detection object of each of the images to be detected and the predicted positions of the key points of the target detection object are determined.

[0015] In one embodiment, for the same target detection object, inputting the predicted positions of the key points of the target detection object in each of the images to be detected into the second target model, and obtaining the category recognition result output by the second target model, includes:

[0016] Obtaining the predicted position of the target key point from the predicted positions of each of the key points;

[0017] Obtaining previous state information of the predicted position of the target key point;

[0018] Determining current state information of the predicted position of the target key point according to the predicted position of the target key point and previous state information of the predicted position of the target key point;

[0019] Based on the current state information and the historical category recognition results, the category recognition results corresponding to the predicted positions of the key points are determined.

[0020] In one embodiment, before inputting the image sequence to be detected into the target behavior category recognition model, the method further includes:

[0021] Acquire a sample image sequence in a power grid production operation; the sample image sequence is annotated with the actual position of each key point of the target detection object and the real power grid operation behavior category of the target detection object;

[0022] The sample image sequence is input into the initial behavior category recognition model to obtain the sample position prediction result and the sample power grid operation behavior category output by the initial behavior category recognition model; wherein the sample position prediction result includes the sample prediction position of each key point of the target detection object of the sample image sequence; the sample power grid operation behavior category is used to determine the power grid operation behavior category of the target detection object of the sample image sequence;

[0023] Determining a total loss value according to the sample predicted position, the sample power grid operation behavior category, the actual position and the real power grid operation behavior category;

[0024] The parameters of the initial behavior category recognition model are adjusted according to the total loss value until the total loss value meets a preset condition, and the initial behavior category recognition model is determined as the target behavior category recognition model.

[0025] In one embodiment, the total loss value includes a first loss value and a second loss value; determining the total loss value according to the sample predicted position, the sample power grid operation behavior category, the actual position and the real power grid operation behavior category includes:

[0026] Determining the first loss value according to the sample predicted position and the actual position;

[0027] Determining the second loss value according to the sample power grid operation behavior category and the real power grid operation behavior category;

[0028] The total loss value is determined based on the first loss value and the second loss value.

[0029] In one embodiment, determining the first loss value according to the sample predicted position and the actual position includes:

[0030] Determining a position loss value according to the sample predicted position and the actual position;

[0031] Obtaining the confidence of the sample predicted position and the confidence of the actual position, and determining a confidence loss value according to the confidence of the sample predicted position and the confidence of the actual position;

[0032] The first loss value is determined based on the position loss value and the confidence loss value.

[0033] In a second aspect, the present application further provides a behavior category identification device, the device comprising:

[0034] A first acquisition module is used to acquire a sequence of images to be detected in a power grid production operation;

[0035] A processing module is used to input the image sequence to be detected into a target behavior category recognition model, and obtain a category recognition result output by the target behavior category recognition model, wherein the category recognition result is used to determine the power grid operation behavior category of the target detection object in the image sequence to be detected, wherein the target behavior category recognition model is trained based on a sample image sequence in power grid production operations.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0038] The above behavior category recognition method, device, computer equipment and computer-readable storage medium can obtain the power grid operation behavior type of the target detection object output by the target behavior category recognition model by inputting the image sequence to be detected in the power grid production operation into the target behavior category recognition model, thereby improving the accuracy of identifying the power grid operation behavior type. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] FIG1 is a diagram of an application environment of a behavior category identification method according to an embodiment;

[0041] FIG2 is a schematic diagram of a flow chart of a behavior category identification method in one embodiment;

[0042] FIG3 is a schematic diagram of a process flow of a target behavior category recognition model processing in one embodiment;

[0043] FIG4 is a schematic diagram of a process flow of a first target model processing in another embodiment;

[0044] FIG5 is a schematic diagram of a process flow of a second target model processing in another embodiment;

[0045] FIG6 is a schematic diagram of a process of training a target behavior category recognition model in one embodiment;

[0046] FIG7 is a schematic diagram of a process for determining a total loss value in one embodiment;

[0047] FIG8 is a schematic diagram of a process for determining a first loss value in one embodiment;

[0048] FIG9 is a structural block diagram of a behavior category identification device in one embodiment;

[0049] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The behavior category recognition method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 1. The terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. Obtain the image sequence to be detected in the power grid production operation; input the image sequence to be detected into the target behavior category recognition model, and obtain the category recognition result output by the target behavior category recognition model, and the category recognition result is used to determine the power grid operation behavior category of the target detection object in the image sequence to be detected, wherein the target behavior category recognition model is trained based on the sample image sequence in the power grid production operation. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0052] In an exemplary embodiment, as shown in FIG2 , a behavior category identification method is provided, and the method is applied to the server in FIG1 as an example for explanation, including the following steps S202 to S204 . Among them:

[0053] Step S202: acquiring a sequence of images to be detected in a power grid production operation.

[0054] The image sequence to be detected includes a group of images to be detected. The image sequence to be detected is obtained by extracting frames from the video to be detected. For example, the server selects multiple video clips containing the behavior to be detected from the video to be detected in response to the instruction, and the server converts the video clips into the image sequence to be detected.

[0055] Optionally, in response to the instruction, the server selects multiple video clips containing the behaviors to be detected from the video to be detected, assuming that each video clip is 10 seconds long; the server extracts two frames of images from each second of the video clip, and generates a 20-frame image sequence to be detected for each video clip of 10 seconds in length, thereby completing the conversion of the video clip into the image sequence to be detected.

[0056] Optionally, the server obtains the video of the power grid production operation from the power grid production operation video acquisition device, crops the video to obtain a video clip; further extracts frames from the video clip to obtain a sequence of images to be detected in the power grid production operation.

[0057] It should be noted that the video clip is not limited here, and can be any video clip of a length that meets the conditions. The number of a group of images to be detected included in the image sequence to be detected is also not limited here.

[0058] Step S204: input the image sequence to be detected into the target behavior category recognition model, and obtain the category recognition result output by the target behavior category recognition model.

[0059] The category recognition result is used to determine the power grid operation behavior category of the target detection object in the image sequence to be detected, such as climbing, crossing and bottom drilling. The target behavior category recognition model is trained based on the sample image sequence in the power grid production operation. The image sequence to be detected can include one detection object or multiple detection objects; the server can select one of them as the target detection object.

[0060] Optionally, before the server inputs the image sequence to be detected into the target behavior category recognition model, the server trains the initial behavior category recognition model through the sample image sequence to obtain the target behavior category recognition model. The image sequence to be detected covers different work scenarios and possible dangerous behaviors to ensure the diversity and representativeness of the training set, thereby ensuring the accuracy of the training results.

[0061] Optionally, the server inputs the image sequence to be detected into the trained target behavior category recognition model, and the target behavior category recognition model can be determined by the YOLOv8 model and the selective state space model. The target behavior category recognition model can also be determined by other machine learning models. The server obtains the category recognition result output by the target behavior category recognition model, such as the behavior of operator A is climbing. Among them, YOLOv8 is the latest generation of target detection model, which inherits the core advantages of the YOLO series of algorithms and shows excellent performance in multiple tasks. The model structure of YOLOv8 is concise and efficient, designed for real-time target detection, and suitable for judging the power grid operation behavior category of the target detection object in power grid operation. YOLOv8 adopts a lightweight network architecture to optimize computing efficiency. It introduces the attention mechanism in the backbone network (Backbone) to improve the model's ability to capture key features. The backbone network and Neck part of the model refer to the design concept of YOLOv7 ELAN, and adopts the C2f structure to replace the C3 structure in YOLOv5, which enhances the gradient fluidity and model performance.

[0062] In the above behavior category recognition method, the image sequence to be detected in the power grid production operation is input into the target behavior category recognition model, and the power grid operation behavior type of the target detection object output by the target behavior category recognition model can be obtained, thereby improving the accuracy of identifying the power grid operation behavior type.

[0063] In an exemplary embodiment, as shown in FIG3 , the target behavior category recognition model includes a first target model and a second target model; inputting the image sequence to be detected into the target behavior category recognition model, and obtaining the category recognition result output by the target behavior category recognition model, including steps S302 to S304. Wherein:

[0064] Step S302: input the image sequence to be detected into the first target model, and obtain the position prediction result output by the first target model.

[0065] The position prediction result includes the target detection object of each image to be detected in the image sequence to be detected, and the predicted position of each key point of the target detection object. The first target model is a target detection model, which can be implemented by a YOLOv8 model or other target detection models.

[0066] Optionally, the server inputs a sequence of images to be detected, such as 20 images to be detected, into a YOLOv8 model, and each image to be detected is processed by the YOLOv8 model, such as extracting features, and outputting the target detection object of each image to be detected, and the predicted positions of each key point of the target detection object, such as the target object output by the YOLOv8 model and the predicted positions of the key points of the target object. If there is only one object in the 20 images to be detected, such as operator A, the YOLOv8 model outputs the anchor point box where operator A is located and the predicted positions of the key points of operator A's nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and outputs the two-dimensional coordinates of the key points and the corresponding confidence scores. If there are multiple objects in the 20 images to be detected, such as operator A, operator B, and operator C, the YOLOv8 model outputs the anchor point box where operator A is located and the predicted positions of key points such as operator A's nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and outputs the two-dimensional coordinates of the key points and the corresponding confidence scores; the anchor point box where operator B is located and the predicted positions of key points such as operator A's nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and outputs the two-dimensional coordinates of the key points and the corresponding confidence scores; similarly, the YOLOv8 model outputs the predicted positions of key points of operator C and the corresponding confidence scores.

[0067] Step S304, for the same target detection object, the predicted positions of the key points of the target detection object in each image to be detected are input into the second target model, and the category recognition result output by the second target model is obtained.

[0068] The second objective model may adopt a selective state space model (Selective SSM).

[0069] Optionally, for the same target detection object, such as operator A, the predicted positions of each key point of operator A in each image to be detected output by the YOLOv8 model are input into the selective state space model. The selective SSM selectively focuses on the important information in the key points and ignores irrelevant or redundant data. For example, if the left knee and the right knee change greatly in the sequence images, the selective SSM will choose to update and predict the "left knee and right knee" as the key states, while the states of other key points remain unchanged temporarily.

[0070] In this embodiment, the target detection object of each image to be detected and the predicted position of each key point of the target detection object are output by the YOLOv8 model, and the YOLOv8 model output is used as the input of the selective SSM. The selective SSM can effectively compress information and improve computational efficiency, thereby achieving efficient sequence modeling and behavior classification. Thus, joint model processing is achieved, the predicted position of each key point of the target detection object is accurately predicted, and the category recognition result of the target detection object is accurately identified according to the predicted position of each key point.

[0071] In an exemplary embodiment, as shown in FIG4 , the image sequence to be detected is input into the first target model, and the position prediction result output by the first target model is obtained, including steps S402 to S406. In which:

[0072] Step S402: extracting initial features of each image to be detected in the sequence of images to be detected, and obtaining initial feature maps.

[0073] Optionally, the server normalizes each image to be detected in the image sequence to adapt to the input requirements of YOLOv8. YOLOv8 performs preliminary feature extraction on each image to be detected through the backbone network.

[0074] Step S404: performing a feature fusion operation on each initial feature map to obtain a target feature map.

[0075] Optionally, the initial feature map output by the server from the backbone network is input into the Feature Pyramid Network (FPN) for multi-scale feature fusion. This step ensures the YOLOv8 model's ability to detect objects of different scales, especially when processing high-resolution surveillance videos, which helps improve the detection accuracy of key points of target detection objects.

[0076] Step S406, determining the target detection object of each image to be detected and the predicted position of each key point of the target detection object according to the target feature map.

[0077] Optionally, the server determines the target detection object of each image to be detected and the predicted position of each key point of the target detection object based on the target feature map. YOLOv8 uses an anchor-free detection head, which simplifies the detection process and improves the accuracy and efficiency of detection compared to traditional anchor-based methods. By dynamically assigning positive and negative samples (Task-Aligned Assigner), the model can better focus on key and difficult samples, thereby improving the overall detection performance.

[0078] In this embodiment, the accuracy of key point detection can be improved by initial features and feature fusion of each image to be detected.

[0079] In an exemplary embodiment, as shown in FIG5 , for the same target detection object, the predicted positions of the key points of the target detection object in each image to be detected are input into the second target model, and the category recognition result output by the second target model is obtained, including steps S502 to S508. Wherein:

[0080] Step S502: Obtain the predicted position of the target key point from the predicted positions of each key point.

[0081] Optionally, before the second target model, i.e., the selective state space model, obtains the predicted position of the target key point from the predicted positions of each key point, the selective SSM performs state initialization. This step initializes the internal state parameters of the model according to the characteristics of the input data to prepare for subsequent sequence modeling.

[0082] Optionally, the server obtains the predicted position of the target key point from the predicted positions of each key point through a selective state space model, such as selecting the left knee and the right knee as the target key points from 17 key points including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and obtains the predicted position of the target key point.

[0083] The number of target key points selected can be any number as long as the conditions are met, and is not limited to a specific number of target key points.

[0084] Step S504, obtaining the previous state information of the predicted position of the target key point.

[0085] Optionally, the server obtains the previous state information of the predicted position of the target key point through the selective state space model .

[0086] Step S506 , determining current state information of the predicted position of the target key point according to the predicted position of the target key point and previous state information of the predicted position of the target key point.

[0087] Optionally, the selective SSM processes the key points of the input image sequence to be detected in a recursive manner. At each time step, the model processes the key points through formula (1).

[0088] Formula (1)

[0089] in, Indicates the current state. is the previous state, is the key point data currently input, is a model parameter. Selective SSM effectively compresses information and improves computational efficiency by focusing on the key points of the input image sequence to be detected and ignoring irrelevant or redundant data.

[0090] Step S508: Determine the category recognition result corresponding to the predicted position of each key point based on the current state information and the historical category recognition results.

[0091] Optionally, after the current state information of the predicted position of the target key point is updated, the selective SSM enters the behavior classification stage. The selective SSM identifies and classifies the current dangerous behavior category, such as climbing, crossing, and bottoming behaviors, based on the updated state sequence and the previously learned behavior patterns.

[0092] In this embodiment, by extracting the key points of the target detection object and constructing a state space model, complex actions can be detected and distinguished more accurately, especially in complex environments such as occlusion and long distance. In addition, the selective state space model focuses on posture changes related to dangerous behaviors, significantly improving detection accuracy and real-time performance.

[0093] In an exemplary embodiment, as shown in FIG6 , before inputting the image sequence to be detected into the target behavior category recognition model, steps S602 to S608 are also included. Among them:

[0094] Step S602, obtaining a sample image sequence in a power grid production operation; the sample image sequence is annotated with the actual position of each key point of the target detection object and the real power grid operation behavior category of the target detection object.

[0095] Optionally, in response to the instruction, the server selects a plurality of sample clip videos containing the behaviors to be detected from the sample video, assuming that each sample clip video is 10 seconds long; the server extracts two frames of images from each second of the sample clip video, and generates a 20-frame sample image sequence for each sample clip video of 10 seconds in length, thereby completing the conversion of the sample clip video into a sample image sequence. The sample clips should cover different operating scenarios and possible dangerous behaviors to ensure the diversity and representativeness of the training set. After obtaining the sample image sequence, the server annotates the sample image sequence. First, annotate the target position of the target object for each sample image in each sample image sequence. Manually mark the position of the target object (which can be an operator) in each sample image, and use a rectangular box to describe the position. The rectangular box is represented by four parameters: the normalized coordinates of the upper left corner point (obtained by dividing the coordinates x and y values ​​of the actual point by the length and width of the image) and the normalized rectangular box size (obtained by dividing the length and width of the actual rectangular box by the length and width of the image).

[0096] In the case where a sample image sequence contains multiple target objects, the sample image sequence is replicated a corresponding number of times, and only the same target object is labeled in each sample image sequence to ensure data consistency and accuracy.

[0097] After marking the target position of the target object, the server further marks the key point position of the target object in each sample image. The key points can include 17 parts: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle. If the part corresponding to the key point is blocked in the video, the key point still needs to be marked as accurately as possible based on the context information.

[0098] The server annotates the behavior categories of the target objects in each sample image sequence according to the annotation instructions, and marks the corresponding power grid operation behavior categories according to the actions in the video. For example, the recognized power grid operation behavior categories include climbing, crossing, and drilling. Each video sequence should clearly annotate the power grid operation behavior categories contained therein so that these behaviors can be effectively learned and identified during subsequent model training.

[0099] Optionally, the server obtains a sample image sequence in a power grid production operation; the sample image sequence is annotated with actual positions of key points of the target detection object and a real power grid operation behavior category of the target detection object.

[0100] Step S604: input the sample image sequence into the initial behavior category recognition model to obtain the sample position prediction result and the sample power grid operation behavior category output by the initial behavior category recognition model.

[0101] Among them, the sample position prediction result includes the sample prediction position of each key point of the target detection object of the sample image sequence; the sample power grid operation behavior category is used to determine the power grid operation behavior category of the target detection object of the sample image sequence.

[0102] Optionally, the server inputs the sample image sequence into an initial behavior category recognition model to obtain sample predicted positions of key points of the target detection object in the image sequence output by the initial behavior category recognition model and the power grid operation behavior category of the target detection object in the sample image sequence.

[0103] Step S606, determining the total loss value according to the sample predicted position, the sample power grid operation behavior category, the actual position and the real power grid operation behavior category.

[0104] Among them, the total loss value represents the difference between the predicted position and the actual position of each key point, the accuracy of the predicted position of each key point, and the difference between the sample power grid operation behavior category and the real power grid operation behavior category.

[0105] Optionally, the server uses a preset total loss function, denoted as L total ,After the initial behavior category recognition model outputs the sample prediction position and the sample grid operation behavior category, the total loss value is determined according to the total loss function and according to the sample prediction position, the sample grid operation behavior category, the actual position and the real grid operation behavior category.

[0106] Step S608, adjusting the parameters of the initial behavior category recognition model according to the total loss value until the total loss value meets the preset conditions, and determining the initial behavior category recognition model as the target behavior category recognition model.

[0107] Among them, the preset conditions include convergence conditions.

[0108] Optionally, after obtaining the first total loss value, if the total loss value does not meet the convergence condition, the server adjusts the parameters of the initial behavior category recognition model according to the total loss value. The server then inputs another sample image sequence into the adjusted initial behavior category recognition model to obtain the second total loss value of the adjusted initial behavior category recognition model, and determines whether the total loss value meets the convergence condition. If so, the initial behavior category recognition model is determined as the target behavior category recognition model. If not, and so on, until the total loss value meets the convergence condition, the initial behavior category recognition model that meets the convergence condition is determined as the target behavior category recognition model.

[0109] In this embodiment, the parameters of the initial behavior category recognition model are adjusted by the total loss value, so that the target behavior category recognition model can be obtained, thereby improving the accuracy of behavior category recognition.

[0110] Continuing from the previous embodiment, as shown in FIG7 , the total loss value includes a first loss value and a second loss value; the total loss value is determined according to the sample predicted position, the sample power grid operation behavior category, the actual position and the real power grid operation behavior category, including steps S702 to S706. Wherein:

[0111] Step S702, determining a first loss value according to the predicted position and the actual position of the sample.

[0112] Among them, the first total loss value represents the difference between the predicted position and the actual position of each key point and the accuracy of the predicted position of each key point.

[0113] Optionally, the server inputs the sample predicted position and actual position into a pre-set first loss function, denoted as ,The first loss function includes the position loss function and the confidence loss function. The position loss function is used to evaluate the error between the key point position predicted by the target behavior category recognition model and the actual position; the confidence loss function is used to measure the accuracy of the target behavior category recognition model's judgment on whether the key point exists.

[0114] As shown in formula (2), the first loss value is determined.

[0115] Formula (2)

[0116] In the formula, the first loss function is recorded as , represents the position loss function; represents the confidence loss function, and αβ represent the weight coefficients respectively.

[0117] Step S704: determining a second loss value according to the sample power grid operation behavior category and the actual power grid operation behavior category.

[0118] Among them, the second total loss value represents the difference between the sample behavior category and the true behavior category.

[0119] Optionally, a cross entropy loss function is usually used to calculate the gap between the model prediction category probability and the true label, the server sample power grid operation behavior category and the true power grid operation behavior category, and the input value is preset for the second loss function, denoted as , as shown in formula (3), determine the second loss value.

[0120] Formula (3)

[0121] In the formula, is the number of categories, Indicates The true category labels of samples, is the predicted class probability.

[0122] Step S706: determine a total loss value based on the first loss value and the second loss value.

[0123] Among them, the total loss value represents the difference between the predicted position and the actual position of each key point, the accuracy of the predicted position of each key point, and the difference between the sample behavior category and the real behavior category.

[0124] Optionally, the server determines the total loss value by formula (4) based on the first loss value calculated by the first loss function and the second loss value calculated by the second loss function.

[0125] Formula (4)

[0126] In the formula, represents the first loss function; represents the second loss function. and is the importance parameter to balance the two loss functions.

[0127] In this embodiment, the total loss value determined by the first loss value and the second loss value characterizes the difference between the predicted position and the actual position of each key point, the accuracy of the predicted position of each key point, and the difference between the sample behavior category and the true behavior category. The initial behavior category recognition model is adjusted by the total loss value, which can improve the accuracy of the target behavior category recognition model.

[0128] In an exemplary embodiment, as shown in FIG8 , determining the first loss value according to the predicted position and the actual position of the sample includes steps S802 to S806 . Among them:

[0129] Step S802, determining the position loss value according to the predicted position and the actual position of the sample.

[0130] Optionally, the position loss is measured by calculating the mean square error between the coordinates of the predicted position and the coordinates of the actual position using formula (5).

[0131] Formula (5)

[0132] In the formula, represents the number of samples, is the number of keypoints in each sample, For the In the sample The predicted position coordinates of key points, is the corresponding actual position coordinate.

[0133] Step S804, obtaining the confidence of the sample predicted position and the confidence of the actual position, and determining the confidence loss value according to the confidence of the sample predicted position and the confidence of the actual position.

[0134] Optionally, the server obtains the confidence of the sample predicted position and the confidence of the actual position, and the confidence of the sample predicted position may be output by the first target model.

[0135] Formula (6)

[0136] in, It is In the sample The true confidence of the key point (1 if there is a key point, otherwise 0), is the confidence level of the prediction.

[0137] Step S806: determine a first loss value based on the position loss value and the confidence loss value.

[0138] Optionally, the server determines the first loss value based on a preset first loss function, as shown in formula (2), based on the position loss value and the confidence loss value.

[0139] Formula (2)

[0140] In this embodiment, the first loss value is determined by the position loss value and the confidence loss value, and the initial behavior category recognition model is adjusted by the first loss value, so as to improve the accuracy of the target behavior category recognition model.

[0141] In one embodiment, it is divided into two stages: one is the training stage of the target behavior category recognition model, and the other is the use stage of the target behavior category recognition model.

[0142] Training phase: In response to the instruction, the server selects multiple sample clip videos containing the behaviors to be detected from the sample videos. Assume that each sample clip video is 10 seconds long. The server extracts two frames of images from each second of the sample clip video, and generates a 20-frame sample image sequence for each 10-second sample clip video, thereby completing the conversion of the sample clip video into a sample image sequence. The sample clips should cover different work scenarios and possible dangerous behaviors to ensure the diversity and representativeness of the training set. After obtaining the sample image sequence, the server annotates the sample image sequence. First, annotate the target position of the target object for each sample image in each sample image sequence. Manually mark the position of the target object (which can be an operator) in each sample image, and use a rectangular box to describe the position. The rectangular box is represented by four parameters: the normalized coordinates of the upper left corner point (obtained by dividing the x and y values ​​of the actual point by the length and width of the image) and the normalized rectangular box size (obtained by dividing the length and width of the actual rectangular box by the length and width of the image). In the case where a sample image sequence contains multiple target objects, the sample image sequence is copied a corresponding number of times, and only the same target object is marked in each sample image sequence to ensure data consistency and accuracy. After marking the target position of the target object, the server further marks the key point position of the target object in each sample image. The key points can include 17 parts: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle. If the part corresponding to the key point is blocked in the video, the key point still needs to be marked as accurately as possible based on the context information.

[0143] The server inputs the sample image sequence into the initial behavior category recognition model to obtain the sample predicted positions of each key point of the target detection object in the sample image sequence output by the initial behavior category recognition model and the power grid operation behavior category of the target detection object in the sample image sequence. The position loss is measured by calculating the mean square error between the coordinates of the predicted position and the coordinates of the actual position using formula (5).

[0144] Formula (5)

[0145] In the formula, represents the number of samples, is the number of keypoints in each sample, For the In the sample The predicted position coordinates of key points, is the corresponding actual position coordinate. The server obtains the confidence of the sample predicted position and the confidence of the real position. The confidence of the sample predicted position can be output by the first target model. And the confidence loss value is determined by the confidence loss function, as shown in formula (6).

[0146] Formula (6)

[0147] in, It is In the sample The true confidence of the key point (1 if there is a key point, otherwise 0), is the confidence level of the prediction.

[0148] The server determines a first loss value based on the position loss value and the confidence loss value.

[0149] Formula (2)

[0150] In the formula, the first loss function is recorded as , represents the position loss function; represents the confidence loss function, and αβ represent the weight coefficients respectively.

[0151] Optionally, a cross entropy loss function is usually used to calculate the gap between the model prediction category probability and the true label, the server sample power grid operation behavior category and the true power grid operation behavior category, and the input value is preset for the second loss function, denoted as , as shown in formula (3), determine the second loss value.

[0152] Formula (3)

[0153] In the formula, is the number of categories, Indicates The true category labels of samples, is the predicted class probability.

[0154] Optionally, the server determines the total loss value by formula (4) based on the first loss value calculated by the first loss function and the second loss value calculated by the second loss function.

[0155] Formula (4)

[0156] In the formula, represents the first loss function; represents the second loss function. and is the importance parameter to balance the two loss functions.

[0157] After obtaining the first total loss value, if the total loss value does not meet the convergence condition, the server adjusts the parameters of the initial behavior category recognition model according to the total loss value. Then the server inputs another sample image sequence into the adjusted initial behavior category recognition model to obtain the second total loss value of the adjusted initial behavior category recognition model, and determines whether the total loss value meets the convergence condition. If it does, the initial behavior category recognition model is determined as the target behavior category recognition model. If it does not meet, and so on, until the total loss value meets the convergence condition, the initial behavior category recognition model that meets the convergence condition is determined as the target behavior category recognition model.

[0158] Usage stage: In response to the instruction, the server selects multiple video clips containing the behaviors to be detected from the video to be detected, assuming that each video clip is 10 seconds long; the server extracts two frames of images from each second of the video clip, and generates a 20-frame image sequence to be detected for each 10-second video clip, thereby completing the conversion of the video clip into the image sequence to be detected.

[0159] The image sequence to be detected is standardized for each image to be detected in the image sequence to adapt to the input requirements of YOLOv8. For example, 20 images to be detected are input to the YOLOv8 model, and the server YOLOv8 performs preliminary feature extraction on each image to be detected through the backbone network. The initial feature map output from the backbone network is input into the Feature Pyramid Network (FPN) for multi-scale feature fusion. This step ensures the detection ability of the YOLOv8 model for targets of different scales, especially when processing high-resolution surveillance videos, which helps to improve the detection accuracy of the key points of the target detection object. The server determines the target detection object of each image to be detected and the predicted position of each key point of the target detection object based on the target feature map. YOLOv8 uses an anchor-free detection head, which simplifies the detection process and improves the accuracy and efficiency of detection compared with the traditional anchor-based method. By dynamically allocating positive and negative samples (Task-Aligned Assigner), the model can better focus on key and difficult samples, thereby improving the overall detection performance. Then the YOLOv8 model outputs the anchor point box where operator A is located and the predicted positions of key points such as operator A's nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and outputs the two-dimensional coordinates of the key points and the corresponding confidence scores.

[0160] Selective SSM will perform state initialization. This step initializes the internal state parameters of the model according to the characteristics of the input data to prepare for subsequent sequence modeling. For the same target detection object, such as operator A, the predicted positions of each key point of operator A in each image to be detected output by the YOLOv8 model are input into the selective state space model. The selective state space model obtains the predicted position of the target key point from the predicted positions of each key point, such as selecting the left knee and the right knee as the target key points from the 17 key points of nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, and obtains the predicted position of the target key point. That is, the selective SSM selectively focuses on the important information in the key points and ignores irrelevant or redundant data. For example, if the left knee and the right knee change greatly in the sequence image, the selective SSM will choose to update and predict the "left knee and the right knee" as the key state, while the states of other key points remain unchanged temporarily. The server obtains the previous state information of the predicted position of the target key point through the selective state space model The selective SSM processes the key points of the input image sequence to be detected in a recursive manner. At each time step, the model processes the key points through formula (1).

[0161] Formula (1)

[0162] in, Indicates the current state. is the previous state, is the key point data currently input, is a model parameter. Selective SSM effectively compresses information and improves computational efficiency by focusing on the key points of the input image sequence to be detected and ignoring irrelevant or redundant data. After the current state information of the predicted position of the target key point is updated, the selective SSM enters the behavior classification stage. The selective SSM identifies and classifies the current dangerous behavior category, such as climbing, crossing, and bottom drilling, based on the updated state sequence and the previously learned behavior pattern.

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a behavior category identification device for implementing the behavior category identification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more behavior category identification device embodiments provided below can refer to the limitations of the behavior category identification method above, and will not be repeated here.

[0165] In an exemplary embodiment, as shown in FIG9 , a behavior category identification device is provided, including: a first acquisition module 901 and a processing module 902, wherein:

[0166] The first acquisition module 901 is used to acquire a sequence of images to be detected in a power grid production operation.

[0167] Processing module 902 is used to input the image sequence to be detected into the target behavior category recognition model, and obtain the category recognition result output by the target behavior category recognition model, and the category recognition result is used to determine the power grid operation behavior category of the target detection object in the image sequence to be detected, wherein the target behavior category recognition model is trained based on the sample image sequence in the power grid production operation.

[0168] In an exemplary embodiment, the target behavior category recognition model includes a first target model and a second target model; the processing module 902 includes:

[0169] The first processing unit is used to input the image sequence to be detected into the first target model and obtain the position prediction result output by the first target model, wherein the position prediction result includes the target detection object of each image to be detected in the image sequence to be detected and the predicted position of each key point of the target detection object.

[0170] The second processing unit is used to input the predicted positions of the key points of the target detection object in each image to be detected into the second target model for the same target detection object, and obtain the category recognition result output by the second target model.

[0171] In an exemplary embodiment, the first processing unit includes:

[0172] The extraction subunit is used to extract the initial features of each image to be detected in the image sequence to obtain each initial feature map.

[0173] The fusion subunit is used to perform feature fusion operation on each initial feature map to obtain a target feature map.

[0174] The position prediction subunit is used to determine the target detection object of each image to be detected and the predicted position of each key point of the target detection object according to the target feature map.

[0175] In an exemplary embodiment, the second processing unit includes:

[0176] The first acquisition subunit is used to acquire the predicted position of the target key point from the predicted positions of each key point.

[0177] The second acquisition subunit is used to acquire the previous state information of the predicted position of the target key point.

[0178] The current state information determination subunit is used to determine the current state information of the predicted position of the target key point according to the predicted position of the target key point and the previous state information of the predicted position of the target key point.

[0179] The category recognition subunit is used to determine the category recognition result corresponding to the predicted position of each key point based on the current state information and the historical category recognition results.

[0180] In an exemplary embodiment, a behavior category identification device is provided, further comprising:

[0181] The second acquisition module is used to acquire a sample image sequence in power grid production operations; the sample image sequence is annotated with the actual position of each key point of the target detection object and the real power grid operation behavior category of the target detection object.

[0182] The training module is used to input the sample image sequence into the initial behavior category recognition model to obtain the sample position prediction results and sample power grid operation behavior categories output by the initial behavior category recognition model; wherein the sample position prediction results include the sample prediction positions of each key point of the target detection object of the sample image sequence; the sample power grid operation behavior category is used to determine the power grid operation behavior category of the target detection object of the sample image sequence. The total loss value is determined based on the sample prediction position, the sample power grid operation behavior category, the actual position and the real power grid operation behavior category. The parameters of the initial behavior category recognition model are adjusted according to the total loss value until the total loss value meets the preset conditions, and the initial behavior category recognition model is determined as the target behavior category recognition model.

[0183] In an exemplary embodiment, the total loss value includes a first loss value and a second loss value; and the training module includes:

[0184] The total loss value determination unit is used to determine the first loss value according to the sample predicted position and the actual position; determine the second loss value according to the sample power grid operation behavior category and the real power grid operation behavior category; and determine the total loss value based on the first loss value and the second loss value.

[0185] In an exemplary embodiment, the total loss value determining unit includes:

[0186] The first loss value determination subunit is used to determine the position loss value based on the sample predicted position and the actual position; obtain the confidence of the sample predicted position and the confidence of the actual position, and determine the confidence loss value based on the confidence of the sample predicted position and the confidence of the actual position; determine the first loss value based on the position loss value and the confidence loss value.

[0187] Each module in the above-mentioned behavior category identification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0188] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store power grid production operation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a behavior category recognition method is implemented.

[0189] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0190] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0191] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0192] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0193] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0194] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A behavior category recognition method, characterized in that: The method comprises: Acquire a sequence of images to be detected in power grid production operations; The image sequence to be detected is input into a target behavior category recognition model, and a category recognition result output by the target behavior category recognition model is obtained, wherein the category recognition result is used to determine the power grid operation behavior category of the target detection object in the image sequence to be detected, wherein the target behavior category recognition model is trained based on a sample image sequence in power grid production operations.

2. The method according to claim 1, characterized in that The target behavior category recognition model includes a first target model and a second target model; the step of inputting the image sequence to be detected into the target behavior category recognition model and obtaining a category recognition result output by the target behavior category recognition model includes: Inputting the image sequence to be detected into a first target model, and obtaining a position prediction result output by the first target model, wherein the position prediction result includes a target detection object of each image to be detected in the image sequence to be detected, and a predicted position of each key point of the target detection object; For the same target detection object, the predicted positions of the key points of the target detection object in each of the images to be detected are input into the second target model, and the category recognition result output by the second target model is obtained.

3. The method according to claim 2, characterized in that The step of inputting the image sequence to be detected into a first target model and obtaining a position prediction result output by the first target model includes: Extracting initial features of each image to be detected in the sequence of images to be detected to obtain initial feature maps; Performing a feature fusion operation on each of the initial feature maps to obtain a target feature map; According to the target feature map, the target detection object of each of the images to be detected and the predicted positions of the key points of the target detection object are determined.

4. The method according to claim 2, characterized in that: For the same target detection object, the predicted positions of the key points of the target detection object in each of the to-be-detected images are input into the second target model, and the category recognition result output by the second target model is obtained, including: Obtaining the predicted position of the target key point from the predicted positions of each of the key points; Obtaining previous state information of the predicted position of the target key point; Determining current state information of the predicted position of the target key point according to the predicted position of the target key point and previous state information of the predicted position of the target key point; Based on the current state information and the historical category recognition results, the category recognition results corresponding to the predicted positions of the key points are determined.

5. The method according to claim 2, characterized in that: Before inputting the image sequence to be detected into the target behavior category recognition model, the method further includes: Acquire a sample image sequence in a power grid production operation; the sample image sequence is annotated with the actual position of each key point of the target detection object and the real power grid operation behavior category of the target detection object; The sample image sequence is input into the initial behavior category recognition model to obtain the sample position prediction result and the sample power grid operation behavior category output by the initial behavior category recognition model; wherein the sample position prediction result includes the sample prediction position of each key point of the target detection object of the sample image sequence; the sample power grid operation behavior category is used to determine the power grid operation behavior category of the target detection object of the sample image sequence; Determining a total loss value according to the sample predicted position, the sample power grid operation behavior category, the actual position and the real power grid operation behavior category; The parameters of the initial behavior category recognition model are adjusted according to the total loss value until the total loss value meets a preset condition, and the initial behavior category recognition model is determined as the target behavior category recognition model.

6. The method according to claim 5, characterized in that The total loss value includes a first loss value and a second loss value; Determining a total loss value according to the sample predicted position, the sample power grid operation behavior category, the actual position and the real power grid operation behavior category includes: Determining the first loss value according to the sample predicted position and the actual position; Determining the second loss value according to the sample power grid operation behavior category and the real power grid operation behavior category; The total loss value is determined based on the first loss value and the second loss value.

7. The method according to claim 6, characterized in that The determining the first loss value according to the sample predicted position and the actual position includes: Determining a position loss value according to the sample predicted position and the actual position; Obtaining the confidence of the sample predicted position and the confidence of the actual position, and determining a confidence loss value according to the confidence of the sample predicted position and the confidence of the actual position; The first loss value is determined based on the position loss value and the confidence loss value.

8. A behavior category recognition device, characterized in that: The device comprises: A first acquisition module is used to acquire a sequence of images to be detected in a power grid production operation; A processing module is used to input the image sequence to be detected into a target behavior category recognition model, and obtain a category recognition result output by the target behavior category recognition model, wherein the category recognition result is used to determine the power grid operation behavior category of the target detection object in the image sequence to be detected, wherein the target behavior category recognition model is trained based on a sample image sequence in power grid production operations.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images