Method for constructing live working unsafe behavior monitoring data set of 10kV boom bucket truck

Abstract

The invention provides a method for constructing an unsafe behavior monitoring data set for live-line work of a 10kV boom lorry, and relates to the technical field of electric power safety, and the method comprises the steps: collecting video data of unsafe behaviors in the live-line work process of the 10kV boom lorry from multiple perspectives; training a first target detection model by using the obtained first basic data set, and performing target identification on the first basic data set by using the model; performing label correction on the first basic data set according to an identification result of the first target detection model and a corresponding label; training a second target detection model by using the obtained second basic data set; carrying out framing processing on the first basic data set by taking each frame as an interval and then sending the processed data into a second target detection model to carry out target identification so as to obtain a third basic data set formed by key frame image data; and merging the second basic data set and the third basic data set to obtain a monitoring data set. According to the scheme, the rich and effective monitoring data set can be obtained, so that the reliability of the trained monitoring model is improved.

Description

Technical Field

[0001] This invention relates to the field of power safety technology, and in particular to a method for constructing a dataset for monitoring unsafe behaviors of 10kV boom trucks operating on energized lines. Background Technology

[0002] With the continuous growth of electricity demand, accelerating the construction and upgrading of power facilities and strengthening their maintenance have become increasingly important. Insulated bucket trucks, with their advantages of high safety, high operating efficiency, and the ability to perform a wide range of tasks, are widely used in the operation and maintenance of power facilities under live-line working conditions. However, although insulated bucket trucks themselves offer high safety, unsafe and non-standard behaviors by operators frequently occur during operation. Therefore, deep learning-based unsafe behavior monitoring systems are typically used to identify the wearing of protective equipment and dangerous behaviors of personnel working in 10kV bucket truck live-line working environments.

[0003] However, current methods for creating datasets for monitoring unsafe behaviors in live-line working scenarios primarily rely on traditional video frame extraction techniques, such as frame-by-frame extraction and fixed-interval frame extraction. These methods have significant limitations in practical applications: they struggle to effectively extract key image samples from videos, leading to insufficient sample validity in the constructed datasets. Furthermore, the low efficiency of video data processing results in poor training performance for deep learning-based unsafe behavior monitoring models. This is especially true in 10kV bucket truck live-line working scenarios, where live-line workers adhere to strict operational standards. Existing dataset construction methods struggle to obtain rich and effective data information, resulting in poor reliability of the monitoring models. Summary of the Invention

[0004] In view of this, and to address the above shortcomings, it is necessary to propose a method for constructing a monitoring dataset for unsafe behaviors of 10kV boom trucks operating on live lines, so as to obtain rich and effective data to construct the monitoring dataset and thereby improve the reliability of the monitoring model trained using the monitoring dataset.

[0005] In a first aspect, the present invention provides a method for constructing a dataset for monitoring unsafe behaviors during live-line operations of a 10kV bucket truck, comprising: The first basic dataset is constructed by collecting video data from multiple perspectives of unsafe behaviors during the live-line operation of a 10kV bucket truck. A first target detection model is trained using the first basic dataset, and the trained first target detection model is used to identify targets in the first basic dataset; wherein, the first target detection model is able to identify protective gear and work tools in the input video data; The first basic dataset is labeled according to the recognition results and corresponding labels of the first object detection model to obtain the second basic dataset; The second object detection model is trained using the second basic dataset; After dividing the first basic dataset into frames at intervals of one frame, each frame image is fed into the second target detection model for target recognition, resulting in a third basic dataset consisting of key frame image data at the time of the unsafe act and before and after the act. The second and third basic datasets are merged to obtain a monitoring dataset of unsafe behaviors of 10kV boom trucks operating on live lines.

[0006] Preferably, the video data collected from multiple perspectives during the live-line operation of the 10kV bucket truck includes: Video data of unsafe behaviors are collected from both ground-level and aerial perspectives; the ground-level perspectives include upward and horizontal views, and the aerial perspectives include downward and horizontal views; data are collected from all frontal and lateral positions of each perspective.

[0007] Preferably, training the first object detection model using the first basic dataset includes: Based on the temporal characteristics of each unsafe behavior, the first basic dataset is segmented into frames using the corresponding frame intervals to obtain the first basic framed dataset. The first basic frame dataset is manually annotated using an image annotation tool; the annotated categories include protective equipment and work tools. The image data and corresponding label data in the first basic frame-segmented dataset are divided into a training set and a validation set according to a certain ratio; The target detection model is trained using the training set and validated using the validation set to obtain the first target detection model.

[0008] Preferably, the labeling categories include: wearing a helmet, not wearing a helmet, wearing goggles, not wearing goggles, wearing a shawl, not wearing a shawl, wearing gloves, not wearing gloves, and bucket truck.

[0009] Preferably, the step of correcting the labels of the first basic dataset based on the recognition results of the first object detection model and the corresponding labels includes: For each image in the first basic dataset, perform the following: The target results identified by the first target detection model are compared with the corresponding manually labeled tags; If the identified target result differs from the label, the current image will be manually re-labeled.

[0010] Preferably, the step of feeding each frame of image into the second target detection model for target recognition to obtain a third basic dataset consisting of keyframe image data at the time of the unsafe act and before and after its occurrence includes: Each frame of image is fed into the second object detection model for object recognition; When a change in a critical state is detected, the current frame image, as well as the image frame corresponding to the previous second and the image frame corresponding to the next second, are saved. The saved current frame image, the previous second image frame, and the next second image frame are determined as the third basic dataset.

[0011] Preferably, before determining the saved current frame image, the previous second image frame, and the next second image frame as the third basic dataset, the method further includes: The recognition results of the current frame image, the previous second image frame, and the next second image frame are compared with the corresponding label information; if there is a difference between the recognition results and the label information, the label is manually corrected.

[0012] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the methods described in the first aspect.

[0013] Thirdly, the present invention provides a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the first aspects.

[0014] As can be seen from the above technical solution, the method for constructing a monitoring dataset for unsafe behaviors during live-line operation of a 10kV bucket truck provided in this embodiment of the invention first collects video data of unsafe behaviors during live-line operation of a 10kV bucket truck from multiple perspectives. Then, a first target detection model is trained using the obtained first basic dataset, and the trained model is used to identify targets in the first basic dataset. Further, the labels of the first basic dataset are corrected according to the identification results and corresponding labels to obtain a second basic dataset. Then, a second target detection model is trained using the second basic dataset, and the first basic dataset is processed by dividing it into frames at intervals of one frame. Each frame image is then fed into the second target detection model for target identification, resulting in a third basic dataset of key image data corresponding to the occurrence of unsafe behaviors and before and after their occurrence. Finally, the obtained second and third basic datasets are merged to construct a monitoring dataset for unsafe behaviors during live-line operation of a 10kV bucket truck. As can be seen from the above technical solution, this solution uses a target detection model to assist in labeling, and the verification method not only effectively saves manpower but also reduces labeling errors, ensuring the accuracy of the dataset. Furthermore, this solution extracts key image data through a second target detection model, making the dataset rich and effective while reducing the amount of invalid data, thereby improving the reliability and efficiency of the monitoring model trained on the monitoring dataset. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a method for constructing a dataset for monitoring unsafe behaviors during live-line operation of a 10kV bucket truck, as provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram illustrating the construction of a dataset for monitoring unsafe behaviors during live-line operation of a 10kV bucket truck, as provided in an embodiment of the present invention. Detailed Implementation

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for constructing a dataset for monitoring unsafe behaviors during live-line operations of a 10kV bucket truck. This method may include the following steps: Step 101: Collect video data of unsafe behaviors during the live-line operation of the 10kV boom truck from multiple perspectives to form the first basic dataset; Step 102: Train a first target detection model using the first basic dataset, and use the trained first target detection model to perform target recognition on the first basic dataset; wherein, the first target detection model is able to identify protective gear and work tools in the input video data; Step 103: Correct the labels of the first basic dataset based on the recognition results and corresponding labels of the first object detection model to obtain the second basic dataset; Step 104: Train the second object detection model using the second basic dataset; Step 105: After dividing the first basic dataset into frames at intervals, each frame image is fed into the second target detection model for target recognition, resulting in a third basic dataset consisting of key frame image data at the time of the unsafe act and before and after the act. Step 106: Merge the second and third basic datasets to obtain the monitoring dataset of unsafe behaviors of 10kV boom trucks operating on live lines.

[0019] like Figure 2 As shown, this solution mainly comprises three parts: video data acquisition, basic dataset construction, optimization, verification, and annotation, and the construction of an unsafe behavior monitoring dataset based on target detection and time-series analysis. The input to the dedicated dataset is video footage containing electrical unsafe work behaviors, and the output is keyframe images of these unsafe behaviors. Specifically, this embodiment uses a target detection model to assist in annotation, and the verification process not only effectively saves manpower but also reduces annotation errors, ensuring the accuracy of the dataset. Furthermore, this solution uses a second target detection model to extract key image data, making the dataset rich and effective while reducing the amount of invalid data, thereby improving the reliability and efficiency of the monitoring model trained on the monitoring dataset.

[0020] For step 101, video data of unsafe behaviors during the live-line operation of the 10kV bucket truck are collected from multiple perspectives to form the first basic dataset. In this step, we will consider using various types of portable cameras to film potential unsafe behaviors of workers during live-line operations with a 10kV bucket truck, such as removing insulated helmets, goggles, and gloves. The captured videos will contain important elements such as protective equipment like insulated helmets, gloves, goggles, and shawls, as well as the bucket truck's operating tools.

[0021] To construct a comprehensive dataset focused on practical operation and capturing unsafe behaviors during live-line work, video data was collected from both ground and aerial perspectives. The ground perspective was further divided into upward and eye-level views, while the aerial perspective included downward and eye-level views. Data was also collected from various frontal and lateral positions within these perspectives to ensure the dataset's multi-angle and comprehensiveness. For the potentially dangerous act of removing personal protective equipment (PPE) while in the boom truck, the process of workers removing PPE from various angles and positions was filmed from the beginning to the complete removal. Furthermore, the footage captured the workers' state before donning any protective gear, the details of donning (the state of each piece of PPE after it was put on), and the overall effect after complete donning.

[0022] For step 102, a first target detection model is trained using the first basic dataset, and the trained first target detection model is used to identify targets in the first basic dataset; wherein, the first target detection model is able to identify protective gear and work tools in the input video data; In this step, training the first object detection model specifically includes: Based on the temporal characteristics of each unsafe behavior, the first basic dataset is segmented into frames using the corresponding framing intervals to obtain the first basic framed dataset. The first basic frame dataset was manually annotated using an image annotation tool; the annotated categories included protective equipment and work tools. The image data and corresponding label data in the first basic frame-segmented dataset are divided into training set and validation set according to a certain ratio; The target detection model is trained using the training set and validated using the validation set to obtain the first target detection model.

[0023] In this embodiment, the construction of the basic dataset provides the data foundation for model training in the subsequent construction of specialized datasets. During video framing, using shorter framing intervals allows for the acquisition of more frames from the video, providing more detailed information and facilitating the analysis of dangerous behavior processes. However, this also generates a large amount of redundant or unnecessary data, significantly increasing the annotation burden. Conversely, while longer framing intervals can significantly reduce the amount of data, they also lead to the loss of keyframe information, especially when the video content involves rapid changes or requires capturing subtle movements; the selection of the framing interval is particularly important in these cases. Therefore, when creating the basic dataset, the framing interval was flexibly adjusted according to the temporal characteristics of different behaviors, ensuring that the image samples in the basic dataset are neither overly redundant nor overly representative. The resulting first basic framing dataset is denoted by D0.

[0024] After acquiring a number of image data, the image data annotation process begins. First, the image data is manually annotated with bounding boxes on the Make Sense platform. The annotation categories cover protective equipment and work tools, specifically including: wearing a helmet (helmet), not wearing a helmet (no_helmet), wearing goggles (goggle), not wearing goggles (no_goggle), wearing a shawl (shawl), not wearing a shawl (no_shawl), wearing gloves (gloves), not wearing gloves (no_gloves), and bucket.

[0025] Next, the verification and annotation process is carried out. The image data and their corresponding label data in the first basic frame-segmented dataset D0 are divided into training and validation sets according to a certain ratio. Then, an effective object detection model (such as YOLOv8) is selected, and the object detection model M is trained using the training set. The model is then validated using the validation set data to determine the parameters of model M.

[0026] In this step, the trained object detection model is used to identify the first basic frame dataset D0. It can identify labeling errors caused by human factors, such as mislabeling or omissions, and then further correct the labels to make the second basic dataset more accurate.

[0027] For step 103, the labels of the first basic dataset are corrected according to the recognition results and corresponding labels of the first target detection model to obtain the second basic dataset; In this step, when correcting the labels of the dataset, for each image in the first base dataset, the target result identified by the first object detection model is compared with the corresponding manually labeled image. If there is a difference between the identified target result and the label, the current image is re-labeled manually. Specifically, this embodiment cleverly utilizes the target model M to extract targets from the images in the base dataset D0, and then compares the predicted target results in the image with the corresponding manually labeled images. If a difference is found between the predicted result and the label, it means that the image needs to be manually verified again to determine if the label is correct. If it is incorrect, the label needs to be corrected. By traversing all images in D0, label verification can be performed efficiently. This method replaces the tedious work of completely manual verification, effectively saving manpower and reducing labeling errors. The second base dataset obtained after verification and correction is denoted as D1.

[0028] Since the basic dataset construction method extracts image frames from video data at certain frame intervals, although this method extracts a large amount of image data, it has a drawback: it may miss keyframes in the video. Therefore, we consider using a keyframe localization method based on object detection and temporal analysis to further obtain the dataset, as detailed in steps 104-105 below.

[0029] For step 104, the second object detection model is trained using the second basic dataset; In this step, we consider retraining the object detection model using the validated and corrected base dataset D1 to obtain a new model Mnew. It should be noted that since D1 is superior to D0, model M... new It will be superior to model M.

[0030] For step 105, after the first basic dataset is divided into frames at intervals of one frame, each frame image is sent to the second target detection model for target recognition, resulting in a third basic dataset consisting of key frame image data when the unsafe behavior occurs and before and after it occurs. In this step, when acquiring the third basic dataset, each frame image can be sent to the second target detection model for target recognition; when a key state change is detected, the current frame image, as well as the image frame corresponding to the previous second and the image frame corresponding to the next second are saved; the saved current frame image, the image frame corresponding to the previous second, and the image frame corresponding to the next second are determined as the third basic dataset.

[0031] In this embodiment, the changes in the target state corresponding to the occurrence of dangerous behaviors are determined in time, based on the specifications for live-line work with 10kV bucket trucks. For example, the process of a worker removing their helmet involves detecting the temporal change from wearing a helmet to not wearing one. First, the video data containing the dangerous behavior is processed frame by frame. Then, each frame is fed into the target detection model M. new Target prediction is performed. When a critical state change is identified, the corresponding image frame (i.e., the one in which the dangerous behavior is occurring), the image frame one second prior (before the dangerous behavior), and the image frame one second later (after the dangerous behavior) are saved. This method significantly reduces useless images in the dataset, retaining only the most critical image data, thus reducing the storage size from dozens of images in traditional methods to just a few. Furthermore, the image data obtained using this method does not require extensive manual annotation; it can be processed using the target detection model M. new The model is used to assist in annotation, and then manually verified. If the verification is successful, the model annotation information is retained; if the verification is incorrect, the annotation information is manually corrected. The third basic dataset of the resulting keyframe image dataset is denoted as D2.

[0032] Of course, it's easy to understand that the first basic dataset used in this step and the basic dataset used in step 102 can be the same set of data. In another, more preferred embodiment, the first basic dataset used in step 102 and the first basic dataset used in this step are merged to form the final first basic dataset. For example, if the first basic dataset includes 100 sets, and step 102 uses the first 40 sets, then this step uses the remaining 60 sets. This makes the constructed monitoring dataset more comprehensive and the trained model more robust.

[0033] For step 106, the second basic dataset and the third basic dataset are merged to obtain the monitoring dataset of unsafe behaviors of 10kV boom trucks operating on live lines.

[0034] In this step, datasets D1 and D2 are merged to obtain a high-quality dedicated image dataset D3 for monitoring unsafe behaviors during live-line operations of 10kV bucket trucks. Table 1 shows the statistical information on the number of each category in dataset D3.

[0035] Table 1. Statistical Information on the Quantity of Protective Equipment and Work Tools by Category

[0036] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods in any of the embodiments of the specification.

[0037] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any of the embodiments of the specification.

[0038] The device embodiments provided by the present invention are based on the same inventive concept as the method embodiments in this specification. For details, please refer to the description in the method embodiments of this specification, which will not be repeated here.

[0039] The modules or units in the device of this invention can be merged, divided, and deleted according to actual needs. The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that implementing all or part of the processes of the above embodiments and making equivalent changes according to the claims of this invention still fall within the scope of the invention.

Claims

1. A method for constructing a 10kV boom truck live working unsafe behavior monitoring data set, characterized in that: include: Video data of unsafe behaviors during live operation of 10kV boom trucks are collected from multiple perspectives to form the first basic data set; Using the first basic data set to train a first target detection model, and using the trained first target detection model to perform target recognition on the first basic data set; wherein the first target detection model is capable of recognizing protective gear and work tools in the input video data; Performing label correction on the first basic data set according to the recognition result of the first target detection model and the corresponding label to obtain a second basic data set; Using the second basic data set to train a second target detection model; After the first basic data set is divided into frames at intervals of each frame, each frame of the image is sent to the second target detection model for target recognition, so as to obtain a third basic data set consisting of key frame image data when the unsafe behavior occurs and before and after the unsafe behavior occurs; The second basic data set and the third basic data set are combined to obtain a monitoring data set of unsafe behaviors of live working of 10kV boom trucks.

2. The method for constructing a 10kV boom truck live working unsafe behavior monitoring data set according to claim 1, characterized in that: The video data of unsafe behaviors during live operation of 10kV boom truck collected from multiple perspectives includes: The video data of unsafe behaviors are collected from the ground perspective and the aerial perspective respectively; wherein the ground perspective includes an upward perspective and a horizontal perspective, and the aerial perspective includes a downward perspective and a horizontal perspective; and data are collected from each frontal orientation and side orientation of each perspective.

3. The method for constructing a 10kV boom truck live working unsafe behavior monitoring data set according to claim 1, characterized in that: The using the first basic data set to train a first target detection model includes: According to the time characteristics of each unsafe behavior, the first basic data set is framed using a corresponding frame interval to obtain a first basic frame data set; Manually annotating the first basic frame data set on an image annotation tool; wherein the annotated categories include protective gear and work tools; Dividing the image data and the corresponding label data in the first basic frame data set into a training set and a validation set according to a certain ratio; The target detection model is trained using the training set, and the model is verified using the verification set to obtain the first target detection model.

4. The method for constructing a 10kV boom truck live working unsafe behavior monitoring data set according to claim 3, characterized in that: The labeling categories include: wearing helmet, not wearing helmet, wearing goggles, not wearing goggles, wearing cape, not wearing cape, wearing gloves, not wearing gloves, and boom truck.

5. The method for constructing a 10kV boom truck live working unsafe behavior monitoring data set according to claim 3, characterized in that: The step of performing label correction on the first basic data set according to the recognition result of the first target detection model and the corresponding label includes: For each picture in the first basic data set, execute: Compare the target result identified by the first target detection model with the corresponding manually annotated label; If the recognized target result is different from the label, the current image will be manually re-labeled.

6. The method for constructing a 10kV boom truck live working unsafe behavior monitoring data set according to claim 1, characterized in that: The step of sending each frame of image to the second target detection model for target recognition to obtain a third basic data set consisting of key frame image data when the unsafe behavior occurs and before and after the unsafe behavior occurs, including: Send each frame of image to the second target detection model for target recognition; When a change in the key state is detected, the current frame image, as well as the image frame corresponding to the previous second and the image frame corresponding to the next second of the frame are saved; The saved current frame image, the previous second image frame and the next second image frame are determined as the third basic data set.

7. The method for constructing a 10kV boom truck live working unsafe behavior monitoring data set according to claim 6, characterized in that: Before determining the saved current frame image, the previous second image frame and the next second image frame as the third basic data set, the method further includes: The recognition results of the current frame image, the image frame of the previous second, and the image frame of the next second are compared with the corresponding label information; if there is a difference between the recognition result and the label information, the label is manually corrected.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

9. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Images