On-pole operation insulation shielding detection method and intelligent hot-line operation evaluation method and system

Abstract

The invention provides a pole operation insulation shielding detection method, which comprises the following steps of: acquiring a field video frame, carrying out key point detection on each frame of video image by utilizing a key point detection model to obtain coordinates of the top of a wire pole, the left end of a cross arm and the right end of the cross arm, generating a key point enclosing rectangle based on coordinate expansion of the top of the wire pole, the left end of the cross arm and the right end of the cross arm; and performing identification and contour extraction of the cable and the cable insulation shielding tool based on the instance segmentation model. On the basis of the on-pole rotating frame detection model, performing pole insulation shielding tool identification on the pole image in the key point external rectangle, and obtaining the insulation shielding state of the pole and the cross arm according to the identification result; and judging the insulation shielding state of the cable according to the cable contour, the cable insulation shielding tool contour and the relative position of the key point external rectangle.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent assessment of live working, and in particular to a method for detecting insulation shielding during pole working, and a method and system for intelligent assessment of live working. Background Art

[0002] Live working is an extremely complex process. The exam primarily takes place on utility poles several meters above the ground, and the assessment relies heavily on the subjective judgment of the instructor. This makes it even more difficult for the instructor to visually observe the trainee's clothing and a series of manual actions, leading to a need for improved assessment efficiency. This method is subject to significant subjectivity, observation errors, difficulty tracking progress, safety concerns, low efficiency, incomplete data records, and delayed feedback.

[0003] CN202010684749.8 provides a safety detection method for live power operations based on a deep learning algorithm. First, a human reference object with a regular shape, obvious features, and easy detection is set on the operator's work clothes; then the camera's configuration file is read to obtain the actual height H of the human reference object, the human activity radius r1, and the equipment parameters; then the historical monitoring video of the camera is extracted, and the human target detection model and the human reference object detection model are trained through the historical monitoring video; finally, the detection model is used to realize human target tracking, find moving human targets, and complete safety distance detection, thereby completing the safety detection of live power operations of operators.

[0004] However, in reality, live working includes ground-based work and pole-based work, and different forms of work have different evaluation standards. Therefore, a method and system that can comprehensively evaluate the entire process of live working is needed.

[0005] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the Invention

[0006] Based on this, it is necessary to provide a deep learning-based pole operation insulation shielding detection method and a live operation intelligent assessment method and system to address the above technical problems, so as to realize automatic evaluation and real-time monitoring of the operating skills of live operation personnel, and improve operation safety and assessment efficiency.

[0007] In order to achieve the above object, the first aspect of the present invention provides a method for detecting insulation shielding during pole work, comprising the following steps: Acquire live video frames. For each frame, use a key point detection model to detect key points, obtain the coordinates of the pole top, left crossarm, and right crossarm, and then expand and generate key point circumscribed rectangles based on the coordinates of the pole top, left crossarm, and right crossarm, respectively. Finally, use an instance segmentation model to identify and extract the contours of cables and cable insulation shielding devices. Based on the rotating frame detection model on the pole, the pole image within the circumscribed rectangle of the key points is used to identify the insulation shielding devices of the pole, and the insulation shielding status of the pole and crossarm is obtained based on the recognition results. The insulation shielding status of the cable is determined based on the relative positions of the cable outline, the cable insulation shielding device outline and the circumscribed rectangle of the key points.

[0008] In a possible embodiment of the first aspect, determining the insulation shielding state of the cable according to the relative positions of the cable contour, the cable insulation shielding device contour, and the circumscribed rectangle of the key points includes: Fitting each cable profile to obtain a fitted cable line; Determine the key point circumscribed rectangle associated with the fitted cable line according to the positional relationship between the fitted cable line and each key point circumscribed rectangle; Calculate the distance from the center point of the cable insulation shielding device outline to the circumscribed rectangle of each key point, and determine the cable where the cable insulation shielding device is located based on the distance.

[0009] In a possible embodiment of the first aspect, the rotating box on the pole detection model uses the PP-YOLOE-R model as a baseline model, predicts four parameters (center point coordinate x, center point coordinate y, target height h, target width w) through a regression branch, and adds an angle prediction branch to predict the target angle.

[0010] In order to achieve the above-mentioned object, the second aspect of the present invention provides a method for intelligent assessment of live working based on deep learning, comprising the following steps: Obtain the live video frames collected when the examinee is working on the pole; Based on the pole operation insulation shielding detection method described in the first aspect, each frame of the on-site video frame is identified and detected to obtain the shielding conditions of the poles, crossarms and cables; Use target tracking algorithm to track the examinee and capture the time sequence of video frames when the examinee is working on the pole; The video frame time series sequence is fed into the action detection model to perform time series action detection and obtain the time series action; The candidate's pole work score is obtained based on the preset pole work scoring rules, the identification results of the shielding conditions of the poles, crossarms and cables, and the candidate's timed actions on the pole.

[0011] In a possible embodiment of the second aspect, before obtaining the live video frames collected when the examinee is working on the pole, the live video frames collected when the examinee is working on the ground are first obtained; Perform target detection on each video frame based on the ground rotating frame detection model to identify tools and protective equipment; Use target tracking algorithm to track the examinee and capture the time sequence of video frames during the examinee's ground work; The video frame time series sequence is fed into the action detection model to perform time series action detection and obtain the time series action; The candidate's ground work score is obtained based on the preset ground work scoring rules, the identification results of tools and protective equipment, and the candidate's ground sequential movements.

[0012] To achieve the above objectives, the present invention provides a third aspect of a live working intelligent assessment system based on deep learning, comprising: The video acquisition module is used to obtain the on-site video frames collected by the candidates during their ground and pole work; The ground rotating frame detection model is used to detect targets in each frame of video images during ground operations and identify tools and protective equipment; A key point detection model is used to detect key points in each frame of video image using the key point detection model to obtain the coordinates of the pole top, the left end of the crossarm, and the right end of the crossarm, and to generate a key point circumscribed rectangle based on the coordinates of the pole top, the left end of the crossarm, and the right end of the crossarm; Instance segmentation module, used to identify and extract the contours of cables and cable insulation shielding devices based on the instance segmentation model; The rotating frame detection model on the pole is used to identify the insulation shielding devices of the pole within the circumscribed rectangle of the key points; The shielding recognition module is used to obtain the insulation shielding status of the pole and crossarm based on the identification results of the pole insulation shielding device, and to determine the insulation shielding status of the cable based on the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points; The tracking video capture module is used to track the examinee using a target tracking algorithm and capture the time sequence of video frames during the examinee's ground and pole work; A real-time video analysis module is used to perform temporal motion detection on the input video frame temporal sequence to obtain temporal motion; The intelligent detection module is used to obtain the candidate's pole work score based on the preset pole work scoring rules, the insulation shielding status identification results of the poles and crossarms, and the candidate's pole-based sequential movements, and to obtain the candidate's ground work score based on the preset ground work scoring rules, the tools and protective equipment identification results, and the candidate's ground-based sequential movements.

[0013] To achieve the above-mentioned object, a fourth aspect of the present invention provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; Memory for storing computer programs; The processor is used to implement the insulation shielding detection method for pole operations as described in the first aspect, or the intelligent assessment method for live operations as described in the second aspect, when executing the program stored in the memory.

[0014] In order to achieve the above-mentioned objectives, the fifth aspect of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the insulation shielding detection method for pole operations as described in the first aspect, or implements the intelligent assessment method for live operations as described in the second aspect.

[0015] In order to achieve the above-mentioned objectives, the sixth aspect of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the pole-mounted operation insulation shielding detection method as described in the first aspect, or implements the live operation intelligent assessment method as described in the second aspect.

[0016] The beneficial effects of the present invention are: The present invention uses a key point detection model to detect key points of utility poles and obtain a circumscribed rectangle of the key points, classifies and identifies utility pole images within the circumscribed rectangle of the key points, and determines whether the utility pole is obscured based on the classification results. Furthermore, based on an instance segmentation model, the present invention identifies and extracts the outline of the cable and the outline of the cable insulation shielding device in each frame of the video image. The present invention determines the insulation shielding status of the cable based on the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points. Through the above operations, it is determined whether the examinee has correctly performed the shielding operation during live work. The present invention also tracks the examinee's sequential movements to identify the examinee; obtains the examinee's pole-based exercise score based on the preset pole-based exercise scoring rules, the identification results of the shielding conditions of the poles and cables, and the examinee's sequential movements on the pole; Furthermore, the present invention also obtains on-site video frames collected when the examinee is working on the ground before the examinee works on the pole; performs target detection on each frame of video image to identify tools and protective equipment; uses a target tracking algorithm to track the examinee and intercepts the time sequence of video frames when the examinee is working on the ground; sends the time sequence of video frames to the action detection model to perform time sequence action detection to obtain time sequence action; obtains the examinee's ground operation score based on preset ground operation scoring rules, tool and protective equipment identification results, and the examinee's ground time sequence action.

[0017] This invention identifies and analyzes personnel protection, tool selection and operation, and job completion confirmation in real time during each operation step, enabling operators to memorize standard operating procedures. Furthermore, the deep learning-based intelligent live work assessment system supports multiple exam topics or types, enabling automated assessment and real-time monitoring of live work personnel's operational skills, improving operational safety and assessment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flow chart of the insulation shielding detection method for pole operation.

[0019] Figure 2 It is the result diagram of key point detection of the present invention.

[0020] Figure 3 This is the result diagram of cable contour identification by the present invention.

[0021] Figure 4 It is a flow chart of the intelligent assessment method for live working of the present invention.

[0022] Figure 5 It is a flow chart of real-time video analysis of the present invention.

[0023] Figure 6 It is a flow chart of the intelligent assessment method for live working of the present invention.

[0024] Figure 7 It is a hardware structure diagram of the present invention.

[0025] Figure 8 It is a structural schematic diagram of the computer device of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0027] Example 1 This embodiment provides a method for detecting insulation shielding during pole operation. Figure 1 As shown, the following steps are included: Acquire live video frames. For each frame, use a key point detection model to detect key points, obtain the coordinates of the pole top, left crossarm, and right crossarm, and then expand and generate key point circumscribed rectangles based on the coordinates of the pole top, left crossarm, and right crossarm, respectively. Finally, use an instance segmentation model to identify and extract the contours of cables and cable insulation shielding devices. Based on the rotating frame detection model on the pole, the pole image within the circumscribed rectangle of the key points is used to identify the insulation shielding devices of the pole, and the insulation shielding status of the pole and crossarm is obtained based on the recognition results. The insulation shielding status of the cable is determined based on the relative positions of the cable outline, the cable insulation shielding device outline and the circumscribed rectangle of the key points.

[0028] The key point detection model refers to the deep learning model used in human posture estimation, facial recognition, gesture recognition, etc. Preferably, the deep learning model uses YOLOv8-pose. It is understandable that other models can also be used, such as the tinypose model of PaddlePaddle.

[0029] It's important to note that before training a keypoint detection model, you need to prepare an image dataset and corresponding annotations. The annotations should include the object's bounding box coordinates and keypoint coordinates. For example, if we want to detect the top and crossarms of a utility pole, we need to assign at least three keypoints to each pole: two at the top and two at each end of the crossarm. The coordinates and visibility information for these keypoints should also be provided. This annotated dataset is then used to train the deep learning model.

[0030] It should be noted that in order to further accurately obtain the relative position relationship of the crossarm, cable and insulation shielding equipment, 5 key points can be set, such as Figure 2 As shown. The five key points include the three key points mentioned above, as well as the intersection of the crossarm and the pole, and an auxiliary key point set on the pole and located below the crossarm. These five key points form a cross shape. Using this key point information, it can be determined whether insulation shielding devices are installed on the pole and crossarm. At the same time, based on this key point information, it can be determined whether the cable insulation shielding device is located on the left or right side of the cable line, or on both sides, thereby determining the target area for the subsequent instance segmentation model of the cable and the cable insulation shielding device. For example, when it is determined that the cable insulation shielding device is located on the left side of the cable line, the image of the left side of the cable line is extracted for instance segmentation.

[0031] Furthermore, to ensure student safety and operational accuracy during the exam, a cable and cable insulation shielding identification step was designed. The core of this step is to utilize an instance segmentation model from deep learning to perform a detailed analysis of the images of the on-site work. This high-precision model accurately identifies the outlines of each cable and cable insulation shielding. It should be noted that any existing instance segmentation model can be used here.

[0032] Specifically, each frame of the acquired live video frame is passed into the instance segmentation model. After the instance segmentation model is inferred, the output result is a grayscale image of the same height and width, in which different categories of cables have different pixel values, and the black one corresponds to the background (0 pixel value). OpenCV is used to extract the contour of the output result of the instance segmentation model to obtain a binary image of each cable, and then a linear regression function is calculated on it to obtain the cable recognition result, as shown below. Figure 3 It should be noted that the above instance segmentation model can also realize the recognition of cable insulation shielding tools on the cable, and then use OpenCV to extract the outline of the cable insulation shielding tools.

[0033] It's important to note that the identification and contour extraction of cables and cable insulation shielding devices provides technical support for the subsequent insulation shielding identification step. This process not only enhances a comprehensive understanding of cable shielding conditions but also provides a solid data foundation and technical support for subsequent insulation shielding identification and judgment. The efficient operation of the instance segmentation model ensures both operational safety for trainees and the fairness of the examination. It also provides an innovative technical solution for standardized training and assessment of power operations.

[0034] Finally, the insulation shielding status of the cable is determined based on the relative positions of the cable outline, the cable insulation shielding device outline and the circumscribed rectangle of the key points.

[0035] Specifically, the insulation shielding status of the cable is judged according to the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points, including: Fitting each cable profile to obtain a fitted cable line; Determine the key point circumscribed rectangle associated with the fitted cable line according to the positional relationship between the fitted cable line and each key point circumscribed rectangle; Calculate the distance from the center point of the cable insulation shielding device outline to the circumscribed rectangle of each key point, and determine the cable where the cable insulation shielding device is located based on the distance.

[0036] Specifically, determine along which fitted cable line the key point circumscribed rectangle lies, i.e., determine that the key point circumscribed rectangle is the key point circumscribed rectangle associated with the corresponding fitted cable line. Alternatively, determine that the highest of the three fitted cable lines is the cable line at the top of the pole, and associate it with the key point circumscribed rectangle formed by the pole vertex; determine that the middle of the three fitted cable lines is the cable line at the right end of the crossarm, and associate it with the key point circumscribed rectangle formed by the right end of the crossarm; determine that the lowest of the three fitted cable lines is the cable line at the left end of the crossarm, and associate it with the key point circumscribed rectangle formed by the right end of the crossarm; and associate it with the key point circumscribed rectangle formed by the right end of the crossarm.

[0037] It can be understood that the cable line associated with the circumscribed rectangle of the key point closest to the outline of the insulation shielding device is the cable line where the outline of the cable insulation shielding device is located, and this cable line is determined to be shielded; while the cable line without the outline of the cable insulation shielding device is determined to be unshielded. In particular, in one embodiment, the cable insulation shielding device is an insulation shielding tube.

[0038] In a specific implementation, the pole rotating frame detection model is a rotating frame detection model based on a convolutional neural network, preferably the PP-YOLOE-R model of YOLOv5, and its training data set is a large number of images of pole insulation shielding devices. Specifically, the pole insulation shielding devices include insulating baffles for shielding objects, lead shielding covers, insulating blankets, tension crossarm shielding covers, insulator shielding covers, special shielding covers, etc.

[0039] Specifically, the PP-YOLOE-R model is a special convolutional neural network.

[0040] The architecture of a conventional convolutional neural network (such as YOLOv5) consists of a backbone, a neck, and a head. The backbone is the main part of the network, responsible for extracting features from the input image. The neck, located between the backbone and the head, connects and transforms features and typically incorporates a feature pyramid network (FPN) or other feature fusion techniques to integrate multi-scale feature maps from the backbone. The head is the output of the network, responsible for making the final prediction based on the extracted features. It typically includes multiple convolutional layers to predict bounding box coordinates, object confidence, and category probabilities. The main difference between the PP-YOLOE-R model and conventional horizontal bounding box detection methods like YOLOv5 is the head network.

[0041] Most rotated target detection algorithms predict five parameters (center point coordinate x, center point coordinate y, target height h, target width w, target angle θ) in the regression branch of the Head to predict a rotated rectangular box; the PP-YOLOE-R model takes into account that θ may require different features from the other four coordinates, so it designs another angle prediction branch and uses Focal Loss to predict the angle; that is, the regression branch predicts four parameters (center point coordinate x, center point coordinate y, target height h, target width w), and the angle prediction branch is used to predict the target angle θ.

[0042] Among them, the formula of Focal Loss is L FL =− α t(1− p t ) γ log( p t ),in p t is the class probability predicted by the model. α t is the weight to balance positive and negative samples, γ It is a parameter that adjusts the weight of easy-to-classify samples.

[0043] It should be noted that the present invention uses a key point detection model to detect the key points of the utility pole and obtain the key point circumscribed rectangle, classifies and identifies the utility pole image within the key point circumscribed rectangle, and determines whether the utility pole is obscured based on the classification result; and based on the instance segmentation model, identifies and extracts the cable contour and the cable insulation shielding device contour in each frame of the video image; judges the insulation shielding status of the cable based on the relative position of the cable contour, the cable insulation shielding device contour, and the key point circumscribed rectangle, thereby reducing the interference of human factors, improving the fairness of the examination and the real-time nature of the assessment; and determines whether the examinee has correctly performed the shielding operation during the live operation through the above operations. In addition, when improper shielding or incorrect item selection occurs, the system will give an error message and record it, and the operator will make shielding corrections according to the prompts, thereby improving the student's operational safety.

[0044] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0045] Example 2 The difference between this embodiment and embodiment 1 is that: a live working intelligent assessment method based on deep learning is provided. Figure 4 As shown, the following steps are included: Obtain the live video frames collected when the examinee is working on the pole; Based on the pole operation insulation shielding detection method described in Example 1, each frame of the on-site video frame is identified and detected to obtain the shielding status of the poles and cables; Use target tracking algorithm to track the examinee and capture the time sequence of video frames when the examinee is working on the pole; The video frame time series sequence is fed into the action detection model to perform time series action detection and obtain the time series action; The candidate's pole work score is obtained based on the preset pole work scoring rules, the identification results of the shielding conditions of the poles, crossarms and cables, and the candidate's timed actions on the pole.

[0046] It should be noted that the intelligent assessment method for live working includes real-time video analysis steps, such as Figure 5 As shown in the figure, first, an advanced target tracking algorithm is used to accurately locate each candidate in the examination room. Through this process, not only can the real-time location information of the candidates be obtained, but also continuous video clips containing the candidates can be dynamically intercepted.

[0047] When eight consecutive frames are collected, they are grouped into a frame sequence, i.e., a time-series video frame sequence, which is fed into a proprietary motion detection model for in-depth analysis. In one embodiment, the motion model used is the PP-TSMv2 model from PaddleVideo.

[0048] It's important to note that the input to the action detection model here is a sequence of video frames containing temporal information; therefore, how to better utilize temporal information in videos is a key focus of video research. Using a 2D network to extract temporal information serves as the input to the action model, ultimately providing a predicted action category probability. Therefore, this model is actually a classification model, predicting a category for the input video frame sequence.

[0049] This action detection model, meticulously designed and trained, efficiently analyzes dynamic information within frame sequences and identifies specific movements performed by examinees. This process not only enables real-time monitoring of examinee behavior but also provides detailed analysis of activities within the examination room, ensuring the fairness and security of the exam process.

[0050] After the real-time video analysis is completed, the candidate's pole work score is obtained based on the preset pole work scoring rules, the identification results of the shielding conditions of the poles, crossarms and cables, and the candidate's timed movements on the pole.

[0051] Specifically, different standard process steps require insulation shielding for different live parts. As each step is performed, images are used to determine the selection and placement of shielding items such as insulation baffles, lead shields, insulation blankets, cable shields, tension crossarm shields, insulator shields, and specialized shields. If improper shielding or incorrect item selection occurs, the system will generate and record an error message, and the operator will follow the prompts to correct the shielding. For more information on insulation shielding detection, refer to Example 1 and will not be elaborated on here.

[0052] By analyzing live work site video streams in real time, the system monitors operator behavior and operational processes. Standardized action recognition: Intelligent AI-powered action recognition and assessment of operator position, tool usage, hardware selection, and action accuracy are performed. The system also records and stores real-time video of the work process for subsequent skill assessment verification.

[0053] In the live working operation steps, different tools (insulating rod wire cutters, insulating rod socket wrenches, wire clamp installation tools, shielding cover operating rods, insulating locking rods, insulating measuring rods, insulating rod-type cable cleaning brushes, insulating cable strippers, insulating cover installation tools, insulating operating rods, insulating clamps, insulating wire tighteners), measuring instruments (insulating wire diameter measuring instruments, insulation testers, current detectors), and used devices (insulating crossarms, insulating drain wires) are selected and used at each stage. Real-time video analysis can determine whether the tools, instruments or hardware used by the operators are correctly selected based on the operation content, and has operation prompts and error correction functions.

[0054] It can be understood that before starting the test, it is necessary to initialize the system first, connect the communication server and the dynamic multi-threaded control test start and stop module; then start and connect the Socket server; then turn on the camera and load the target model to execute the deep learning-based intelligent assessment method for live working described in this embodiment.

[0055] It is understood that the deep learning-based intelligent assessment method for live working described in this embodiment utilizes a multi-threaded approach: one thread executes the test, while the remaining threads are used to send and receive messages and intelligently delete images. The test thread first determines whether the camera is on. If so, it captures a video frame. It then executes the insulation shielding detection method for pole work described in Example 1, performs real-time video analysis, and sends the test results to the intelligent detection platform for scoring.

[0056] Example 3 The difference between this embodiment and embodiment 2 is that: Figure 6 As shown, the intelligent assessment method for live working also includes a ground stage.

[0057] Specifically, before obtaining the on-site video frames collected when the examinee is working on the pole, first obtain the on-site video frames collected when the examinee is working on the ground; Perform target detection on each video frame based on the ground rotating frame detection model to identify tools and protective equipment; Use target tracking algorithm to track the examinee and capture the time sequence of video frames during the examinee's ground work; The video frame time series sequence is fed into the action detection model to perform time series action detection and obtain the time series action; The candidate's ground work score is obtained based on the preset ground work scoring rules, the identification results of tools and protective equipment, and the candidate's ground sequential movements.

[0058] It should be noted that, similarly, the ground stage includes a real-time video analysis step. The specific analysis steps refer to the analysis steps in Example 2 and will not be repeated here.

[0059] The ground rotating frame detection model uses the same PP-YOLOE-R model as the baseline model of the pole rotating frame detection model, but the training dataset is different. The dataset here contains a large number of tools and protective equipment of various angles and types.

[0060] It's important to note that rotating frame detection models offer significant advantages over conventional horizontal frame detection models in certain scenarios due to their more accurate target positioning, adaptability to various shapes and angles, reduced occlusion errors, and improved detection accuracy. They are particularly well-suited for narrow, long objects such as electroscopes and insulated operating rods. A deep neural network is trained using a large amount of tool image data to identify different types of tools. AI image recognition of workers' insulating shielding and protective gear (such as insulating gloves, helmets, double-protection insulating belts, suits, and blankets) ensures compliance with safety requirements during standard operating procedures. In the event of improper protection or incorrect item selection, the system displays an error message, prompting the worker to make corrective adjustments.

[0061] Example 4 Based on the same inventive concept, embodiments of the present application also provide a deep learning-based intelligent assessment system for live working, which is used to implement the deep learning-based intelligent assessment method for live working. The solution provided by this system is similar to the solution described in the method described above. Therefore, the specific limitations of one or more embodiments of the deep learning-based intelligent assessment system for live working provided below can be found in the limitations of the deep learning-based intelligent assessment method for live working above, and will not be repeated here.

[0062] Specifically, the live working intelligent assessment system based on deep learning includes: The video acquisition module is used to obtain the on-site video frames collected by the candidates during their ground and pole work; The ground rotating frame detection model is used to detect targets in each frame of video images during ground operations and identify tools and protective equipment; A key point detection model is used to detect key points in each frame of video image using the key point detection model to obtain the coordinates of the pole top, the left end of the crossarm, and the right end of the crossarm, and to generate a key point circumscribed rectangle based on the coordinates of the pole top, the left end of the crossarm, and the right end of the crossarm; Instance segmentation module, used to identify and extract the contours of cables and cable insulation shielding devices based on the instance segmentation model; The rotating frame detection model on the pole is used to identify the insulation shielding devices of the pole within the circumscribed rectangle of the key points; The shielding recognition module is used to obtain the insulation shielding status of the pole and crossarm based on the identification results of the pole insulation shielding device, and to determine the insulation shielding status of the cable based on the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points; The tracking video capture module is used to track the examinee using a target tracking algorithm and capture the time sequence of video frames during the examinee's ground and pole work; A real-time video analysis module is used to perform temporal motion detection on the input video frame temporal sequence to obtain temporal motion; The intelligent detection module is used to obtain the candidate's pole work score based on the preset pole work scoring rules, the insulation shielding status identification results of the poles and crossarms, and the candidate's pole-based sequential movements, and to obtain the candidate's ground work score based on the preset ground work scoring rules, the tools and protective equipment identification results, and the candidate's ground-based sequential movements.

[0063] Furthermore, in some embodiments, the deep learning-based intelligent assessment system for live working also includes a data recording and return assessment deduction certificate image module and an intelligent image deletion module.

[0064] The workflow of this embodiment includes: (1) System initialization like Figure 7 As shown, 12 cameras are set up to capture video information of the examination assignments. The cameras used are Dahua DH-SD-6A9630UA-HNI (6-megapixel) infrared network high-speed smart dome camera, which supports 30x optical zoom and 16x digital zoom; HUAWEI MatePad Air 11.5-inch tablet computer; an AI algorithm deployment server configured with an i9 processor, 128G memory, and RTX 4090 NVIDIA graphics card; and an examination platform deployment server.

[0065] First, start this system. After the system service is started, the configuration table and various models needed will be loaded. The configuration table includes the connection server IP, port, rtsp stream addresses of multiple cameras, the path for loading models, and some other control parameters; the loaded models include the ground rotating box detection model, the pole rotating box detection model, the key point detection model, the instance segmentation model, and the video real-time action detection model.

[0066] After initialization is completed, communication with the intelligent detection module must be established to facilitate receiving signals indicating the start and end of the test, as well as sending information about the test process; Create multiple threads, including an exam thread, a message sending and receiving thread, and an automatic image deletion thread. The message sending and receiving thread is used to receive and send exam-related messages; the automatic image deletion thread detects the number of files saved in each folder in the current directory and automatically deletes them. The examination plan is created on the PC examination platform. The examination plan is selected on the tablet, and the examination venue, candidates and other information are confirmed. After clicking to start the examination, the intelligent detection module will receive the examination start information from the tablet, and then send the information to this system.

[0067] When the test start signal from the intelligent detection module is received, the message receiving thread will receive the message and parse it to obtain test-related information, including the test ID and subject ID. The test ID is an identifier of the test number, and the subject ID is used to distinguish which test it is. After distinguishing the test subject, it enters the test thread of the corresponding subject. After the test thread is included, the core algorithm starts to start; it first detects whether the camera is turned on, attempts to connect, and obtains the video frame; and loads the key:value file of the detection items and the algorithm return values; in this way, the target or action finally detected by the system can be returned to the image detection module in the form of the algorithm return value.

[0068] The standard process identification module then identifies the examination process.

[0069] The system sends the image to the standard process recognition module and detects the current operation process, which includes three processes: ground operation, pole operation, and end operation.

[0070] (2) Ground operations Entering the ground tool detection and personal protective equipment detection steps: the acquired video frames are fed into the ground rotating frame target detection model to detect personal protective equipment, insulating tools, and tools used for exam assignments. Combined with the subsequent real-time video analysis module, it determines whether the personal protection is complete and the tools are comprehensive, and determines whether points should be deducted; Next, the real-time video analysis step begins. First, the system applies advanced target tracking algorithms to precisely locate each examinee within the examination room. This process not only captures the examinee's real-time location but also dynamically captures continuous video clips containing the examinee. Once the system collects eight consecutive frames, these are organized into a sequence and fed into a proprietary motion detection model for in-depth analysis, identifying the examinee's specific movements.

[0071] The ground operation detection process ends here, and the data recording module sends the detected valid information (the algorithm code of the detection item and the value of the algorithm code) to the intelligent detection module.

[0072] (3) Pole work First, the acquired video frames are sent to the key point detection model for key point detection; the model will output multiple key points including the top of the pole, the left end of the crossarm, and the right end of the crossarm to determine the position information of the top of the pole and the crossarm. Using this key point information, the position and relative relationship of the crossarm and insulator can be determined, and based on this key point information, it can be judged whether the cable insulation shielding device is on the left or right side of the cable line.

[0073] Next, the test proceeds to the cable identification module. To ensure student safety and operational accuracy during the exam, deep learning-based instance segmentation technology is used to perform detailed analysis of on-site images. This high-precision model accurately identifies the outlines of each cable and cable insulation shielding device.

[0074] Specifically, each frame of the acquired live video frame is passed into the instance segmentation model. After the instance segmentation model is inferred, the output result is a grayscale image of the same height and width, in which different categories of cables have different pixel values, and the black one corresponds to the background (0 pixel value). OpenCV is used to extract the contour of the output result of the instance segmentation model to obtain a binary image of each cable, and then a linear regression function is calculated on it. Specifically, Figure 8 It should be noted that the above instance segmentation model can also realize the recognition of cable insulation shielding tools on the cable, and then use OpenCV to extract the outline of the cable insulation shielding tools.

[0075] Then enter the insulation shielding recognition module to determine the insulation shielding status of the cable based on the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points; Specifically, the insulation shielding status of the cable is judged according to the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points, including: Fitting each cable profile to obtain a fitted cable line; Determine the key point circumscribed rectangle associated with the fitted cable line according to the positional relationship between the fitted cable line and each key point circumscribed rectangle; Calculate the distance from the center point of the cable insulation shielding device outline to the circumscribed rectangle of each key point, and determine the cable where the cable insulation shielding device is located based on the distance.

[0076] After the insulation shielding recognition is completed, the system enters the real-time video analysis module. The system uses the action detection model to analyze the image sequence to determine whether the trainee's action is correct. The on-pole detection process ends here, and the detection results are sent back to the intelligent detection end (server).

[0077] (4) End the work When the standard process identification module recognizes the end of the work process, it means that the exam is over.

[0078] After the exam, the intelligent evaluation report generation module will generate an intelligent scoring report; The scoring report can be viewed on both the tablet and PC examination platforms.

[0079] Example 5 This embodiment provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication, which can be achieved via Wi-Fi, mobile cellular networks, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements the insulation shielding detection method for pole-mounted work described in Example 1, or the intelligent assessment method for live work described in Example 2 or Example 3. The display unit of the computer device is used to produce visual images and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0080] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0081] Example 6 Based on the above embodiments, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the pole-mounted operation insulation shielding detection method as described in Example 1, or implements the live operation intelligent assessment method as described in Example 2 or Example 3.

[0082] Example 7 Based on the above embodiments, this embodiment provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the pole-based work insulation shielding detection method as described in Example 1, or implements the live work intelligent assessment method as described in Example 2 or Example 3.

[0083] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to a memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0084] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solutions of the present invention. They should all be included in the scope of the technical solutions claimed for protection by the present invention.

Claims

1. A method for detecting insulation shielding during pole work, characterized in that: The following steps are involved: Acquire live video frames. For each frame, use a key point detection model to detect key points, obtain the coordinates of the pole top, left crossarm, and right crossarm, and then expand and generate key point circumscribed rectangles based on the coordinates of the pole top, left crossarm, and right crossarm, respectively. Finally, use an instance segmentation model to identify and extract the contours of cables and cable insulation shielding devices. Based on the rotating frame detection model on the pole, the pole image within the circumscribed rectangle of the key points is used to identify the insulation shielding devices of the pole, and the insulation shielding status of the pole and crossarm is obtained based on the recognition results. The insulation shielding status of the cable is determined based on the relative positions of the cable outline, the cable insulation shielding device outline and the circumscribed rectangle of the key points.

2. A method for detecting insulation shielding during pole work according to claim 1, characterized in that: The insulation shielding status of the cable is determined based on the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points, including: Fitting each cable profile to obtain a fitted cable line; Determine the key point circumscribed rectangle associated with the fitted cable line according to the positional relationship between the fitted cable line and each key point circumscribed rectangle; Calculate the distance from the center point of the cable insulation shielding device outline to the circumscribed rectangle of each key point, and determine the cable where the cable insulation shielding device is located based on the distance.

3. The method for detecting insulation shielding during pole work according to claim 1, wherein: The rotating box detection model on the pole uses the PP-YOLOE-R model as the baseline model. The regression branch predicts four parameters (center point coordinate x, center point coordinate y, target height h, target width w) and adds an angle prediction branch to predict the target angle. The key point detection model uses YOLOv8-pose as the baseline model.

4. A method for intelligent assessment of live working based on deep learning, characterized in that: The following steps are involved: Obtain the live video frames collected when the examinee is working on the pole; Based on the pole operation insulation shielding detection method according to any one of claims 1 to 3, each frame of the on-site video frame is identified and detected to obtain the shielding conditions of the poles, crossarms and cables; Use target tracking algorithm to track the examinee and capture the time sequence of video frames when the examinee is working on the pole; The video frame time series sequence is fed into the action detection model to perform time series action detection and obtain the time series action; The candidate's pole work score is obtained based on the preset pole work scoring rules, the identification results of the shielding conditions of the poles, crossarms and cables, and the candidate's timed actions on the pole.

5. The method for intelligent assessment of live working based on deep learning according to claim 4 is characterized in that: Before obtaining the live video frames collected when the candidate is working on the pole, first obtain the live video frames collected when the candidate is working on the ground; Perform target detection on each video frame based on the ground rotating frame detection model to identify tools and protective equipment; Use target tracking algorithm to track the examinee and capture the time sequence of video frames during the examinee's ground work; The video frame time series sequence is fed into the action detection model to perform time series action detection and obtain the time series action; The candidate's ground work score is obtained based on the preset ground work scoring rules, the identification results of tools and protective equipment, and the candidate's ground sequential movements.

6. The method for intelligent assessment of live working based on deep learning according to claim 4, characterized in that: The action detection model uses the PP-TSMv2 model, and the ground rotation box detection model uses the PP-YOLOE-R model as the baseline model. The regression branch predicts four parameters (center point coordinate x, center point coordinate y, target height h, target width w) and adds an angle prediction branch to predict the target angle.

7. An intelligent assessment system for live working based on deep learning, characterized in that: include: The video acquisition module is used to obtain the on-site video frames collected by the candidates during their ground and pole work; The ground rotating frame detection model is used to detect targets in each frame of video images during ground operations and identify tools and protective gear. A key point detection model is used to detect key points in each frame of video image using the key point detection model to obtain the coordinates of the pole top, the left end of the crossarm, and the right end of the crossarm, and to generate a key point circumscribed rectangle based on the coordinates of the pole top, the left end of the crossarm, and the right end of the crossarm; Instance segmentation module, used to identify and extract the contours of cables and cable insulation shielding devices based on the instance segmentation model; The rotating frame detection model on the pole is used to identify the insulation shielding devices of the pole within the circumscribed rectangle of the key points; The shielding recognition module is used to obtain the insulation shielding status of the pole and crossarm based on the identification results of the pole insulation shielding device, and to determine the insulation shielding status of the cable based on the relative positions of the cable outline, the cable insulation shielding device outline, and the circumscribed rectangle of the key points; The tracking video capture module is used to track the examinee using a target tracking algorithm and capture the time sequence of video frames during the examinee's ground and pole work; A real-time video analysis module is used to perform temporal motion detection on the input video frame temporal sequence to obtain temporal motion; The intelligent detection module is used to obtain the candidate's pole work score based on the preset pole work scoring rules, the insulation shielding status identification results of the poles and crossarms, and the candidate's pole-based sequential movements, and to obtain the candidate's ground work score based on the preset ground work scoring rules, the tools and protective equipment identification results, and the candidate's ground-based sequential movements.

8. A computer device, characterized in that: The processor, the communication interface, the memory and the communication bus are connected to each other via the communication bus. Memory for storing computer programs; The processor is used to implement the insulation shielding detection method for pole operations as described in any one of claims 1 to 3, or the intelligent assessment method for live operations as described in any one of claims 4 to 6 when executing the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the insulation shielding detection method for pole operations as described in any one of claims 1 to 3 is implemented, or the intelligent assessment method for live operations as described in any one of claims 4 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting insulation shielding during pole work as described in any one of claims 1 to 3 is implemented, or the method for intelligent assessment of live work as described in any one of claims 4 to 6 is implemented.

Images