Electric power high-altitude operation dangerous point identification method and system based on combination of large and small models

Abstract

The invention provides an electric power high-altitude operation dangerous point identification method and system based on combination of large and small models, and relates to the technical field of electric power safety, and the method comprises the steps: obtaining high-altitude operation picture data in an electric power high-altitude operation process in real time; inputting the aerial work picture data into a pre-trained target detection small model, identifying the protective equipment in the aerial work picture data, and outputting auxiliary identification information of the protective equipment; wherein the auxiliary identification information comprises box coordinates of the protective equipment; according to the auxiliary identification information and a preset cue word template, determining a large model cue word used for carrying out auxiliary identification on the dangerous point of the electric power high-altitude operation; and inputting the aerial work picture data and the large model cue word into the visual language large model, and outputting a dangerous point identification result in the electric power aerial work process. According to the scheme, the accuracy of identifying the high-altitude operation dangerous point can be improved, and then the safety of high-altitude operation personnel is improved.

Description

Technical Field

[0001] This invention relates to the field of power safety technology, and in particular to a method and system for identifying hazards in high-altitude power operations based on a combination of large and small models. Background Technology

[0002] High-altitude power operations are a crucial link in supporting the entire lifecycle of the power system, spanning core operations such as transmission line construction, substation operation and maintenance, and distribution network fault repair. They directly impact national energy security, the continuity of industrial production, and the stability of electricity supply for residential use. However, while vital to the power sector, high-altitude power operations are also considered among the most dangerous operations in the industry due to their high correlation with other sectors, high hazards, and unpredictability. For example, violations of protective equipment usage regulations, such as not wearing a safety helmet, not fastening a safety belt, incorrect safety belt attachment points, or improper helmet wearing, all pose significant safety risks to workers.

[0003] Hazard identification, as the first line of defense in preventing accidents during high-altitude operations, transforms passive accident response into proactive risk prevention, playing a crucial role in ensuring the safety of workers at height. However, current methods for identifying hazard points in high-altitude operations employ two main approaches. One relies on visual inspection by ground supervisors or monitoring center personnel. Limited by field of vision, energy, and experience, this method suffers from a high rate of missed detections of subtle risks, resulting in poor accuracy. The other approach, based on traditional target detection models, can only identify the presence of safety helmets or safety belts but cannot confirm whether they are worn correctly. Furthermore, it is highly susceptible to environmental influences, leading to a high false detection rate and poor identification effectiveness.

[0004] In summary, existing technologies have low accuracy in identifying hazards during high-altitude power operations, making it difficult to effectively ensure the safety of workers at heights. Summary of the Invention

[0005] In view of this, and to address the above shortcomings, it is necessary to propose a method and system for identifying hazards in high-altitude power operations based on a combination of large and small models, so as to improve the accuracy of identifying hazards in high-altitude operations and thus enhance the safety of high-altitude workers.

[0006] In a first aspect, the present invention provides a method for identifying hazardous points in high-altitude power operations based on a combination of large and small models, including:

[0007] Real-time acquisition of high-altitude operation images and data during power high-altitude operations;

[0008] The high-altitude operation image data is input into a pre-trained target detection mini-model to identify the protective equipment in the high-altitude operation image data and output auxiliary identification information of the protective equipment; wherein, the auxiliary identification information includes the box coordinates of the protective equipment;

[0009] Based on the auxiliary identification information and the preset prompt word template, determine the large model prompt words used to assist in the identification of dangerous points in high-altitude power operations;

[0010] The high-altitude operation image data and the large model prompts are input into the visual language large model, and the result of hazard point identification in the power high-altitude operation process is output.

[0011] Preferably, the training method of the target detection small model includes:

[0012] Acquire historical high-altitude operation images during historical power line high-altitude operations;

[0013] The historical high-altitude operation image data is labeled to form sample training data; wherein, the labels include at least: with safety helmet, without safety helmet, with safety belt, without safety belt, and bucket;

[0014] Configure model training parameters and train the target detection mini-model using the sample training data based on a three-stage refinement mechanism; wherein, the three-stage refinement mechanism includes: a first training stage, a second training stage, and a third training stage; the first training stage is used to focus on feature extraction of the core target, the second training stage is used to improve the detection accuracy and generalization ability of the mini-model, and the third training stage is used to optimize the head detection layer.

[0015] Preferably, the step of labeling the historical high-altitude operation image data includes:

[0016] The LabelImg software platform was used to manually label each historical high-altitude operation image.

[0017] After annotation, the data is converted to the standard COCO format and divided into a training set and a test set according to a preset ratio.

[0018] Preferably, the step of training the target detection small model using the sample training data based on a three-stage refinement mechanism includes:

[0019] The iterative training rounds in the model training parameters are divided into three parts according to a preset rule: the first training stage, the second training stage, and the third training stage; each training stage includes at least one iterative training round.

[0020] In the first training phase, the focus is on feature extraction of the core target and optimization of bounding box regression accuracy. After the first training phase has completed its iterative training rounds, if the average accuracy (AP) of the test set is not less than the preset first average accuracy threshold, the second training phase is initiated. Otherwise, the learning rate in the model training parameters is lowered and training continues until the average accuracy (AP) requirement is met.

[0021] In the second training phase, the global optimal localization self-distillation module is enabled to improve the model's detection accuracy and generalization ability for small targets. At the same time, after the iterative training rounds of the second training phase, if the average accuracy AP of the test set is not less than the preset second average accuracy threshold, the third training phase is entered; otherwise, the iterative training rounds of this training phase are extended. The second average accuracy threshold is greater than the first average accuracy threshold.

[0022] In the third training phase, the backbone network parameters of the model are frozen, and only the head detection layer is optimized. After training, it is verified whether the average precision (AP) meets the requirements of the third average precision threshold. If it does, training ends and the final target detection mini-model is output. Otherwise, training continues. The third average precision threshold is greater than the second average precision threshold.

[0023] Preferably, the box coordinates of the protective equipment are [x1, y1; x2, y2], where [x1, y1] are the coordinates of the upper left corner of the rectangle that selects the identified protective equipment, and [x2, y2] are the coordinates of the lower right corner of the rectangle that selects the identified protective equipment.

[0024] Preferably, the large model prompt word has a structure consisting of an auxiliary information embedding area, a task instruction area, and an output format area;

[0025] The auxiliary information embedding area is used to embed auxiliary recognition information output by the target detection mini-model; the task instruction area is used to clarify the core tasks that the visual language large model needs to complete; the output format area is used to specify the structured format of the output of the visual language large model; wherein, the core tasks include: determining the type of work scenario, recognizing the status of protective equipment, and classifying the hazard level.

[0026] Preferably, the large visual language model is DeepSeek-VL2.

[0027] Preferably, after outputting the hazard identification results during high-altitude power operations, the process further includes:

[0028] Based on the hazard level output by the visual language big data model, a warning method corresponding to that hazard level is used to issue a warning.

[0029] Secondly, the present invention provides a power high-altitude operation hazard identification system based on a combination of large and small models. The system includes: a field data acquisition module, a protective equipment identification module, a prompt word construction module, and a hazard identification module.

[0030] The on-site data acquisition module is deployed at the power high-altitude operation site to collect high-altitude operation image data in real time during the power high-altitude operation process, and transmit the collected high-altitude operation image data to the protective equipment identification module and the hazard point identification module.

[0031] The protective equipment recognition module is equipped with a pre-trained target detection mini-model, which is used to identify protective equipment in the input high-altitude operation image data and output auxiliary recognition information of the protective equipment; and to transmit the output auxiliary recognition information to the prompt word construction module; wherein, the auxiliary recognition information includes the box coordinates of the protective equipment;

[0032] The prompt word construction module is used to determine a large model prompt word for assisting in the identification of dangerous points in high-altitude power operations based on the auxiliary identification information and the preset prompt word template, and input the large model prompt word into the dangerous point identification module;

[0033] The hazard identification module is equipped with a visual language model, which is used to output the hazard identification results in the process of high-altitude power operations based on the input high-altitude operation image data and the prompt words of the model.

[0034] Preferably, the system also includes: a hazard warning module;

[0035] The danger warning module is used to issue warnings based on the danger level output by the visual language large model, and to use a warning method corresponding to that danger level.

[0036] As described above, the power high-altitude operation hazard identification method based on a combination of large and small models provided in this solution first acquires high-altitude operation image data in real time. Then, this image data is input into a pre-trained target detection small model to identify protective equipment in the image data and outputs auxiliary identification information including the box coordinates of the protective equipment. Further, based on the auxiliary identification information and a preset prompt word template, a large model prompt word is determined for identifying hazard points in power high-altitude operations. Then, the high-altitude operation image data and the large model prompt word are input into a visual language large model, ultimately outputting the hazard point identification result during the power high-altitude operation. Therefore, after the collected high-altitude operation image is input into the target detection small model, the small model provides the corresponding detection box for the protective equipment of the workers in the image. The coordinates of the box are then incorporated into the prompt text to form the large model prompt word, assisting the large model in determining dangerous behaviors. The system uses prompts containing auxiliary information from the small object detection model to guide the large visual language model in judging the behavior of images. By leveraging the accurate detection of protective equipment in high-altitude power operations by the small object detection model, the system can improve the description accuracy of the large visual language model, thereby enhancing the accuracy of identifying dangerous behaviors in high-altitude power operation scenarios and ensuring the safety of high-altitude workers. Attached Figure Description

[0037] Figure 1 The flowchart illustrates a method for identifying hazardous points in high-altitude power operations based on a combination of large and small models, as provided in this embodiment of the invention.

[0038] Figure 2 This is a schematic diagram of a framework for identifying hazards in high-altitude power operations based on a combination of large and small models, provided as an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of a danger warning mechanism provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of a power high-altitude operation hazard identification system based on a combination of large and small models, provided as an embodiment of the present invention.

[0041] Figure 5 The total training loss of the D-FINE small model for object detection in the training experiment.

[0042] Figure 6 The bounding box evaluation results of COCO object detection in the training experiment of the D-FINE small object detection model. Detailed Implementation

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] like Figure 1 and 2 As shown, this invention provides a method for identifying hazards in high-altitude power operations based on a combination of size models. This method may include the following steps:

[0045] Step 101: Acquire high-altitude operation image data in real time during the power high-altitude operation process;

[0046] Step 102: Input the high-altitude operation image data into a pre-trained target detection mini-model to identify the protective equipment in the high-altitude operation image data and output auxiliary identification information of the protective equipment; wherein, the auxiliary identification information includes the box coordinates of the protective equipment;

[0047] Step 103: Based on the auxiliary identification information and the preset prompt word template, determine the large model prompt words used to assist in the identification of dangerous points in high-altitude power operations;

[0048] Step 104: Input the high-altitude operation image data and the large model prompt words into the visual language large model, and output the hazard point identification results in the power high-altitude operation process;

[0049] Step 105: Based on the danger level output by the visual language big data model, issue a warning using the warning method corresponding to that danger level.

[0050] In this embodiment, after the collected images of high-altitude operations are input into the target detection mini-model for processing, the mini-model outputs corresponding detection boxes for the protective equipment worn by the workers in the images. The coordinates of these boxes are then incorporated into the prompt text to form prompts for the larger model, assisting it in determining dangerous behaviors. In other words, prompts containing auxiliary information output by the target detection mini-model guide the visual language larger model in judging behavior in the images. For example, in a scenario where an electrician is working at height without a helmet and safety belt, this behavior is described as a high-risk violation. By leveraging the precise detection of protective equipment in high-altitude power operations by the target detection mini-model, the accuracy of the visual language larger model's description is improved, thereby enhancing the accuracy of identifying dangerous behaviors in high-altitude power operation scenarios and ultimately ensuring the safety of workers at height.

[0051] The steps of the present invention will be further described in detail below.

[0052] Step 101: Acquire high-altitude operation image data in real time during the power high-altitude operation process.

[0053] When acquiring high-altitude operation images during power line work, high-definition camera equipment can be deployed on-site. This equipment can also be equipped with nighttime infrared imaging capabilities to accommodate image acquisition during periods of low light. In some embodiments, a multi-dimensional data acquisition system can be constructed using drone inspection equipment equipped with a full-frame camera and high-speed mobile storage devices to obtain comprehensive, multi-dimensional data from the site.

[0054] Step 102: Input the high-altitude operation image data into a pre-trained target detection mini-model to identify the protective equipment in the high-altitude operation image data and output auxiliary identification information of the protective equipment; wherein, the auxiliary identification information includes the box coordinates of the protective equipment.

[0055] In this step, the small object detection model can be trained as follows:

[0056] S21: Obtain historical high-altitude operation image data during historical power high-altitude operations;

[0057] In this step, we can consider using 4K high-definition camera equipment with nighttime infrared imaging function, drone inspection equipment equipped with full-frame camera and high-speed mobile storage equipment to build a multi-dimensional data acquisition system. The acquisition scope needs to cover three core scenarios of high-altitude power operations: (1) Pole tower operation scenario: covering climbing, maintenance, line connection and other operation links of pole towers of different voltage levels of 110kV, 220kV and 500kV; (2) Bucket operation scenario: focusing on the operation process of high-altitude work vehicle bucket lifting, personnel posture adjustment, tool passing and other operation processes; (3) Line inspection scenario: collecting data on scenarios such as personnel working in the air and crossing obstacles along the transmission line by drone. At the same time, it is necessary to collect data in four typical environments: sunny day (including strong light and side light periods), cloudy day, light rain, and night (with work lighting on) to ensure that the data contains environmental diversity characteristics and finally obtain no less than 5,000 valid image data.

[0058] Furthermore, based on frame extraction algorithms commonly used in computer vision, the acquired operation videos are processed into frames at 1-second intervals, with the output format being JPG, retaining the original 4K (3840×2160) resolution. An image sharpness evaluation algorithm (by calculating the Laplacian variance value, setting a threshold ≥100 as the sharpness standard) is used to remove blurry frames, while frames of empty scenes without workers or protective equipment are filtered out. Finally, valid images are retained and categorized and stored according to the naming rule of "scene type-environment-date," resulting in historical high-altitude operation image data.

[0059] S22: Label the historical high-altitude operation image data to form sample training data; wherein, the labels include at least: with safety helmet, without safety helmet, with safety belt, without safety belt, and bucket;

[0060] In this step, we consider labeling historical high-altitude operation images. Specifically, we can first use the LabelImg software platform to manually label each historical high-altitude operation image; then, after labeling, we convert the data to the standard COCO format and divide the data into a training set and a test set according to a preset ratio. For example, we can first deploy image labeling tools such as LabelImg, configure labeling shortcut keys to improve labeling efficiency, and uniformly use a pixel-level coordinate system with the top left corner as the origin (coordinate format: x1, y1, x2, y2), where [x1, y1] is the coordinate of the top left corner of the rectangle that selects the identified protective equipment, and [x2, y2] is the coordinate of the bottom right corner of the rectangle that selects the identified protective equipment. Then, a professional labeling team was formed to conduct a two-day standardized training, which included: core labels such as: with safety helmet, without safety helmet, with safety belt, without safety belt, bucket, etc.; labeling accuracy requirements (the bounding box must completely cover the target, the error ≤ 2 pixels, and there must be no mislabeling or omissions); and 50 standard labeled examples were provided to ensure the consistency of the team's labeling standards.

[0061] Furthermore, a three-tiered annotation verification process can be designed: individual annotation, cross-validation, and final review. In individual annotation, each annotator is responsible for annotating one type of scenario data, with a daily annotation volume not exceeding 800 images. Annotator identifiers are embedded in the image filenames after annotation. In cross-validation, a 1-annotator + 2-verifier model is adopted. After the annotator completes self-checking, two other personnel conduct cross-checking, recording erroneous annotations. Batches with an error rate >5% need to be re-annotated. In the final review, the technical lead conducts a 10% sampling review of the verified annotation data. After confirming there are no errors, the annotation files are locked and backed up to a cloud storage system with redundancy protection.

[0062] Furthermore, we are considering developing a batch conversion program from XML to COCO format. This program will have the following functions: automatically reading image paths, dimensions, target labels, and coordinate information from XML files; and generating five core fields according to the COCO dataset specification: info (dataset information), licenses (license information), images (image information), annotations (annotation information), and categories (category information). Simultaneously, it will include a built-in coordinate validity verification module to automatically filter out abnormal data with coordinate values ​​exceeding the image dimensions and output error logs. After conversion, the COCO API tool will be used to verify the integrity of the converted files, ensuring no missing fields or format errors.

[0063] Finally, stratified sampling can be used to divide the dataset into a training set (4000 images) and a test set (1000 images) in an 8:2 ratio, ensuring that the proportion of each scene and label class is consistent across the datasets. A structured storage directory is designed, storing image files according to dataset type / scene type paths, and using JPEG compression format (85% compression quality) to balance storage efficiency and image clarity; COCO format annotation files and image folders are placed in the same root directory for easy model access.

[0064] S23: Configure model training parameters and train the target detection mini-model using the sample training data based on a three-stage refinement mechanism; wherein, the three-stage refinement mechanism includes: a first training stage, a second training stage, and a third training stage; the first training stage is used to focus on feature extraction of the core target, the second training stage is used to improve the detection accuracy and generalization ability of the mini-model, and the third training stage is used to optimize the head detection layer.

[0065] This step aims to train a small object detection model. Specifically, the iterative training rounds in the model training parameters can be divided into three parts according to a preset rule: a first training phase, a second training phase, and a third training phase; each training phase includes at least one iterative training round.

[0066] In the first training phase, the focus is on feature extraction of the core target and optimization of bounding box regression accuracy. After the first training phase has completed its iterative training rounds, if the average accuracy (AP) of the test set is not less than the preset first average accuracy threshold, the second training phase is initiated. Otherwise, the learning rate in the model training parameters is lowered and training continues until the average accuracy (AP) requirement is met.

[0067] In the second training phase, the global optimal localization self-distillation module is enabled to improve the model's detection accuracy and generalization ability for small targets. At the same time, after the iterative training rounds of the second training phase, if the average accuracy AP of the test set is not less than the preset second average accuracy threshold, the third training phase is entered; otherwise, the iterative training rounds of this training phase are extended. The second average accuracy threshold is greater than the first average accuracy threshold.

[0068] In the third training phase, the backbone network parameters of the model are frozen, and only the head detection layer is optimized. After training, it is verified whether the average precision (AP) meets the requirements of the third average precision threshold. If it does, training ends and the final target detection mini-model is output. Otherwise, training continues. The third average precision threshold is greater than the second average precision threshold.

[0069] In this embodiment, when training the small object detection model, it is considered to first build a high-performance model training hardware environment, which may include: (1) computing module: two NVIDIA A100 80GB GPUs supporting NVLink interconnection, paired with an Intel Xeon Platinum 8375C (32 cores and 64 threads) CPU and 256GB DDR4 3200MHz memory; (2) storage module: 10TB NVMe SSD, used to store datasets, model files and training logs; (3) system software: install Ubuntu 22.04LTS operating system, configure SSH remote connection function; deploy NVIDIA driver (version 535.104.05), CUDA 12.2, CuDNN 8.9.4, and verify the normal loading of the driver through GPU status detection tool. (4) Create a Python virtual environment (Python 3.10), install PyTorch 2.1.0 (supports CUDA 12.2), torchvision 0.16.0, pycocotools 2.0.7, opencv-python 4.8.1.78, tensorboard 2.15.1 and other dependent libraries, and ensure environment consistency through the dependency package version locking file (requirements.txt).

[0070] Then, the local dataset is transferred to the training server using a remote file transfer protocol. After the transfer is complete, the D-FINE model code is retrieved from the open-source repository, switched to the latest stable version, and the model configuration file is modified. Specifically, the dataset path configuration specifies the storage paths for the training and test sets, as well as the path to the annotation files; the training parameter configuration sets the training epochs (Epoch=120), batch size (batch_size=16, supporting multi-GPU data parallelism), initial learning rate (lr=2.5e-05), and weight decay coefficient (0.0001); and the output configuration specifies the storage paths for the model checkpoint and training logs, and sets the log output interval.

[0071] Furthermore, the model training process is initiated based on the PyTorch framework, enabling mixed-precision training technology to accelerate the training process. The training command requires specifying multi-GPU collaborative training parameters. Specifically, model training can be divided into three stages, with the target operations for each stage as follows:

[0072] The first training phase, namely the basic feature learning phase (rounds 1-30): focuses on feature extraction of core targets (helmets, seat belts), optimizes bounding box regression accuracy, and saves the model checkpoint once every 5 rounds; after the 30th round, if the validation set AP ≥ 0.65, it enters the next phase; otherwise, it lowers the learning rate (to 1.25e-05) and continues training.

[0073] The second training phase, namely the refined training phase (rounds 31-90): The global optimal localization self-distillation module is enabled to improve the model's detection accuracy and generalization ability for small targets (such as tools). A checkpoint is saved every 10 rounds. After the 90th round, the validation set AP is required to be ≥0.70. Otherwise, the training rounds of this phase are extended.

[0074] The third training phase, namely the fine-tuning and optimization phase (rounds 91-120): freeze the parameters of the model backbone network (such as ResNet50), optimize only the head detection layer, reduce the learning rate to 5e-06, and save the checkpoint once every 5 rounds; after training, select the checkpoint with the highest AP (≥0.72) and lowest loss (≤12) on the validation set as the final small model for object detection.

[0075] In one embodiment, the object detection mini-model can be the D-FINE model, a real-time object detection model based on the Transformer architecture proposed by a team from the University of Science and Technology of China. Its core innovation lies in redefining the bounding box regression task. Through two main methods—Fine-grained distribution optimization (FDR) and global optimal localization self-distillation (GO-LSD)—it significantly improves localization accuracy and detection efficiency without increasing additional training costs. On the COCO dataset, this model achieves an average accuracy (AP) of 59.3% at 78 FPS, surpassing the latest versions of the YOLO and RT-DETR series.

[0076] D-FINE's innovations on the existing DETR architecture are twofold: First, it innovates the regression task methodology by proposing fine-grained distribution optimization, transforming the traditional bounding box regression task from predicting fixed coordinates to layer-by-layer optimization of probability distributions. Second, it innovates the training strategy methodology by optimizing the training process through global optimal localization self-distillation, a knowledge distillation method that requires no additional computational cost. The training data for the small model is primarily obtained by segmenting high-altitude work videos collected at power plant sites. After data cleaning, 3306 images were selected. The sample labels for each image were manually annotated and validated using the LabelImg software platform, and the XML format was converted to the standard COCO format. The dataset is divided into a 7:3 ratio: 2316 images for training and 993 images for testing, including five categories of sample labels: with safety helmet, without safety helmet, with safety belt, without safety belt, and bucket. Specific sample label IDs and corresponding category names are shown in Table 1.

[0077] Table 1. Label Description for Power Safety Equipment Dataset Tag ID 0 1 2 3 4 Category Name Wearing a safety helmet No helmet Seat belts are available No seat belt Bucket

[0078] In addition, commonly used tools and equipment during the operation are also marked. This information can be used for further analysis during reasoning, employing methods such as location distribution and regional intersection / union comparison to reserve space for in-depth analysis of the positional relationship between tools and personnel, as well as the interaction between different tools. For example, the statistical count of the number of marked protective equipment and tools is shown in Table 2.

[0079] Table 2 Quantity Statistics for Each Category of Protective Equipment Category Name Wearing a safety helmet No helmet Seat belts are available No seat belt Bucket quantity 660 670 650 642 634

[0080] Furthermore, in one embodiment, the preset prompt word template can also include multiple prompt word templates for different scenarios, such as tower climbing scenarios, bucket hoisting scenarios, and high-altitude inspection scenarios, etc., and different prompt words can be set for these scenarios respectively. Further, a power scene recognition module can be embedded in the target detection small model to first determine the corresponding power scene based on the high-altitude operation image data. Then, before determining the large model prompt words for assisting in the identification of dangerous points in high-altitude power operations based on the auxiliary recognition information and the preset prompt word template, the corresponding prompt word template is matched according to the identified power scene, and then the auxiliary recognition information is embedded into the matched prompt word template to obtain the large model prompt words, thus adapting to different power scenarios.

[0081] Step 103: Based on the auxiliary identification information and the preset prompt word template, determine the large model prompt words used to assist in the identification of dangerous points in high-altitude power operations;

[0082] This step aims to construct a large-scale model prompt. In one embodiment, the large-scale model prompt is a structure consisting of an auxiliary information embedding area, a task instruction area, and an output format area; wherein, the auxiliary information embedding area is used to embed auxiliary recognition information output by the target detection small model; the task instruction area is used to clarify the core tasks that the visual language large model needs to complete; the output format area is used to specify the structured format of the visual language large model's output; wherein, the core tasks include: determining the type of work scenario, recognizing the status of protective equipment, and classifying the hazard level.

[0083] In this embodiment, a three-segment prompt word framework is designed, comprising an auxiliary information embedding area, a task instruction area, and an output format area. The specific structure is as follows:

[0084] Auxiliary information embedding area: Automatically embeds the target detection information output by the small model, including label category and bounding box coordinates (x1,y1,x2,y2), in the format "label name: box (x1,y1,x2,y2)".

[0085] Task Instruction Area: Clearly define the three core tasks that the large model needs to complete, namely, determining the type of work scenario (such as tower operation / bucket operation / line inspection), identifying the status of protective equipment (whether the safety helmet is worn, whether the safety belt is worn and attached), and classifying the risk level (low / medium / high risk).

[0086] Output format area: Specifies the structured format for the output of the large model, which must include four fields: job scenario type, helmet wearing status, safety belt wearing and attachment status, and hazard level. No additional content is allowed.

[0087] Furthermore, we could consider developing an auxiliary information generation module. This module would extract valid information from the detection results (including labels, boxes, and scores) of the D-FINE model, concatenate them into auxiliary information text according to a preset format, and embed a prompt word framework.

[0088] In this embodiment, prompt engineering primarily focuses on optimizing language models (including the language interaction module in multimodal models). Its goal is to guide the large model to generate more accurate and targeted output text by designing and adjusting input prompts. When interacting with the visual language model DeepSeek-VL2, the given prompts significantly impact the model's response content and quality. Prompt engineering focuses on creating the most effective prompts that enable the model to understand and meet task response requirements. This involves understanding the high-altitude power operation scenario, using correct vocabulary and grammatical structures, and trying different prompting strategies to achieve the best response from the large model. This solution designs guiding prompts where the target object's box coordinate information is automatically filled in from the detection results of the small model, thus forming the fused prompts, as detailed below:

[0089] "We currently have auxiliary information: safety helmet (box1), safety belt (box2), ... Please analyze and determine the type of work scenario in this diagram, whether the worker is wearing a safety helmet and safety belt, and assess the risk level based on this information. Please strictly adhere to the following output format:"

[0090] Job scenario type:

[0091] Helmet wearing status:

[0092] Seatbelt wearing status:

[0093] Hazard level: (Select from low risk, medium risk, and high risk. Low risk means wearing a helmet and seat belt correctly and properly; medium risk means wearing only a seat belt; high risk means not wearing a seat belt or not wearing either a seat belt or helmet.)

[0094] Step 104: Input the high-altitude operation image data and the large model prompt words into the visual language large model, and output the hazard point identification results in the power high-altitude operation process.

[0095] In this step, the large visual language model may include DeepSeek-VL2. This patent is based on the DeepSeek-VL2-tiny large visual language model for image data inference and judgment. DeepSeek-VL2 is an advanced large hybrid expert (MoE) visual language model, which is a significant improvement over its predecessor, DeepSeek-VL. DeepSeek-VL2 demonstrates superior capabilities in various tasks, including but not limited to visual question answering, optical character recognition, document / table / chart understanding, and visual localization. The model series consists of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and DeepSeek-VL2, differing in their underlying LLM. DeepSeek-VL2-tiny is built based on DeepSeekMoE-3B (total activation parameters are 1.0B); DeepSeek-VL2-small is built based on DeepSeekMoE-16B (total activation parameters are 2.8B); DeepSeek-VL2 is built based on DeepSeekMoE-27B (total activation parameters are 4.5B).

[0096] Specifically, you can obtain the DeepSeek-VL2-Tiny model weights from the official platform (4-bit quantization version recommended, approximately 1.2GB in size) and store them in a specified directory. Install the large model dependency libraries in the Python virtual environment, including transformers 4.35.2, accelerate 0.24.1, and pillow 10.1.0, ensuring that the dependency library versions are compatible with the model. Then, design a standardized process for loading and inference of the large model, specifically including:

[0097] Model loading module: Loads the DeepSeek-VL2-Tiny model and its corresponding processor, enables GPU acceleration, and sets the inference mode (disables the Dropout layer).

[0098] Image preprocessing module: Reads the image to be detected, converts it to RGB format, and performs preprocessing such as size adjustment and normalization according to the requirements of the large model;

[0099] Prompt word generation module: Calls the auxiliary information generation module to generate prompt words containing the detection results of the small model;

[0100] Inference Execution Module: Input the preprocessed image and prompt words into the large model, set the inference parameters (maximum output length 200 characters, temperature 0.1 to reduce randomness), and execute the inference calculation;

[0101] Results parsing module: Extracts structured information (scene type, protective equipment status, hazard level) from the output text of the large model and outputs standardized results that can be used for subsequent processing.

[0102] When conducting large-scale model inference tests, typical test images (such as pole tower operations + standardized wearing of dual equipment scenarios) can be selected to execute the inference process and verify the accuracy of the output results. The expected output must conform to the preset format, and the judgment results for each field must be consistent with the manual annotation. If the results are inconsistent, check whether the small-scale model detection results correctly identify the target and whether the prompts clearly describe the task requirements; correct the errors and retest.

[0103] Step 105: Based on the danger level output by the visual language big data model, issue a warning using the warning method corresponding to that danger level.

[0104] In this step, after the visual language model accurately outputs the hazard description text, the hazard level is determined by matching the output text with the safety operation manual for high-altitude power operations. The final assessment result is presented visually, such as through information notifications and visual warnings, to alert on-site personnel. Specific hazard warning mechanisms are as follows: Figure 3 As shown. For example, hazard levels can be divided into high risk corresponding to a red alert, medium risk corresponding to a yellow alert, and low risk corresponding to a blue alert. High risk includes not wearing a seatbelt or not wearing both a seatbelt and helmet; medium risk includes wearing only a seatbelt; and low risk includes correctly wearing a helmet and necessary protective equipment. When an alarm is triggered, the warning information can be sent to the control center via text notification, and the control center can issue corrective instructions; alternatively, visual warnings can be used, such as flashing lights, to alert workers to dangerous behavior.

[0105] In addition, such as Figure 4 As shown, the present invention also provides a power high-altitude operation hazard identification system based on a combination of large and small models. The system includes: a field data acquisition module 401, a protective equipment identification module 402, a prompt word construction module 403, and a hazard identification module 404.

[0106] The on-site data acquisition module 401 is deployed at the power high-altitude operation site to collect high-altitude operation image data in real time during the power high-altitude operation process, and transmit the collected high-altitude operation image data to the protective equipment identification module 402 and the hazard point identification module 404.

[0107] The protective equipment identification module 402 is equipped with a pre-trained target detection small model, which is used to identify protective equipment in the input high-altitude operation image data and output auxiliary identification information of the protective equipment; and to transmit the output auxiliary identification information to the prompt word construction module 403; wherein, the auxiliary identification information includes the box coordinates of the protective equipment.

[0108] The prompt word construction module 403 is used to determine a large model prompt word for assisting in the identification of dangerous points in high-altitude power operations based on the auxiliary identification information and the preset prompt word template, and input the large model prompt word into the dangerous point identification module 404;

[0109] The hazard identification module 404 is equipped with a visual language large model, which is used to output the hazard identification results in the process of high-altitude power operations based on the input high-altitude operation image data and the prompt words of the large model.

[0110] In one embodiment, the system further includes a hazard warning module 405; the hazard warning module is used to issue a warning based on the hazard level output by the visual language large model, using a warning method corresponding to the hazard level.

[0111] The system embodiments provided by this invention are based on the same inventive concept as the method embodiments in this specification. For details, please refer to the description in the method embodiments of this specification, which will not be repeated here.

[0112] The beneficial effects of the present invention will be further explained below with reference to specific experiments.

[0113] This experimental platform is based on the following: Linux system, Python programming language, Visual Studio Code programming software, PyTorch training framework, and TensorBoard data visualization.

[0114] (1) Training results and experimental analysis of the D-FINE small model for object detection

[0115] Training parameters: Training epochs: Epoch=100; Learning rate: lr=2.5e-05; Training and validation batches: batch_size=8

[0116] The model training total loss and the COCO object detection eval_bbox metric (boundary box evaluation) results are as follows: Figure 5 , Figure 6As shown in the figure, the overall training loss curve of the model exhibits a healthy convergence trend with a continuous decline, gradually decreasing from an initial 30+ to around 15, indicating that the model is continuously and effectively learning and optimizing during training. The initial stage sees a rapid decrease in loss, representing a highly efficient learning period where the model quickly masters basic detection features; the mid-term convergence fluctuates between 20 and 25, which is normal gradient oscillation influenced by data randomness and optimizer updates, indicating that the model is still continuously fine-tuning and improving; in the later stage, the loss converges more rapidly and tends to stabilize, gradually decreasing to around 15, reflecting the model's ability to learn complex features with fine precision. Figure 6 The main bounding box evaluation curve metrics are shown in Table 3 below.

[0117] Table 3 Training Indicators and Their Meanings COCO index Precision value Analysis and Interpretation AP (IoU=0.50:0.95) 0.6933 Average precision at multiple IoU thresholds (from 0.5 to 0.95, with a step size of 0.05) AP (IoU=0.50) 0.927 AP under a relatively lenient IoU threshold (0.5) AP (IoU=0.75) 0.7914 AP under a stricter IoU threshold (0.75) AP (small) 0.364 AP for small objects (area < 322 pixels) AP (medium) 0.5439 AP for medium-sized objects (322 < area < 962 pixels) AP (large) 0.7704 AP for large objects (area > 962 pixels) AR (maxDets=100) 0.8188 Average recall rate when detecting a maximum of 100 targets per image AR (small) 0.515 Average recall rate for small objects AR (large) 0.877 Average recall rate for large objects

[0118] The data shows that the model's core mean precision (AP) is as high as 0.6933, demonstrating its strong overall performance. Under standard IoU (0.5), the AP reaches 0.927, indicating strong object recognition capabilities. Furthermore, the model maintains a high precision of 0.7914 under standard IoU (0.75), showcasing its accurate localization ability. Meanwhile, the model's high recall rate of over 81% ensures an extremely low false negative rate, indicating that the model's training results are exceptionally good in terms of accuracy, recognition rate, and stability.

[0119] (2) DeepSeek-VL2-Tiny model inference experiment

[0120] The experiment was divided into two cases: with and without small model auxiliary information. The same set of test images were input into the visual language large model for experimental inference and its text accuracy was calculated. The total number of test images was 200.

[0121] Experiment 1: Without small model auxiliary information, the following is an example of using no information fusion prompt words:

[0122] "Please analyze and determine the type of work scenario in the diagram, whether the worker is wearing a safety helmet and safety belt, and assess the risk level. Please strictly follow the following format for output: Work Scenario Type: Safety Helmet Wearing Status: Safety Belt Wearing Status: Risk Level: (Select from low risk, medium risk, and high risk. Low risk means correctly and properly wearing a safety helmet and safety belt; medium risk means only wearing a safety belt; high risk means not wearing a safety belt or not wearing either a safety belt or a safety helmet)."

[0123] The large model description output is guided, and the model output results are as follows:

[0124] "Work scenario type: High-altitude power work Safety helmet wearing status: Wearing safety belt wearing status: Not wearing Hazard level: High risk."

[0125] Experiment 2: With small model auxiliary information, the following is a sample case of information fusion prompt words:

[0126] "We currently have auxiliary information: safety helmet (box1), safety belt (box2), ... Please analyze and determine the type of work scenario in the diagram and whether the worker is wearing a safety helmet and safety belt, and assess the risk level based on this information. Please strictly adhere to the following output format: Work Scenario Type: Safety Helmet Wearing Status: Safety Belt Wearing Status: Risk Level: (Select from low risk, medium risk, and high risk. Low risk means correctly and properly wearing the necessary protective equipment such as a safety helmet and safety belt; medium risk means only wearing a safety belt; high risk means not wearing a safety belt or neither wearing a safety belt nor a safety helmet)."

[0127] The large model description output is guided, and the model output results are as follows:

[0128] Work scenario type: High-altitude power work; Helmet wearing status: Wearing; Safety belt wearing status: Wearing; Risk level: Low risk

[0129] Through comparative experiments and test sample analysis, the reasoning accuracy rates of Experiment 1 and Experiment 2 were 93% and 96%, respectively. The accuracy rate of reasoning with small model-assisted information was 3% higher than that without small model-assisted information. Specific data are shown in Table 4.

[0130] Table 4. Experimental data for Experiments 1 and 2 experiment Number of test images Text accuracy No small model assistance 200 93% Small model assistance 200 96%

[0131] It is evident that the DeepSeek-VL2-Tiny visual language large model lacks inference accuracy without the assistance of the small model, but its inference is more accurate with the assistance of the small model, demonstrating the feasibility of the power high-altitude operation hazard identification method based on the combination of large and small models.

[0132] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods in any of the embodiments of the specification.

[0133] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any of the embodiments of the specification.

[0134] The modules or units in the device of this invention can be merged, divided, and deleted according to actual needs. The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that implementing all or part of the processes of the above embodiments and making equivalent changes according to the claims of this invention still fall within the scope of the invention.

Claims

1. A method for identifying hazards in high-altitude power operations based on a combination of large and small models, characterized in that, include: Real-time acquisition of high-altitude operation images and data during power high-altitude operations; The high-altitude operation image data is input into a pre-trained target detection mini-model to identify the protective equipment in the high-altitude operation image data and output auxiliary identification information of the protective equipment; wherein, the auxiliary identification information includes the box coordinates of the protective equipment; Based on the auxiliary identification information and the preset prompt word template, determine the large model prompt words used to assist in the identification of dangerous points in high-altitude power operations; The high-altitude operation image data and the large model prompts are input into the visual language large model, and the result of hazard point identification in the power high-altitude operation process is output.

2. The method for identifying hazardous points in high-altitude power operations based on a combination of large and small models as described in claim 1, characterized in that, The training methods for the object detection mini-model include: Acquire historical high-altitude operation images during historical power line high-altitude operations; The historical high-altitude operation image data is labeled to form sample training data; wherein, the labels include at least: with safety helmet, without safety helmet, with safety belt, without safety belt, and bucket; Configure model training parameters and train the target detection mini-model using the sample training data based on a three-stage refinement mechanism; wherein, the three-stage refinement mechanism includes: a first training stage, a second training stage, and a third training stage; the first training stage is used to focus on feature extraction of the core target, the second training stage is used to improve the detection accuracy and generalization ability of the mini-model, and the third training stage is used to optimize the head detection layer.

3. The method for identifying hazardous points in high-altitude power operations based on a combination of large and small models according to claim 2, characterized in that, The labeling of the historical high-altitude operation image data includes: The LabelImg software platform was used to manually label each historical high-altitude operation image. After annotation, the data is converted to the standard COCO format and divided into a training set and a test set according to a preset ratio.

4. The method for identifying hazardous points in high-altitude power operations based on a combination of large and small models as described in claim 3, characterized in that, The process of training the target detection mini-model using the sample training data based on a three-stage refinement mechanism includes: The iterative training rounds in the model training parameters are divided into three parts according to a preset rule: the first training stage, the second training stage, and the third training stage; each training stage includes at least one iterative training round. In the first training phase, the focus is on feature extraction of the core target and optimization of bounding box regression accuracy. After the first training phase has completed its iterative training rounds, if the average accuracy (AP) of the test set is not less than the preset first average accuracy threshold, the second training phase is initiated. Otherwise, the learning rate in the model training parameters is lowered and training continues until the average accuracy (AP) requirement is met. In the second training phase, the global optimal localization self-distillation module is enabled to improve the model's detection accuracy and generalization ability for small targets. At the same time, after the iterative training rounds of the second training phase, if the average accuracy AP of the test set is not less than the preset second average accuracy threshold, the third training phase is entered; otherwise, the iterative training rounds of this training phase are extended. The second average accuracy threshold is greater than the first average accuracy threshold. In the third training phase, the backbone network parameters of the model are frozen, and only the head detection layer is optimized. After training, it is verified whether the average precision (AP) meets the requirements of the third average precision threshold. If it does, training ends and the final target detection mini-model is output. Otherwise, training continues. The third average precision threshold is greater than the second average precision threshold.

5. The method for identifying hazardous points in high-altitude power operations based on a combination of large and small models as described in claim 1, characterized in that, The box coordinates of the protective equipment are [x1, y1; x2, y2], where [x1, y1] are the coordinates of the upper left corner of the rectangle that selects the identified protective equipment, and [x2, y2] are the coordinates of the lower right corner of the rectangle that selects the identified protective equipment.

6. The method for identifying hazardous points in high-altitude power operations based on a combination of large and small models as described in claim 1, characterized in that, The large model prompt word has a structure consisting of an auxiliary information embedding area, a task instruction area, and an output format area; The auxiliary information embedding area is used to embed auxiliary recognition information output by the target detection mini-model; the task instruction area is used to clarify the core tasks that the visual language large model needs to complete; the output format area is used to specify the structured format of the output of the visual language large model; wherein, the core tasks include: determining the type of work scenario, recognizing the status of protective equipment, and classifying the hazard level.

7. The method for identifying hazardous points in high-altitude power operations based on a combination of large and small models according to any one of claims 1-6, characterized in that, The large-scale visual language model is DeepSeek-VL2.

8. The method for identifying hazardous points in high-altitude power operations based on a combination of large and small models according to claim 1, characterized in that, Following the identification results of hazards during high-altitude power operations, the following further applies: Based on the hazard level output by the visual language big data model, a warning method corresponding to that hazard level is used to issue a warning.

9. A hazard identification system for high-altitude power operations based on a combination of large and small models, the system comprising: The module includes a field data acquisition module, a protective equipment identification module, a prompt word construction module, and a hazard point identification module. The on-site data acquisition module is deployed at the power high-altitude operation site to collect high-altitude operation image data in real time during the power high-altitude operation process, and transmit the collected high-altitude operation image data to the protective equipment identification module and the hazard point identification module. The protective equipment recognition module is equipped with a pre-trained target detection mini-model, which is used to identify protective equipment in the input high-altitude operation image data and output auxiliary recognition information of the protective equipment; and to transmit the output auxiliary recognition information to the prompt word construction module; wherein, the auxiliary recognition information includes the box coordinates of the protective equipment; The prompt word construction module is used to determine a large model prompt word for assisting in the identification of dangerous points in high-altitude power operations based on the auxiliary identification information and the preset prompt word template, and input the large model prompt word into the dangerous point identification module; The hazard identification module is equipped with a visual language model, which is used to output the hazard identification results in the process of high-altitude power operations based on the input high-altitude operation image data and the prompt words of the model.

10. The power high-altitude operation hazard identification system based on a combination of large and small models according to claim 9, further comprising: Danger warning module; The danger warning module is used to issue warnings based on the danger level output by the visual language large model, and to use a warning method corresponding to that danger level.

Images