Electric power field operation safety detection method and system based on hybrid expert model

Abstract

The invention belongs to the field of electric power and the technical field of artificial intelligence, and particularly relates to an electric power field operation safety detection method and system based on a hybrid expert model. According to the method, the visual language model and the YOLO target detection model are combined, and efficient and accurate electric power field operation safety detection is realized under the condition of few samples through a hybrid expert model architecture. In the training stage, through multi-model cooperative training, an expert model set for different scenes and targets is constructed. In the detection stage, a two-stage detection mechanism is adopted, YOLO is firstly used for rapid preliminary detection, then VLM is used for rechecking a low-confidence target, and finally detection results are integrated to ensure the accuracy of detection. The method aims to effectively cope with complex environments and potential safety hazards in electric power field operation, safety detection efficiency and accuracy are remarkably improved, and compared with a single model architecture, the method shows wider application scenes and higher detection quality in an electric power field operation scene.

Description

Technical Field

[0001] The present invention belongs to the field of electric power and artificial intelligence technology, and specifically relates to a method and system for detecting safety of electric power field operations based on a hybrid expert model. Background Art

[0002] The safety of power field operations plays a vital role in the power industry. In recent years, with the continuous expansion of the scale of power systems and the increase in operational complexity, safety accidents in power field operations have occurred frequently, posing a major threat to the safety of life and property. Traditional safety monitoring measures mainly rely on manual inspections, but this model has obvious limitations. Manual inspections are inefficient and easily affected by environmental changes and work fatigue, resulting in a high rate of missed detection of safety hazards, making it difficult to meet the growing safety needs of the power system.

[0003] Driven by artificial intelligence technology, intelligent safety detection systems based on image recognition and target detection have gradually become an important means of safety monitoring in the power industry. In particular, the deep learning model YOLO has attracted widespread attention due to its fast and real-time detection capabilities and its ability to process large-scale image data. The YOLO model can complete target detection with a single image scan, greatly improving the detection speed.

[0004] Nevertheless, existing technologies still face the following challenges in the field of power field operation safety detection:

[0005] First, the power field operation scenes are complex and changeable, and some specific types of safety hazards occur less frequently, making it difficult to collect enough sample data for model training. Existing target detection models perform poorly under small sample conditions and must be trained on a large number of samples to achieve good results, which brings difficulties in data collection and labeling in practical applications.

[0006] Secondly, even if a certain number of samples can be collected, the labeling work will require a lot of manpower and time, especially in on-site power operations, which requires the detection model to be able to quickly identify safety hazards in complex environments, but the real-time performance of existing models is often difficult to meet.

[0007] Thirdly, new safety hazards may appear in power field operation scenarios, which need to be identified and warned in a timely manner. However, the existing target detection model lacks open vocabulary detection capabilities and can only detect labels seen during training. For newly appeared targets, the model needs to be retrained, which has obvious deficiencies in real-time and application flexibility. The power field operation environment is changeable, and the types and manifestations of safety hazards in different scenarios are different. The existing models have poor adaptability in specific scenarios, and it is difficult to maintain consistent detection effects under different environmental conditions, which limits their scope of application in complex power field operations.

[0008] In response to the above challenges, the present invention proposes a real-time safety detection system for power field operations based on a hybrid expert model architecture of a multimodal large model to improve the accuracy of detection and real-time response capability, while reducing dependence on a large amount of sample data, enhancing the generalization ability and adaptability of the model, and providing the power industry with a more intelligent and efficient safety monitoring solution. Summary of the invention

[0009] In order to overcome the problems in the prior art, the present invention proposes a method and system for detecting safety of electric power field operations based on a hybrid expert model.

[0010] The technical solution of the present invention to solve the above technical problems is as follows:

[0011] In a first aspect, the present invention provides a method for detecting safety of electric power field operations based on a hybrid expert model, comprising the following steps:

[0012] In the training phase, a set of expert models for different scenarios and targets of power field operation safety detection is constructed through multi-model collaborative training, wherein the expert model set includes a scenario classification model, a general target detection model, a scenario-specific target detection model, a first multimodal target detection model, and a second multimodal target detection model;

[0013] In the detection stage, if in real-time detection mode, a two-stage detection mechanism is adopted, firstly, a general target detection model performs a quick preliminary detection and returns a first detection result, wherein the first detection result includes a first detection frame and its confidence, and the confidence of the first detection frame is referred to as a first confidence; the first confidence is compared with a preset confidence threshold; if the first confidence is higher than the preset confidence threshold, the high confidence result is retained; if the first confidence is lower than the preset confidence threshold, the first multimodal target detection model is used for re-inspection to obtain a re-inspection result, and the re-inspection result is merged with the first detection result; the detection result is integrated and output;

[0014] In the detection stage, if in non-real-time detection mode, a scene classification model is used to perform scene classification to obtain a scene classification result; based on the scene classification result, a scene-specific target detection model is used to perform detection and return a second detection result, the second detection result includes a second detection box and its confidence, and the confidence of the second detection box is called the second confidence, and at the same time, a general target detection model is used for global detection to return a third detection result, the third detection result includes a third detection box and its global confidence; the second confidence and the global confidence are compared with the preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, the high confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, it is regarded as a low confidence target, and the second multimodal target detection model is used to re-check the low confidence target; the detection results of all models are merged to output the final detection result.

[0015] Furthermore, before the training stage, it also includes constructing a multimodal dataset, wherein the multimodal dataset includes a scene classification dataset and a target detection dataset.

[0016] Furthermore, the first multimodal target detection model is used to detect all labeled targets in all scenarios; and the second multimodal target detection model is used to detect specific labeled targets in specific scenarios.

[0017] Furthermore, in the detection stage, if in real-time detection mode, the detection results are integrated and output, specifically including: in the merging process, checking the overlap of the detection frames, if the overlap is greater than the preset overlap threshold, that is, for the frames with high overlap and the same category, they are retained according to priority:

[0018] First priority: the re-test results of the first multimodal object detection model;

[0019] Second priority: the first detection result returned by the general object detection model that is higher than the preset confidence threshold;

[0020] Third priority: the first detection result returned by the general object detection model that is lower than the preset confidence threshold, except for those that have passed the re-inspection of the first multimodal object detection model;

[0021] In the merged result, only one optimal detection box is retained for each target.

[0022] Furthermore, in the detection stage, if in non-real-time detection mode, the detection results of all models are merged and the final detection results are output, including: during the merging process, the overlap of the detection frames is checked. If the overlap is greater than the preset overlap threshold, that is, for frames with high overlap and the same category, they are retained according to priority:

[0023] First priority: re-test results of the second multimodal object detection model;

[0024] Second priority: high-confidence detection results returned by the scene-specific object detection model and the general object detection model, where the high-confidence detection results are detection results that are higher than a preset confidence threshold;

[0025] Third priority: low confidence results returned by the scene-specific target detection model and the general target detection model, where the low confidence detection results are detection results below a preset confidence threshold, except for those that have passed the re-inspection of the second multimodal target detection model;

[0026] In the merged and filtered results, only one optimal detection box is retained for each target.

[0027] Furthermore, the scene classification model is a VLM-Classify model, the general target detection model is a YOLO-All model, the scene-specific target detection model is a YOLO1-5 model, the first multimodal target detection model is a VLM-Detection1 model, and the second multimodal target detection model is a VLM-Detection2 model.

[0028] In a second aspect, the present invention also provides a power field operation safety detection system based on a hybrid expert model, comprising: a model training unit, a real-time detection unit and a non-real-time detection unit;

[0029] The model training unit is used for multimodal data set construction and training evaluation of hybrid expert model architecture;

[0030] The real-time detection unit is used for safety detection of power field operations in real time and outputs the detection results;

[0031] The non-real-time detection unit is used for safety detection of power field operations in a non-real-time state and outputs detection results.

[0032] Furthermore, the model training unit includes a data set construction module, a prompt engineering test module, a scene classification training module, a target detection model training module, an optimization strategy application module and a model evaluation module;

[0033] The data set construction module is used to construct a multimodal data set, and the multimodal data set includes a scene classification data set and a target detection data set; the prompt engineering test module is used to perform prompt engineering testing to determine the applicability of the model and the required prompt word type; the scene classification training module is used to train the scene classification training model; the target detection model training module is used to train a general target detection model, a scene-specific target detection model, a first multimodal target detection model and a second multimodal target detection model; the optimization strategy application module is used to apply data enhancement, transfer learning, mixed precision training and learning rate scheduling optimization strategies; the model evaluation module is used to evaluate the performance of the trained model.

[0034] Further, the real-time detection unit includes a first image acquisition module, a YOLO detection module, a first detection result storage module, a first confidence check module, a first VLM re-check module, a first result merging module, a first overlapping frame processing module and a first output module;

[0035] Among them, the first image acquisition module is used to acquire image data to be detected; the YOLO detection module is used to input the acquired image data into a general target detection model for target detection; the first detection result saving module is used to save the first detection result of the general target detection model, the first detection result includes a first detection frame and its confidence, and the confidence of the first detection frame is called the first confidence; the first confidence checking module is used to check the confidence and filter out the detection frames whose confidence is lower than a preset threshold; the first VLM re-inspection module is used to input the detection frames lower than the preset threshold into the first confidence checking model for re-inspection to obtain the re-inspection result; the first result merging module is used to merge the re-inspection result with the first detection result; the first overlapping frame processing module is used to check the overlap of the merged detection frames, and filter the detection frames whose overlap is higher than the preset overlap threshold and whose categories are the same; the first output module is used to output the final detection result after screening and merging.

[0036] Furthermore, the non-real-time detection unit includes a second image acquisition module, a scene classification module, a detection model selection module, a second detection result storage module, a second confidence check module, a second VLM re-check module, a second result merging module, a second overlapping frame processing module and a second output module; wherein the scene classification module is used to input the acquired image data into the scene classification for scene classification; the detection model selection module is used to select the corresponding scene-specific target detection model according to the scene classification result, and at the same time input the general target detection model for global detection; the second detection result storage module is used to save the scene-specific target detection model and the general target detection model The method comprises the following steps: a first detection frame is used to detect the detection result of the scene-specific target detection model and the second detection frame processing module; a second confidence checking module is used to check the confidence of each detection frame and filter out the detection frames with confidence lower than the preset confidence threshold; a second VLM re-checking module is used to input the detection frames with confidence lower than the preset confidence threshold into the second multimodal target detection model for re-checking; a second result merging module is used to merge the re-checking result with the detection results of the scene-specific target detection model and the general target detection model; a second overlapping frame processing module is used to check the overlap of the merged detection frames and filter out the frames with overlap higher than the preset threshold and the same category; a second output module is used to output the final detection result after screening and merging.

[0037] Compared with the prior art, the present invention has the following technical effects:

[0038] (1) The present invention designs a unique hybrid expert model (MoE) architecture, which organically integrates the advantages of the visual language model (VLM) and the YOLO model to achieve a breakthrough improvement in the safety detection of power field operations. This architecture uses a scene-adaptive expert model selection mechanism to flexibly call the most appropriate model according to different operation scenarios and detection targets, giving full play to the zero-sample and few-sample learning capabilities of the VLM and the real-time detection characteristics of the YOLO model. This architecture effectively solves the challenges of scarce sample data, insufficient detection accuracy, poor real-time response, and limited generalization ability in traditional detection systems, and achieves high-precision and high-real-time safety detection of power operations, providing intelligent protection for complex field operations.

[0039] (2) The present invention innovatively proposes a multi-model collaborative training strategy. By training expert models for different scenarios and targets, an efficient expert model set is constructed, including the scene classification model VLM-Classify, the general target detection model YOLO-All, the scene-specific target detection model YOLO1-5, and the multimodal target detection models VLM-Detection1 and VLM-Detection2. Each model is collaboratively optimized during training, which effectively improves the system's adaptability in complex and changeable power operation environments. Through this collaborative training method, the system can achieve fast and accurate detection in different scenarios, greatly improving the accuracy and efficiency of safety detection.

[0040] (3) The present invention innovatively designs a two-stage detection mechanism, combining the rapid initial detection capability of the YOLO model and the precise re-inspection capability of the VLM model, significantly improving the overall detection effect. In the first stage, the YOLO model quickly detects potential targets; in the second stage, the VLM model re-inspects low-confidence targets detected by the YOLO model. This mechanism not only ensures the detection speed of the system, but also shows excellent performance in processing complex scenes and low-confidence targets, overcoming the limitation of traditional methods in their insufficient ability to identify new safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is the overall framework of the model of the present invention;

[0043] Figure 2 It is a schematic diagram of the detection method of the present invention;

[0044] Figure 3 It is a schematic diagram of the real-time detection process of the present invention;

[0045] Figure 4 It is a schematic diagram of the non-real-time detection process of the present invention. DETAILED DESCRIPTION

[0046] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the technical solutions proposed by the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. The specific features, structures or characteristics in one or more embodiments may be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by technicians in the technical field of the present invention.

[0047] This paper proposes a method for power field operation safety detection based on a hybrid expert model, aiming to solve the problems existing in the existing power field safety detection technology, such as insufficient sample data, poor detection accuracy and real-time performance, and limited generalization ability. This method cleverly combines the visual language model VLM and the YOLO model to achieve efficient and accurate power field operation safety detection under few sample conditions.

[0048] The present invention designs a novel hybrid expert model architecture MoE, which makes full use of the zero-sample and few-sample learning capabilities of VLM and the real-time detection characteristics of YOLO model. This architecture can not only improve the detection accuracy, but also maintain the real-time response capability of the system, which is particularly suitable for complex and changeable environments such as power field operations.

[0049] The implementation process of the present invention can be divided into two main parts: the training phase and the detection phase. In the training phase, the system builds a set of expert models for different scenarios and targets through multi-model collaborative training. In the detection phase, the system adopts a two-stage detection mechanism, first with YOLO for rapid preliminary detection, then VLM for re-inspection of low-confidence targets, and finally integrates the detection results to ensure the accuracy of the detection.

[0050] In the embodiment of the present invention, refer to Figure 1-Figure 4 , discloses a method for safety detection of power field operations based on a hybrid expert model, comprising the following steps:

[0051] In the training phase, a set of expert models for different scenarios and targets of power field operation safety detection is constructed through multi-model collaborative training, wherein the expert model set includes a scenario classification model, a general target detection model, a scenario-specific target detection model, a first multimodal target detection model, and a second multimodal target detection model;

[0052] In the detection stage, if in real-time detection mode, a two-stage detection mechanism is adopted, firstly, a general target detection model performs a quick preliminary detection and returns a first detection result, wherein the first detection result includes a first detection frame and its confidence, and the confidence of the first detection frame is referred to as a first confidence; the first confidence is compared with a preset confidence threshold; if the first confidence is higher than the preset confidence threshold, the high confidence result is retained; if the first confidence is lower than the preset confidence threshold, the first multimodal target detection model is used for re-inspection to obtain a re-inspection result, and the re-inspection result is merged with the first detection result; the detection result is integrated and output;

[0053] In the detection stage, if in non-real-time detection mode, a scene classification model is used to perform scene classification to obtain a scene classification result; based on the scene classification result, a scene-specific target detection model is used to perform detection and return a second detection result, the second detection result includes a second detection box and its confidence, and the confidence of the second detection box is called the second confidence, and at the same time, a general target detection model is used for global detection to return a third detection result, the third detection result includes a third detection box and its global confidence; the second confidence and the global confidence are compared with the preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, the high confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, it is regarded as a low confidence target, and the second multimodal target detection model is used to re-check the low confidence target; the detection results of all models are merged to output the final detection result.

[0054] The following is a detailed explanation of each of the above steps:

[0055] Step 100: Construct a multimodal dataset, including a scene classification dataset and an object detection dataset.

[0056] The scene classification dataset contains system prompts, user instructions, correct answers, and images to be detected. The object detection dataset contains system prompts, user instructions, correct answers (including labels and detection box positions), and images to be detected.

[0057] Step 200: During the training phase, a set of expert models for different scenarios and targets is constructed through multi-model collaborative training, wherein the expert model set includes a scene classification model, a general target detection model, a scene-specific target detection model, a first multimodal target detection model, and a second multimodal target detection model.

[0058] The scene classification model is used to classify scenes and guide the selection and parameter adjustment of subsequent models; the general target detection model is used for preliminary target detection in all scenes; the scene-specific target detection model is used for accurate target detection in specific scenes; the first multimodal target detection model and the second multimodal target detection model are used for multimodal target detection to further improve detection accuracy. This multi-model collaborative training method combines the advantages of different models, enabling the system to adapt to the changing power field operation scenes and improve detection accuracy and efficiency.

[0059] As a specific example, this step 200 may include the following sub-steps:

[0060] Step 210: training a scene classification model, namely a VLM-Classify model, for quickly learning scene features on limited power operation data, providing scene information for subsequent detection models, and guiding subsequent model selection and parameter adjustment;

[0061] The VLM-Classify model is a pre-trained model based on the internvl2 series. It is fine-tuned to adapt to the power field operation dataset and becomes a visual language multimodal model specifically for the classification of power operation scenes.

[0062] The improved Lora efficient fine-tuning technology used enables the model to quickly learn scene features on limited power operation data, thereby significantly improving the model training efficiency while ensuring detection accuracy. This step provides accurate scene information for subsequent detection models, guides model selection and parameter adjustment, and ensures that the entire system can select the most appropriate expert model according to different on-site operation scenarios, thereby achieving improved detection accuracy and efficiency.

[0063] Among them, the improved Lora efficient fine-tuning technology refers to assigning different learning rates to different modules of Lora during training, thereby improving the convergence speed of Lora fine-tuning, in order to get closer to the performance of full fine-tuning.

[0064] Step 220: training a general target detection model, namely the YOLO-All model, for preliminary detection of targets in all scenarios;

[0065] For full-scenario detection, a general target detection model, namely the YOLO-All model, is introduced. The YOLO-All model is based on YOLO v10-X. The training of the YOLO-All model not only focuses on hyperparameter tuning and model architecture optimization, but also pays special attention to improving the generalization ability and robustness of the model, which enables the model to stably detect various types of targets in a variety of operating scenarios. Under the framework of collaborative training, the YOLO-All model can quickly perform preliminary detection of targets in all scenarios, providing basic results for subsequent model re-inspection.

[0066] Step 230: training a scene-specific target detection model, namely a YOLO1-5 model, for detecting corresponding label targets in a specific scene;

[0067] For typical power field operation scenarios, scenario-specific target detection models, namely YOLO1-5 models, were trained. These models are based on YOLO v10-M and are trained with data from specific scenarios to more accurately detect corresponding labeled targets in specific scenarios. Through this specialized training, the detection performance of the model in specific scenarios is significantly improved while maintaining a relatively fast detection speed.

[0068] For example, five scene-specific object detection YOLO1-5 models were trained for five typical power site operation scenarios, namely: aerial work, ladder work, dress code, crossing safety fences, and hole detection at the work site. In fact, the categories of power site operation scenarios can be increased or decreased accordingly depending on the specific tasks.

[0069] Step 240: training a multimodal target detection model, including a first multimodal target detection model and a second multimodal target detection model, wherein the first multimodal target detection model VLM-Detection1 is used to detect all label targets in all scenarios, and the second multimodal target detection model VLM-Detection2 is used to detect specific label targets in specific scenarios.

[0070] Two multimodal target detection models, VLM-Detection1 and VLM-Detection2, are introduced. These two models are based on the internvl2-8B pre-trained model and have been specially fine-tuned to handle the complex environment of power field operations. The special fine-tuning refers to targeted training on the power field operation safety detection dataset, and through transfer learning, the general target detection capability of the model is generalized to the power field operation safety detection.

[0071] Among them, the VLM-Detection1 model is used to detect all labeled targets in all scenarios, while the VLM-Detection2 model is used to detect specific labeled targets in specific scenarios. Considering the strong generalization performance of the VLM model and the high deployment cost, all tasks of the VLM-Detection2 model are completed by training a model for multi-task learning. The main difference from the VLM-Detection1 model is that the training samples encountered by the VLM-Detection2 model during training clearly state the scene to which the sample belongs.

[0072] During the training process, a hybrid training strategy was adopted, which included training different combinations of LLM (Large Language Model) modules, visual modules, and projection layers. While maintaining detection efficiency, it achieved a significant improvement in accuracy, especially in few-sample learning and open vocabulary detection.

[0073] The LLM model is generally composed of an image encoder (visual module, ViT), a projection layer (aligning visual features with text features, generally MLP), and an LLM backbone (large language model, LLM).

[0074] In VLM, the LLM module refers to the large language model backbone. The main function of this module is to process and generate text information, thereby enhancing the ability of VLM in multimodal tasks.

[0075] The functions of the LLM module are as follows: The LLM module can integrate information from visual input and language input and generate language descriptions related to them. This makes VLM a generative artificial intelligence model. By standardizing and parsing the output of VLM, it can handle a variety of tasks, including image classification, target detection, etc. In addition, the LLM module also enables VLM to support prompt engineering, that is, guiding the model to complete specific tasks through specific instructions. This method can improve the effect and adaptability of the model without changing the core parameters of the model.

[0076] During the training process, depending on the training order and task difficulty, one or two of the three modules can be frozen (i.e., parameters are not updated) when necessary. This can bring two benefits: avoid catastrophic forgetting: prevent the model from learning new knowledge and forgetting old knowledge, which reduces generalization performance; reduce resource requirements: due to freezing certain parameters, the resource requirements during model training will also decrease.

[0077] The pre-trained VLM model uses the internvl2-8B model, which uses InternViT-300M-448px as the visual module and internlm2_5-7b-chat as the LLM module, and connects the visual features with the text features by training the visual projection layer.

[0078] Step 300: In the detection phase, if in real-time detection mode, a two-stage detection mechanism is adopted, firstly, the general target detection model performs a quick preliminary detection, then the first multimodal target detection model performs a re-inspection on low-confidence targets, and finally integrates the detection results.

[0079] During the detection phase, if in non-real-time detection mode, a scene classification model is used to perform scene classification; based on the scene classification results, a scene-specific target detection model is used for detection, and at the same time, the detection results are passed into a general target detection model for global detection; the confidence level is checked, and low-confidence targets are passed into the first multimodal target detection model for re-inspection; the detection results of all models are merged, and the final detection results are output.

[0080] This step may include the following sub-steps:

[0081] Step 310: In the detection phase, if in real-time detection mode, a two-stage detection mechanism is adopted, and the general target detection model first performs a quick preliminary detection to return the first detection result, the first detection result includes the first detection frame and its confidence, and the confidence of the first detection frame is called the first confidence; the first confidence is compared with the preset confidence threshold; if the first confidence is higher than the preset confidence threshold, the high confidence result is retained; if the first confidence is lower than the preset confidence threshold, the first multimodal target detection model is used for re-inspection to obtain a re-inspection result, and the re-inspection result is merged with the first detection result; the detection result is integrated and output. Specifically including:

[0082] Step 311: Obtain the image to be detected, and pass it into the general target detection model for fast preliminary detection to return a first detection result, wherein the first detection result includes a detection box and its attribute description and confidence. The confidence of the first detection box is called a first confidence.

[0083] The attribute description refers to the name of the category. The confidence is the degree of certainty of the existence of the target object in the detection box output by the YOLO model during detection. If the confidence is close to 1, it means that the model is very sure that the detection box contains the target object; if the confidence is close to 0, it means that the model believes that the detection box may not contain the target object.

[0084] Step 312: Preset a confidence threshold and evaluate the confidence of the detection box, that is, compare the first confidence with the preset confidence threshold; regard the detection results below the preset confidence threshold as low-confidence detection results, and regard the detection results above the preset confidence threshold as high-confidence detection results, and retain the high-confidence detection results; pass the low-confidence detection results to the first multimodal target detection model for re-inspection to obtain re-inspection results, thereby obtaining more accurate labeling results.

[0085] Step 313: merging the re-detection result of the first multimodal object detection model with the first detection result of the general object detection model;

[0086] During the merging process, check the overlap of the detection frames. If the overlap is greater than the preset overlap threshold, that is, for frames with a high overlap and the same category, they are retained according to priority:

[0087] First priority: the re-test results of the first multimodal object detection model;

[0088] Second priority: the first detection result returned by the general object detection model that is higher than the preset confidence threshold;

[0089] Third priority: the first detection result returned by the general object detection model that is lower than the preset confidence threshold, except for those that have passed the re-inspection of the first multimodal object detection model;

[0090] That is, first retain the re-test results of the VLM-Detection1 model; if the VLM-Detection1 model does not detect a certain target, retain the first detection result with higher confidence in the general target detection model; if the general target detection model also does not detect it or the confidence is low, consider whether to retain the first detection result of the general target detection model with low confidence;

[0091] In the merged result, only one optimal detection box is retained for each target.

[0092] Step 314: Output the combined final detection result.

[0093] According to the screening results, the final detection results are output, including detection box, attribute description, confidence level and other information.

[0094] Step 320: In the detection stage, if in non-real-time detection mode, a scene classification model is used to perform scene classification to obtain a scene classification result; based on the scene classification result, a scene-specific target detection model is used to perform detection and return a second detection result, the second detection result includes a second detection box and its confidence, and the confidence of the second detection box is called the second confidence, and at the same time, a general target detection model is used to perform global detection and return a third detection result, the third detection result includes a third detection box and its global confidence; the second confidence and the global confidence are compared with a preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, the high confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, it is regarded as a low confidence target, and the second multimodal target detection model is used to re-check the low confidence target; the detection results of all models are merged to output the final detection result.

[0095] The processing flow of step 320 is more complex and comprehensive. Specifically, it includes the following sub-steps:

[0096] Step 321: Obtain the image to be detected and pass it into the scene classification model, that is, the VLM-Classify model, to perform scene classification, and the scene classification model returns the scene classification result;

[0097] Step 322: According to the scene classification result, a model that best matches the current scene is selected from the scene-specific object detection model for detection. The scene-specific object detection model returns a detection result, which includes a second detection box, a property description thereof, a confidence level, etc. The confidence level of the second detection box is regarded as a second confidence level.

[0098] At the same time, the image to be detected is passed into the general target detection model, namely the YOLO-All model, for global detection, and the general target detection model returns the global detection result.

[0099] Step 323: traverse the detection results of the general target detection model and the scene-specific target detection model, check the confidence of each detection box, that is, preset the confidence threshold, and compare the second confidence and the global confidence with the preset confidence threshold respectively; if the first confidence or the global confidence is higher than the preset confidence threshold, retain the high confidence result; if the second confidence or the global confidence is lower than the preset confidence detection threshold, pass it to the second multimodal target detection model VLM-Detection2 for re-inspection.

[0100] The second multimodal object detection model, VLM-Detection2, takes into account scene classification information when performing detection and returns more accurate detections.

[0101] Step 324: Merge the detection results of all models.

[0102] Perform overlap check on the merged detection results: set an overlap threshold, and treat detection frames with overlap higher than the threshold and the same category as duplicate detection results.

[0103] That is, for boxes with high overlap and the same category, they are retained according to priority:

[0104] First priority: re-test results of the second multimodal object detection model;

[0105] Second priority: high-confidence detection results returned by the scene-specific object detection model and the general object detection model, where the high-confidence detection results are detection results that are higher than a preset confidence threshold;

[0106] Third priority: low confidence results returned by the scene-specific target detection model and the general target detection model, where the low confidence detection results are detection results below a preset confidence threshold, except for those that have passed the re-inspection of the second multimodal target detection model;

[0107] In the merged and filtered results, only one optimal detection box is retained for each target.

[0108] That is, first retain the re-inspection results of the VLM-Detection2 model; if VLM-Detection2 does not detect a target, retain the detection results with higher confidence in the scene-specific target detection model and the general target detection model; if the scene-specific target detection model and the general target detection model also do not detect the target or have low confidence, consider whether to retain the detection results of the scene-specific target detection model and the general target detection model with low confidence.

[0109] Step 325: Output the final detection result:

[0110] According to the screening results, the final detection results are output. The output results include information such as detection box, attribute description and confidence, which can be presented in the form of image annotation or text list.

[0111] This multi-stage, multi-model detection strategy fully leverages the strengths of each model. The YOLO model provides fast preliminary detection results, while the VLM model plays a key role in dealing with complex scenarios and low-confidence targets. In this way, the system can not only quickly identify common safety hazards, but also accurately detect rare or emerging safety issues.

[0112] Based on the same inventive concept, an embodiment of the present invention also provides a hybrid expert model-based power field operation safety detection system for implementing the above-mentioned hybrid expert model-based power field operation safety detection method. The implementation solution provided by the system to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more system embodiments provided below can refer to the above-mentioned limitations on a hybrid expert model-based power field operation safety detection method, which will not be repeated here.

[0113] In one embodiment, a power field operation safety detection system based on a hybrid expert model is provided, comprising a model training unit, a real-time detection unit and a non-real-time detection unit; the model training unit is used for multimodal data set construction and training evaluation of the hybrid expert model architecture; the real-time detection unit is used for power field operation safety detection in real-time state; the non-real-time detection unit is used for power field operation safety detection in non-real-time state.

[0114] In this embodiment, the model training unit includes a data set construction module, a prompt engineering test module, a scene classification training module, a target detection model training module, an optimization strategy application module and a model evaluation module;

[0115] The data set construction module is used to construct a multimodal data set, and the multimodal data set includes a scene classification data set and a target detection data set; the prompt engineering test module is used to perform prompt engineering testing to determine the applicability of the model and the required prompt word type; the scene classification training module is used to train the scene classification training model; the target detection model training module is used to train a general target detection model, a scene-specific target detection model, a first multimodal target detection model and a second multimodal target detection model; the optimization strategy application module is used to apply data enhancement, transfer learning, mixed precision training and learning rate scheduling optimization strategies; the model evaluation module is used to evaluate the performance of the trained model.

[0116] In a specific embodiment, the data set construction module is used to construct a multimodal data set, which includes a scene classification data set and a target detection data set. The scene classification data set includes system prompt words, user instructions, correct answers, and images to be detected; the target detection data set includes system prompt words, user instructions, correct answers, and images to be detected, wherein the correct answer of the target detection data set includes the label of the target category and the position information of the detection box, the label is used to identify the category of the target, and the detection box position describes the exact position of the target in the image.

[0117] In a specific embodiment, the prompt engineering test module is used to perform prompt engineering testing, the purpose of which is to determine whether the model is suitable for the task and to understand the type of prompt words required by the model to complete a specific task. The testing process includes designing clear prompt words, requiring the model to output results in a specific format, and analyzing the output of the model.

[0118] When using a large multimodal model for scene classification and object detection tasks, the format of the sample construction often determines its actual effect. Before constructing the sample, do some prompt engineering tests on the model for two purposes:

[0119] First, determine whether the model is suitable for performing scene classification and target detection tasks, that is, determine whether the model can be guided to complete scene classification and target detection tasks by constructing prompt words.

[0120] The method adopted is to clearly and unambiguously state the purpose of the task in the prompt words, provide the background information needed to complete the task, and require the model to output the detection results in a specific format string such as Json and XML to facilitate parsing and comparison of the answers.

[0121] At this point, the accuracy of the model in performing related tasks is not required, and considering the randomness of the AIGC (AI Generated Content) model generation, the results given by the model at this stage may not perfectly meet the requirements, and there will definitely be a large number of unparseable outputs.

[0122] Second, through testing, we can understand the types of prompt words that the model needs to complete specific tasks, which will be used as system prompt words to be filled in when constructing training samples, that is:

[0123] (1) Use data engineering methods to understand the model's preferences as much as possible;

[0124] (2) construct training samples in the input and output format preferred by the model;

[0125] (3) The dataset constructed by this method can achieve twice the result with half the effort during training because it fits the model generation preference. The model loss is often lower at the beginning of training.

[0126] For the target detection task, you need to know in advance how the pre-trained model perceives the input image, which refers to the reference coordinate system of the annotation box given by the target detection task. Specifically for the multimodal large model, its generation method is consistent with that of the large language model, which is generated token by token, and the result is in string format. Therefore, you need to understand how the model tends to represent categories and how to annotate the box diagram when performing the target detection task.

[0127] After testing the prompt engineering, it was found that the Internvl2-8B model tends to describe objects or areas in the image in XML tag format, the format is as follows:

[0128] <ref> human< / ref> <box> [[221,223,940,652]]< / box> ;

[0129] in, <ref>Tag represents the label or name of the object. <box>The labels represent the detection boxes of objects in the image, expressed in coordinate form. In particular, the model prefers <box>The label treats any incoming image as 1000*1000 in size, and the given annotation boxes represent the Xmin, Ymin, Xmax, and Ymax after the size is scaled.

[0130] Scene classification, this task is for scene classification, corresponding to the VLM-Classify model. The dataset structure of the scene classification task includes system (system prompt words), query (user instructions), response (correct answer), and images (images to be detected). System prompt words include preference prompt words to guide the model to complete the task, such as role positioning, description of the scene to be classified, output format, etc.; response (correct answer), converted from label information, contains the correct category of the scene, represented by a string. Each image data contains a variety of typical scenes. For example, workers at different work sites may have irregularities in their outfits, so the labeled data may contain multiple categories. This task is a typical multi-classification task.

[0131] After the above processing, the sample data for the image classification task is constructed in a generated way.

[0132] An example is as follows:

[0133] "system":"

[0134] #Role

[0135] You are an object detector.

[0136] #Task

[0137] You need to detect the content of the pictures uploaded by users and return the detection results.

[0138] The pictures focus on five typical scenes of power sites:

[0139] - Safety belts are not worn properly when working at heights. Check the workers and safety belts as a whole. - No one is holding the ladder when working on the ladder. Check; workers on the ladder.

[0140] -Hole at the work site is not covered or protected (holes refer to square or round holes on the ground at the work site). Detect the holes.

[0141] -Staff are not dressed properly. All staff who are not dressed properly will be checked.

[0142] - Workers are crossing or under the safety fence, detect workers who are crossing the fence.

[0143] The five tags are: "safetyBelt", "ladderSupport", "holeProtection", "workwear", and "barrierCrossing";

[0144] You need to <ref> class_name< / ref> <box> [[x,x,x,x]]< / box> Returns the detection result, where 'class_name' must be one of the five label names.

[0145] Just return the detection results in the given format without any other interpretation.

[0146] There may be multiple violations in one picture. For example, workers at different work sites may not be dressed properly. Please provide all eligible targets to be detected.

[0147] If there is no typical violation scene in the picture, return <ref> class_name< / ref> <box> [[0,0,0,0]]< / box> ",

[0148] "query":

[0149] "Please check the illegal content in this picture",

[0150] "response":

[0151] " <ref> holeProtection< / ref> <box> [[221,223,940,652]]< / box> ",

[0152] "images":

[0153] [{images_path}].

[0154] In a specific embodiment, the scene classification model training module is used to train the VLM-Classify model. This model is based on the internvl2 series and is used to accurately classify power field operation scenes. The training process uses the improved Lora efficient fine-tuning technology to improve the training efficiency while ensuring model performance by training different combinations of LLM modules, visual modules and projection layers. This step provides accurate scene information for subsequent detection models, guides model selection and parameter adjustment, and ensures that the entire system can select the most appropriate expert model according to different field operation scenarios, thereby achieving a dual improvement in detection accuracy and efficiency.

[0155] In a specific embodiment, the target detection model training module trains multiple target detection models, including: YOLO-All: a general target detection model based on YOLO v10-X; YOLO1-5: five scene-specific target detection models based on YOLO v10-M; VLM-Detection1 and VLM-Detection2: two multimodal target detection models based on the internvl2 series.

[0156] In a specific embodiment, the optimization strategy application module applies multiple optimization strategies during the training process, including data enhancement, transfer learning, mixed precision training, and learning rate scheduling.

[0157] The training set is expanded through data augmentation technology to improve the generalization ability of the model: (1) Image rotation: randomly rotate the image to increase the model's ability to recognize objects at different angles; (2) Flip: flip the image horizontally and vertically to improve the model's directional invariance; (3) Scaling: randomly scale the image to improve the model's ability to recognize objects of different sizes; (4) Color jittering: adjust the brightness, contrast, and saturation of the image to improve the model's performance under different lighting conditions.

[0158] Leverage pre-trained models for transfer learning to speed up training and improve model performance: (1) Use the InternVL2-8B model pre-trained on a large-scale dataset as the base model; (2) Freeze some underlying feature extraction layers and only fine-tune the high-level semantic understanding layers; (3) Use a progressive unfreezing strategy to gradually unfreeze more layers to achieve better feature adaptation.

[0159] InternVL2 uses the ViTMLP-LLM architecture, which is similar to the popular multimodal large language model (MLLM). This architecture combines the pre-trained InternViT-300M visual model (ViT) with the Internlm2.5-7b-chat language model (LLM) through a multi-layer perceptron (MLP) projection layer to achieve semantic alignment of visual-text features. Therefore, InternVL2-8B mainly consists of three parts: ViT+MLP+LLM, where ViT is the visual model module, MLP is the projection layer, and LLM is the language model module. On this basis, the transfer learning strategy is:

[0160] First, freeze some of the underlying feature extraction layers and only fine-tune the high-level semantic understanding layer:

[0161] Freeze the InternViT-300M module, which is the visual encoder part of the model and is responsible for low-level feature extraction. Freezing this module can preserve the pre-trained general visual features.

[0162] Frozen MLP Projector, this module is responsible for projecting visual features into language space, keeping it frozen can maintain visual-language alignment;

[0163] Unfreeze the Internlm2.5-7b-chat language model layer and train the model to perform semantic understanding and task-related reasoning;

[0164] Second, a progressive unfreezing strategy is adopted to gradually unfreeze more layers to achieve better feature adaptation:

[0165] Phase 1: Fine-tune only the Internlm2.5-7b-chat language model layer and train it for a certain number of rounds to allow the model to initially adapt to the new task;

[0166] Stage 2: Unfreeze the MLP Projector and optimize the visual-language alignment;

[0167] Stage 3: Selectively unfreeze the high layers of InternViT-6B and fine-tune the visual features for specific tasks;

[0168] Gradually adjust the learning rate, gradually reducing it as more layers are unfrozen to prevent overfitting and stabilize the training process.

[0169] Through this incremental approach, the model can gradually adapt to new tasks while retaining the strong features of the pre-trained model. This strategy balances the efficiency of transfer learning and the adaptability of the model to new tasks.

[0170] Mixed precision training is used to improve training speed and reduce memory usage while ensuring model accuracy: (1) Use FP16 and FP32 mixed precision; (2) Use FP16 in forward propagation and backpropagation, and use FP32 when updating weights; (3) Dynamically adjust the loss scaling factor to prevent gradient underflow.

[0171] Use appropriate learning rate scheduling strategies to improve the model convergence speed and final performance: (1) Use the warmup strategy to gradually increase the learning rate in the early stage of training; (2) Use cosine annealing learning rate scheduling to gradually reduce the learning rate in the later stage of training; (3) Implement periodic changes in the learning rate to help the model escape from the local optimum.

[0172] Finally, the model evaluation module evaluates the performance of the trained model. Evaluation indicators include accuracy, recall, F1 score, etc. to ensure that the model meets the expected performance standards.

[0173] The method of this embodiment successfully trains a series of models suitable for safety detection of power field operations by constructing a special multimodal data set and combining a variety of advanced training strategies and optimization techniques. These models include scene classification models and multiple target detection models, which together constitute a powerful hybrid expert model system. The advantage of this training method is that it makes full use of the zero-sample and few-sample learning capabilities of the visual language model and the real-time detection characteristics of the YOLO model. Through multi-model collaborative training, the system can adapt to the complex and changing environment of power field operations, improve detection accuracy while maintaining real-time response capabilities.

[0174] In this embodiment, the real-time detection unit includes a first image acquisition module, a YOLO detection module, a first detection result storage module, a first confidence check module, a first VLM re-check module, a first result merging module, a first overlapping frame processing module and a first output module;

[0175] Among them, the first image acquisition module is used to acquire image data to be detected; the YOLO detection module is used to input the acquired image data into a general target detection model for target detection; the first detection result saving module is used to save the first detection result of the general target detection model, the first detection result includes a first detection frame and its confidence, and the confidence of the first detection frame is called the first confidence; the first confidence checking module is used to check the confidence and filter out the detection frames whose confidence is lower than a preset threshold; the first VLM re-inspection module is used to input the detection frames lower than the preset threshold into the first confidence checking model for re-inspection to obtain the re-inspection result; the first result merging module is used to merge the re-inspection result with the first detection result; the first overlapping frame processing module is used to check the overlap of the merged detection frames, and filter the detection frames whose overlap is higher than the preset overlap threshold and whose categories are the same; the first output module is used to output the final detection result after screening and merging.

[0176] In a specific embodiment, the image acquisition module is used to acquire images to be detected. These images can come from various sources, such as pictures captured by a camera in real time, pre-stored image files, etc.

[0177] In a specific embodiment, the YOLO detection module obtains the image to be detected and inputs it into the YOLO-All model for target detection to obtain a first detection result. The YOLO-All model is a general target detection model that can quickly detect multiple targets in an image.

[0178] In a specific embodiment, the first detection result saving module is used to save the first detection result of the YOLO-All model, and the first detection result includes information such as the detection box coordinates, category label, attribute description, and detection confidence of each detected target.

[0179] In a specific embodiment, the first confidence checking module is used to check the confidence of each detection frame. For detection frames whose confidence is lower than a preset threshold, they will be marked as needing to be rechecked.

[0180] For those detection boxes that need to be rechecked, the VLM recheck module inputs them into the VLM-Detection1 model for secondary detection. The VLM-Detection1 model is a visual language-based detection model that can combine image and text information for more accurate target detection.

[0181] In a specific embodiment, the first result merging module merges the re-detection result of the VLM-Detection1 model with the first detection result of YOLO-All. This process comprehensively considers the detection results of the two models to obtain a more comprehensive and accurate detection result.

[0182] In a specific embodiment, the first overlapping frame processing module is used to check the merged detection frames and find frames with high overlap and the same category. For these overlapping frames, screening and deduplication are performed. The priority retention rule is as follows: the VLM-Detection1 model re-check result takes precedence over the YOLO high confidence result, and the YOLO high confidence result takes precedence over the YOLO low confidence result.

[0183] That is, for boxes with high overlap and the same category, they are retained according to priority:

[0184] First priority: the re-test results of the first multimodal object detection model;

[0185] Second priority: the first detection result returned by the general object detection model that is higher than the preset confidence threshold;

[0186] Third priority: the first detection result returned by the general object detection model that is lower than the preset confidence threshold, except for those that have passed the re-inspection of the first multimodal object detection model;

[0187] In the merged result, only one optimal detection box is retained for each target.

[0188] In a specific embodiment, the first output module outputs the final detection results after the above processing, which include high-quality target detection results after multi-model detection, re-testing and screening.

[0189] The real-time detection module of this embodiment significantly improves the accuracy of detection while ensuring real-time performance by combining the rapid detection capability of the YOLO model and the accurate re-inspection capability of the VLM model. The YOLO model can quickly detect most targets, while the VLM model can perform more detailed analysis and confirmation on those targets that the YOLO model is not sure about. This method of combining different types of models effectively balances the requirements of detection speed and accuracy. In addition, the accuracy of detection is further improved by processing overlapping frames. By removing redundant detection frames, the possibility of false detection is reduced, and the final detection result is also made clearer and easier to understand.

[0190] In this embodiment, refer to Figure 4 The non-real-time detection unit includes a second image acquisition module, a scene classification module, a detection model selection module, a second detection result storage module, a second confidence check module, a second VLM re-check module, a second result merging module, a second overlapping frame processing module and a second output module; wherein the second image acquisition module is used to acquire image data to be detected; the scene classification module is used to input the acquired image data into the scene classification for scene classification; the detection model selection module is used to select a corresponding scene-specific target detection model according to the scene classification result, and simultaneously input the general target detection model for global detection; the second detection result storage module is used to save the scene-specific target detection model. The first embodiment of the present invention is to combine the detection results of the scene-specific target detection model and the general target detection model; the second confidence checking module is used to check the confidence of each detection box, and screen out the detection boxes with confidence lower than the preset confidence threshold; the second VLM re-checking module is used to input the detection boxes with confidence lower than the preset confidence threshold into the second multimodal target detection model for re-checking; the second result merging module is used to merge the re-checking results with the detection results of the scene-specific target detection model and the general target detection model; the second overlapping box processing module is used to check the overlap of the merged detection boxes, and screen out the boxes with overlap higher than the preset threshold and the same category; the second output module is used to output the final detection results after screening and merging.

[0191] In a specific embodiment, the second image acquisition module is used to acquire image data to be detected from multiple sources. These images may come from on-site photos taken by the monitoring system or historical images that have been stored in the system. Due to non-real-time detection, the system can process a larger set of images and perform more detailed analysis at each stage.

[0192] In a specific embodiment, the scene classification module uses the VLM-Classify model to classify each image to be detected. The model is based on visual language multimodal technology and can classify images into different power field operation scenes, such as aerial work, ground work, ladder work, etc. Based on the classification results, the system can select the expert detection model that best suits the scene.

[0193] In a specific embodiment, the detection model selection module is used to select the YOLO1-5 model that best matches the scene for target detection according to the classification result of the VLM-Classify model to obtain the second detection result. At the same time, the system will also use the YOLO-All model for detection to ensure that no potential safety hazards are missed.

[0194] In a specific embodiment, the second detection result saving module saves the second detection results, which include information such as the detected target detection box, category label, attribute description, and detection confidence.

[0195] In a specific embodiment, the second confidence checking module is used to check the confidence of each detection frame, and mark the detection frame with a confidence lower than a preset threshold as an object that needs to be rechecked.

[0196] In a specific embodiment, the second VLM re-inspection module is used to input the detection box that needs to be re-inspected into the VLM-Detection2 model for secondary detection. The VLM-Detection2 model is a model specially optimized according to scene classification information, and can more accurately detect low-confidence targets in specific scenes.

[0197] In a specific embodiment, the second result merging module merges the re-detection result of VLM-Detection2 with the second detection result. This process comprehensively considers the detection results of all models to obtain the most comprehensive and accurate detection result.

[0198] In a specific embodiment, the second overlapping frame processing module checks the merged detection frames and looks for frames with high overlap and the same category. For these overlapping frames, the system will screen and remove duplicates according to certain rules. The priority retention rules are as follows: VLM-Detection2 results take precedence over YOLO high confidence results, and YOLO high confidence results take precedence over YOLO low confidence results.

[0199] For boxes with high overlap and the same category, they are retained according to priority:

[0200] First priority: re-test results of the second multimodal object detection model;

[0201] Second priority: high-confidence detection results returned by the scene-specific object detection model and the general object detection model, where the high-confidence detection results are detection results that are higher than a preset confidence threshold;

[0202] Third priority: low confidence results returned by the scene-specific target detection model and the general target detection model, where the low confidence detection results are detection results below a preset confidence threshold, except for those that have passed the re-inspection of the second multimodal target detection model;

[0203] In the merged and filtered results, only one optimal detection box is retained for each target.

[0204] In a specific embodiment, the second output module outputs the final detection results after the above processing, which include high-quality target detection results after scene classification, multi-model detection, re-inspection and screening.

[0205] The non-real-time detection module of this embodiment achieves more sophisticated and comprehensive detection in non-real-time mode by combining the scene classification model and the multimodal target detection model. The scene classification model can ensure the selection of the most appropriate detection model, while the multimodal target detection model can perform more accurate re-inspection of low-confidence targets in complex scenes. This strategy of using different models in combination can significantly improve the accuracy and reliability of detection, and is very suitable for application scenarios with higher requirements for detection accuracy.

[0206] Effect verification: This example reports in detail the effect verification results of the method proposed in the present invention in the safety detection of power field operations. In terms of target detection effect, the detection accuracy of the YOLO model, VLM model and MoE model in five typical power field scenarios is compared.

[0207] Table 1 shows the detection accuracy of the YOLO model, VLM model and MoE (mixed expert model) on different categories:

[0208] Table 1 Detection accuracy of different models in different categories

[0209]

[0210]

[0211] As can be seen from Table 1 above, the detection accuracy of the MoE architecture on all categories exceeds that of the individual YOLO and VLM models. This shows that through the synergy of the MoE architecture, the system can more effectively handle complex safety detection tasks in power field operations, significantly improving the accuracy and reliability of detection.

[0212] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.< / box> < / box> < / ref>

Claims

1. A method for safety detection of power field operations based on a hybrid expert model, characterized in that: The following steps are involved: In the training phase, through multi-model collaborative training, an expert model set for different scenarios and targets of power field operation safety detection is constructed, wherein the expert model set includes a scenario classification model, a general target detection model, a scenario-specific target detection model, a first multimodal target detection model, and a second multimodal target detection model; In the detection stage, if in real-time detection mode, a two-stage detection mechanism is adopted, firstly, a general target detection model performs a quick preliminary detection and returns a first detection result, wherein the first detection result includes a first detection frame and its confidence, and the confidence of the first detection frame is referred to as a first confidence; the first confidence is compared with a preset confidence threshold; if the first confidence is higher than the preset confidence threshold, the high confidence result is retained; if the first confidence is lower than the preset confidence threshold, the first multimodal target detection model is used for re-inspection to obtain a re-inspection result, and the re-inspection result is merged with the first detection result; the detection result is integrated and output; In the detection stage, if in non-real-time detection mode, a scene classification model is used to perform scene classification to obtain a scene classification result; based on the scene classification result, a scene-specific target detection model is used to perform detection and return a second detection result, the second detection result includes a second detection box and its confidence, and the confidence of the second detection box is called the second confidence, and at the same time, a general target detection model is used for global detection to return a third detection result, the third detection result includes a third detection box and its global confidence; the second confidence and the global confidence are compared with the preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, the high confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, it is regarded as a low confidence target, and the second multimodal target detection model is used to re-check the low confidence target; the detection results of all models are merged to output the final detection result.

2. According to claim 1, a method for detecting safety of electric power field operations based on a hybrid expert model is characterized in that: Before the training phase, a multimodal dataset is constructed, where the multimodal dataset includes a scene classification dataset and a target detection dataset.

3. The method for detecting safety of electric power field operations based on a hybrid expert model according to claim 1, characterized in that: The first multimodal target detection model is used to detect all labeled targets in all scenarios; the second multimodal target detection model is used to detect specific labeled targets in specific scenarios.

4. The method for detecting safety of electric power field operations based on a hybrid expert model according to claim 1, characterized in that: In the detection stage, if in real-time detection mode, the detection results are integrated and output, specifically including: in the merging process, check the overlap of the detection frames. If the overlap is greater than the preset overlap threshold, that is, for frames with high overlap and the same category, they are retained according to priority: First priority: the re-test results of the first multimodal object detection model; Second priority: the first detection result returned by the general object detection model that is higher than the preset confidence threshold; Third priority: the first detection result returned by the general object detection model that is lower than the preset confidence threshold, except for those that have passed the re-inspection of the first multimodal object detection model; In the merged result, only one optimal detection box is retained for each target.

5. The method for detecting safety of electric power field operations based on a hybrid expert model according to claim 1, characterized in that: In the detection stage, if in non-real-time detection mode, the detection results of all models are merged and the final detection results are output, including: During the merging process, the overlap of the detection frames is checked. If the overlap is greater than the preset overlap threshold, that is, for frames with high overlap and the same category, they are retained according to priority: First priority: re-inspection results of the second multimodal object detection model; Second priority: high-confidence detection results returned by the scene-specific object detection model and the general object detection model, where the high-confidence detection results are detection results that are higher than a preset confidence threshold; Third priority: low confidence results returned by the scene-specific target detection model and the general target detection model, where the low confidence detection results are detection results below a preset confidence threshold, except for those that have passed the re-inspection of the second multimodal target detection model; In the merged and filtered results, only one optimal detection box is retained for each target.

6. The method for detecting safety of electric power field operations based on a hybrid expert model according to claim 1, characterized in that: The scene classification model is a VLM-Classify model, the general target detection model is a YOLO-All model, the scene-specific target detection model is a YOLO1-5 model, the first multimodal target detection model is a VLM-Detection1 model, and the second multimodal target detection model is a VLM-Detection2 model.

7. A power field operation safety detection system based on a hybrid expert model, characterized in that: include: Model training unit, real-time detection unit and non-real-time detection unit; The model training unit is used for multimodal data set construction and training evaluation of hybrid expert model architecture; The real-time detection unit is used for safety detection of power field operations in real time and outputs the detection results; The non-real-time detection unit is used for safety detection of power field operations in a non-real-time state and outputs detection results.

8. The electric power field operation safety detection system based on hybrid expert model according to claim 7 is characterized in that: The model training unit includes a data set construction module, a prompt engineering test module, a scene classification training module, a target detection model training module, an optimization strategy application module and a model evaluation module; The data set construction module is used to construct a multimodal data set, and the multimodal data set includes a scene classification data set and a target detection data set; the prompt engineering test module is used to perform prompt engineering tests to determine the model applicability and the required prompt word type; The scene classification training module is used to train the scene classification training model; the target detection model training module is used to train the general target detection model, the scene-specific target detection model, the first multimodal target detection model and the second multimodal target detection model; the optimization strategy application module is used to apply data enhancement, transfer learning, mixed precision training and learning rate scheduling optimization strategies; the model evaluation module is used to evaluate the performance of the trained model.

9. The electric power field operation safety detection system based on hybrid expert model according to claim 8 is characterized in that: The real-time detection unit includes a first image acquisition module, a YOLO detection module, a first detection result storage module, a first confidence check module, a first VLM re-check module, a first result merging module, a first overlapping frame processing module and a first output module; Among them, the first image acquisition module is used to acquire image data to be detected; the YOLO detection module is used to input the acquired image data into a general target detection model for target detection; the first detection result saving module is used to save the first detection result of the general target detection model, the first detection result includes a first detection frame and its confidence, and the confidence of the first detection frame is called the first confidence; the first confidence checking module is used to check the confidence and filter out the detection frames whose confidence is lower than a preset threshold; the first VLM re-inspection module is used to input the detection frames lower than the preset threshold into the first confidence checking model for re-inspection to obtain the re-inspection result; the first result merging module is used to merge the re-inspection result with the first detection result; the first overlapping frame processing module is used to check the overlap of the merged detection frames, and filter the detection frames whose overlap is higher than the preset overlap threshold and whose categories are the same; the first output module is used to output the final detection result after screening and merging.

10. The electric power field operation safety detection system based on hybrid expert model according to claim 9, characterized in that: The non-real-time detection unit includes a second image acquisition module, a scene classification module, a detection model selection module, a second detection result storage module, a second confidence check module, a second VLM re-check module, a second result merging module, a second overlapping frame processing module and a second output module; wherein the scene classification module is used to input the acquired image data into the scene classification for scene classification; the detection model selection module is used to select the corresponding scene-specific target detection model according to the scene classification result, and at the same time input the general target detection model for global detection; the second detection result storage module is used to save the scene-specific target detection model and the general target detection model The detection result of the detection model; the second confidence checking module is used to check the confidence of each detection frame, and filter out the detection frames with confidence lower than the preset confidence threshold; the second VLM re-inspection module is used to input the detection frames with confidence lower than the preset confidence threshold into the second multimodal target detection model for re-inspection; the second result merging module is used to merge the re-inspection result with the detection results of the scene-specific target detection model and the general target detection model; the second overlapping frame processing module is used to check the overlap of the merged detection frames, and filter out the frames with overlap higher than the preset threshold and the same category; the second output module is used to output the final detection result after screening and merging.

Images