Power field operation safety detection method and system based on hybrid expert model

Abstract

The present application belongs to the field of electric power and the field of artificial intelligence technology, and specifically relates to a power field operation safety detection method and system based on a mixed expert model. The present application combines a visual language model and a YOLO target detection model, and realizes efficient and accurate power field operation safety detection under the condition of few samples through the mixed expert model architecture. In the training stage, through multi-model collaborative training, an expert model set for different scenes and targets is constructed. In the detection stage, a two-stage detection mechanism is adopted, first YOLO performs rapid preliminary detection, then VLM rechecks the low confidence targets, and finally the detection results are integrated to ensure the accuracy of the detection. The present application aims to effectively deal with the complex environment and safety hazards in the power field operation, significantly improve the safety detection efficiency and accuracy, and compared with a single model architecture, the present application shows a wider application scene and higher detection quality in the power field operation scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of electric power and artificial intelligence technology, and specifically relates to a power field operation safety detection method and system based on a hybrid expert model. BACKGROUND

[0002] Electric power field operation safety problems occupy a crucial position in the electric power industry. In recent years, with the continuous expansion of the scale of the electric power system and the increase of the operation complexity, electric power field operation safety accidents occur frequently, which poses a significant threat to personnel life and property safety. Traditional safety monitoring measures mainly rely on manual inspection, however, this mode has obvious limitations. Manual inspection is inefficient and is easily affected by environmental changes and work fatigue, resulting in a high rate of missed detection of safety hazards, which makes it difficult to meet the growing safety needs of the electric power system.

[0003] Under the impetus of artificial intelligence technology, intelligent safety detection systems based on image recognition and target detection have gradually become an important means of safety monitoring in the electric power industry. In particular, the deep learning model YOLO has received widespread attention due to its fast real-time detection capability and processing capability for large-scale image data. The YOLO model can complete target detection through a single image scan, greatly improving the detection speed.

[0004] Despite this, existing technologies in the field of electric power field operation safety detection still face the following challenges:

[0005] Firstly, the electric power field operation scene is complex and variable, and certain specific types of safety hazards occur less frequently, making it difficult to collect enough sample data for model training. Existing target detection models perform poorly under small sample conditions and must be trained on a large number of samples to achieve good results, which poses a challenge in data collection and labeling in practical applications.

[0006] Secondly, even if a certain number of samples can be collected, the labeling work requires a large amount of manpower and time, especially in electric power field operations, which require detection models to quickly identify safety hazards in complex environments. However, the real-time performance of existing models often fails to meet the requirements.

[0007] Thirdly, new safety hazards may occur in the electric power field operation scene, which need to be identified and warned in a timely manner. However, existing target detection models lack open vocabulary detection capability and can only detect labels seen during training. For newly emerging targets, the model needs to be retrained, which has obvious shortcomings in real-time performance and application flexibility. The electric power field operation environment is variable, and the types and forms of safety hazards in different scenarios differ. Existing models have poor adaptability in specific scenarios and are difficult to maintain consistent detection performance in different environmental conditions, limiting their application range in complex electric power field operations.

[0008] To address the above challenges, the present application proposes a power field operation real-time safety detection system based on a hybrid expert model architecture of a multi-modal large model, to improve the accuracy and real-time response capability of detection, while reducing the dependence on a large amount of sample data, enhancing the generalization ability and adaptability of the model, and providing a more intelligent and efficient safety monitoring solution for the power industry. SUMMARY

[0009] To overcome the problems in the prior art, the present application proposes a power field operation safety detection method and system based on a hybrid expert model.

[0010] The technical solution of the present application to solve the above technical problems is as follows:

[0011] In a first aspect, the present application provides a power field operation safety detection method based on a hybrid expert model, comprising the following steps:

[0012] In the training phase, through multi-model collaborative training, a set of expert models for power field operation safety detection in different scenes and targets is constructed, wherein the set of expert models includes a scene classification model, a general target detection model, a scene-specific target detection model, a first multi-modal target detection model, and a second multi-modal target detection model.

[0013] In the detection phase, if in real-time detection mode, a two-stage detection mechanism is adopted, first a general target detection model is used for rapid preliminary detection to return a first detection result, the first detection result including a first detection frame and its confidence, the confidence of the first detection frame being referred to as a first confidence; the first confidence is compared with a preset confidence threshold; if the first confidence is higher than the preset confidence threshold, the high-confidence result is retained; if the first confidence is lower than the preset confidence threshold, a first multi-modal target detection model is used for re-inspection to obtain a re-inspection result, the re-inspection result is combined with the first detection result, and the output detection result is integrated.

[0014] In the detection stage, if in the non-real-time detection mode, a scene classification model is used for scene classification to obtain a scene classification result; according to the scene classification result, a scene-specific target detection model is used for detection to return a second detection result, the second detection result including a second detection frame and a confidence thereof, the confidence of the second detection frame being referred to as a second confidence, and a general target detection model is used for global detection to return a third detection result, the third detection result including a third detection frame and a global confidence thereof; the second confidence and the global confidence are compared with a preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, a high-confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, a low-confidence target is considered, and a second multi-modal target detection model is used for re-inspection of the low-confidence target; detection results of all models are merged, and a final detection result is output.

[0015] Further, before the training stage, a multi-modal data set is constructed, the multi-modal data set including a scene classification data set and a target detection data set.

[0016] Further, the first multi-modal target detection model is used for detecting all label targets in all scenes; and the second multi-modal target detection model is used for detecting specific label targets in specific scenes.

[0017] Further, in the detection stage, if in the real-time detection mode, an output detection result is integrated, specifically including: in the merging process, the coincidence degree of the detection frame is checked, if the coincidence degree is greater than a preset coincidence degree threshold, that is, for the frames with high coincidence degree and same class, the frames are retained according to the priority:

[0018] The first priority: the re-inspection result of the first multi-modal target detection model;

[0019] The second priority: the first detection result returned by the general target detection model and higher than the preset confidence threshold;

[0020] The third priority: the first detection result returned by the general target detection model and lower than the preset confidence threshold, but except for the result that has been re-inspected by the first multi-modal target detection model;

[0021] In the merged result, only one optimal detection frame is retained for each target.

[0022] Further, in the detection stage, if in the non-real-time detection mode, detection results of all models are merged, and a final detection result is output, including: in the merging process, the coincidence degree of the detection frame is checked, if the coincidence degree is greater than a preset coincidence degree threshold, that is, for the frames with high coincidence degree and same class, the frames are retained according to the priority:

[0023] The first priority: the review result of the second multi-modal target detection model;

[0024] The second priority: the high confidence detection result returned by the scene-specific target detection model and the general target detection model, the high confidence detection result being a detection result higher than a preset confidence threshold;

[0025] The third priority: the low confidence result returned by the scene-specific target detection model and the general target detection model, the low confidence detection result being a detection result lower than a preset confidence threshold; but the result reviewed by the second multi-modal target detection model is excluded;

[0026] In the result after the screening, only one optimal detection box is reserved for each target.

[0027] Further, the scene classification model is a VLM-Classify model, the general target detection model is a YOLO-All model, the scene-specific target detection model is a YOLO1-5 model, the first multi-modal target detection model is a VLM-Detection1 model, and the second multi-modal target detection model is a VLM-Detection2 model.

[0028] In a second aspect, the present application further provides a power field operation safety detection system based on a hybrid expert model, comprising: a model training unit, a real-time detection unit and a non-real-time detection unit;

[0029] The model training unit is used for multi-modal data set construction and training and evaluation of a hybrid expert model architecture;

[0030] The real-time detection unit is used for power field operation safety detection in a real-time state, and outputs a detection result;

[0031] The non-real-time detection unit is used for power field operation safety detection in a non-real-time state, and outputs a detection result.

[0032] Further, the model training unit comprises a data set construction module, a prompt engineering test module, a scene classification training module, a target detection model training module, an optimization strategy application module and a model evaluation module;

[0033] The data set construction module is configured to construct a multi-modal data set, the multi-modal data set comprising a scene classification data set and a target detection data set; the prompt engineering test module is configured to perform prompt engineering test to determine model applicability and required prompt word types; the scene classification training module is configured to train a scene classification training model; the target detection model training module is configured to train a general target detection model, a scene-specific target detection model, a first multi-modal target detection model and a second multi-modal target detection model; the optimization strategy application module is configured to apply data enhancement, transfer learning, mixed precision training and learning rate scheduling optimization strategies; and the model evaluation module is configured to evaluate the performance of the trained model.

[0034] Further, the real-time detection unit comprises a first image acquisition module, a YOLO detection module, a first detection result saving module, a first confidence checking module, a first VLM rechecking module, a first result merging module, a first overlapping box processing module and a first output module.

[0035] The first image acquisition module is configured to acquire image data to be detected; the YOLO detection module is configured to input the acquired image data into a general target detection model for target detection; the first detection result saving module is configured to save the first detection result of the general target detection model, the first detection result comprising a first detection box and a confidence thereof, the confidence of the first detection box being referred to as a first confidence; the first confidence checking module is configured to check the confidence and filter out detection boxes with a confidence lower than a preset threshold; the first VLM rechecking module is configured to input the detection boxes with a confidence lower than the preset threshold into a first confidence checking model for rechecking to obtain a rechecking result; the first result merging module is configured to merge the rechecking result with the first detection result; the first overlapping box processing module is configured to check the coincidence degree of the merged detection boxes and filter out detection boxes with a coincidence degree higher than a preset coincidence degree threshold and the same class; and the first output module is configured to output the final detection result after filtering and merging.

[0036] Further, the non-real-time detection unit comprises a second image acquisition module, a scene classification module, a detection model selection module, a second detection result saving module, a second confidence check module, a second VLM review module, a second result merging module, a second overlapping box processing module, and a second output module; the scene classification module is configured to input the acquired image data into scene classification for scene classification; the detection model selection module is configured to select a corresponding scene-specific target detection model according to the scene classification result, and simultaneously input a general target detection model for global detection; the second detection result saving module is configured to save the detection results of the scene-specific target detection model and the general target detection model; the second confidence check module is configured to check the confidence of each detection box and filter out the detection boxes with a confidence lower than a preset confidence threshold; the second VLM review module is configured to input the detection boxes with a confidence lower than the preset confidence threshold into a second multi-modal target detection model for review; the second result merging module is configured to merge the review results with the detection results of the scene-specific target detection model and the general target detection model; the second overlapping box processing module is configured to check the coincidence degree of the merged detection boxes and filter out the boxes with a coincidence degree higher than a preset threshold and the same class; and the second output module is configured to output the final detection result after filtering and merging.

[0037] Compared with the prior art, the present application has the following technical effects:

[0038] (1) The present application designs a unique mixed expert model (MoE) architecture that organically integrates the advantages of visual language model (VLM) and YOLO model, achieving a breakthrough improvement in power field operation safety detection. The architecture flexibly calls the most suitable model according to different operation scenes and detection targets through a scene-adaptive expert model selection mechanism, fully utilizing the zero-shot and few-shot learning capabilities of VLM and the real-time detection characteristics of YOLO model. The architecture effectively solves the challenges of sample data scarcity, insufficient detection accuracy, poor real-time response, and limited generalization ability in traditional detection systems, achieving high-precision and high-real-time power operation safety detection and providing intelligent support for complex field operations.

[0039] (2) The application innovatively proposes a multi-model collaborative training strategy, by training expert models for different scenes and targets, an efficient expert model set is constructed, including scene classification model VLM-Classify, general target detection model YOLO-All, scene-specific target detection model YOLO1-5, and multi-modal target detection model VLM-Detection1 and VLM-Detection2. Each model is optimized in training, which effectively improves the adaptability of the system in the complex and changeable power operation environment. Through this collaborative training method, the system can realize rapid and accurate detection in different scenes, greatly improving the accuracy and efficiency of safety detection.

[0040] (3) The application innovatively designs a two-stage detection mechanism, combining the rapid preliminary detection ability of the YOLO model and the accurate re-inspection ability of the VLM model, which significantly improves the overall detection effect. In the first stage, the YOLO model quickly detects potential targets; in the second stage, the VLM model re-inspects the low confidence targets detected by the YOLO model. This mechanism not only ensures the detection speed of the system, but also shows excellent performance in complex scenes and low confidence target processing, overcoming the limitations of traditional methods in identifying new safety hazards. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0042] Figure 1 The model overall framework of the application;

[0043] Figure 2 The detection method flowchart of the application;

[0044] Figure 3 The real-time detection flowchart of the application;

[0045] Figure 4 The non-real-time detection flowchart of the application. DETAILED DESCRIPTION

[0046] To further clarify the technical means and effects taken by the present application to achieve the intended purpose, the specific implementation, structure, features and effects of the technical solutions proposed according to the present application are described in detail below in combination with the drawings and preferred embodiments. The specific features, structures or characteristics in one or more embodiments can be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by those skilled in the art to which the present application belongs.

[0047] The present application proposes a power field operation safety detection method based on a mixed expert model, aiming to solve the problems existing in the current power field safety detection technology, such as insufficient sample data, poor detection accuracy and real-time performance, limited generalization ability, etc. This method combines the visual language model VLM and the YOLO model skillfully, achieving efficient and accurate power field operation safety detection under the condition of few samples.

[0048] The present application designs a novel mixed expert model architecture MoE, which fully utilizes the zero-shot and few-shot learning ability of VLM and the real-time detection characteristics of YOLO model. This architecture not only improves the detection accuracy, but also maintains the real-time response ability of the system, especially suitable for the complex and variable environment of power field operation.

[0049] The implementation process of the present application can be divided into two main parts: training phase and detection phase. In the training phase, the system constructs a set of expert models for different scenes and targets through multi-model collaborative training. In the detection phase, the system adopts a two-stage detection mechanism, first performs rapid preliminary detection by YOLO, then VLM rechecks the low confidence targets, and finally integrates the detection results to ensure the accuracy of the detection.

[0050] In the embodiment of the present application, with reference to Figures 1-4 , a power field operation safety detection method based on a mixed expert model is disclosed, comprising the following steps:

[0051] In the training phase, a set of expert models for different scenes and targets of power field operation safety detection is constructed through multi-model collaborative training, wherein the set of expert models includes a scene classification model, a general target detection model, a scene-specific target detection model, a first multi-modal target detection model and a second multi-modal target detection model;

[0052] In the detection stage, if in real-time detection mode, a two-stage detection mechanism is adopted, first a general target detection model is used for rapid preliminary detection to return a first detection result, which includes a first detection box and its confidence, and the confidence of the first detection box is referred to as a first confidence; the first confidence is compared with a preset confidence threshold; if the first confidence is higher than the preset confidence threshold, the high-confidence result is retained; if the first confidence is lower than the preset confidence threshold, a first multi-modal target detection model is used for re-inspection to obtain a re-inspection result, and the re-inspection result is combined with the first detection result; and the output detection result is integrated;

[0053] In the detection stage, if in non-real-time detection mode, a scene classification model is used for scene classification to obtain a scene classification result; according to the scene classification result, a scene-specific target detection model is used for detection to return a second detection result, which includes a second detection box and its confidence, and the confidence of the second detection box is referred to as a second confidence, and at the same time, a general target detection model is used for global detection to return a third detection result, which includes a third detection box and its global confidence; the second confidence and the global confidence are compared with a preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, the high-confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, it is considered as a low-confidence target, and a second multi-modal target detection model is used for re-inspection of the low-confidence target; the detection results of all models are combined to output the final detection result.

[0054] The above steps are described in detail as follows:

[0055] Step 100: Construct a multi-modal data set, including a scene classification data set and a target detection data set.

[0056] The scene classification data set contains system prompt words, user instructions, correct answers and images to be detected. The target detection data set contains system prompt words, user instructions, correct answers (including labels and detection box positions) and images to be detected.

[0057] Step 200: In the training stage, a multi-model collaborative training is used to build a set of expert models for different scenes and targets, wherein the set of expert models includes a scene classification model, a general target detection model, a scene-specific target detection model, a first multi-modal target detection model and a second multi-modal target detection model.

[0058] The scene classification model is used for scene classification, guiding the selection and parameter adjustment of subsequent models; the general object detection model is used for preliminary object detection in all scenes; the scene-specific object detection model is used for accurate object detection in specific scenes; and the first and second multi-modal object detection models are used for multi-modal object detection, further improving detection accuracy. This multi-model collaborative training method combines the advantages of different models, enabling the system to adapt to various power field operation scenes and improving detection accuracy and efficiency.

[0059] As a specific example, the step 200 can include the following sub-steps:

[0060] Step 210: Train the scene classification model, i.e., the VLM-Classify model, for quickly learning scene features on limited power operation data, providing scene information for subsequent detection models, and guiding the selection and parameter adjustment of subsequent models.

[0061] The VLM-Classify model is a pre-trained model based on the internvl2 series, which is fine-tuned to adapt to the power field operation data set and becomes a visual language multi-modal model specifically for power operation scene classification.

[0062] The improved Lora efficient fine-tuning technology adopted enables the model to quickly learn scene features on limited power operation data, significantly improving the training efficiency of the model while ensuring detection accuracy. This step provides accurate scene information for subsequent detection models, guiding model selection and parameter adjustment, ensuring that the entire system can select the most suitable expert model according to different field operation scenes, thereby achieving improved detection accuracy and efficiency.

[0063] The improved Lora efficient fine-tuning technology refers to assigning different learning rates to different modules of Lora during training to improve the convergence speed of Lora fine-tuning, with the aim of achieving performance closer to that of full-scale fine-tuning.

[0064] Step 220: Train the general object detection model, i.e., the YOLO-All model, for preliminary object detection in all scenes.

[0065] For all-scene detection, a general object detection model, YOLO-All model, is introduced. The YOLO-All model is based on YOLO v10-X. The training of the YOLO-All model not only focuses on hyperparameter tuning and model architecture optimization, but also pays special attention to the improvement of the generalization ability and robustness of the model, which enables the model to stably detect various targets in multiple job scenes. Under the framework of collaborative training, the YOLO-All model can quickly preliminarily detect targets in all scenes, providing a basic result for the subsequent model review.

[0066] Step 230: Train scene-specific object detection models, YOLO1-5 models, for detecting corresponding labeled targets in specific scenes.

[0067] For typical power field job scenes, scene-specific object detection models, YOLO1-5 models, are trained. These models are based on YOLO v10-M and trained with specific scene data, which can more accurately detect corresponding labeled targets in specific scenes. Through this specialized training, the detection performance of the model in specific scenes is significantly improved while maintaining a fast detection speed.

[0068] For example, for five typical power field job scenes, five scene-specific object detection YOLO1-5 models are trained, and the five scenes are high-altitude work, ladder work, dress code, crossing safety fence, and job site hole detection. In fact, according to different specific tasks, the number of power job scene categories can also be increased or decreased accordingly.

[0069] Step 240: Train multi-modal object detection models, including a first multi-modal object detection model and a second multi-modal object detection model. The first multi-modal object detection model VLM-Detection1 is used to detect all labeled targets in all scenes, and the second multi-modal object detection model VLM-Detection2 is used to detect specific labeled targets in specific scenes.

[0070] Two multi-modal object detection models, VLM-Detection1 and VLM-Detection2, are introduced. These two models are based on the internvl2-8B pre-trained model and have the ability to handle complex environments in power field job through special fine-tuning. The special fine-tuning refers to targeted training on the power field job safety detection dataset, and through transfer learning, the general object detection ability of the model is generalized to power field job safety detection.

[0071] Among them, the VLM-Detection1 model is used to detect all label targets in all scenes, and the VLM-Detection2 model is used to detect specific label targets in specific scenes. Considering that the VLM model has strong generalization performance and high deployment cost, all tasks of the VLM-Detection2 model are completed through multi-task learning of a trained model. The main difference between the VLM-Detection1 model and the VLM-Detection2 model is that the VLM-Detection2 model is explicitly informed of the scene to which the training sample belongs in the training process.

[0072] During the training process, a hybrid training strategy is adopted, which includes different combinations of training the LLM (Large Language Model) module, the visual module, and the projection layer. This strategy not only maintains the detection efficiency but also significantly improves the precision, especially in few-shot learning and open-vocabulary detection.

[0073] The LLM model generally consists of an image encoder (visual module, ViT), a projection layer (aligning visual features with text features, usually an MLP), and an LLM backbone (large language model, LLM).

[0074] In the VLM, the LLM module refers to the large language model backbone. The main function of this module is to process and generate text information, thereby enhancing the VLM's ability in multi-modal tasks.

[0075] The LLM module functions as follows: the LLM module can integrate information from visual and language inputs and generate relevant language descriptions. This makes the VLM a generative artificial intelligence model that can handle a variety of tasks, including image classification and object detection, by standardizing and analyzing the VLM's output. In addition, the LLM module also enables prompt engineering, which guides the model to complete specific tasks through specific instructions. This method can improve the model's effectiveness and adaptability without changing the core parameters of the model.

[0076] During the training process, according to the training sequence and task difficulty, one or two of the three modules can be frozen (i.e., not updated) if necessary. This brings two benefits: avoiding catastrophic forgetting: preventing the model from learning new knowledge and forgetting old knowledge, which reduces the generalization performance; and reducing resource requirements: since some parameters are frozen, the resource requirements of the model during training will also decrease.

[0077] The pre-trained VLM model uses an internvl2-8B model, which uses InternViT-300M-448px as a visual module, internlm2_5-7b-chat as an LLM module, and connects visual features with text features through training of a visual projection layer.

[0078] Step 300: In the detection stage, if in real-time detection mode, a two-stage detection mechanism is adopted, first a general target detection model is used for rapid preliminary detection, and then a first multi-modal target detection model is used for re-inspection of low confidence targets, and finally the detection results are integrated.

[0079] In the detection stage, if in non-real-time detection mode, a scene classification model is used for scene classification; according to the scene classification result, a scene-specific target detection model is used for detection, and at the same time a general target detection model is used for global detection; check the confidence, low confidence targets are transmitted to the first multi-modal target detection model for re-inspection; merge the detection results of all models, and output the final detection result.

[0080] This step can include the following sub-steps:

[0081] Step 310: In the detection stage, if in real-time detection mode, a two-stage detection mechanism is adopted, first a general target detection model is used for rapid preliminary detection to return a first detection result, the first detection result includes a first detection frame and its confidence, the confidence of the first detection frame is called the first confidence; compare the first confidence with the preset confidence threshold; if the first confidence is higher than the preset confidence threshold, keep the high confidence result; if the first confidence is lower than the preset confidence threshold, use the first multi-modal target detection model for re-inspection to obtain a re-inspection result, merge the re-inspection result with the first detection result; integrate and output the detection result. Specifically, it includes:

[0082] Step 311: Obtain the image to be detected and transmit it to the general target detection model for rapid preliminary detection to return a first detection result, the first detection result includes a detection frame and its attribute description and confidence, the confidence of the first detection frame is called the first confidence.

[0083] The attribute description refers to the name of the category. The confidence is the degree of certainty of the YOLO model when detecting that there is a target object in the detection frame, if the confidence is close to 1, it means that the model is very confident that the detection frame contains the target object; if the confidence is close to 0, it means that the model thinks that the detection frame may not contain the target object.

[0084] Step 312: a confidence threshold is preset to evaluate the confidence of the detection box, that is, the first confidence is compared with the preset confidence threshold; the detection result lower than the preset confidence threshold is regarded as a low-confidence detection result, and the detection result higher than the preset confidence threshold is regarded as a high-confidence detection result, and the high-confidence detection result is retained; the low-confidence detection result is transmitted into the first multi-modal target detection model for re-inspection to obtain a re-inspection result, so that a more accurate labeling result is obtained.

[0085] Step 313: the re-inspection result of the first multi-modal target detection model is merged with the first detection result of the general target detection model.

[0086] In the merging process, the coincidence degree of the detection box is checked, and if the coincidence degree is greater than a preset coincidence degree threshold, that is, for the boxes with high coincidence degree and same class, the boxes are retained according to the priority:

[0087] First priority: the re-inspection result of the first multi-modal target detection model.

[0088] Second priority: the first detection result higher than the preset confidence threshold returned by the general target detection model.

[0089] Third priority: the first detection result lower than the preset confidence threshold returned by the general target detection model, but excluding the re-inspection result of the first multi-modal target detection model.

[0090] That is, the re-inspection result of the VLM-Detection1 model is first retained; if the VLM-Detection1 model does not detect a certain target, the first detection result with higher confidence in the general target detection model is retained; if the general target detection model also does not detect or the confidence is low, whether to retain the first detection result of the general target detection model with low confidence is considered.

[0091] In the merged result, only one optimal detection box is retained for each target.

[0092] Step 314: output the final detection result after merging.

[0093] According to the screened result, the final detection result is output, and the output result includes information such as detection box, attribute description and confidence.

[0094] Step 320: In the detection stage, if in the non-real-time detection mode, scene classification is performed using a scene classification model to obtain a scene classification result; according to the scene classification result, a scene-specific target detection model is used for detection to return a second detection result, the second detection result including a second detection frame and a confidence thereof, the confidence of the second detection frame being referred to as a second confidence, and a general target detection model is used for global detection to return a third detection result, the third detection result including a third detection frame and a global confidence thereof; the second confidence and the global confidence are compared with a preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, a high-confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, it is considered as a low-confidence target, and a second multi-modal target detection model is used for re-inspection of the low-confidence target; and detection results of all models are merged to output a final detection result.

[0095] Step 320: In the detection stage, if in the non-real-time detection mode, scene classification is performed using a scene classification model to obtain a scene classification result; according to the scene classification result, a scene-specific target detection model is used for detection to return a second detection result, the second detection result including a second detection frame and a confidence thereof, the confidence of the second detection frame being referred to as a second confidence, and a general target detection model is used for global detection to return a third detection result, the third detection result including a third detection frame and a global confidence thereof; the second confidence and the global confidence are compared with a preset confidence threshold respectively; if the second confidence or the global confidence is higher than the preset confidence threshold, a high-confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence threshold, it is considered as a low-confidence target, and a second multi-modal target detection model is used for re-inspection of the low-confidence target; and detection results of all models are merged to output a final detection result.

[0096] Step 321: An image to be detected is obtained and input into a scene classification model, i.e., a VLM-Classify model, for scene classification, and the scene classification model returns a scene classification result;

[0097] Step 322: According to the scene classification result, a scene-specific target detection model most matched with the current scene is selected from the scene-specific target detection model for detection, and the scene-specific target detection model returns a detection result, the detection result including a second detection frame, attribute description thereof, confidence, etc., and the confidence of the second detection frame is referred to as a second confidence;

[0098] Meanwhile, the image to be detected is input into a general target detection model, i.e., a YOLO-All model, for global detection, and the general target detection model returns a global detection result.

[0099] Step 323: Detection results of the general target detection model and the scene-specific target detection model are traversed, and the confidence of each detection frame, i.e., a preset confidence threshold, is checked; the second confidence and the global confidence are compared with the preset confidence threshold respectively; if the first confidence or the global confidence is higher than the preset confidence threshold, a high-confidence result is retained; if the second confidence or the global confidence is lower than the preset confidence detection threshold, it is input into a second multi-modal target detection model VLM-Detection2 for re-inspection.

[0100] The second multi-modal target detection model VLM-Detection2 considers scene classification information when performing detection to return more accurate detection.

[0101] Step 324: Detection results of all models are merged.

[0102] The degree of coincidence of the merged detection results is checked: a coincidence threshold is set, and detection boxes with a high degree of coincidence and the same category are considered as repeated detection results.

[0103] That is, for boxes with a high degree of coincidence and the same category, the priority is retained:

[0104] First priority: the review result of the second multi-modal target detection model;

[0105] Second priority: high confidence detection results returned by the scene-specific target detection model and the general target detection model, which are detection results higher than the pre-set confidence threshold;

[0106] Third priority: low confidence results returned by the scene-specific target detection model and the general target detection model, which are detection results lower than the pre-set confidence threshold; but the review result of the second multi-modal target detection model is excluded;

[0107] In the merged and screened results, only one optimal detection box is retained for each target.

[0108] That is, first, the review result of the VLM-Detection2 model is retained; if VLM-Detection2 does not detect a certain target, the detection result with higher confidence from the scene-specific target detection model and the general target detection model is retained; if neither the scene-specific target detection model nor the general target detection model detects or has low confidence, whether to retain the low confidence detection result of the scene-specific target detection model and the general target detection model is considered.

[0109] Step 325: output the final detection result:

[0110] According to the screened results, the final detection result is output, and the output result includes detection boxes, attribute descriptions, and confidence information, which can be presented in the form of image annotation or text list.

[0111] This multi-stage and multi-model detection strategy fully utilizes the advantages of each model. The YOLO model provides fast preliminary detection results, while the VLM model plays a key role in handling complex scenes and low confidence targets. In this way, the system not only can quickly identify common safety hazards, but also can accurately detect rare or newly emerging safety problems.

[0112] Based on the same inventive concept, the embodiments of the present application also provide a hybrid expert model-based power field operation safety detection system for implementing the hybrid expert model-based power field operation safety detection method. The system provides a solution to the implementation scheme as described in the above method, and therefore the specific limitations in one or more system embodiments provided below can refer to the limitations of the hybrid expert model-based power field operation safety detection method described above, which will not be repeated here.

[0113] In one embodiment, a hybrid expert model-based power field operation safety detection system is provided, comprising a model training unit, a real-time detection unit and a non-real-time detection unit; the model training unit is used for multi-modal data set construction and training and evaluation of hybrid expert model architecture; the real-time detection unit is used for real-time power field operation safety detection; the non-real-time detection unit is used for non-real-time power field operation safety detection.

[0114] In the present embodiment, the model training unit comprises a data set construction module, a prompt engineering test module, a scene classification training module, a target detection model training module, an optimization strategy application module and a model evaluation module;

[0115] The data set construction module is used to construct a multi-modal data set, which comprises a scene classification data set and a target detection data set; the prompt engineering test module is used to perform prompt engineering test to determine model applicability and required prompt word types; the scene classification training module is used to train a scene classification training model; the target detection model training module is used to train a general target detection model, a scene-specific target detection model, a first multi-modal target detection model and a second multi-modal target detection model; the optimization strategy application module is used to apply data enhancement, transfer learning, mixed precision training and learning rate scheduling optimization strategies; and the model evaluation module is used to evaluate the performance of the trained model.

[0116] In a specific embodiment, the data set construction module is used to construct a multi-modal data set, which comprises a scene classification data set and a target detection data set. The scene classification data set comprises system prompt words, user instructions, correct answers and images to be detected; and the target detection data set comprises system prompt words, user instructions, correct answers and images to be detected, wherein the correct answers of the target detection data set comprise labels of target categories and position information of detection boxes, the labels are used to identify the categories of the targets, and the position information of the detection boxes describes the exact positions of the targets in the images.

[0117] In specific embodiments, the prompt engineering test module is used to conduct prompt engineering tests, the purpose of which is to determine whether the model is suitable for this type of task and to understand the type of prompt words required by the model to complete a specific task. The testing process includes designing clear prompt words, requiring the model to output results in a specific format, and analyzing the model's output.

[0118] When a multi-modal large model is used for scene classification and object detection tasks, the format of the constructed samples often determines its actual effect. Before constructing the samples, some prompt engineering tests are performed on the model, with two purposes:

[0119] First, determine whether the model is suitable for performing scene classification and object detection tasks, i.e., determine whether the model can be guided to complete scene classification and object detection tasks through the construction of prompt words.

[0120] The approach adopted is to clearly and explicitly state the task purpose in the prompt words, provide background information needed to complete the task, and require the model to output the detection results in a specific format such as Json, XML, etc. string, which facilitates the analysis and comparison of the answers.

[0121] At this time, the model is not required to perform the related task with accuracy, and considering the randomness of AIGC (AI Generated Content) model generation, the results given by the model at this stage may not be perfect and there will be a large number of outputs that cannot be parsed.

[0122] Second, understand the type of prompt words required by the model to complete a specific task through testing, which are used as system prompt words when constructing training samples, i.e.:

[0123] (1) Understand the model's preferences as much as possible through data engineering methods;

[0124] (2) Construct training samples in the input and output format preferred by the model;

[0125] (3) The data set constructed by this method can achieve a multiplier effect in training because it fits the model's generation preferences, and the loss of the model when starting training is often lower.

[0126] For the object detection task, it is necessary to understand the pre-trained model's understanding of the input image, which refers to the reference coordinate system of the annotation box given by the object detection task. Specifically for multi-modal large models, their generation method is consistent with that of large language models, i.e., generating one Token at a time, and the result is a string format, therefore, it is necessary to understand how the model tends to represent categories and how to annotate box graphs when performing object detection tasks.

[0127] After prompt engineering testing, it was found that the Internvl2-8B model tends to describe objects or regions in images in XML markup format, as follows:

[0128] <ref>human< / ref> <box> [[221,223,940,652]]< / box> ;

[0129] wherein, <ref>a label representing a label or name of an object, <box>The label represents the bounding box of the object in the image, in the form of coordinates. In particular, the model prefers <box>The label takes any incoming image as 1000*1000 size, and the given annotation box represents the scaled Xmin, Ymin, Xmax, Ymax respectively.

[0130] Scene classification, which is oriented to scene classification, corresponds to the VLM-Classify model. The dataset construction of the scene classification task includes system (system prompt words), query (user instructions), response (correct answers), and images (images to be detected). The system prompt words include preference prompt words that guide the model to complete the task, such as role positioning, scene description to be classified, output format, etc. The correct answer (response) is converted from label information and contains the correct category of the scene, represented by a string. Each image data contains multiple typical scenes, for example, workers in different work sites may have non-standard dress, so the annotation data may contain multiple categories. This task is a typical multi-classification task.

[0131] After the above processing, the sample data of the image classification task is constructed in a generated manner.

[0132] An example is as follows:

[0133] "system":

[0134] # Role

[0135] You are a target detector.

[0136] # Task

[0137] You need to detect the content contained in the user-uploaded picture and return the detection result.

[0138] The picture focuses on five typical scenes of power sites:

[0139] -Non-standard wearing of safety belts during high-altitude work, detect high-altitude workers and safety belts as a whole. -No one to help on the ladder, detect; ladder workers.

[0140] -Worksite holes are not covered or have no protective measures (holes refer to square or circular holes on the ground of the work site), detect holes.

[0141] -Workers are not dressed in a standard manner, detect all workers who are not dressed in a standard manner.

[0142] -Workers are crossing or under the safety fence, detect workers who are crossing the fence.

[0143] The five labels are "safetyBelt", "ladderSupport", "holeProtection", "workwear", and "barrierCrossing".

[0144] You need to return the detection results in <ref>class_name< / ref> <box>[[x,x,x,x]]< / box> Return the detection results, where 'class_name' must be one of the five label names.

[0145] Only return the detection results in the given format, without giving any other explanations.

[0146] There may be multiple violations in a picture, such as workers in different work sites not wearing standard clothing. Please provide all eligible targets for detection.

[0147] If there is no typical violation scene in the picture, return <ref>class_name< / ref> <box> [[0,0,0,0]]< / box> That's it.

[0148] "query" is:

[0149] "Please detect the violations in this picture",

[0150] "response" is:

[0151] " <ref>holeProtection< / ref> <box> [[221,223,940,652]]< / box> ",

[0152] "images" is:

[0153] [{images_path}].

[0154] In specific embodiments, the scene classification model training module is used to train the VLM-Classify model. This model is based on the internvl2 series and is used to accurately classify power site work scenes. The training process uses an improved Lora efficient fine-tuning technique to train different combinations of LLM modules, visual modules, and projection layers, improving training efficiency while ensuring model performance. This step provides accurate scene information for the subsequent detection model, guiding model selection and parameter adjustment, ensuring that the entire system can select the most suitable expert model according to different site work scenes, thereby achieving both detection accuracy and efficiency.

[0155] In specific embodiments, the target detection model training module trains multiple target detection models. This includes: YOLO-All: a general target detection model based on YOLO v10-X; YOLO1-5: five scene-specific target detection models based on YOLO v10-M; VLM-Detection1 and VLM-Detection2: two multi-modal target detection models based on the internvl2 series.

[0156] In specific embodiments, the optimization strategy application module applies multiple optimization strategies during training, including applying data augmentation, transfer learning, mixed precision training, and learning rate scheduling, etc.

[0157] Expand the training set through data augmentation techniques to improve the generalization ability of the model: (1) Image rotation: randomly rotate the image to increase the model's ability to recognize objects at different angles; (2) Flip: horizontally and vertically flip the image to improve the model's direction invariance; (3) Scaling: randomly scale the image to improve the model's ability to recognize objects of different sizes; (4) Color jitter: adjust the brightness, contrast, and saturation of the image to improve the model's performance under different lighting conditions.

[0158] Use pre-trained models for transfer learning to speed up training and improve model performance: (1) Use the InternVL2-8B model pre-trained on a large-scale dataset as the base model; (2) Freeze part of the bottom feature extraction layer and only fine-tune the high-level semantic understanding layer; (3) Adopt a progressive unfreezing strategy to gradually unfreeze more layers to achieve better feature adaptation.

[0159] InternVL2 adopts a ViTMLP-LLM architecture similar to the popular multi-modal large language model (MLLM). This architecture combines a pre-trained InternViT-300M visual model (ViT) with an Internlm2.5-7b-chat language model (LLM) through a multi-layer perceptron (MLP) projection layer to achieve semantic alignment of visual-text features. Therefore, InternVL2-8B mainly consists of ViT+MLP+LLM, where ViT is the visual model module, MLP is the projection layer, and LLM is the language model module. Based on this, the transfer learning strategy is:

[0160] First, freeze part of the bottom feature extraction layer and only fine-tune the high-level semantic understanding layer:

[0161] Freeze the InternViT-300M module, which is the visual encoder part of the model, responsible for bottom-level feature extraction. Freezing this module can preserve the pre-trained general visual features;

[0162] Freeze MLP Projector, which is responsible for projecting visual features into language space. Keeping it frozen helps maintain visual-language alignment.

[0163] Thaw Internlm2.5-7b-chat language model layers, train the model to understand semantics and perform task-related reasoning.

[0164] Second, adopt a gradual thawing strategy, gradually thaw more layers to achieve better feature adaptation:

[0165] Phase 1: Only fine-tune Internlm2.5-7b-chat language model layers, train for a certain number of rounds to allow the model to adapt to the new task;

[0166] Phase 2: Thaw MLP Projector, optimize visual-language alignment;

[0167] Phase 3: Selectively thaw high-level InternViT-6B, fine-tune specific task visual features;

[0168] Gradually adjust the learning rate, gradually reduce the learning rate as more layers are thawed, to prevent overfitting and stabilize the training process.

[0169] Through this gradual method, the model can gradually adapt to new tasks while retaining the powerful features of the pre-trained model. This strategy balances the efficiency of transfer learning and the adaptability of the model to new tasks.

[0170] Use mixed precision training to improve training speed and reduce memory usage while ensuring model accuracy: (1) Use FP16 and FP32 mixed precision; (2) Use FP16 in forward propagation and backward propagation, and use FP32 in weight update; (3) Dynamically adjust the loss scaling factor to prevent gradient underflow.

[0171] Use appropriate learning rate scheduling strategies to improve model convergence speed and final performance: (1) Use the warmup strategy to gradually increase the learning rate at the beginning of training; (2) Use cosine annealing learning rate scheduling to gradually reduce the learning rate in the later training period; (3) Implement periodic learning rate changes to help the model escape local optima.

[0172] Finally, the model evaluation module evaluates the performance of the trained model. Evaluation metrics include accuracy, recall, F1 score, etc. to ensure that the model meets the expected performance standards.

[0173] The method of the embodiment successfully trains a series of models suitable for safety detection of power field operations by constructing a special multimodal dataset, combining various advanced training strategies and optimization techniques. These models include a scene classification model and multiple target detection models, which together form a powerful hybrid expert model system. The advantage of this training method is that it fully utilizes the zero-shot and few-shot learning capabilities of visual language models and the real-time detection characteristics of YOLO models. Through multi-model collaborative training, the system can adapt to the complex and variable environment of power field operations, improving detection accuracy while maintaining real-time response capabilities.

[0174] In the embodiment, the real-time detection unit includes a first image acquisition module, a YOLO detection module, a first detection result saving module, a first confidence check module, a first VLM review module, a first result merging module, a first overlapping box processing module, and a first output module.

[0175] The first image acquisition module is configured to acquire image data to be detected. The YOLO detection module is configured to input the acquired image data into a general target detection model for target detection. The first detection result saving module is configured to save the first detection result of the general target detection model, which includes a first detection box and its confidence. The first confidence check module is configured to check the confidence and filter out detection boxes with a confidence lower than a preset threshold. The first VLM review module is configured to input the detection boxes with a confidence lower than the preset threshold into a first confidence check model for review to obtain a review result. The first result merging module is configured to merge the review result with the first detection result. The first overlapping box processing module is configured to check the overlap degree of the merged detection boxes and filter out detection boxes with an overlap degree higher than a preset overlap degree threshold and the same class. The first output module is configured to output the final detection result after filtering and merging.

[0176] In specific embodiments, the image acquisition module is configured to acquire images to be detected. These images can come from various sources, such as real-time captured images by a camera, pre-stored image files, etc.

[0177] In specific embodiments, the YOLO detection module inputs the acquired images to be detected into a YOLO-All model for target detection to obtain a first detection result. The YOLO-All model is a general target detection model that can quickly detect multiple targets in an image.

[0178] In specific embodiments, the first detection result saving module is configured to save the first detection result of the YOLO-All model, which includes the detection box coordinates, class labels, attribute descriptions, and detection confidence of each detected target, etc.

[0179] In specific embodiments, the first confidence checking module is configured to check the confidence of each bounding box. For bounding boxes with confidence lower than a preset threshold, they will be marked as needing re-inspection.

[0180] For those bounding boxes needing re-inspection, the VLM re-inspection module inputs them into the VLM-Detection1 model for secondary detection. The VLM-Detection1 model is a visual language-based detection model that can combine image and text information for more accurate target detection.

[0181] In specific embodiments, the first result merging module merges the re-inspection results of the VLM-Detection1 model with the first detection results of YOLO-All. This process considers the detection results of both models to obtain more comprehensive and accurate detection results.

[0182] In specific embodiments, the first overlapping box processing module is configured to check the merged bounding boxes for those with high overlap and same class. For these overlapping boxes, screening and deduplication are performed. The priority rules are as follows: VLM-Detection1 model re-inspection results are preferred to YOLO high confidence results, and YOLO high confidence results are preferred to YOLO low confidence results.

[0183] That is, for bounding boxes with high overlap and same class, they are retained according to priority:

[0184] First priority: re-inspection results of the first multi-modal target detection model;

[0185] Second priority: first detection results returned by the general target detection model that are higher than the preset confidence threshold;

[0186] Third priority: first detection results returned by the general target detection model that are lower than the preset confidence threshold, except those that have been re-inspected by the first multi-modal target detection model;

[0187] In the merged results, only one optimal detection box is retained for each target.

[0188] In specific embodiments, the first output module outputs the final detection results after the above processing. These results include high-quality target detection results after multi-model detection, re-inspection, and screening.

[0189] The real-time detection module of the embodiment combines the rapid detection capability of the YOLO model and the accurate re-checking capability of the VLM model, significantly improving the accuracy of detection while ensuring real-time performance. The YOLO model can quickly detect most targets, while the VLM model can perform more detailed analysis and confirmation on those targets that the YOLO model is not sure about. This combination of different types of models effectively balances the needs of detection speed and accuracy. In addition, the detection accuracy is further improved through the processing of overlapping boxes. By removing redundant detection boxes, the likelihood of false positives is reduced, and the final detection results are more clear and easy to understand.

[0190] In the embodiment, the non-real-time detection unit comprises a second image acquisition module, a scene classification module, a detection model selection module, a second detection result saving module, a second confidence check module, a second VLM re-checking module, a second result merging module, a second overlapping box processing module, and a second output module. Figure 4 The second image acquisition module is configured to acquire image data to be detected. The scene classification module is configured to input the acquired image data to the scene classification for scene classification. The detection model selection module is configured to select a corresponding scene-specific target detection model according to the scene classification result, and simultaneously input a general target detection model for global detection. The second detection result saving module is configured to save the detection results of the scene-specific target detection model and the general target detection model. The second confidence check module is configured to check the confidence of each detection box and filter out detection boxes with a confidence lower than a preset confidence threshold. The second VLM re-checking module is configured to input the detection boxes with a confidence lower than the preset confidence threshold into a second multi-modal target detection model for re-checking. The second result merging module is configured to merge the re-checking results with the detection results of the scene-specific target detection model and the general target detection model. The second overlapping box processing module is configured to check the overlap degree of the merged detection boxes and filter out boxes with an overlap degree higher than a preset threshold and the same class. The second output module is configured to output the final detection results after filtering and merging.

[0191] In specific embodiments, the second image acquisition module is configured to acquire image data to be detected from multiple sources. These images can come from live photos taken by a monitoring system or historical images already stored in the system. Since it is non-real-time detection, the system can process a larger set of images and perform more detailed analysis at each stage.

[0192] In specific embodiments, the scene classification module uses the VLM-Classify model to classify the scene of each image to be detected. Based on the visual language multi-modal technology, this model can classify images into different power field work scenes, such as aerial work, ground work, ladder work, etc. According to the classification result, the system can select the expert detection model most suitable for this scene.

[0193] In specific embodiments, the detection model selection module is used to select the YOLO1-5 model that best matches the scene according to the classification result of the VLM-Classify model for target detection, obtaining the second detection result. At the same time, the system also uses the YOLO-All model for detection to ensure that no potential safety hazards are missed.

[0194] In specific embodiments, the second detection result saving module saves the second detection result, which includes information such as the detected target detection frame, class label, attribute description, and detection confidence.

[0195] In specific embodiments, the second confidence check module is used to check the confidence of each detection frame. For detection frames with a confidence lower than the preset threshold, mark them as objects that need to be reviewed.

[0196] In specific embodiments, the second VLM review module is used to input the detection frames that need to be reviewed into the VLM-Detection2 model for secondary detection. The VLM-Detection2 model is a model specially optimized according to the scene classification information, which can more accurately detect low-confidence targets in a specific scene.

[0197] In specific embodiments, the second result merging module merges the review results of VLM-Detection2 with the second detection results. This process considers all model detection results to obtain the most comprehensive and accurate detection results.

[0198] In specific embodiments, the second overlapping frame processing module checks the merged detection frames to find frames with high overlap and the same class. For these overlapping frames, the system will filter and remove duplicates according to certain rules. The priority rules are as follows: VLM-Detection2 results are preferred to YOLO high-confidence results, and YOLO high-confidence results are preferred to YOLO low-confidence results.

[0199] For frames with high overlap and the same class, the following priority rules are used to retain them:

[0200] First priority: review results of the second multi-modal target detection model;

[0201] Second priority: high confidence detection results returned by the scene-specific object detection model and the general object detection model, which are higher than the pre-set confidence threshold;

[0202] Third priority: low confidence results returned by the scene-specific object detection model and the general object detection model, which are lower than the pre-set confidence threshold; but except for those that have been reviewed by the second multi-modal object detection model;

[0203] In the merged and screened results, only one optimal detection box is retained for each target.

[0204] In specific embodiments, the second output module outputs the final detection results after the above processing. These results include high-quality object detection results after scene classification, multi-model detection, review, and screening.

[0205] The non-real-time detection module of this embodiment achieves more detailed and comprehensive detection in non-real-time mode by combining scene classification models and multi-modal object detection models. The scene classification model can ensure the selection of the most suitable detection model, while the multi-modal object detection model can perform more accurate review of low-confidence targets in complex scenes. This strategy of using different models in combination can significantly improve the accuracy and reliability of detection, and is very suitable for application scenarios that have higher requirements for detection accuracy.

[0206] Effect verification: This embodiment reports the effect verification results of the method proposed in this invention in the detection of power field operation safety. In terms of target detection effect, the detection accuracy of YOLO model, VLM model and MoE model in five typical power field scenes is compared.

[0207] Table 1 shows the detection accuracy of YOLO model, VLM model and MoE (mixed expert model) in different categories:

[0208] Table 1 Detection accuracy of different models in different categories

[0209]

[0210]

[0211] From Table 1 above, it can be seen that the detection accuracy of MoE architecture in all categories exceeds that of YOLO model and VLM model alone. This shows that through the synergistic effect of MoE architecture, the system can more effectively handle complex safety detection tasks in power field operations, significantly improving the accuracy and reliability of detection.

[0212] The above examples are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing examples, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.< / box> < / box> < / ref>

Claims

1. A method for detecting safety in electric field work based on a hybrid expert model, characterized by, The method comprises the following steps: In the training stage, a plurality of models are cooperatively trained to construct an expert model set for different scenes and targets of power field operation safety detection, wherein the expert model set comprises a scene classification model, a general target detection model, a scene-specific target detection model, a first multi-modal target detection model, and a second multi-modal target detection model; In the detection stage, if in a real-time detection mode, a two-stage detection mechanism is adopted, a general target detection model is used for rapid preliminary detection to return a first detection result, the first detection result comprises a first detection frame and a confidence thereof, the confidence of the first detection frame is referred to as a first confidence, the first confidence is compared with a preset confidence threshold, if the first confidence is higher than the preset confidence threshold, a high-confidence result is retained, if the first confidence is lower than the preset confidence threshold, a first multi-modal target detection model is used for re-inspection to obtain a re-inspection result, the re-inspection result is combined with the first detection result, and a detection result is output; In the detection stage, if in a non-real-time detection mode, a scene classification model is used for scene classification to obtain a scene classification result, according to the scene classification result, a scene-specific target detection model is used for detection to return a second detection result, the second detection result comprises a second detection frame and a confidence thereof, the confidence of the second detection frame is referred to as a second confidence, and a general target detection model is used for global detection to return a third detection result, the third detection result comprises a third detection frame and a global confidence thereof, the second confidence and the global confidence are compared with a preset confidence threshold respectively, if the second confidence or the global confidence is higher than the preset confidence threshold, a high-confidence result is retained, if the second confidence or the global confidence is lower than the preset confidence threshold, a low-confidence target is considered, and a second multi-modal target detection model is used for re-inspection of the low-confidence target, detection results of all models are combined, and a final detection result is output.

2. The method for safety detection of electric power field operation based on hybrid expert model according to claim 1, characterized in that, Before the training stage, a multi-modal data set is constructed, the multi-modal data set comprises a scene classification data set and a target detection data set.

3. The method of claim 1, wherein the method is based on a hybrid expert model. The first multi-modal target detection model is used for detecting all label targets in all scenes, and the second multi-modal target detection model is used for detecting specific label targets in specific scenes.

4. The method of claim 1, wherein the method is based on a hybrid expert model. In the detection stage, if in a real-time detection mode, a detection result is integrated and output, specifically, in the combination process, the coincidence degree of the detection frame is checked, if the coincidence degree is greater than a preset coincidence degree threshold, that is, for the frames with high coincidence degree and same class, the frames are retained according to priorities: A first priority: re-inspection result of the first multi-modal target detection model; A second priority: first detection result returned by the general target detection model and higher than the preset confidence threshold; A third priority: first detection result returned by the general target detection model and lower than the preset confidence threshold, but except for the result that has been re-inspected by the first multi-modal target detection model; In the combined result, only one optimal detection frame is retained for each target.

5. The method of claim 1, wherein the method is based on a hybrid expert model. In the detection stage, if in the non-real-time detection mode, the detection results of all models are combined to output the final detection result, including: in the merging process, the coincidence degree of the detection box is checked, if the coincidence degree is greater than the preset coincidence degree threshold, that is, for the high coincidence degree and the same class box, the priority is retained: The first priority: the recheck result of the second multi-modal target detection model; The second priority: the high confidence detection result returned by the scene-specific target detection model and the general target detection model, the high confidence detection result is the detection result higher than the preset confidence threshold; The third priority: the low confidence result returned by the scene-specific target detection model and the general target detection model, the low confidence detection result is the detection result lower than the preset confidence threshold; but the recheck by the second multi-modal target detection model is excluded; In the combined and screened result, only one optimal detection box is retained for each target.

6. The method of claim 1, wherein the method is a hybrid expert model-based safety detection method for electric power field operation. The scene classification model is VLM-Classify model, the general target detection model is YOLO-All model, the scene-specific target detection model is YOLO1-5 model, the first multi-modal target detection model is VLM-Detection1 model, and the second multi-modal target detection model is VLM-Detection2 model.

7. A power field work safety detection system based on a hybrid expert model, characterized by, A power field operation safety detection method based on a hybrid expert model is used to realize any one of claims 1-6, comprising: a model training unit, a real-time detection unit and a non-real-time detection unit; The model training unit is used for multi-modal data set construction and training and evaluation of the hybrid expert model architecture; The real-time detection unit is used for real-time power field operation safety detection, and outputs the detection result; The non-real-time detection unit is used for non-real-time power field operation safety detection, and outputs the detection result.

8. The hybrid expert model based power field operation safety detection system according to claim 7, wherein, The model training unit includes a data set construction module, a prompt engineering test module, a scene classification training module, a target detection model training module, an optimization strategy application module and a model evaluation module; The data set construction module is used to construct a multi-modal data set, which includes a scene classification data set and a target detection data set; the prompt engineering test module is used to perform prompt engineering test to determine the model applicability and the required prompt word type; The scene classification training module is used to train the scene classification training model; the target detection model training module is used to train the general target detection model, the scene-specific target detection model, the first multi-modal target detection model and the second multi-modal target detection model; the optimization strategy application module is used to apply data enhancement, transfer learning, mixed precision training and learning rate scheduling optimization strategies; and the model evaluation module is used to evaluate the performance of the trained model.

9. The hybrid expert model based power field operation safety detection system according to claim 8, wherein, The real-time detection unit includes a first image acquisition module, a YOLO detection module, a first detection result saving module, a first confidence checking module, a first VLM recheck module, a first result merging module, a first overlapping box processing module and a first output module; The first image acquisition module is configured to acquire image data to be detected; the YOLO detection module is configured to input the acquired image data into a general object detection model for object detection; the first detection result saving module is configured to save a first detection result of the general object detection model, the first detection result including a first detection frame and a confidence thereof, and the confidence of the first detection frame is referred to as a first confidence; the first confidence checking module is configured to check the confidence and screen out a detection frame with a confidence lower than a preset threshold; the first VLM review module is configured to input the detection frame with the confidence lower than the preset threshold into a first confidence checking model for review to obtain a review result; the first result merging module is configured to merge the review result and the first detection result; the first overlapping frame processing module is configured to check an overlapping degree of the merged detection frames, and screen out detection frames with an overlapping degree higher than a preset overlapping degree threshold and the same category; and the first output module is configured to output a final detection result after screening and merging.

10. The hybrid expert model based power field operation safety detection system according to claim 9, wherein, The non-real-time detection unit includes a second image acquisition module, a scene classification module, a detection model selection module, a second detection result saving module, a second confidence checking module, a second VLM review module, a second result merging module, a second overlapping frame processing module, and a second output module; the scene classification module is configured to input acquired image data into scene classification for scene classification; the detection model selection module is configured to select a corresponding scene-specific object detection model according to a scene classification result, and simultaneously input a general object detection model for global detection; the second detection result saving module is configured to save detection results of the scene-specific object detection model and the general object detection model; the second confidence checking module is configured to check a confidence of each detection frame, and screen out a detection frame with a confidence lower than a preset confidence threshold; the second VLM review module is configured to input the detection frame with the confidence lower than the preset confidence threshold into a second multi-modal object detection model for review; the second result merging module is configured to merge the review result and the detection results of the scene-specific object detection model and the general object detection model; the second overlapping frame processing module is configured to check an overlapping degree of the merged detection frames, and screen out frames with an overlapping degree higher than a preset threshold and the same category; and the second output module is configured to output a final detection result after screening and merging.

Images