A power safety violation identification and perception planning method based on a large language model

Abstract

The application discloses a power safety violation identification and perception planning method based on a large language model, and belongs to the field of power safety intelligent supervision and multi-modal intelligent reasoning. The method relies on the visual perception and cross-modal reasoning capability of the multi-modal large language model, and decomposes the complex power violation identification process into a structured process of scene understanding, professional end-side detection model scheduling and violation determination. Through scene analysis of the power operation video segment by the multi-modal large language model, structured semantic features are extracted to call a special detection model to complete fine-grained detection, and then a traceable criterion chain is generated through bidirectional evidence fusion, and finally the accurate determination of the violation type is completed in combination with the scene semantics and the detection evidence. Meanwhile, the application constructs a PowerPAct data set containing multiple power violation scenes. The method effectively realizes high accuracy of power safety violation identification and verifiability of the reasoning process.

Description

Technical Field

[0001] This invention belongs to the field of intelligent supervision of power safety and multimodal intelligent reasoning, specifically involving a power safety violation identification and perception planning method based on a large language model. Background Technology

[0002] Power operation scenarios are characterized by complex environments, concealed violations, and high safety risks. Identifying violations during the operation process requires a combination of accurate visual perception and logical reasoning based on power industry standards to complete the entire process of "scene recognition - tool scheduling - violation determination." This places extremely high demands on the professionalism, accuracy, and interpretability of intelligent detection methods.

[0003] In recent years, large language models (LLMs) and multimodal large language models (MLLMs) have been widely used in visual task planning and target recognition due to their powerful reasoning and cross-modal interaction capabilities. However, their research has mostly focused on general scenario adaptation, and there is a lack of in-depth exploration in the field of power safety violation identification, which is highly specialized and subject to strong risk constraints.

[0004] In the existing technology, there are many defects when general MLLM is applied to power violation detection: (1) it separates visual perception from decision planning, fails to give full play to end-to-end multimodal interaction capabilities, and is prone to semantic deviation from video to text, resulting in inaccurate understanding of power operation scenarios; (2) it lacks dedicated data support for power professional scenarios, and the general pre-trained model is difficult to conform to power safety regulations, and has insufficient ability to capture fine-grained violation criteria in operations; (3) it cannot realize the fine-grained scheduling of professional end-side detection models, and is prone to problems such as omission of focus points and incomplete tool calls, resulting in missed violation detection and misjudgment of risks.

[0005] In addition, the field of power safety violation identification currently suffers from a lack of dedicated datasets, which cannot support fine-tuning training and performance verification of MLLMs in professional scenarios. The problem of limited computing power and token resources in the process of processing massive video data has also not been effectively solved.

[0006] Therefore, there is an urgent need for a perception planning method specifically designed for power safety violation identification scenarios to solve the problems of untraceable reasoning process and low violation identification rate of general MLLMs in power safety violation scenarios, and to achieve accurate and interpretable identification of power operation violations. Summary of the Invention

[0007] The problem this invention aims to solve in power safety violation identification scenarios is that general-purpose MLLMs suffer from low accuracy due to the lack of explicit traceable criteria in the inference process, resulting in insufficient reliability and interpretability of power operation violation detection and failing to meet the high safety and professional regulatory requirements of power operations. Therefore, this invention breaks down the complex violation identification task into multiple sub-tasks, achieving accurate scenario understanding and traceable violation determination for power safety violation identification, thereby improving the model's detection accuracy and inference interpretability in professional power scenarios.

[0008] The specific technical solution is as follows: S1. Collect video clips containing potential power safety violations and preprocess them; use two multimodal large language models as questioner agents and answerer agents respectively, and construct a first question for analyzing visual elements in the video clips and a second question for identifying the power safety violations to which the video clips belong; S2, the responder agent extracts visual elements from the preprocessed video clips and performs semantic parsing based on the first question raised by the questioner agent, generating a structured semantic representation; S3, based on the detection targets contained in the potential risk cues in the structured semantic representation, call the target detection model corresponding to the detection target for the preprocessed video segment, and obtain the structured detection evidence output by each target detection model respectively; S4, combine all structured detection evidence to obtain the detection evidence set; S5, the responder agent, based on the second question posed by the questioner agent, and using structured semantic representation and a set of detected evidence, maximizes the output of the power safety violation corresponding to the video segment through conditional probability.

[0009] Furthermore, in S2, the visual elements include the working environment, key equipment, operators, and work behaviors; The process of generating structured semantic representations is as follows: the responder agent extracts visual elements from video clips and performs semantic analysis on the visual elements in conjunction with professional knowledge of power safety to obtain potential risk clues containing key information of concern; the visual elements and potential risk clues are further integrated to generate structured semantic representations.

[0010] Furthermore, S3 specifically refers to: S301, Construct a power safety violation detection toolkit containing multiple target detection models; S302, Generate a binary gating vector for calling different target detection models based on the detection targets contained in the potential risk cues in the structured semantic representation; S303, based on the binary gating vector, calls the corresponding target detection model from the power safety violation detection tool library; S304: Input the preprocessed video segments into all the object detection models called in S303, and each object detection model outputs structured detection evidence.

[0011] Furthermore, S301 specifically includes: S3011, taking different detection targets as units, constructs training datasets corresponding to each detection target; and constructs a power safety violation detection tool library containing multiple target detection models; S3012, For all the object detection models in the tool library, train them using training datasets corresponding to different detection targets to obtain dedicated object detection models for extracting the corresponding detection targets.

[0012] Furthermore, each dimension of the binary gating vector corresponds to a target detection model in the power safety violation detection toolkit. If the first dimension of the binary gating vector... If the value is 1, it means that the nth target detection model in the power safety violation detection tool library is called; if the value is 0, it means that the target detection model is not called.

[0013] Furthermore, before entering S303, the validity of the binary gating vector needs to be verified to ensure that at least one target detection model is invoked.

[0014] Furthermore, in S4, the structured detection evidence includes the bounding box of the detected target, the confidence level, and the determination result of whether the detected target has entered the danger zone.

[0015] Furthermore, in step S4, in addition to obtaining the set of detection evidence, a security violation criterion chain is further obtained to facilitate subsequent tracing by the user; the security violation criterion chain is a triple structure, which includes the index of all target detection models called by the video segment, the parameter configuration of each target detection model, and all normalized structured detection evidence.

[0016] Furthermore, in S5, the electrical safety violation corresponding to the video clip specifically refers to: ; in, This is a probability estimate for the responder agent of each category of violation under given semantic conditions of the scenario. For different violations of power safety regulations, s is a structured semantic representation. To detect the evidence set, The video clip output by the respondent AI represents a violation of power safety regulations.

[0017] The beneficial effects of this invention are: First, this invention constructs the PowerPAct dataset, a dedicated dataset for identifying violations in the power safety field, filling the gap in dedicated data for violation identification in the power industry and providing reliable data support for fine-tuning and performance verification of multimodal large language models.

[0018] Secondly, this invention decomposes the complex process of identifying power violations into a structured process of "scene understanding - evidence verification - violation determination", which realizes the synergistic integration of visual perception and decision planning and improves the detection accuracy of the model.

[0019] Finally, this invention constructs a dedicated power-side detection tool library, which provides objective and complete evidence to support violation judgments, enabling the reasoning process to be traceable and verifiable, and significantly improving the reliability and interpretability of the judgment results. Attached Figure Description

[0020] Figure 1 This is an overall framework diagram of the present invention.

[0021] Figure 2 This is a detailed flowchart of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0023] like Figure 1 and Figure 2 As shown, this invention proposes a power safety violation identification and perception planning method based on a large language model. Through multimodal scene understanding, edge tool scheduling, and traceable criterion chain generation, it improves the accuracy, interpretability, and reliability of power safety violation identification, providing an efficient and reliable technical solution for intelligent power safety supervision.

[0024] Includes the following steps: Step 1: Collect and preprocess videos of illegal power operation scenarios. First, videos of violations in power operations were collected, covering 10 typical power safety violation scenarios (e.g., workers staying under a crane during operation). For each scenario, approximately 300 video clips of about 10 seconds each were collected, and the data was anonymized and standardized. Next, video footage is manually screened to remove samples with unclear violation characteristics or no clear operator, ensuring the validity of the samples.

[0025] Step 2: Perform multimodal scene understanding on power operation video clips. First, two multimodal large language models are used as the questioner and the answerer, respectively (in a specific embodiment of the present invention, GPT-4o is used as the questioner and GPT-5.2 is used as the answerer). The video clip of the power operation to be detected is input into the multimodal large language model, and visual frames are extracted at a fixed frame rate and converted into visual features that the model can recognize. At the same time, the number of tokens in the text output is limited to ensure the efficiency and accuracy of scene understanding. Then, the model analyzes the core visual elements in the video, such as the working environment, key equipment, workers, and work behaviors. Utilizing the model's cross-modal reasoning capabilities and combining power safety expertise, it performs semantic analysis on the visual elements to identify potential risk clues. Finally, by integrating core elements and risk clues, a structured semantic representation containing key information of concern is generated. This serves as the semantic basis for scheduling edge-side tools.

[0026] Through the collaboration of two models, the video can be analyzed for scenarios and risks can be identified, guiding respondents to output content according to the logic of "scenario description - extraction of key information - preliminary risk assessment"; Step 3: Schedule edge detection tools based on scene semantic representation First, a pre-built edge tool library for detecting power safety violations is retrieved. This library contains 28 pre-trained detection models targeting key objectives in power operations, with standardized parameter configurations. Each model is pre-trained using a dataset corresponding to the detection target, resulting in a model with good detection performance for that target. For example, the YOLOv11 target detection model is trained on a dataset of 3000 images of workers to determine whether they are wearing safety helmets. The WIoU loss function is used to update the YOLOv11 target detection model parameters, resulting in a model with a detection accuracy exceeding 90% in detecting whether safety helmets are worn. This model is then used as a model in the tool library.

[0027] Then, a tool selection function is introduced to transform the scene structured semantic representation generated in step 2 into a structured semantic representation based on the key information extracted by the respondent. As input to this function, based on the key interest information in the semantic representation, it is matched with the target detected by the tool library model, and the output is a binary gating vector. The first vector dimension This indicates calling the nth corresponding edge detection tool in the tool library. , This indicates that the tool will not be invoked; Finally, the binary gating vector is validated to ensure that at least one corresponding edge detection tool is triggered for key information of interest in the video, thus guaranteeing the integrity of subsequent evidence verification.

[0028] Step 4: Generate detection evidence and construct a traceable chain of violation criteria. First, based on the tool scheduling results, the corresponding end-side detection model is invoked. The power operation video clips are input into the model for fine-grained visual detection. The end-side detection model, according to the preset parameter configuration, outputs structured detection evidence including the bounding box of the detected target, confidence level, and hazard area determination. Among them, the danger zone determination is based on the end-side detection model to determine whether personnel or objects have entered the pre-defined prohibited / restricted area according to the preset spatial range (such as the area around live equipment, under high-voltage lines, the activity area of ​​the robotic arm, etc.) and the real-time position or posture of the operator.

[0029] Then, all test evidence is normalized to unify the format and data dimensions, which facilitates subsequent evidence integration and analysis.

[0030] Finally, the indexes of each called tool are determined according to the order in which the edge detection tools are invoked. Tool parameter configuration Normalized test evidence Integrate into a triplet Arrange all triples in sequence to generate a traceable chain of criteria for judging power safety violations. To facilitate post-monitoring by users, the formula is as follows: in, The total number of times the detection tool is called for a video segment is recorded. The criterion chain fully records the entire process of tool scheduling and detection, enabling the reasoning process to be traceable and verifiable.

[0031] First, all structured detection evidence is extracted from the violation judgment chain and integrated into a unified detection evidence set. Conflicting information in the detection evidence set is screened and verified, and valid and accurate detection evidence is retained, thus obtaining a valid detection evidence set.

[0032] Then, the structured semantic representation With valid test evidence set The module takes into account the input of respondents and combines professional power safety standards with violation judgment logic. It performs deep fusion reasoning on scene semantics and detection evidence, calculates the probability of various power safety violation types by maximizing conditional probability, and outputs the final power safety violation category. Its formula is: in, This model estimates the probability of various types of power safety violations under given scenario semantic conditions. The parameter matrix of the model, Different categories of electrical safety violations.

[0033] Meanwhile, this invention further integrates all collected video clips of power operation violation scenarios, criterion chains, and corresponding violation results to form the PowerPAct dataset.

[0034] For existing open-source multimodal large language models, the PowerPAct dataset can be further divided into training, validation, and test sets. The training set can be used to fine-tune the basic multimodal large language model corresponding to the respondent using LoRA, and the validation set can be used to optimize the model parameters in real time, thereby achieving professional adaptation of the model to the power scenario.

[0035] To further verify the effectiveness of the proposed method, based on the self-developed PowerPAct power safety violation identification dataset (which covers 10 typical power safety violation scenarios, totaling 3000 10-second power operation video clips, with each scenario accompanied by a corresponding violation criterion chain and judgment result), and combined with the actual operational needs of the power industry, three core application verification scenarios were identified: Power construction site operations: Detecting violations such as personnel staying in dangerous areas while cranes / excavators are in operation, not wearing fall arrest ropes when climbing poles, and not wearing safety goggles when welding. Substation inspection: Identify safety hazards such as inspection personnel not wearing insulated boots or gloves, and improper equipment inspection operations; Transmission line work: violations included lack of protective measures during voltage testing, failure to wear safety helmets during pole erection, and lack of dedicated supervision during distribution box maintenance.

[0036] The specific experimental setup is as follows: The base model uses open-source mainstream multimodal large language models such as InternVL3.5-14B and Qwen3-8B, which have strong visual perception and cross-modal reasoning capabilities. The edge tool library integrates 28 power-specific edge detection models, covering key detection targets such as cranes, hooks, safety helmets, insulating protective equipment, and workers, with a uniform detection confidence threshold of 0.85. During fine-tuning, BF16 mixed precision training is used, with 5 LoRA fine-tuning rounds and a learning rate of 5e. -5 Only a small number of parameters are added to the core layer of the training model, thereby reducing computational consumption.

[0037] During the training phase, the PowerPAct dataset was divided into training, validation, and test sets in a 7:1:2 ratio. The training set was used to fine-tune the basic multimodal large language model using LoRA, and the validation set was used to optimize the model parameters in real time, thereby achieving professional adaptation of the model to the power scenario. During the inference phase, the test set's detected videos are evaluated using inference. The input is a 10-second video clip of power operation, and the maximum number of tokens for the text output is limited to 1024. The hardware environment consists of four NVIDIA RTX 5090 GPUs.

[0038] The following indicators are used to evaluate the performance of safety risk detection: (1) ToolAct-Acc: measures the ability of the model to accurately trigger edge detection tools based on scene semantics.

[0039] in, For the first Number of tool scheduling hits in a power scenario For the first The total number of times the tool should be triggered in each power scenario.

[0040] (2) Violation identification accuracy (Vio-Acc): measures the accuracy of the model in the final determination of the type of violation in power operations.

[0041] in, For the first The number of video samples that correctly predicted violations in power-related scenarios. For the first Total number of video clips in power-related scenarios.

[0042] (3) Scene understanding semantic similarity (BERTScore): measures the accuracy of the model in describing the scene of power operation video. It includes three sub-indicators: Precision, Recall, and F1. The higher the value, the more accurate the scene understanding.

[0043] Finally, the experimental results are as follows: This method was compared with the performance of a general pre-trained multimodal large language model on the PowerPAct dataset. The experimental results are shown in Tables 1-4. The results show that after fine-tuning with LoRA on the PowerPAct dataset, the model achieves significant improvements in all three core metrics.

[0044] As shown in Table 1, during the pre-training phase, the word-level similarity scores of different MLLMs were generally low, indicating that the descriptions generated by the models lacked accuracy and completeness, making it difficult to meet the needs of actual operations. Among them, InternVL3.5-14B was relatively superior in BLEU-4 and ROUGE-L metrics, reaching 0.0584 and 0.2034 respectively, indicating that models with a larger number of parameters in the same series are more likely to generate more accurate descriptions. On the Token-Acc scale, LLaVA-1.5-13B reached 0.4715, indicating that it has relatively accurate descriptions of some high-frequency keywords in the power sector.

[0045] However, keyword matching alone is insufficient to support criterion chain reasoning. As shown in Table 2, during the pre-training phase, InternVL3.5-14B achieved the highest BERTS score of 0.6308, while the other models ranged from 0.54 to 0.62. This indicates that the generated text still suffers from "missing similarity information or unclear relationship descriptions," which may hinder the model from extracting a complete set of concerns, thus affecting the trigger coverage of edge tools.

[0046] After LoRA fine-tuning, the model's performance on both word-level and semantic-level metrics for video scene understanding was significantly improved. Specifically, Qwen2.5-7B showed the most significant improvements in BLEU-4 and Token-Acc, reaching 0.1629 and 0.5652 respectively, and ROUGE-L improved to 0.3333. Qwen3-8B achieved the highest value of 0.3402 in ROUGE-L, indicating a significant improvement in key information coverage and descriptive coherence of its generated text. In terms of semantic consistency, the fine-tuned InternVL3.5-14B achieved the highest value of 0.7227, a significant improvement compared to its pre-training stage value of 0.6308, indicating that aligning the operational behavior and risk cues in the reference description at the semantic level is equally important for the model. Overall, LoRA fine-tuning enhanced the model's scene awareness capabilities, providing solid evidence for subsequent tool invocation and final violation determination.

[0047] To evaluate the intermediate inference capability of the model after instruction decomposition, this invention uses the ToolAct-Acc metric to measure the scene semantic representation based on video clips, accurately extracting points of interest and triggering corresponding edge tools to complete verification. The experimental results are shown in Table 3. Since power violations rely on multiple verifiable criteria (such as entering a danger zone or walking under a crane), ToolAct-Acc can effectively measure whether the model has the ability to translate "language understanding" into "executable detection".

[0048] As shown in Table 3, the ToolAct-Acc of the pre-trained model is generally low, indicating that when relying solely on a general pre-trained model, the model is prone to overlooking key points or incomplete tool triggering in the "which tools to call" stage, making it difficult to close the criterion chain. After LoRA fine-tuning, ToolAct-Acc has significantly improved, indicating that PowerPAct greatly enhances the model's ability to describe power operation videos, enabling the model to generate verifiable criterion chains more stably and providing reliable evidence for subsequent violation identification.

[0049] To evaluate the accuracy of MLLMs, this invention uses the Vio-Acc metric for violation identification accuracy to demonstrate the final decision-making capability of the method in actual power safety supervision scenarios, as shown in Table 4. The overall performance of the pre-trained models is weak, with most results below 0.35, indicating that general pre-trained models lack expertise in power safety violation scenarios and are prone to inaccurate judgments. Notably, GPT-4o achieved a relatively high accuracy (0.6950), thanks to its strong generalization ability on large-scale general data, enabling it to directly identify key features in power scenarios. However, its high invocation cost limits its practical deployment capability in power field applications.

[0050] In contrast, the InternVL3.5 series exhibits better recognition capabilities as the scale increases, with InternVL3.5-14B reaching 0.4516. This indicates that stronger representation capabilities can alleviate inference uncertainty in complex scenarios to some extent, but it is still difficult to meet the actual needs in power scenarios.

[0051] After LoRA fine-tuning, the Vio-Acc of different MLLMs significantly improved, with InternVL3.5-14B reaching the highest at 0.8033 and Qwen3-8B reaching 0.7533. This indicates that PowerPAct can improve the model's recognition of power scenario elements and facilitate instruction decomposition reasoning. Overall, PowerPAct can effectively improve the verifiable decision-making ability of MLLMs in power violation scenarios, enabling the final violation identification to be based on evidence. Table 1. Comparison of caption generation quality for video scene understanding models Table 2. Comparison of semantic similarity of different MLLMs for video scene understanding Table 3. Comparison of tool invocation hit rates for different MLLMs Table 4 Comparison of violation detection accuracy of different MLLMs Based on the same inventive concept, this invention proposes a power safety violation identification and planning system based on a large language model, comprising: The data acquisition module is used to acquire and preprocess video clips containing potential power safety violations; two multimodal large language models are used as questioner and responder agents respectively, and a first question is constructed to analyze the visual elements in the video clips and a second question is constructed to identify the power safety violations to which the video clips belong; The semantic parsing module is used by the responder agent to extract visual elements from the preprocessed video clips based on the first question posed by the questioner agent and perform semantic parsing to generate a structured semantic representation. The target detection model invocation module is used to invoke the target detection model corresponding to the target detection target for the preprocessed video segment based on the detection target contained in the potential risk cues in the structured semantic representation, and obtain the structured detection evidence output by each target detection model. The evidence combination module is used to combine all structured detection evidence to obtain a detection evidence set; The violation classification module is used by the responder agent to output the power safety violation corresponding to the video segment based on the second question raised by the questioner agent, according to the structured semantic representation and the detection evidence set, by maximizing the conditional probability.

[0052] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0053] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0054] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A power safety violation identification and perception planning method based on a large language model, characterized in that, include: S1, Collect video clips containing potential violations of power safety regulations and preprocess them; Two multimodal large language models were used as the questioner agent and the responder agent, respectively, and a first question was constructed to analyze the visual elements in the video clip and a second question was constructed to identify the power safety violation of the video clip. S2, the responder agent extracts visual elements from the preprocessed video clips and performs semantic parsing based on the first question raised by the questioner agent, generating a structured semantic representation; S3, based on the detection targets contained in the potential risk cues in the structured semantic representation, call the target detection model corresponding to the detection target for the preprocessed video segment, and obtain the structured detection evidence output by each target detection model respectively; S4, combine all structured detection evidence to obtain the detection evidence set; S5, the responder agent, based on the second question posed by the questioner agent, and using structured semantic representation and a set of detected evidence, maximizes the output of the power safety violation corresponding to the video segment through conditional probability.

2. The power safety violation identification and perception planning method based on a large language model according to claim 1, characterized in that, In S2, the visual elements include the working environment, key equipment, workers, and work behaviors; The process of generating structured semantic representations is as follows: the responder agent extracts visual elements from video clips and performs semantic analysis on the visual elements in conjunction with professional knowledge of power safety to obtain potential risk clues containing key information of concern; the visual elements and potential risk clues are further integrated to generate structured semantic representations.

3. The power safety violation identification and perception planning method based on a large language model according to claim 1, characterized in that, Specifically, S3 is: S301, Construct a power safety violation detection toolkit containing multiple target detection models; S302, Generate a binary gating vector for calling different target detection models based on the detection targets contained in the potential risk cues in the structured semantic representation; S303, based on the binary gating vector, calls the corresponding target detection model from the power safety violation detection tool library; S304: Input the preprocessed video segments into all the object detection models called in S303, and each object detection model outputs structured detection evidence.

4. The power safety violation identification and perception planning method based on a large language model of claim 3, wherein, Specifically, S301 is as follows: S3011, taking different detection targets as units, constructs training datasets corresponding to each detection target; and constructs a power safety violation detection tool library containing multiple target detection models; S3012, For all the object detection models in the tool library, train them using training datasets corresponding to different detection targets to obtain dedicated object detection models for extracting the corresponding detection targets.

5. The power safety violation identification and perception planning method based on a large language model according to claim 3, characterized in that, Each dimension of the binary gating vector corresponds to a target detection model in the power safety violation detection tool library. If the value of the nth dimension of the binary gating vector is 1, it indicates that the nth target detection model in the power safety violation detection tool library is called, and if the value is 0, it indicates that the target detection model is not called. The value of the nth dimension of the binary gating vector is 1, indicating that the nth target detection model in the power safety violation detection tool library is called, and the value is 0, indicating that the target detection model is not called.

6. The power safety violation identification and perception planning method based on a large language model according to claim 3, characterized in that, Before entering S303, the validity of the binary gating vector must be verified to ensure that at least one target detection model is called.

7. The power safety violation identification and perception planning method based on a large language model of claim 1, wherein, In S4, the structured detection evidence includes the bounding box of the detected target, the confidence level, and the determination result of whether the detected target has entered the danger zone.

8. The power safety violation identification and perception planning method based on a large language model of claim 1, wherein, In step S4, in addition to obtaining the set of detection evidence, a security violation criterion chain is further obtained to facilitate subsequent tracing by the user. The security violation criterion chain is a triple structure, which includes the index of all target detection models called by the video segment, the parameter configuration of each target detection model, and all normalized structured detection evidence.

9. The power safety violation identification and perception planning method based on a large language model of claim 1, wherein, In S5, the power safety violation corresponding to the video clip is specifically as follows: ； in, This is a probability estimate for the responder agent of each category of violation under given semantic conditions of the scenario. For different violations of power safety regulations, s is a structured semantic representation. To detect the evidence set, The video clip output by the respondent AI represents a violation of power safety regulations.

10. A power safety violation identification and perception planning system based on a large language model, used to implement the power safety violation identification and perception planning method based on a large language model as described in claim 1, characterized in that, include: The data acquisition module is used to acquire and preprocess video clips containing potential violations of power safety regulations. Two multimodal large language models were used as the questioner agent and the responder agent, respectively, and a first question was constructed to analyze the visual elements in the video clip and a second question was constructed to identify the power safety violation of the video clip. The semantic parsing module is used by the responder agent to extract visual elements from the preprocessed video clips based on the first question posed by the questioner agent and perform semantic parsing to generate a structured semantic representation. The target detection model invocation module is used to invoke the target detection model corresponding to the target detection target for the preprocessed video segment based on the detection target contained in the potential risk cues in the structured semantic representation, and obtain the structured detection evidence output by each target detection model. The evidence combination module is used to combine all structured detection evidence to obtain a detection evidence set; The violation classification module is used by the responder agent to output the power safety violation corresponding to the video segment based on the second question raised by the questioner agent, according to the structured semantic representation and the detection evidence set, by maximizing the conditional probability.

Images