Electric power operation target detection method based on multi-mode large model knowledge distillation

Abstract

The invention relates to the field of target detection, and particularly discloses an electric power work target detection method based on multi-modal large model knowledge distillation, which utilizes a vision-language multi-modal large model as a teacher model, and improves the target detection efficiency by expanding prompt word guidance. A high-quality pseudo label and a region-text pair are generated for an unlabeled electric power work image as a supervision signal, and on this basis, through joint optimization of detection loss, feature distillation loss, logic distillation loss and multi-modal contrast learning loss, a lightweight YOLO student model is guided to learn positioning and classification knowledge and to learn a multi-modal contrast learning loss. And deep alignment with the open vocabulary understanding ability of the teacher model is carried out on the feature space and semantic level, so that a semantic gap between closed category detection and open world perception is effectively bridged. Through the mode, the detection precision and generalization ability of the student model on common, rare and even unseen targets in the electric power work scene are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of target detection, and more specifically, to a method for power operation target detection based on multimodal large model knowledge distillation. Background Technology

[0002] In the field of safety monitoring at power operation sites, real-time target detection technology is crucial for ensuring personnel safety and the stable operation of the power grid. While traditional lightweight detectors based on architectures such as YOLO offer high inference speeds and can meet the real-time requirements of edge deployments, their training process relies on labeled data with closed category sets. This makes it difficult to effectively identify rare tools, specific safety equipment, or unseen targets in power operation scenarios, limiting the model's generalization ability in complex real-world environments. Meanwhile, large-scale vision-language multimodal models, with their powerful open-vocabulary understanding capabilities, can identify a wide range of target categories guided by prompts. However, their massive computational resource requirements prevent direct application to resource-constrained edge devices, hindering their ability to support real-time inference tasks in the field.

[0003] While knowledge distillation has achieved success in image classification and general detection tasks, its efficient application in safety-critical, domain-specific scenarios such as power operations remains to be explored. Existing detectors, limited by the diversity of training categories and scenarios, often struggle to identify rare or unseen targets; while relying directly on manual annotation for training faces the dual challenges of high annotation costs and long-tailed distribution. Although large multimodal models can provide rich semantic supervision, many current distillation methods neglect the semantic alignment between region-level features and textual descriptions; furthermore, they typically focus only on bounding box regression and classification, failing to balance multimodal feature consistency and the flexibility of open vocabularies in practical deployments.

[0004] Therefore, an optimized method for power operation target detection based on multimodal large model knowledge distillation is needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a method for detecting power operation targets using multimodal large-model knowledge distillation.

[0006] According to one aspect of this application, a method for detecting power operation targets based on multimodal large model knowledge distillation is provided, comprising:

[0007] The training phase includes the following steps:

[0008] Obtain the set of unlabeled power operation images and the set of extended prompt words;

[0009] The unlabeled set of power operation images and the set of extended prompt words are input as a multimodal large model of the teacher model to obtain filtered pseudo-supervised data;

[0010] Input unlabeled power operation images into a lightweight YOLO student model to obtain the student model output;

[0011] Based on the filtered pseudo-supervised data and the student model output, the detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss are determined.

[0012] Based on detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss, the model parameters of the lightweight YOLO student model are optimized and updated to obtain the updated YOLO student model.

[0013] The detection phase includes the following steps:

[0014] Acquire images of the power operation to be detected;

[0015] The image of the power operation to be detected is input into the updated YOLO student model to obtain the target detection results of the power operation.

[0016] Compared with existing technologies, this application provides a multimodal large-model knowledge distillation method for power operation target detection. It utilizes a visual-linguistic multimodal large-model as the teacher model, and through expanded cue words, generates high-quality pseudo-labels and region-text pairs as supervision signals for unlabeled power operation images. Based on this, by jointly optimizing the detection loss, feature distillation loss, logistic distillation loss, and multimodal contrastive learning loss, it guides a lightweight YOLO student model not only to learn localization and classification knowledge, but also to deeply align with the teacher model's open-vocabulary understanding capabilities at the feature space and semantic levels. This effectively bridges the semantic gap between closed-category detection and open-world perception. In this way, it significantly improves the student model's detection accuracy and generalization ability for common, rare, and even unseen targets in power operation scenarios. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 This is a learning model framework diagram of the power operation target detection method based on multimodal large model knowledge distillation according to the embodiments of this application;

[0019] Figure 2 This is a flowchart of the training phase of the power operation target detection method based on multimodal large model knowledge distillation according to embodiments of this application;

[0020] Figure 3 This is a flowchart of the detection phase of the power operation target detection method based on multimodal large model knowledge distillation according to embodiments of this application;

[0021] Figure 4 This is a data architecture diagram of the training phase of the power operation target detection method based on multimodal large model knowledge distillation according to embodiments of this application;

[0022] Figure 5 This is a schematic diagram of an ablation experiment using the Safety Helmet Detection dataset.

[0023] Figure 6 A schematic diagram of the ablation experiment using the VOC 2008 dataset;

[0024] Figure 7 Parameters in the Safety Helmet Detection dataset A diagram illustrating the impact on the model;

[0025] Figure 8 Parameters in the VOC 2008 dataset A diagram illustrating the impact on the model. Detailed Implementation

[0026] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0027] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0028] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0029] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0030] To address the aforementioned challenges, this application proposes a real-time target detection knowledge distillation framework guided by a multimodal large model, designed for power operation environments. Figure 1 This is a learning model framework diagram of the power operation target detection method based on multimodal large model knowledge distillation according to embodiments of this application, such as... Figure 1 As shown, this framework transfers the open vocabulary perception capabilities of a large multimodal model to a lightweight YOLO network detector for real-time perception in power operation scenarios. Specifically, the prompt words and input image are first fed into a large multimodal teacher model; the teacher outputs pseudo-labels (such as categories) and high-level features; simultaneously, the same image is input into a lightweight YOLO student model to obtain edge prediction results; based on this, the supervision module is trained through two types of knowledge distillation: one is label-level distillation based on teacher pseudo-labels, and the other is feature-level distillation by aligning intermediate representations, and the entire model is updated based on the joint loss composed of the two.

[0031] Specifically, the technical solution of this application proposes a power operation target detection method based on multimodal large model knowledge distillation. Figure 2 This is a flowchart of the training phase of the power operation target detection method based on multimodal large model knowledge distillation according to an embodiment of this application. Figure 4 This is a data architecture diagram of the training phase of the power operation target detection method based on multimodal large model knowledge distillation according to embodiments of this application. Figure 2 and Figure 4 As shown, the power operation target detection method based on multimodal large model knowledge distillation according to an embodiment of this application includes a training phase, the specific steps of which include: S1, obtaining an unlabeled power operation image set and an extended prompt word set; S2, inputting the unlabeled power operation image set and the extended prompt word set into a multimodal large model as a teacher model to obtain filtered pseudo-supervised data; S3, inputting the unlabeled power operation images into a lightweight YOLO student model to obtain student model output; S4, determining detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss based on the filtered pseudo-supervised data and student model output; S5, optimizing and updating the model parameters of the lightweight YOLO student model based on the detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss to obtain an updated YOLO student model.

[0032] Specifically, in S1, an unlabeled set of power operation images and an extended cue word set are acquired. It should be understood that while traditional real-time detectors (such as YOLO) are efficient, they are limited by closed vocabularies and struggle to generalize to rare or unseen targets (such as specific power operation tools or safety equipment). While multimodal large-scale models possess open-vocabulary understanding capabilities, their computational overhead prevents deployment at edge computing environments. Therefore, this method acquires unlabeled images and extended cue words, using a multimodal large-scale model as a teacher model to generate pseudo-supervised data, thereby transferring open-vocabulary capabilities to a lightweight student model, achieving efficient and deployable detection. This step forms the basis of the knowledge distillation framework, ensuring the model can cover the diversity and long-tail distribution of power operation scenarios while avoiding costly manual annotation.

[0033] In practice, the first step is to acquire an unlabeled set of power operation images. This set refers to a collection of RGB images directly collected from power operation sites, without the need for manual bounding box or category labeling. The images contain information about workers, safety equipment, and specialized tools, and are characterized by complex scenes, diverse targets, and variable conditions, serving as the raw data source for model training. Specifically, the system collects unlabeled RGB image sets from power operation sites (such as complex scenarios like substations, overhead lines, and switchyards). ,in, These images cover different lighting and weather conditions, feature large depth of field and domain-specific tools, and present real and demanding visual challenges (such as small target helmets, frequent occlusion, cluttered backgrounds, and motion blur).

[0034] Secondly, an expanded cue word set is obtained. This expanded cue word set refers to a set of text prompts constructed and expanded based on safety operation specifications, used to guide the multimodal large model in open-vocabulary detection. Expansion methods include adding synonyms and domain aliases to cover rare or variant targets, serving as key input for semantic supervision generated by the teacher model; specifically, the system constructs a cue word set P based on power safety operation specifications. Furthermore, it expands upon this by introducing synonyms and domain aliases (such as "safety helmet" and its variants) to improve recall for low-frequency tools or identifiers. The expansion process aims to enhance the coverage of the open vocabulary, enabling the teacher model to respond to a wider range of semantic queries.

[0035] Specifically, in step S2, the unlabeled power operation image set and the extended cue word set are input as the multimodal large model of the teacher model to obtain filtered pseudo-supervised data. It should be understood that traditional methods rely on manual annotation, which is costly and difficult to cover the long-tail distribution and rare targets in power operation scenarios; while multimodal large models can provide detection results with strong generalization, their raw output may contain noise, redundancy, or low-confidence pseudo-labels, and direct use would reduce the learning efficiency of the student model. Therefore, generating and filtering pseudo-supervised data through the teacher model can ensure the stability and reliability of the distillation process, enabling the student model to inherit the semantic prior of the teacher model while adapting to the deployment constraints at the edge. This step is a key link in the knowledge distillation framework, bridging the theoretical advantages of large-scale models with the practical needs of lightweight models, providing purified input for subsequent loss calculation and parameter optimization.

[0036] In practice, firstly, multimodal encoding, cross-modal fusion, and decoding are performed on each unlabeled power operation image and the extended prompt word set in the unlabeled power operation image set to obtain the original detection result set. During this process, the teacher model receives image I and the prompt word set P as input, extracts image features through a visual encoder, and simultaneously extracts the embedded representations of the prompt words through a text encoder. Subsequently, the cross-modal fusion module interacts with the visual and text features, and the decoder outputs the original detection result set. This process is expressed by the following formula:

[0037]

[0038] Each result tuple in the original detection result set contains five key elements: bounding box. This represents the location information output by the teacher model; category index. , represents the index of the corresponding hint in the hint set P; confidence level This indicates the degree of certainty the model has regarding the detection result; region-level visual embedding. This represents the high-dimensional visual features of the target region in the image; and text embedding. , representing the prompt word semantic features;

[0039] Next, the original detection result set is filtered for high-confidence pseudo-labels to obtain a high-confidence detection result set. Specifically, this is done by setting a confidence threshold. When the confidence level of the test results If a false label is present, it is retained; otherwise, it is filtered out. This is to balance the noise of false labels with coverage, remove low-quality or unreliable predictions, and ensure the accuracy of subsequent distillation data.

[0040] Furthermore, redundant pseudo-label suppression based on prompt words is applied to the high-confidence detection result set to obtain filtered pseudo-supervised data. It should be understood that although the preceding steps have filtered out some low-quality predictions through a confidence threshold, large models often generate multiple slightly offset but semantically identical bounding boxes for the same target. Without suppression, these overlapping bounding boxes will act as noise signals, misleading the training of lightweight YOLO student models, causing the model to fail to converge or produce duplicate detections. Therefore, in the technical solution of this application, redundant pseudo-label suppression based on prompt words is applied to the high-confidence detection result set to eliminate duplicate detection boxes and retain the most representative pseudo-supervised data, thereby improving the efficiency and effectiveness of knowledge distillation.

[0041] In this process, firstly, the high-confidence detection result set is grouped by category to obtain a category-grouped detection result set. That is, each detection result in the high-confidence detection result set is grouped according to its corresponding category index. Specifically, each detection result in the high-confidence detection result set is traversed, its category index is read, and then all detection results belonging to the same category are merged into the same subset. This means that for each specific prompt word, the system constructs an independent candidate box list, laying the foundation for subsequent processing of redundant boxes within the same category.

[0042] Next, based on the NMS threshold, independent non-maximum suppression is performed on the detection result sets grouped by category to obtain the final detection result set. That is, after grouping by category, the system independently performs non-maximum suppression on each subset of detection results grouped by category, based on a preset NMS threshold. Non-maximum suppression is a classic algorithm whose core principle is to sort detection boxes within the same category according to their confidence scores and calculate the overlap area (usually using the Intersection over Union (IoU) metric) between all boxes and the highest-scoring box. Boxes with an overlap area exceeding the NMS threshold are considered redundant and suppressed (i.e., deleted), retaining only the detection box with the highest confidence score. This process is performed independently within each category group, ensuring that targets from different categories are not incorrectly suppressed, while effectively eliminating redundant detection boxes within the same category.

[0043] Next, the final detection result set is formatted to obtain filtered pseudo-supervised data. That is, after obtaining the final detection result set through independent nonmaximum suppression, these results are transformed into a standard format that the student model can recognize during training, thus obtaining the filtered pseudo-supervised data. The specific content of data formatting typically includes organizing the bounding box coordinates, class index, and any other possible related information (such as text embeddings, visual embeddings, etc.) of each retained detection result into a specific data structure, such as a list or tensor, to facilitate matching and calculation with the student model's output in subsequent loss calculations.

[0044] Specifically, in step S3, unlabeled power operation images are input into a lightweight YOLO student model to obtain the student model output. Considering that while the large multimodal model serving as the teacher model possesses powerful open-vocabulary understanding capabilities, its large model size and computational overhead make real-time deployment at the edge difficult, failing to meet the stringent real-time requirements of intelligent monitoring of power operation sites. To address these issues, the technical solution of this application introduces a lightweight YOLO network as the student model. By inputting unlabeled power operation images into this student model, an efficient detector suitable for edge deployment is obtained, producing its original output. This output is then compared and distilled with the rich semantic supervision signals provided by the teacher model during the training phase. Ultimately, this allows the lightweight student model to inherit the open-vocabulary perception capabilities of the teacher model while maintaining its own efficiency, thereby achieving real-time and accurate target detection on edge devices with limited computing resources.

[0045] In practice, firstly, a one-stage YOLO style detector with a feature pyramid is used as the student model. During the forward propagation of the model, unlabeled power operation images are input into the network. At each feature layer with a step size of s, the input image is discretized into segments of size s. The grid, for each grid cell With anchor frame The detection head outputs the raw prediction. ;

[0046] Next, these raw predictions are converted to predictions at the actual image scale using standard YOLO network parameterization formulas. In this process, the center coordinates are first calculated. Specifically, the original coordinate offset is processed using the logical sigmoid function, combined with grid indexing. With a step size s, calculate the center coordinates of the prediction box. This process can be expressed by the following formula:

[0047]

[0048]

[0049] in, This represents the logical Sigmoid function. Integer grid index, The pixel stride of the feature layer;

[0050] Next, the bounding box size is calculated. Specifically, the system is based on the anchor box prior. The width of the prediction box is calculated using the predicted size value processed by the exponential function. and height This process can be expressed by the following formula:

[0051]

[0052]

[0053] in, and These are the widths of the prediction boxes. and height ;

[0054] Next, the confidence score and category probability are calculated. Specifically, the sigmoid function is used to convert the raw output into probability values, yielding the target score. and the category probability vector for N prompt words This process can be expressed by the following formula:

[0055]

[0056]

[0057] in, The number of prompt words, This is the category probability vector for N prompt words. For target-oriented scoring;

[0058] Finally, after the above calculations, the final output set of the student model is represented as a series of prediction tuples, which contain axis-aligned bounding boxes and corresponding probability information, expressed by the formula:

[0059]

[0060] Taking the scheme in this application as an example, firstly, an anchor box matching mechanism is established between the teacher model and the student model. The system uses a shape prior strategy based on the intersection-over-union (IoU) ratio to match the bounding boxes output by the teacher model. Match the anchor frames with the preset anchor frames to divide the anchor frames into a set of positive anchor frames. and negative anchor box set For each positive sample anchor box on the feature layer with a step size of s This corresponds to a teacher box with a hint index and confidence level. Building upon this, the system further calculates supervised targets for regression, targeting, and classification. For the center coordinate regression of the bounding box, the system calculates the relative offset of the target's center point within the grid cells. Specifically, the target's horizontal coordinate... and ordinate The results are obtained by calculating the non-integer part (i.e., the decimal part) of the coordinates of the teacher's frame center divided by the step size. This process can be expressed by the formula:

[0061]

[0062]

[0063] in, Operator for decimal part and These are the target's x and y coordinates, respectively; for the bounding box size regression, the system calculates the logarithmic scaling ratio of the target size relative to the anchor box size. Specifically, the target width... and height It is determined by calculating the natural logarithm of the ratio of the teacher frame size to the scaled prior anchor frame size, which is expressed by the formula:

[0064]

[0065]

[0066] in, and These represent the target width and height, respectively. For both target-oriented and classification tasks, the system directly specifies the supervision signal for positive samples. The target label of the positive sample anchor box is set to 1, and the classification label is set to the one-hot basis vector of the corresponding class index. This process is expressed by the formula:

[0067]

[0068]

[0069] in, For category index One-hot basis vectors, where each element is 1 and the rest are 0, for negative samples. ,set up This is then excluded from the bounding box and classification loss. This assignment mechanism consistently maps the semantics of the teacher model to the grid parameterization of YOLO.

[0070] Specifically, in step S4, based on the filtered pseudo-supervised data and the student model output, the detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss are determined. It should be understood that simply relying on the pseudo-labels provided by the teacher model for standard object detection training is insufficient to fully exploit and transfer the deep semantic representation capabilities of a multimodal large model. The teacher model can not only provide the bounding box and category of the target, but also generate region-level visual feature embeddings and text embeddings corresponding to the prompt words. This information together constitutes a richer supervisory signal than traditional annotation. To enable the student model to not only learn localization and classification, but also internalize a language-based, open-ended visual concept understanding, a set of multi-angle loss functions is introduced in the technical solution of this application. Specifically, detection loss ensures the student model acquires basic detection accuracy; feature distillation loss forces the student's intermediate layer features to mimic the teacher's feature distribution, learning its powerful representational capabilities; logistic distillation loss uses the soft-class distribution calculated by the teacher model based on visual-language similarity as the optimization objective to calibrate the student's classification logic; and contrastive learning loss strengthens the consistency between visual and linguistic modalities by bringing visual regions closer to their corresponding text descriptions and distancing them from other unrelated text descriptions. Through the synergistic effect of these four losses, the system can achieve deep, high-quality knowledge transfer from a large teacher model to a lightweight student model.

[0071] In practice, firstly, sample allocation and matching are performed on the filtered pseudo-supervised data and student model output to obtain a set of positive sample anchor boxes, a set of negative sample anchor boxes, and a mapping between the positive sample anchor boxes and their matching teacher pseudo-label indices. That is, the filtered pseudo-supervised data and student model output are associated. Specifically, an IoU-based strategy is used to divide the student model's anchor boxes into a set of positive sample anchor boxes and a set of negative sample anchor boxes, and a positive sample anchor box is established. Teacher pseudo-label index that matches it The mapping relationship is used to provide a clear supervisory relationship for the calculation of subsequent losses;

[0072] Next, based on the filtered pseudo-supervised data, student model output, and the mapping between positive sample anchor boxes and their matching teacher pseudo-label indices, the detection loss for the positive sample anchor box set and the negative sample anchor box set is calculated. This includes bounding box regression loss (measuring the geometric misalignment between the predicted box and the target box) and multi-label classification loss (optimizing the consistency between the class log odds and the target vector) calculated for positive samples, and targetability loss calculated for all samples. The total detection loss is a weighted sum of the three.

[0073] In this process, firstly, the bounding box regression loss for positive samples is calculated. This step only considers samples in the positive sample anchor box set. To measure the geometric localization error between the student model's predicted boxes and the pseudo-label boxes generated by the teacher model, the intersection-union (CIoU) loss is used. The CIoU loss not only considers the overlap area of ​​the two bounding boxes but also introduces the consistency of the center point distance and aspect ratio, thus achieving more accurate and faster-converging bounding box regression. This process is expressed by the formula:

[0074]

[0075]

[0076]

[0077] in, Indicates the center of the box. The Euclidean distance between the center points is... To surround simultaneously and The length of the diagonal of the minimum bounding rectangle. Used to penalize aspect ratio mismatch. Used to adjust the weight of aspect ratio terms. The standard intersection-union ratio of the two frames;

[0078] Next, the multi-label classification loss for positive samples is calculated, again only for the set of anchor boxes for positive samples. To enable the student model to learn the semantics of open-ended words, this loss function optimizes the consistency between the predicted log odds and the target vector along the cue word (category) dimension. Specifically, it uses the form of binary cross-entropy (BCE) loss, summing and averaging over all N cue words. This allows the model to handle multi-label classification tasks, where one target may correspond to multiple related cue descriptions simultaneously. This process is expressed by the formula:

[0079]

[0080] in, The number of positive sample anchors. express right Predicting probability The corresponding target;

[0081] Next, the target-oriented loss for all samples is calculated. This step is performed on all anchor boxes (including both positive and negative sample sets). The target-oriented branch determines whether a candidate box contains any target, thus decoupling foreground localization from specific class assignment. Given that the number of negative samples far exceeds that of positive samples in dense detection, Focal Loss (Focus Binary Cross-Entropy) is employed to mitigate this extreme imbalance. A modulating factor is introduced... and This loss function reduces the weight of simple negative samples, allowing the model to focus on samples that are difficult to classify, thereby achieving better background discrimination ability. This process can be expressed by the formula:

[0082]

[0083] in, Candidate boxes Predictive target score, This indicates its binary target label. , Represents the parameters used for targeted loss;

[0084] Subsequently, the three losses are weighted and summed to obtain the total detection loss. Specifically, the bounding box regression loss and classification loss are accumulated only over the positive sample set, while the target detection loss is accumulated over the entire set of both positive and negative samples. This is achieved by introducing non-negative balanced weights. , and The process can be tailored to specific deployment needs by balancing positioning accuracy, classification accuracy, and target presence determination. This process can be expressed by the following formula:

[0085]

[0086] in, This represents the non-negative balanced weights applied to the three components of the detection objective function;

[0087] Furthermore, based on the filtered pseudo-supervised data, student model output, and the mapping between positive sample anchor boxes and their matching teacher pseudo-label indices, the feature distillation loss and logistic distillation loss of the positive sample anchor box set are calculated. It should be understood that simple bounding box regression and hard label classification cannot fully capture the rich open-vocabulary knowledge of a multimodal large model (LMM). Feature distillation, by aligning intermediate representations, can reduce the distributional bias between students and teachers, thus stabilizing the learning process in power field environments with drastic changes in lighting and weather. Logistic distillation loss (i.e., cue-aware logistic distillation) is responsible for conveying the teacher model's soft target distribution regarding cue words, which is crucial for improving the model's calibration ability in open-vocabulary settings and its recall rate for rare tools or labels.

[0088] In this process, firstly, for the calculation of feature distillation loss, the system uses a linear projector to extract the features from the student model. The model maps the data to the feature space of the teacher model to address the inconsistency in dimensionality between the two. Subsequently, for each positive sample anchor box, the system searches for its matching teacher region index. And obtain the corresponding teacher region embedding representation. The feature distillation loss is the mean square error between the projected student features and the teacher region embedding. This process is expressed by the formula:

[0089]

[0090] in, Indicates and Index of the matching teacher region, In order to be in Student characteristics extracted from the site Embed a representation for the corresponding teacher region;

[0091] Secondly, for calculating the logistic distillation loss, the system first needs to construct the soft target distribution of the teacher model on the prompt words. Since the teacher model outputs region embeddings and text embeddings, the system can generate the teacher's probability distribution by calculating the cosine similarity between the two and smoothing it using a temperature parameter. This distribution reflects the semantic similarity between the current region and each prompt word. This process is expressed by the formula:

[0092]

[0093] After obtaining the teacher distribution, the system also processes the class log odds output by the student model through a temperature parameter to obtain the soft prediction distribution of students. This process is expressed by the formula:

[0094]

[0095] in, Represents cosine similarity. These are the temperature parameters used to control teacher objectives and student predicted softness, respectively. It is a region The distribution of teacher prompts. During training, classification is still optimized as a multi-label task;

[0096] Subsequently, based on the filtered pseudo-supervised data, student model output, and the mapping between positive sample anchor boxes and their matching teacher pseudo-label indices, the contrastive learning loss of the positive sample anchor box set is calculated. It should be understood that simple category classification loss can easily lead to model confusion between objects with similar visual features but completely different semantics. In the technical solution of this application, the projected student region is associated with the correct cue embedding through visual-linguistic contrast alignment to strengthen the consistency between vision and language, thereby reducing category confusion. In this way, the student model not only learns the category label of the target, but more importantly, it learns the deep semantic matching relationship between the target region and the text description. This is crucial for improving the model's generalization ability to unseen targets in open-vocabulary scenarios.

[0097] In this process, firstly, since the feature dimensions extracted by the student model are usually inconsistent with the text embedding dimensions of the teacher model, feature projection is performed first. Specifically, a linear projector is used to map the student features to the teacher space.

[0098] Next, the system calculates the cosine similarity between the projected student features and the text embeddings. For each positive anchor box, the system not only calculates the similarity between its projected features and the correct cue embedding it matches (positive sample pair), but also calculates the similarity between that feature and all other cue embeddings in the cue set (negative sample pair).

[0099] Furthermore, the system calculates the contrastive learning loss based on the InfoNCE objective function to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. Specifically, the system iterates through all positive sample anchor boxes, calculates the logarithm of the ratio of the exponent value of the similarity of positive sample pairs to the sum of the exponent values ​​of the similarity of all possible pairs, and takes the negative average. This process is expressed by the formula:

[0100]

[0101] in, Indicates the comparison temperature. This indicates that student characteristics are projected onto the teacher's space. Indicates the index of the matching suggestion Embedded.

[0102] Specifically, in step S5, the model parameters of the lightweight YOLO student model are optimized and updated based on detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss to obtain the updated YOLO student model. It should be understood that traditional model parameter optimization mechanisms use static, fixed hyperparameter weights to linearly combine the standard detection loss and multiple knowledge distillation losses when calculating the total loss. The fundamental technical flaw of this method lies in its context-independent nature; that is, it completely ignores the special relationship between semantic hierarchy and safety criticality among different detection targets in the specific scenario of power operation safety monitoring. At power operation sites, not all targets have the same safety level. For example, accurately identifying a high-risk behavior such as a person not wearing a safety helmet has far greater safety significance than identifying a traffic cone on the roadside. Similarly, when workers and critical elements such as high-voltage transmission lines appear together in the same scenario, the detection and knowledge distillation of related personal protective equipment such as insulated gloves and safety belts should be given higher learning priority. The original fixed-weighting scheme cannot dynamically adjust the contribution of each knowledge distillation component based on the real-time semantic importance of the image content within the current training batch. This may result in the model's learning resources being ineffectively distributed across non-critical background targets, while insufficient investment is made in learning core knowledge related to life safety. This static weighting mechanism inherently limits the model's learning efficiency and reduces its reliability in complex, high-risk scenarios.

[0103] To address the aforementioned technical shortcomings, a dynamic loss weighting mechanism based on real-time semantic importance perception is proposed. This mechanism dynamically generates the weights of the knowledge distillation loss based on the inherent security criticality of the target identified by the teacher model in the current training batch of data, thereby guiding the student model to prioritize and deeply learn the knowledge most relevant to the security rules.

[0104] In practice, firstly, based on the filtered pseudo-supervised data, a batch-level semantic importance score is generated. This step aims to quantify the overall safety criticality of the image content contained in the current training batch. Specifically, the system receives the pseudo-supervised data for the current batch, which contains the category indices of all targets identified by the teacher model. Simultaneously, the system also loads a predefined semantic priority vector representing the basic safety priority of various targets, and a key co-occurrence matrix defining the special safety significance of key target pairs (e.g., people and exposed wires) when they co-occur. By aggregating the predefined priority scores of all targets appearing in the batch and introducing a logarithmic enhancement term to amplify the risk of key target pairs co-occurring, a scalar value quantifying the overall safety criticality of the current batch, i.e., the batch-level semantic importance score, is finally calculated. This process is expressed by the formula:

[0105]

[0106] in, This is the batch-level semantic importance score, which is the semantic importance score of the current training batch; The set of categories within a batch is a collection of all unique target category indices appearing in the current batch. The number of categories within the batch; the number of elements in this set. Semantic priority represents the basic security importance score for category j; The key co-occurrence matrix is ​​a predefined binary matrix, and... The first in the key co-occurrence matrix For elements at a given position, if the simultaneous appearance of categories j and k has special security significance, then... It is 1 if it is true, otherwise it is 0; The co-occurrence enhancement coefficient is a hyperparameter used to control the degree of influence of co-occurrence relationships on the overall importance score. The batch-level semantic importance score calculated in this step not only reflects the average importance of high-priority targets within the batch, but also pays special attention to the co-occurrence relationship of key target pairs through the logarithmic enhancement term. This makes the loss weights more sensitive to training samples containing highly dangerous combinations, thereby achieving the goal and effect of accurate and dynamic profiling of the risk level of batch data.

[0107] Next, based on the batch-level semantic importance score, the basic distillation weights are dynamically adjusted to obtain dynamic distillation weights. That is, the abstract importance score generated in the previous step is transformed into specific weight values ​​that can directly affect the loss function. During execution, the system uses the batch-level semantic importance score calculated in the previous step to dynamically adjust the basic weights of each knowledge distillation loss through a non-linear modulation function. This process acts like an intelligent regulator, aiming to allocate the model's learning attention according to the importance of the data. Specifically, a non-linear modulator composed of a hyperbolic tangent function (tanh) amplifies the distillation weights in response to batch importance exceeding a baseline threshold, and moderately reduces them otherwise. For the first... The calculation process for the dynamic weight of the distillation loss is expressed by the following formula:

[0108]

[0109] in, The dynamic distillation weight is the k-th distillation loss (including characteristic distillation). Logic distillation and comparative learning Dynamic weights; The basic distillation weight is the basic weight set by the user. This is the modulation amplitude factor, used to control the maximum proportion by which the dynamic weights deviate from the base weights; This is a modulation sensitivity factor used to control the sensitivity of weight changes to changes in importance scores; It is the hyperbolic tangent function, which is a smooth activation function; The importance benchmark threshold is used to determine whether a batch's importance is above or below the average level. It should be understood that when a batch's semantic importance score exceeds the benchmark threshold, the tanh function will output a positive value, making the final dynamic weight higher than the base value, thus increasing the knowledge distillation intensity for this critical batch. Conversely, if the batch content is relatively conventional, the weight will be appropriately reduced. The purpose and effect of this step is to achieve adaptive adjustment of the learning intensity for different batches of data, ensuring that training resources are prioritized for learning the most critical security knowledge.

[0110] Furthermore, based on the dynamic distillation weights, the detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss are fused to obtain the total loss. Specifically, the dynamic distillation weights generated in the previous step are linearly combined with the standard detection loss and each knowledge distillation loss to form the final total loss function. This total loss function is then used to drive the standard gradient backpropagation algorithm to update all trainable parameters of the student model, a process expressed by the formula:

[0111]

[0112] in, The total loss is the final objective function used for model optimization. To detect losses, For characteristic distillation losses, For logical distillation loss and To compare the learning loss, we have the various loss components calculated in the previous steps. Among them, Aligning intermediate representations with linear projectors reduces student-teacher distribution shift and stabilizes learning under varying lighting and weather conditions. By conveying teacher-provided semantic prompts, we can enhance open-ended vocabulary calibration and recall of rare tools / symbols. By associating visual regions with their textual meanings, visual-linguistic consistency is strengthened, thereby reducing category confusion (e.g., yellow objects and safety helmets). By using dynamically computed weights instead of fixed weights, the total loss function becomes highly sensitive to the semantic content of the training data. For batches containing high-risk scenarios, the proportion of knowledge distillation in the total loss dynamically increases, forcing the student model to invest more effort in learning the teacher model's deeper knowledge about these specific hazardous scenarios. This achieves an adaptive, safety-oriented knowledge transfer process, ultimately solidifying this learning preference into the updated student model network parameters.

[0113] In summary, this mechanism can intelligently identify batches of samples with higher safety criticality in the training data and dynamically amplify the effect of knowledge distillation on these batches. Specifically, the model's learning resources are intelligently and preferentially allocated to high-priority knowledge that is crucial to power operation safety. This not only overcomes the technical shortcomings of the original fixed-weight mechanism, such as low learning efficiency and insufficient learning of key scenarios, but also enables the final student model to exhibit stronger reliability and higher accuracy when facing complex power operation environments, especially in identifying and warning of critical safety violations. Ultimately, a lightweight target detection model that can be deployed at the edge is obtained. This model not only inherits the open-vocabulary understanding ability of the teacher model, but also internalizes a deep and prioritized understanding of safety rules in the power operation field, thereby more effectively serving the intelligent and high-reliability safety monitoring tasks of power operations and significantly improving the safety assurance level of on-site operations.

[0114] Subsequently, based on the total loss, the model parameters of the lightweight YOLO student model are optimized and updated using backpropagation via gradient descent to obtain the updated YOLO student model. It should be understood that the lightweight YOLO student model, in its initial state, does not possess the ability to accurately detect targets in power operations; its parameters are randomly initialized or pre-trained only on general datasets, making it unsuitable for specific and complex power operation scenarios. To enable it to perform safety monitoring tasks at power operation sites, an automated mechanism must be used to adjust its internal parameters. The total loss function quantitatively characterizes the gap between the student model's current prediction and the ideal target (the teacher model's output). Based on this gap, the gradient descent backpropagation algorithm provides an efficient and computationally achievable method to determine the direction and magnitude of adjustment for each model parameter to minimize this gap. By iteratively executing this update step, the student model's parameters are continuously optimized, and its behavior gradually approaches that of the powerful teacher model, ultimately achieving open-vocabulary detection capabilities and high accuracy close to the teacher model while maintaining lightweight and high speed.

[0115] In this process, the total loss is calculated first. This is the foundation of parameter optimization. The system linearly combines the previously calculated losses according to the set weights to form the final overall objective function. As seen from the above steps, this step uses dynamically calculated weights instead of fixed weights. These dynamic weights are modulated by a batch-level semantic importance score, which integrates the basic security priority of targets appearing in the current batch and the co-occurrence relationship of key targets. The higher the score, the stronger the semantic importance or security criticality of the data in that batch. The modulation process uses a hyperbolic tangent function, which allows the weights to increase or decrease smoothly and non-linearly according to the importance score relative to a certain benchmark threshold, thereby achieving adaptive adjustment of the learning intensity for batches of data of different importance.

[0116] Next, after the total loss is calculated, the system initiates the backpropagation algorithm. This algorithm, based on the chain rule, calculates the gradient of the total loss with respect to each trainable parameter of the student model (including the weights and biases of convolutional layers, the parameters of batch normalization layers, and the parameters of the projection matrix) layer by layer, starting from the output layer of the network. The gradient is a vector, where each component represents the rate of change of the total loss when the corresponding parameter changes slightly. The gradient direction indicates the direction in which the loss function grows fastest; therefore, to minimize the loss, the parameters should be adjusted in the opposite direction of the gradient.

[0117] Then, the optimizer updates the parameters. During this process, the system uses a pre-defined optimization algorithm (such as AdamW) to update the model parameters based on the calculated gradients. Taking the AdamW optimizer as an example, its update rule is more complex than simple gradient descent. It combines the ideas of momentum (first-order moment estimation) and RMSProp (second-order moment estimation) and incorporates weight decay. Specifically, the optimizer maintains the exponential moving averages of the first and second moments of the gradient, corrects for biases, and then uses the corrected moment estimates, learning rate, and weight decay coefficients to calculate the final parameter update. This process essentially adjusts the parameters intelligently based on gradient information, enabling the model to converge steadily on the training set.

[0118] Furthermore, after the above steps (forward propagation to calculate the loss, backpropagation to calculate the gradient, and optimizer to update parameters) are completed on one batch of training data, they are repeated on the next batch until all training data has been traversed (one epoch). This process is repeated for multiple epochs until the model performance stabilizes (converges) on the validation set, or the preset maximum number of training epochs is reached. The resulting model is the updated YOLO student model, which has internalized the knowledge distilled from the teacher model.

[0119] Figure 3 This is a flowchart of the detection phase of the power operation target detection method based on multimodal large model knowledge distillation according to embodiments of this application. Figure 3 As shown, the power operation target detection method based on multimodal large model knowledge distillation according to an embodiment of this application further includes a detection stage, the specific steps of which include: S11, acquiring a power operation image to be detected; S12, inputting the power operation image to be detected into the updated YOLO Student model to obtain a power operation target detection result.

[0120] Specifically, in S11, an image of the power operation to be detected is acquired. It should be understood that the ultimate purpose of power operation target detection is to serve on-site safety monitoring, which requires the model to be able to analyze images captured in real-world scenarios. Therefore, in the technical solution of this application, acquiring an image of the power operation to be detected transforms the actual, analyzable operation scenario into a digital format that the student model can process. This step ensures that the detection system can interact with the real world and provide detection results for specific, unknown on-site conditions, thereby realizing its core value of early warning and monitoring.

[0121] In practice, the first step is to determine the image data source. The power operation images to be detected typically originate from specific work sites requiring safety monitoring. These sites include, but are not limited to, complex scenarios such as substations, overhead lines, and switchyards. These environments are characterized by large depth of field, domain-specific tools and equipment, and shooting conditions encompassing varying lighting and weather conditions to ensure the model's robustness under diverse real-world conditions. Image data may come from high-definition surveillance cameras pre-deployed in the work area, or from photos or video streams captured by mobile devices (such as smartphones or inspection recorders) held by inspection personnel. The second step is to acquire the power operation images to be detected. Specifically, image acquisition equipment (such as CCD or CMOS sensors) converts the optical information in the scene into continuous electronic signals, which are then sampled and quantized into discrete digital signals by an analog-to-digital converter, ultimately forming a digital image that a computer can process. This digital image is typically represented in RGB color mode, where each pixel consists of values ​​from the red, green, and blue color channels. In practical applications, this process is automatic and continuous to meet the needs of real-time monitoring, forming the image sequence or stream to be detected.

[0122] Specifically, in S12, the image of the power operation to be detected is input into the updated YOLO student model to obtain the power operation target detection result. The core requirement of power operation safety monitoring is to perform real-time analysis of on-site video streams or images on edge computing devices, promptly identify targets such as workers, safety equipment, and key equipment, and determine their status (e.g., whether they are wearing safety helmets). In the technical solution of this application, the specific image to be detected is further input into the prepared model, triggering the model's forward inference process. This transforms the model's potential capabilities into specific detection results that can guide safety decisions, verifying the distillation effect and ultimately providing an automated visual perception tool that can be deployed in resource-constrained environments and understand complex scenarios, significantly improving the intelligence level and response speed of power operation safety monitoring.

[0123] In practice, firstly, before the detection phase begins, the system needs to load the parameters of the YOLO Student model, optimized and updated during the training phase, from the storage medium into memory and initialize the model's computational graph. Next, the image to be detected is preprocessed. This mainly includes adjusting the image size (e.g., scaling) to the model's preset input size and normalizing pixel values ​​to maintain consistency with the data distribution during training. Then, the preprocessed image tensor is input into the loaded and updated YOLO Student model. The model performs a series of convolution, pooling, and upsampling operations according to its architecture (e.g., Backbone, Neck, Head), i.e., forward propagation. During this process, image features are extracted and fused layer by layer. Finally, in the detection head, the model outputs the raw prediction tensor. For each grid cell and each anchor box on the feature map, the output typically includes the original offset of the bounding box, the target score, and the class logical value for each cue word. Finally, the raw prediction tensor output by the model is decoded to convert it into an intuitive detection result, i.e., the power operation target detection result.

[0124] In summary, the power operation target detection method based on multimodal large-scale model knowledge distillation according to the embodiments of this application is explained. It utilizes a visual-linguistic multimodal large-scale model as the teacher model, and through expanded cue words, generates high-quality pseudo-labels and region-text pairs as supervision signals for unlabeled power operation images. Based on this, by jointly optimizing the detection loss, feature distillation loss, logistic distillation loss, and multimodal contrastive learning loss, it guides the lightweight YOLO student model not only to learn localization and classification knowledge, but also to deeply align with the teacher model's open-vocabulary understanding ability at the feature space and semantic levels, thereby effectively bridging the semantic gap between closed-category detection and open-world perception. In this way, the detection accuracy and generalization ability of the student model for common, rare, and even unseen targets in power operation scenarios are significantly improved.

[0125] In the practical application of this application, firstly, two benchmark datasets were selected during the data preparation and model configuration phase. The first is the Safety Helmet Detection dataset, designed for compliance detection in industrial and power operation scenarios. This dataset presents images of workers in complex outdoor and semi-structured environments, labeled with categories such as Helmet, Head, and Person, making it particularly suitable for determining compliance through "head-helmet" pairing. The second is the PASCAL VOC 2008 dataset, serving as a complementary general detection benchmark. It covers 20 everyday categories, aiming to enrich background diversity and help student models suppress common false alarms in power operation scenarios, such as objects similar in color and shape to safety helmets, thereby improving the model's domain generalization ability.

[0126] Next, the Grounding-DINO model, which has open-vocabulary capabilities, was used as the teacher model. A set of prompt words was constructed and expanded based on electrical safety operation specifications, including synonyms such as "safety helmet," "personnel," and "insulating gloves," as well as domain aliases. A confidence threshold was set during the teacher model's runtime. Pseudo-label filtering was performed, and non-maximum suppression (NMS) was applied independently to each prompt word with an IoU threshold of 0.6. Region and text embeddings were extracted and projected onto a 256-dimensional model for distillation. The student model used a lightweight YOLOv8-s configuration (width factor 0.50, depth factor 0.33), with a backbone network and a Feature Pyramid Network (PAN-FPN) as the neck. The detection head was adapted for multi-scale prediction (stride sizes of 4, 8, 16, and 32, with a high-resolution P2 layer enabled for better detection of small targets such as safety helmets). The model input size was uniformly adjusted to 640x640 pixels.

[0127] Next, the model training and optimization phase begins. Training uses the AdamW optimizer with an initial learning rate of 1e-4 and weight decay of 5e-2. A cosine annealing scheduler is used to dynamically adjust the learning rate. The training process includes a 5-epoch warm-up phase. The training batch size is set to 16, and mixed precision (FP16) is used to accelerate training and save GPU memory. A total of 200 epochs are trained. The loss function is configured as follows: In the detection loss, bounding box regression uses CIoU loss, with weights... The target loss uses Focal Loss (parameters). , ), weight The classification loss uses binary cross-entropy (BCE), with weights... Knowledge distillation section, characteristic distillation loss. weight Logical distillation loss weight , Comparative learning loss weight The temperature parameters during the distillation process were set as follows: Teacher similarity Softmax temperature Student knowledge distillation Softmax temperature Comparison of learning temperatures During training, the system uses the teacher model to generate pseudo-supervised data for unlabeled images in parallel, calculates the output of the student model, and then iteratively optimizes the parameters of the student model through the aforementioned combined loss function and optimization strategy.

[0128] Subsequently, in the performance verification and comparative analysis phase, the trained model underwent a systematic evaluation. For example... Figure 5As shown, on the Safety Helmet Detection dataset, the mean average accuracy (mAP) of our method (LMM-KD) is significantly higher than the baselines (such as FR-CNN, Meta R-CNN, DeFRCN, GLIP, etc.) in 1-shot, 3-shot, 5-shot, and 10-shot settings, especially reaching 73.23% in the 10-shot setting, which fully demonstrates its superiority in scenarios where labeled data is scarce. Figure 6 As shown, experiments on the PASCAL VOC 2008 dataset also observed a consistent trend. Our method achieved leading performance across all sample settings, with a 10-shot mAP of 66.93%, validating its strong generalization ability. Training data results are shown in Table 1 below. Ablation experiments further confirmed the importance of each component: removing the LMM teacher or YOLO student model led to a significant performance drop, and removing knowledge distillation (KD) also caused a significant performance loss. This indicates that the open-vocabulary semantics provided by LMM, the dense detection capabilities of YOLO, and the transfer effect of KD collectively constitute the effectiveness of our approach. Furthermore, as... Figure 7 and Figure 8 As shown, the parameter analysis experiment also explored the contrastive loss weights. The study found that visual-language alignment can stably improve model performance in the range of 0.3 to 0.5, indicating that appropriate visual-language alignment is crucial for knowledge transfer.

[0129]

[0130] Table 1

[0131] Furthermore, in the model deployment and visualization analysis phase, the trained lightweight YOLO student model is deployed to edge computing devices. In actual substation or overhead line monitoring scenarios, this model can efficiently process real-time video streams and output bounding boxes, categories, and confidence scores in real time. Visual analysis of the results (such as the confusion matrix) reveals that the model has high recognition accuracy for key targets such as "safety helmets." Although there is some "whole-component" confusion in the overall recall of "personnel," it effectively meets the dual requirements of accuracy and real-time performance for on-site safety monitoring. This ultimately demonstrates that this embodiment successfully distills the knowledge of a large-scale multimodal teacher model into a deployable lightweight model, providing a practical technical path for power operation safety monitoring.

[0132] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting power operation targets using multimodal large-model knowledge distillation, characterized in that, include: Training phase and testing phase; The training phase includes the following steps: Obtain the set of unlabeled power operation images and the set of extended prompt words; The unlabeled set of power operation images and the set of extended prompt words are input as a multimodal large model of the teacher model to obtain filtered pseudo-supervised data; Input unlabeled power operation images into a lightweight YOLO student model to obtain the student model output; Based on the filtered pseudo-supervised data and the student model output, the detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss are determined. Based on detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss, the model parameters of the lightweight YOLO student model are optimized and updated to obtain the updated YOLO student model. The detection phase includes the following steps: Acquire images of the power operation to be detected; The image of the power operation to be detected is input into the updated YOLO student model to obtain the target detection results of the power operation.

2. The power operation target detection method based on multimodal large model knowledge distillation according to claim 1, characterized in that, The unlabeled power operation image set and the extended cue word set are input as a multimodal large model of the teacher model to obtain filtered pseudo-supervised data, including: Multimodal encoding, cross-modal fusion, and decoding are performed on each unlabeled power operation image and extended prompt word set in the unlabeled power operation image set to obtain the original detection result set; High-confidence pseudo-labels are filtered from the original detection result set to obtain a high-confidence detection result set; Redundant pseudo-labels are suppressed based on prompt words in the high-confidence detection result set to obtain filtered pseudo-supervised data.

3. The power operation target detection method based on multimodal large model knowledge distillation according to claim 2, characterized in that, The raw detection results in the raw detection result set include bounding boxes, category indexes, text embeddings, region-level visual embeddings, and confidence scores.

4. The power operation target detection method based on multimodal large model knowledge distillation according to claim 2, characterized in that, Redundant pseudo-labels based on prompt words are suppressed on the high-confidence detection result set to obtain filtered pseudo-supervised data, including: The high-confidence detection result set is grouped by category to obtain a category-grouped detection result set; Based on the NMS threshold, independent nonmaximum suppression is applied to the detection result set grouped by category to obtain the final detection result set; The final detection result set is formatted to obtain filtered pseudo-supervisory data.

5. The power operation target detection method based on multimodal large model knowledge distillation according to claim 1, characterized in that, Based on the filtered pseudo-supervised data and the student model output, the detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss are determined, including: Sample allocation and matching are performed on the filtered pseudo-supervised data and student model output to obtain a set of positive sample anchor boxes, a set of negative sample anchor boxes, and a mapping of positive sample anchor boxes to the teacher pseudo-label indexes that they match; Based on the filtered pseudo-supervised data, student model output, and the mapping between positive sample anchor boxes and their matching teacher pseudo-label indices, the detection loss for the positive sample anchor box set and the negative sample anchor box set is calculated. Based on the mapping of filtered pseudo-supervised data, student model output, and positive sample anchor boxes to their matching teacher pseudo-label indices, the feature distillation loss and logistic distillation loss of the positive sample anchor box set are calculated. Based on the filtered pseudo-supervised data, the student model output, and the mapping between the positive sample anchor boxes and their matching teacher pseudo-label indices, the contrastive learning loss of the positive sample anchor box set is calculated.

6. The power operation target detection method based on multimodal large model knowledge distillation according to claim 1, characterized in that, Based on detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss, the model parameters of the lightweight YOLO Student model are optimized and updated to obtain the updated YOLO Student model, including: Based on the filtered pseudo-supervised data, batch-level semantic importance scores are generated; Based on batch-level semantic importance scores, the basic distillation weights are dynamically adjusted to obtain dynamic distillation weights; Based on dynamic distillation weights, the detection loss, feature distillation loss, logistic distillation loss, and contrastive learning loss are fused to obtain the total loss; The model parameters of the lightweight YOLO Student model are optimized and updated based on the total loss and through backpropagation of gradient descent to obtain the updated YOLO Student model.

7. The power operation target detection method based on multimodal large model knowledge distillation according to claim 6, characterized in that, Based on the filtered pseudo-supervised data, batch-level semantic importance scores are generated, including: generating batch-level semantic importance scores using the following formula, where the formula is: in, Batch-level semantic importance score; The set of categories within a batch is a collection of all unique target category indices appearing in the current batch. This refers to the number of categories within the batch. Semantic priority represents the basic security importance score for category j; It is the key co-occurrence matrix, and The first in the key co-occurrence matrix The element at the specified position; This is the co-occurrence enhancement coefficient.

Images