Electric power image-text large model device and hierarchical deployment method thereof

Abstract

The invention belongs to the technical field of electric power image-text large models, and discloses an electric power image-text large model device and a hierarchical deployment method thereof. The electric power image-text large model device comprises an edge model used for being deployed on an edge device and a cloud model used for being deployed on a cloud. The edge model comprises a post-distillation visual coding layer obtained after knowledge distillation is carried out on a visual coding layer of the original electric power image-text large model, and a head layer added to the post-distillation visual coding layer; the cloud model comprises a text coding layer and a modal interaction layer in the original electric power image-text large model. According to the technical scheme, the potential of a cloud large model can be further exerted, and the recognition capability of complex scenes is improved; besides, the calculation complexity of the model can be reduced, and the core performance can be ensured, so that the model can be efficiently deployed and operated on edge equipment with limited calculation capability, and a series of technical problems caused by frequent data transmission can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power graphic large model, and in particular relates to an electric power graphic large model device and a hierarchical deployment method thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal large models have demonstrated powerful performance in various fields, but such complex models usually require huge computing resources and are difficult to deploy directly on edge devices with limited computing power. In order to solve the above deployment problems, researchers have proposed knowledge distillation technology, which can transfer the knowledge of complex multimodal large models (i.e., teacher models) to smaller and more efficient models (i.e., student models). While maintaining high performance, it can significantly reduce the demand for computing resources and reasoning time, and can provide important support for the actual implementation of large models.

[0003] In actual application scenarios, especially in key infrastructure fields such as power systems, cloud-edge collaborative deployment strategies are becoming increasingly mainstream; explanatory, the above-mentioned cloud-edge collaborative deployment strategy deploys large multimodal models with more parameters, higher accuracy, richer functions, and greater resource consumption on the cloud platform, and at the same time deploys small models that have been compressed on the edge side of the station. This collaborative operation of large and small models can alleviate the contradiction between computing resource utilization efficiency and system performance to a certain extent, but there are still some problems that need to be solved.

[0004] At present, in the current digital intelligent applications in the power field, there is significant room for optimization in the deployment mode of large power graphic and text models; among them, computationally intensive and large-parameter deep learning models are mainly deployed on cloud servers, while the side models are responsible for calling terminal devices for image acquisition, and transmitting the collected image samples and related text information back to the cloud for processing and analysis by the large graphic and text models on the cloud. However, the above-mentioned centralized processing mode has many problems: edge computing resources are not fully utilized. Edge devices usually have certain computing capabilities, but in the current mode, these potential computing resources have not been effectively utilized and are only used as data transfer stations, resulting in resource waste; network bandwidth consumption is huge, and the real-time transmission of a large number of high-definition images and related data poses severe challenges to network infrastructure, increases operating costs, and may cause network congestion; model reasoning delay increases, and data is transmitted from the terminal device to the cloud, and then the results are returned after model processing. This process prolongs the overall response time. In some power system scenarios that require real-time decision-making, this delay will affect the timeliness and effectiveness of operations; data security risks increase, and a large amount of sensitive power system data is transmitted in the network, increasing the risk of data leakage or tampering, posing a potential threat to the safe operation of the power grid; data reliability is reduced, the process relies on cloud processing and network transmission, and network fluctuations increase, which may lead to interruption of key functions. In summary, the existing cloud-edge deployment architecture of multimodal large models still has many limitations, mainly including: the potential of large cloud models is not fully utilized, and unnecessary compromises are made between real-time response and reasoning accuracy; transmitting site-side samples back to the cloud for inference not only increases system latency, but also occupies a large amount of bandwidth, leading to network congestion; frequent data transmission may also cause security and privacy issues. Summary of the invention

[0005] The purpose of the present invention is to provide a large-scale model device for electric power graphics and its hierarchical deployment method to solve one or more of the above-mentioned technical problems. The technical solution disclosed in the present invention can further unleash the potential of large cloud models and improve the recognition ability of complex scenes; in addition, it can reduce the computational complexity of the model and ensure the core performance, so that the model can be efficiently deployed and run on edge devices with limited computing power, solving a series of problems caused by frequent data transmission.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a large-scale model device for electric power graphics, including: an edge model for deployment on an edge device and a cloud model for deployment on a cloud; wherein: The edge model includes a distilled visual coding layer obtained by performing knowledge distillation on the visual coding layer of the original power graphic model, and a head layer added to the distilled visual coding layer; wherein the head layer is used to detect a predetermined task based on the features extracted by the distilled visual coding layer and to feed back the detection result; The cloud-based model includes a text encoding layer and a modal interaction layer in the original power graphic model; wherein the text encoding layer is used to input structured prompts, and the modal interaction layer is used to input the output results of the distilled visual encoding layer and the output results of the text encoding layer and interactively fuse them; the structured prompts are obtained by integrating the target bounding box detected by the head layer, the category information detected and identified, and the relevant text description.

[0007] A further improvement of the present invention is that The head layer performs performance evaluation and fine-tuning updates by calculating the intersection-over-union ratio between the detection box generated by the head layer and the detection box output by the cloud model.

[0008] A further improvement of the present invention is that In the step of obtaining the distilled visual coding layer after performing knowledge distillation on the visual coding layer of the original power graphic model, when performing knowledge distillation, the KL divergence with a temperature hyperparameter is used to calculate the loss to achieve knowledge transfer between the teacher network and the student network; Among them, the teacher network is the visual encoding layer of the original power graphic model, and the student network is the visual encoding layer after distillation. When calculating the distillation loss, the Z-score method is used to standardize Logit before applying Softmax, and the value of the temperature hyperparameter is adaptively and dynamically adjusted according to the difficulty of the sample knowledge.

[0009] A further improvement of the present invention is that In the step of using the Z-score method to standardize Logit before applying Softmax, for An array of , the standardized formula is expressed as: ; ; In the formula, is the mean of the array; Indicates the length of the array; Indicates the first Number; Indicates the value of Z-score; represents the temperature hyperparameter.

[0010] A further improvement of the present invention is that In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the calculation of the difficulty takes into account two factors: the output distribution characteristics of the teacher network and the similarity between the outputs of the teacher network and the student network.

[0011] A further improvement of the present invention is that In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the calculation expression of the difficulty is: ; In the formula, Indicates the calculated value of sample difficulty; represents the balance factor, ; represents the difficulty score based on the teacher network distribution; Represents the score based on the similarity of the teacher network and the student network Logit output; ; In the formula, and They are the maximum and second largest Logit values ​​output by the teacher network, respectively; ; In the formula, represents the Logit value of the teacher network, Represents the Logit value of the student network.

[0012] A further improvement of the present invention is that Calculate the score based on the similarity of the teacher network and the student network Logit output When , Similarity is calculated using cosine similarity, using dot product or KL divergence.

[0013] A further improvement of the present invention is that In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the temperature hyperparameter The expression is: ; in, and are the preset minimum and maximum values ​​of the temperature hyperparameter, respectively.

[0014] A further improvement of the present invention is that In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the temperature hyperparameter The expression is: ; In the formula, and They are the minimum and maximum values ​​of the preset temperature hyperparameters respectively; epoch is the round of the current iteration, and max_epoch is the total round of model training iterations.

[0015] A second aspect of the present invention provides a hierarchical deployment method for a large power graphic model device, comprising the following steps: Perform knowledge distillation on the visual coding layer of the original power graphic model to be deployed to obtain a distilled visual coding layer; add a header layer to the distilled visual coding layer to obtain a lightweight model of the visual coding module, and use the lightweight model of the visual coding module as an edge model and deploy it on the edge device; wherein the header layer is used to detect the predetermined task based on the features extracted by the distilled visual coding layer and feedback the detection results; The text encoding layer and modal interaction layer in the original power graphic model to be deployed are used as cloud models and deployed on the cloud; the text encoding layer is used to input structured prompts, and the modal interaction layer is used to obtain the output results of the distilled visual encoding layer and the output results of the text encoding layer and interactively fuse them; the structured prompts are obtained by integrating the target bounding box detected by the head layer, the category information detected and identified, and the relevant text description.

[0016] Compared with the prior art, the present invention has the following beneficial effects: In view of the many key challenges in the current deployment of large-scale power graph models, including insufficient utilization of edge computing resources, excessive consumption of network bandwidth, increased model inference delay, increased data security risks, and reduced system reliability, the present invention proposes a new cloud-edge collaborative deployment scheme for large-scale power graph models. It uses knowledge distillation technology to lightweight compress the visual encoding module of the existing large-scale power graph model, reducing the computational complexity of the model and ensuring the core performance, so that the edge model can be efficiently deployed and run on edge devices with limited computing power; by adding a special header layer (for example, the model can be fine-tuned on key tasks such as personnel violation behavior recognition and equipment defect detection), the high performance of the edge model in specific application scenarios is ensured; the image features, bounding boxes, recognition category information and user-entered text extracted by the edge model are uploaded to the cloud model for further accurate recognition. In summary, the present invention not only balances real-time and analysis depth, but also improves data security and privacy protection by reducing raw data transmission, and can provide a flexible framework for future functional expansion and performance optimization. With the improvement of edge device computing power and the advancement of algorithms, more complex tasks can be gradually migrated to the edge side. The present invention provides a scalable, safe and efficient technical solution for the intelligent transformation of the power industry and provides strong support for the intelligent monitoring and early warning of the power system.

[0017] In the present invention, the performance of the model after deployment is evaluated by IoU. If the IoU is low, false positive samples are automatically collected for fine-tuning of the edge model head layer, ensuring continuous learning of the edge model. If the IoU is high, the cloud model is used for in-depth analysis to integrate multimodal information to improve recognition capabilities in complex scenarios (explanatory, particularly suitable for behavior recognition such as "throwing objects from high places" that requires combining image and scene information).

[0018] In existing knowledge distillation methods, the teacher network and the student network are forced to share temperature hyperparameters, resulting in a mandatory match in the output feature (Logit) range between the two. Considering the ability gap between the teacher network and the student network, this mandatory match will limit the distillation effect of the student network. During the distillation process, the teacher-student network generally uses a fixed temperature hyperparameter. However, in different data sets or teacher-student network combinations, it is necessary to repeatedly adjust the parameters to obtain the optimal temperature hyperparameter. , resulting in low efficiency and time-consuming. In view of the problems existing in the above-mentioned prior art solutions, the preferred technical solution of the present invention further proposes an adaptive distillation method, which aims to solve the inefficiency problem caused by fixed temperature hyperparameter training in the existing knowledge distillation algorithm, and attempts to enhance the effect of model distillation. The adaptive distillation method proposed in the present invention effectively solves the two main limitations existing in the traditional knowledge distillation technology (explanatory, one is that the temperature hyperparameter remains constant in the global range, and the other is that the teacher network and the student network use the same temperature hyperparameter), and shows significant advantages in processing complex tasks and deep network architectures. Specifically, the method of the present invention introduces a dynamic adjustment mechanism, so that the temperature hyperparameter can be adjusted in real time according to the difficulty of the sample knowledge, by fully considering the output distribution characteristics of the teacher network and the similarity between the outputs of the teacher network and the student network. This flexibility enables the model to more accurately capture the essential knowledge contained in the teacher network, while fully considering the learning ability and characteristics of the student network. In summary, the method of the present invention not only improves the efficiency and accuracy of knowledge distillation, but also enhances the understanding and generalization ability of the student network for complex features.

[0019] In view of the problems existing in the existing technical solutions, the power industry urgently needs to explore more efficient, secure and responsive large-scale graphic model deployment strategies; in this context, the present invention proposes a new layered deployment strategy that fully utilizes the computing power of edge-side devices, sinks part of the model reasoning capabilities to the edge model deployed by edge devices, realizes localized preliminary data processing and decision-making, and transmits the recognition and decision results back to the cloud model for further accurate recognition. The technical solution of the present invention can not only reduce the network burden and improve the system response speed, but also make full use of distributed computing resources while ensuring data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a schematic diagram of a large electric power graphic model device in an embodiment of the present invention.

[0022] Figure 2 Schematic diagram of the effect of KL divergence on model evaluation accuracy before and after using Z-score in an embodiment of the present invention.

[0023] Figure 3 It is a schematic diagram of the framework of the adaptive knowledge distillation algorithm in an embodiment of the present invention.

[0024] Figure 4 It is a flow chart of a hierarchical deployment method of a large power graphic model device in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0026] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] See also Figure 1 , an electric power graphic model device disclosed in an embodiment of the present invention includes: an edge model for deployment on an edge device and a cloud model for deployment on the cloud; wherein, The edge model includes a distilled visual coding layer obtained by performing knowledge distillation on the visual coding layer of the large power graphics model, and a head layer added to the distilled visual coding layer; wherein the head layer is used to detect and feedback the detection results for a predetermined task based on the features extracted by the distilled visual coding layer; further explanatory, the head layer can be fine-tuned for a specific task, and the fine-tuning mainly has the following purposes: adapting to new tasks and adjusting the pre-trained model to a specific task; improving performance and obtaining better results in the target task; saving resources and avoiding the high cost of training from scratch; The cloud model includes a text encoding layer and a modal interaction layer in the power graphic model; wherein the text encoding layer is used to input structured prompts, and the modal interaction layer is used to obtain the output results of the distilled visual encoding layer and the output results of the text encoding layer and perform interactive fusion; the structured prompts are obtained by integrating the target bounding box detected by the head layer, the category information detected and identified, and the relevant text description of the external (user) input.

[0028] In the technical solution disclosed in the embodiment of the present invention, the edge model significantly reduces the computational complexity and resource requirements of the model by performing knowledge distillation on the visual coding layer in the large electric power graphic model, while maintaining its key performance, thereby achieving edge model optimization. This optimization enables the edge model to be efficiently deployed and run on edge devices with limited computing power. In addition, on the basis of ensuring that the parameters of the visual coding layer remain unchanged after distillation, a head layer specifically for predetermined tasks is added, which can be fine-tuned for specific downstream tasks. Different tasks can use different head layers, thereby achieving fine-tuning of specific tasks of the power station.

[0029] To explain the principle, in the technical solution disclosed in the embodiment of the present invention, when a user uploads an image to the edge device, the edge model deployed on the edge device first extracts image features and performs preliminary detection (i.e., uses the distilled visual coding layer to generate a high-level feature representation of the image, and uses the fine-tuned head layer to perform preliminary detection of violations and equipment defects on the image. If the edge model detects violations or equipment defects, it immediately feeds back the results to the user). This rapid response mechanism not only improves the real-time performance of the system, but also serves as a preliminary screening mechanism, effectively reducing the number of samples that need to be reported to the cloud model, thereby optimizing overall efficiency.

[0030] For cases that require further analysis, the image features extracted by the edge model deployed on the edge device, the detected target bounding box, the preliminary identified category information, and the relevant text description entered by the user are uploaded to the cloud model to achieve cloud-edge collaborative processing. Further explanatory, after the data is uploaded to the cloud model deployed on the cloud, the detection box generated by the head layer of the edge model is compared with the detection box output by the cloud model, and the performance of the edge model is evaluated by calculating the intersection over union (IoU) between them; if the IoU is lower than the preset threshold, the sample is judged as a potential false positive, and these false positive samples are automatically collected and accumulated for subsequent regular fine-tuning of the head layer. This mechanism ensures that the edge model can continue to learn and adapt to new data distributions, thereby gradually improving its recognition accuracy; if the IoU is higher than the preset threshold, the preliminary recognition result of the edge model is considered to have a high degree of credibility. In this case, the powerful capabilities of the cloud model are further utilized, especially for in-depth analysis of complex scenes; specifically, the detected target bounding box (Bounding Box) is compared with the detection box output by the cloud model. The fusion of multimodal information enables the large-scale electric power graphic model to focus on the area around the detection box, thereby greatly improving the ability to understand complex scenes. It is particularly helpful for identifying complex violations such as "throwing objects from high places", which usually require the combination of image context and scene information to accurately judge.

[0031] To sum up, the technical solution disclosed in the embodiment of the present invention can not only fully utilize the real-time advantages of edge computing, but also give full play to the powerful analysis capabilities of large cloud models, which can significantly improve the recognition accuracy of complex violations. At the same time, the performance of the edge model can be continuously optimized through a continuous learning mechanism.

[0032] Further explanation is given by example. The fine-tuning of the head layer can focus on two key tasks, namely, the identification of personnel violations and the detection of equipment defects. This targeted fine-tuning ensures that the model can achieve excellent performance on these specific tasks. In the model fine-tuning stage, considering that the model is mainly used for preliminary screening, a high recall priority strategy is adopted when designing the model structure. In the power station, more attention is paid to identifying all potential defects or violations. Even if this may lead to a certain degree of false positive increase, this trade-off strategy ensures that key problems will not be missed, and also provides comprehensive candidate samples for subsequent fine analysis. Further explanation is given that in the initial screening stage of the power scene, the risk of mistakenly marking normal situations as potential problems is more acceptable than the risk of missing real defects. Although this method may increase the workload of subsequent processing, it can significantly improve the ability of the overall system to capture key scenes and provide more reliable guarantees for the safe and stable operation of the power station.

[0033] In the existing technical solutions, the current mainstream knowledge distillation method usually adopts KL divergence calculation loss with temperature hyperparameter to realize the knowledge transfer between the teacher network and the student network. This method has two main limitations: first, the traditional method defaults to keep the temperature hyperparameter of the teacher-student network consistent in the global scope, forcing the student to imitate the specific Logit value of the teacher network instead of capturing the essential relationship; secondly, the fixed temperature hyperparameter fails to take into account that the knowledge output by the teacher network may have different difficulty levels for the student model. This simplification ignores the complexity and dynamics in the model learning process; both problems may limit the effect of knowledge distillation, especially when dealing with complex tasks and deep networks. The adaptive distillation method proposed in the embodiment of the present invention effectively solves the two main limitations of traditional knowledge distillation technology. By overcoming these limitations, the present invention shows significant advantages in dealing with complex tasks and deep network architectures. The method of the present invention introduces a dynamic adjustment mechanism, so that the temperature hyperparameter can be adjusted in real time according to the difficulty of the sample knowledge, by fully considering the output distribution characteristics of the teacher network and the similarity between the output of the teacher network and the student network. This flexibility enables the model to more accurately capture the essential knowledge contained in the teacher network, while fully considering the learning ability and characteristics of the student network. The method of the present invention not only improves the efficiency and accuracy of knowledge distillation, but also enhances the student network's understanding and generalization ability of complex features.

[0034] In one embodiment of the present invention, in order to solve the problems caused by the teacher network and the student network being forced to share temperature parameters and the global use of fixed temperature parameters in the existing knowledge distillation algorithm, the embodiment of the present invention proposes an adaptive distillation method. The method draws on the core concept of curriculum learning and evaluates the difficulty of training samples to achieve dynamic and adaptive adjustment of the temperature parameters of the teacher network and the student network during the model training process. This flexible parameter adjustment mechanism enables the model to more accurately capture and extract the essential features and key knowledge contained in the teacher network, while fully considering the learning ability and characteristics of the student network. Explanatory, curriculum learning (CL) is a training strategy that imitates the meaningful learning sequence in human courses and trains machine learning models from easier data to more difficult data. The curriculum learning strategy has been widely used in application scenarios such as computer vision and natural language processing, which can improve the performance and generalization ability of various models and accelerate the training process. Further explanatorily, the existing analysis explains the essence of curriculum learning from the perspective of optimization problems and data distribution, with its main motivations being guidance and denoising; application scenarios based on guidance motivations often involve difficult target tasks, and directly training these tasks may lead to poor performance or slow convergence. To address this problem, curriculum learning strategies are used to guide training and transition from simpler or smoother tasks to the target tasks; application scenarios based on denoising motivations often have noisy or heterogeneous training datasets, and curriculum learning strategies can help denoising, making training faster, more robust, and more general.

[0035] The embodiment of the present invention provides a flexible and adaptive distillation method that can better capture the essence of teacher knowledge and adapt to the learning ability of the student network, which specifically includes: First, when calculating the distillation loss, if the Logit is not preprocessed, the KL divergence may not accurately describe the student's distillation performance and learning effect, such as Figure 2 As shown in the figure, in a knowledge distillation experiment, there are two student networks Stu1 and Stu2, which learn from the teacher network Teach at the same time. The experimental results show that the Logit output range of Stu1 is in a similar numerical range to the output range of Teach, while the Logit output range of Stu2 is significantly smaller than that of Teach. Therefore, according to the calculation results of the following formula (to simplify the calculation, (set to 1), the KL divergence between Stu1 and Teach is smaller than that between Stu2 and Teach; however, this observation is somewhat inconsistent with the actual prediction results of the two student networks. Specifically, although Stu1 has a smaller KL divergence with the teacher network, its prediction result is category 3, which is inconsistent with the actual label; on the contrary, although Stu2 has a larger KL divergence with the teacher network, it correctly predicts category 2.

[0036] ; ; ; ; In the formula, P and Q represent two probability distributions respectively. For every possible event, and Respectively represent events The probability of being in the distribution P, Q, represents the Logit value of the teacher network at i, represents the Logit value of the student network at i, Represents an exponential function with base e.

[0037] To solve the above problem, the technical solution of the embodiment of the present invention uses the Z-score method to standardize Logit before applying Softmax, and the formula is as follows: ; ; In the formula, let is an array, is the index in the array, Indicates the length of the array, is the mean of the array.

[0038] Explanatorily, the above Z-score based preprocessing method has multiple advantages: First, it maps any numerical range of Logit to a standardized scale, so that the Logit outputs of the teacher network and the student network can be compared under the same dimension, ensuring that the mean of the processed Logit is 0 and the variance is 1, so that the teacher and student Logits are mapped to a common space that approximates a Gaussian distribution after standardization; since this mapping process is essentially many-to-one, this means that its reverse mapping is non-unique, and this feature effectively prevents excessive constraints on the original Logit of the student network and avoids the performance degradation of the student network due to mandatory matching with the teacher network Logit; in addition, Z-score standardization can be proved to be a linear transformation. This important property ensures that the standardized Logit of the student network maintains the same relative order as its original Logit, ensuring that the relative relationship between categories contained in the teacher Logit can be effectively retained and passed to the student network.

[0039] Secondly, compared with strictly matching the Logit prediction results of the teacher network, the relative relationship between learning categories is sufficient to enable the student network to achieve similar performance to the teacher network; however, the practice of sharing temperature parameters in the KL divergence calculation between the teacher and student networks in the traditional method will implicitly force the student network to overfit the specific Logit value of the teacher network; considering the inherent ability differences between the teacher and student networks, this practice will limit the distillation effect of the student network, especially when the network level of the student network is much smaller than that of the teacher network; in addition, during the distillation process, it is not optimal for the student model to keep the temperature hyperparameter statically fixed. This method fails to take into account that the knowledge output by the teacher network may have different levels of difficulty for the student model. The present invention draws on the idea of ​​course learning to enable the student network to learn from the teacher network from easy to difficult.

[0040] In the existing knowledge distillation methods, taking the classification task as an example, the training loss of the student network usually consists of two parts: the conventional classification loss and the knowledge distillation loss. It plays a key role in regulating the smoothness of the output distribution of the teacher network and the student network. The larger ( ), the output distribution tends to be smooth, the difference in prediction probability of each category decreases, and a more uniform distribution characteristic is presented. The smaller ( ), the output distribution becomes steeper, with some categories having significantly higher predicted probabilities and others having relatively lower predicted probabilities; The value of has a significant impact on the difficulty of the student network in the distillation process. The current existing technical solutions generally adopt fixed value, which may limit the further improvement of the student network distillation effect to some extent; In view of the above problems of the prior art solution, the present invention proposes an adaptive method to dynamically adjust A formula is designed to quantify the difficulty of sample knowledge, which takes into account two key factors: the output distribution characteristics of the teacher network and the similarity between the outputs of the teacher network and the student network; based on this difficulty index, it can be dynamically adjusted for different samples. value, thereby optimizing the entire knowledge distillation process.

[0041] ; In the formula, is a balancing factor used to adjust the relative importance of the two parts ( ); is a difficulty score based on the teacher network distribution; It is a score based on the similarity of the teacher-student network Logit output; ; In the formula, and They are the maximum and second largest Logit values ​​output by the teacher network, respectively. When the gap between the two largest Logit values ​​is large, Close to 0, it means the sample is simple; when the gap is small, Close to 1, it means the sample is complex; ; Similarity can be calculated by cosine similarity, dot product or KL divergence. Here, cosine similarity is used as the standard to measure the similarity between the teacher and student network outputs. When the similarity is high, Close to 1; when the similarity is low, Close to 0; Based on the difficulty score, the present invention designs a function to dynamically adjust , the expression is: ; in, and are the minimum and maximum values ​​of the preset temperature hyperparameters, respectively; This formula ensures that when the gap between the two maximum Logits of the teacher network is large and the similarity between the teacher and the student network is low, the difficulty score is low, resulting in a smaller , which is conducive to learning steeper knowledge; when the two maximum Logit gaps of the teacher network are small and the teacher-student network similarity is high, the difficulty score is high, resulting in a larger , which is conducive to learning the relationship between categories; The parameter allows flexible adjustment of the relative importance of the two factors. Restricted to within the scope to ensure rationality.

[0042] In summary, in view of the significant difference in model capacity between the teacher network and the student network, the present invention adopts a differentiated temperature parameter setting strategy; specifically, the teacher network is usually assigned a higher temperature value to give full play to its powerful representation ability, thereby generating a more delicate and rich probability distribution; relatively speaking, the temperature parameter of the student network is slightly lower than that of the teacher network. This design aims to enable the student network to produce a relatively steep distribution, which not only avoids the risk of over-smoothing, but also effectively captures key category discrimination information. It can ensure that while retaining the rich knowledge of the teacher network, the student network can focus more on more essential and key features, thereby achieving the best learning effect under limited model capacity.

[0043] ; Where epoch is the current iteration, and max_epoch is the total number of model training iterations.

[0044] The logarithmic function can be used to achieve The smooth growth of the training is fast in the early stage; as the training progresses, the growth rate gradually slows down, and the formula can be flexibly adjusted. range of change to avoid excessive growth.

[0045] The knowledge distillation algorithm described above in the embodiment of the present invention can realize efficient edge deployment and cloud-edge collaboration of large power graph and text models.

[0046] See also Figure 3 , Figure 3 The adaptive knowledge distillation algorithm framework is shown, and the specific implementation may include the following steps: Input: Dataset Contains N samples ,here , H, W are the height and width of the image; given an input ,teacher and students Predict the Logit vectors respectively, and the Z-score function is , currently epochs, a total of epochs; Output: The trained student model

[0047] 1: for in do 2:

[0048] 3:

[0049] 4:

[0050] 5:

[0051] 6:

[0052] 7:

[0053] 8:

[0054] 9:

[0055] 10:

[0056] 11: Minimize the following formula to update : 12:

[0057] 13: end In summary, the embodiment of the present invention uses the knowledge distillation technology proposed above to lightweight compress the visual encoding module of the large power graphics model, reduce the computational complexity of the model and ensure the core performance, so that the model can be efficiently deployed and run on edge devices with limited computing power. By adding a special header layer, the high performance of the model in specific application scenarios is ensured. At the same time, the image features, bounding boxes, recognition category information and user-entered text extracted by the edge device are uploaded to the cloud model to achieve further accurate recognition.

[0058] See also Figure 4 , a hierarchical deployment method of a large power graphic model device disclosed in an embodiment of the present invention includes the following steps: Step 1: Perform knowledge distillation on the visual coding layer of the large electric power graphics model to be deployed to obtain a distilled visual coding layer; add a header layer to the distilled visual coding layer to obtain a lightweight model of the visual coding module, and use the lightweight model of the visual coding module as an edge model and deploy it on the edge device; wherein the header layer is used to detect the predetermined task based on the features extracted by the distilled visual coding layer and feedback the detection results; Step 2: Use the text encoding layer and modal interaction layer in the large electric power graphic model to be deployed as cloud models and deploy them on the cloud; wherein the text encoding layer is used to input structured prompts, and the modal interaction layer is used to obtain the output results of the distilled visual encoding layer and the output results of the text encoding layer and perform interactive fusion; the structured prompts are obtained by integrating the target bounding box detected by the head layer, the category information detected and identified, and the relevant text description of the external (user) input.

[0059] In view of the problems existing in the existing technical solutions, the power industry urgently needs to explore more efficient, secure and responsive large-scale graphic model deployment strategies; in this context, the embodiment of the present invention proposes a new layered deployment strategy that fully utilizes the computing power of edge devices, sinks part of the model reasoning capabilities to edge devices, realizes localized preliminary data processing and decision-making, and transmits the recognition and decision results back to the cloud model for further accurate recognition. The technical solution of the present invention can not only reduce the network burden and improve the system response speed, but also make full use of distributed computing resources while ensuring data privacy.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A large-scale model device of electric power graphics, characterized in that: include: An edge model for deployment on edge devices and a cloud model for deployment on the cloud; wherein, The edge model includes a distilled visual coding layer obtained by performing knowledge distillation on the visual coding layer of the original power graphic model, and a head layer added to the distilled visual coding layer; wherein the head layer is used to detect a predetermined task based on the features extracted by the distilled visual coding layer and to feed back the detection result; The cloud-based model includes a text encoding layer and a modal interaction layer in the original power graphic model; wherein the text encoding layer is used to input structured prompts, and the modal interaction layer is used to input the output results of the distilled visual encoding layer and the output results of the text encoding layer and interactively fuse them; the structured prompts are obtained by integrating the target bounding box detected by the head layer, the category information detected and identified, and the relevant text description.

2. The large-scale model device of electric power graphics and text according to claim 1, characterized in that: The head layer performs performance evaluation and fine-tuning updates by calculating the intersection-over-union ratio between the detection box generated by the head layer and the detection box output by the cloud model.

3. The large-scale model device of electric power graphics according to claim 1, characterized in that: In the step of obtaining the distilled visual coding layer after performing knowledge distillation on the visual coding layer of the original power graphic model, when performing knowledge distillation, the KL divergence with a temperature hyperparameter is used to calculate the loss to achieve knowledge transfer between the teacher network and the student network; Among them, the teacher network is the visual encoding layer of the original power graphic model, and the student network is the visual encoding layer after distillation. When calculating the distillation loss, the Z-score method is used to standardize Logit before applying Softmax, and the value of the temperature hyperparameter is adaptively and dynamically adjusted according to the difficulty of the sample knowledge.

4. The large-scale model device of electric power graphics and text according to claim 3, characterized in that: In the step of using the Z-score method to standardize Logit before applying Softmax, for An array of , the standardized formula is expressed as: ； ； In the formula, is the mean of the array; Indicates the length of the array; Indicates the first Number; Indicates the value of Z-score; represents the temperature hyperparameter.

5. The large-scale model device of electric power graphics and text according to claim 3, characterized in that: In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the calculation of the difficulty takes into account two factors: the output distribution characteristics of the teacher network and the similarity between the outputs of the teacher network and the student network.

6. The large-scale model device of electric power graphics according to claim 3, characterized in that: In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the calculation expression of the difficulty is: ； In the formula, Indicates the calculated value of sample difficulty; represents the balance factor, ; represents the difficulty score based on the teacher network distribution; Represents the score based on the similarity of the teacher network and the student network Logit output; ； In the formula, and They are the maximum and second largest Logit values ​​output by the teacher network, respectively; ； In the formula, represents the Logit value of the teacher network, Represents the Logit value of the student network.

7. The large-scale model device of electric power graphics and text according to claim 6, characterized in that: Calculate the score based on the similarity of the teacher network and the student network Logit output When , Similarity is calculated using cosine similarity, using dot product or KL divergence.

8. The large-scale model device of electric power graphics and text according to claim 6, characterized in that: In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the temperature hyperparameter The expression is: ； in, and are the preset minimum and maximum values ​​of the temperature hyperparameter, respectively.

9. The large-scale model device of electric power graphics and text according to claim 6, characterized in that: In the step of adaptively and dynamically adjusting the value of the temperature hyperparameter according to the difficulty of the sample knowledge, the temperature hyperparameter The expression is: ； In the formula, and They are the minimum and maximum values ​​of the preset temperature hyperparameters respectively; epoch is the round of the current iteration, and max_epoch is the total round of model training iterations.

10. A hierarchical deployment method for a large power graphic model, characterized in that: The following steps are involved: Perform knowledge distillation on the visual coding layer of the original power graphic model to be deployed to obtain a distilled visual coding layer; add a header layer to the distilled visual coding layer to obtain a lightweight model of the visual coding module, and use the lightweight model of the visual coding module as an edge model and deploy it on the edge device; wherein the header layer is used to detect the predetermined task based on the features extracted by the distilled visual coding layer and feedback the detection results; The text encoding layer and modal interaction layer in the original power graphic model to be deployed are used as cloud models and deployed on the cloud; the text encoding layer is used to input structured prompts, and the modal interaction layer is used to obtain the output results of the distilled visual encoding layer and the output results of the text encoding layer and interactively fuse them; the structured prompts are obtained by integrating the target bounding box detected by the head layer, the category information detected and identified, and the relevant text description.

Images