A method and system for incremental learning of defect images under UAV-cloud-edge collaboration

Abstract

The present application provides a method and system for incremental learning of defect images under the collaboration of drones, cloud and edge, to build a distributed cloud-edge collaborative transmission line defect recognition platform, deploy an inference and semi-automatic annotation system on the edge, and build a multi-model incremental training system on the cloud through digital twin technology; the central cloud and the edge nodes cooperate with each other, and the annotation efficiency and model accuracy can be improved through incremental learning. At the same time, samples are assisted by manual or other systems, and then the model is incrementally trained again. If the incrementally trained model has a significant improvement in accuracy over the original model, it can be deployed to the edge node to update the original model. The mechanism for pipeline tasks is implemented through crowdsourcing, and the semi-automatic annotation tasks are divided into multiple small pipeline tasks. The pipeline operation not only makes the annotation work more efficient, but also integrates the characteristics of different models to make the obtained annotation results more accurate.

Description

Technical Field

[0001] The present application relates to the technical field of incremental learning and annotation of defect images, and in particular to a method and system for incremental learning of defect images under UAV cloud-edge collaboration. Background Art

[0002] With the development of drone technology, the use of drones to replace manual inspections of power transmission channels has become increasingly common. Currently, the mainstream approach to power transmission drone inspections is to send images and video data captured by the equipment to an artificial intelligence platform, which then uses the platform's model to detect transmission faults and identify defects. However, this approach presents the following issues: 1. The artificial intelligence platform model is static and does not currently support self-learning and iterative improvements in defect identification capabilities. 2. Cloud computing and channel networks are under excessive pressure. Currently, the cloud computing within artificial intelligence platforms cannot meet the needs of applications with demanding real-time and security requirements. For example, defect detection, such as forest fire detection, requires high real-time performance, which current cloud-based artificial intelligence platforms are unable to meet.

[0003] Accurate and efficient object detection and annotation are critical steps in the task of identifying transmission line defects. Traditional manual annotation methods are not only time-consuming and labor-intensive, but also prone to human error. Furthermore, with the expansion of transmission line scale and the increase in data volume, the construction of a distributed cloud-edge collaborative identification platform has become an urgent need.

[0004] Therefore, designing an efficient data labeling and model updating mechanism to conduct timely and accurate detection and defect troubleshooting of transmission lines is an important means and urgent task to ensure the safe and stable operation of the power grid. Summary of the Invention

[0005] In view of this, it is necessary to provide a defect image incremental learning method under UAV cloud-edge collaboration, realize real-time defect detection through edge computing nodes, and use the powerful computing power of the cloud to perform incremental model updates. While ensuring real-time detection, it can effectively adapt to newly emerging defect types, significantly improve the accuracy of defect recognition and system adaptability, be easy to promote and apply, and be relatively simple to operate, making it suitable for promotion and application in more scenarios.

[0006] The present application provides a method and system for incremental learning of defect images under the collaboration of drones, cloud and edge, to build a distributed cloud-edge collaborative transmission line defect recognition platform, deploy an inference and semi-automatic annotation system on the edge, and build a multi-model incremental training system on the cloud through digital twin technology; the central cloud and the edge nodes cooperate with each other, and the annotation efficiency and model accuracy can be improved through incremental learning. At the same time, samples are assisted by manual or other systems, and then the model is incrementally trained again. If the incrementally trained model has a significant improvement in accuracy over the original model, it can be deployed to the edge node to update the original model. The mechanism for pipeline tasks is implemented through crowdsourcing, and the semi-automatic annotation tasks are divided into multiple small pipeline tasks. The pipeline operation not only makes the annotation work more efficient, but also integrates the characteristics of different models to make the obtained annotation results more accurate.

[0007] In the first aspect, the embodiment of the present application provides a method for incremental learning of defect images under UAV cloud-edge collaboration.

[0008] S1: Build a distributed cloud-edge collaborative transmission line defect identification platform, deploy inference and semi-automatic annotation systems on the edge, and build a multi-model incremental training system on the cloud using digital twin technology;

[0009] S2: Using drones and low-power wide-area IoT technology, the system collects line temperature, sag, and vibration status in real time and uploads the collected data to the edge.

[0010] S3: The edge-side inference and semi-automatic annotation system provides the required pre-trained models. It uses crowdsourcing to distribute the uploaded collected data, performs pre-annotation tasks, and aggregates samples with low confidence in the automatic identification and inference results, packaging them and sending them to the central cloud.

[0011] S4: Samples are labeled manually or with the assistance of Label Studio software in the central cloud, and then multiple incremental training models built using digital twin technology are retrained;

[0012] S5: Calculate the accuracy of the incrementally trained models and compare them with the accuracy of the original model;

[0013] S6: If the comparison result is higher than the threshold, the trained model is deployed to each edge node to update the original model.

[0014] Optionally, in an implementation of the first aspect of the present invention, in step S1, a distributed cloud-edge collaborative transmission line defect identification platform is constructed, an inference and semi-automatic annotation system is deployed on the edge, and a multi-model incremental training system is constructed on the cloud using digital twin technology, including:

[0015] The distributed cloud-edge collaborative transmission line defect identification platform includes: a central cloud, edge terminals, and drones. The edge terminals include field devices, edge gateways, and edge clouds. A multi-level distributed management architecture is used to interconnect edge nodes. By converting the proprietary communication protocols used by various field devices into the standardized OPC UA protocol, computing tasks are distributed to multiple nodes for parallel processing, enabling access to heterogeneous devices and data collection, transmission, and processing.

[0016] The reasoning and semi-automatic annotation system includes a data input module, a model annotation module, an RTMDet reasoning service module, and an annotation output module; the data input module is used to receive image data to be annotated; the model annotation module uses different pre-trained models to perform preliminary annotation on the image; the RTMDet reasoning service module optimizes and adjusts the preliminary annotation results; and the annotation output module saves and outputs the final annotation results.

[0017] The multi-model incremental training system includes a digital twin layer, a model management layer, and an incremental training engine; the digital twin layer is used to mirror the model physical structure according to the model of the model annotation module to obtain multiple mirror models; the model management layer is used to perform version control, performance monitoring and model dependency management on the multiple mirror models; the incremental training engine is used to adaptively adjust the learning rate through an online learning algorithm library and a distributed training framework.

[0018] Optionally, in an implementation of the first aspect of the present invention, S3: carrying out an inference and semi-automatic annotation system on the edge side to provide the required multiple pre-trained models, distributing the uploaded collected data in a first crowdsourcing manner, performing pre-annotation tasks, and summarizing samples with low confidence in the automatic identification reasoning results, packaging and sending them to the central cloud, including:

[0019] The multiple pre-trained models include at least three models: a CNN model, an improved dual Unet model, and a Logits distillation model; the first crowdsourcing method is a pipeline task implementation mechanism;

[0020] The semi-automatic labeling task is divided into a plurality of pipeline tasks; a corresponding number of pre-trained models are selected from the plurality of pre-trained models according to the number of divided small tasks, and the pre-trained models are arranged and combined to obtain a plurality of candidate model combinations, and the pipeline tasks are executed respectively;

[0021] The optimal model combination is selected based on the confidence of the automatic recognition reasoning results, and the samples with low confidence are filtered out from the recognition results corresponding to the optimal model combination, summarized, packaged and sent to the central cloud.

[0022] Optionally, in an implementation of the first aspect of the present invention, a corresponding number of pre-trained models are selected from the multiple pre-trained models according to the number of divided small tasks, and the pre-trained models are arranged and combined to obtain multiple candidate model combinations, and the pipeline tasks are respectively executed; the optimal model combination is selected according to the confidence of the automatic recognition reasoning result, and the samples with low confidence are screened from the recognition results corresponding to the optimal model combination, summarized, packaged and sent to the central cloud, including:

[0023] The number of the divided small tasks is three;

[0024] Selecting, from the plurality of candidate model combinations, a first model in the combination to perform the task of labeling the type of the target object, a second model to perform the task of labeling the number and position of the target object, and a third model to perform the task of labeling the contour boundary of the target object;

[0025] The optimal candidate model combination is selected based on the weighted sum of the accuracy of the three execution results as the confidence level;

[0026] The optimal candidate model combination is used as the target model combination, and a dependency relationship is established between the models in the target model combination. The first model in the target model combination is used again to perform the task of labeling the type of target object, the second model performs the task of labeling the number and position of target objects, and the third model performs the task of labeling the contour boundary of the target object based on the execution result of labeling the type of target object combined with the execution result of labeling the number and position of target objects. The recognition results corresponding to the optimal model combination are filtered out and the samples with low confidence are summarized, packaged and sent to the central cloud.

[0027] Optionally, in an implementation of the first aspect of the present invention, S4: labeling samples manually or with the assistance of LabelStudio software in the central cloud, and then retraining multiple incremental training models constructed using digital twin technology, includes:

[0028] Collect an image dataset containing transmission line defects and divide it into training and test sets;

[0029] Preprocess the image, resize it according to different pre-trained models, and perform normalization operations to meet the input requirements of different models; at the same time, perform preliminary screening and compression on the collected images at the edge to reduce data transmission volume;

[0030] Select different pre-trained models and use the interfaces provided by MMDETECTION to perform preliminary image annotation, including:

[0031] Load the pre-trained model: Use the MMDETECTION configuration file and pre-trained weights to load the model; input the pre-processed image into the model for inference to obtain preliminary annotation boxes; manually review the inference results, manually drag the box, and correct the position of the box to obtain the corrected annotation results; save the annotation results;

[0032] Use labeled data to retrain multiple incremental training models built using digital twin technology.

[0033] Optionally, in an implementation of the first aspect of the present invention, the improved dual Unet model structure is an asymmetric Unet model, including UNet1 and UNet2, wherein the low-level and deep-level semantic features are extracted by the encoders of UNet1 and UNet2 respectively, and the low-level and deep-level semantic features are extracted at the decoding layer for interaction and fusion, and the prediction results are output, and the loss function is the cross entropy loss function and Dice loss function The joint loss function composed of is:

[0034] ;

[0035] in, Representation sample The true distribution of Representation sample The predicted distribution of represents the total number of samples;

[0036] ;

[0037] in, Respectively The probability value of the class prediction and the true label value, is the number of categories, is the total number of categories, Indicates the Class weights;

[0038] ;

[0039] The Logits distillation model structure consists of three parallel network models, namely the standard teacher network model, the student network model, and the adversarial teacher network model. The standard teacher network model adopts the UNet network structure based on the attention mechanism as the backbone network, and embeds the attention mechanism into the decoding part of the UNet network. The student network model adopts the lightweight UNet network structure. The adversarial teacher network model adopts the U-CliqueNet model structure, and replaces the convolution layer of UNet with the Clique Block. The loss function consists of hard loss and soft loss, wherein the hard loss function for:

[0040] ;

[0041] in, Representation sample The true distribution of Representation sample The predicted distribution of represents the total number of samples;

[0042] The soft loss function is :

[0043] ;

[0044] in, represents the temperature parameter, They represent the probability distributions of the standard teacher network model, the student network model, and the adversarial teacher network model after high temperature softening, is the total number of categories, is the divergence;

[0045] The total loss function is :

[0046] ;

[0047] in, represents the hyperparameters for adjusting the weights of hard loss and soft loss, .

[0048] Optionally, in an implementation of the first aspect of the present invention, respectively calculating the accuracy of the incrementally trained models and comparing them with the accuracy of the original models includes:

[0049] The accuracy of each model is evaluated by the average intersection-over-union ratio, and the formula is:

[0050] ;

[0051] in, represents the total number of categories, The true category is And the predicted category is The number of samples, The true category is And the predicted category is The number of samples.

[0052] In a second aspect, an embodiment of the present application provides a system for incremental learning of defect images under drone-cloud-edge collaboration, which is applied to the method for incremental learning of defect images under drone-cloud-edge collaboration as described in the first aspect, and is characterized by comprising:

[0053] Cloud-edge collaborative platform construction module: Build a distributed cloud-edge collaborative transmission line defect identification platform, deploy inference and semi-automatic annotation systems on the edge, and build a multi-model incremental training system on the cloud using digital twin technology;

[0054] Data acquisition module: Using low-power wide-area IoT technology via drones, it can collect line temperature, sag, and vibration status in real time, and upload the collected data to the edge side.

[0055] The first labeling module: The edge side is equipped with an inference and semi-automatic labeling system to provide the required multiple pre-trained models. It uses the first crowdsourcing method to distribute the uploaded collected data, perform pre-labeling tasks, and summarize samples with low confidence in the automatic identification reasoning results, package them and send them to the central cloud;

[0056] Second labeling module: Samples are labeled manually or with the assistance of Label Studio software in the central cloud, and then multiple incremental training models built using digital twin technology are retrained;

[0057] Accuracy calculation module: calculates the accuracy of the incrementally trained model and compares it with the accuracy of the original model;

[0058] Model update module: If the comparison result is higher than the threshold, the trained model is deployed to each edge node to update the original model.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, characterized by including:

[0060] processor;

[0061] a memory for storing processor-executable instructions;

[0062] Among them, the processor is configured to implement the incremental learning method of defect images under drone cloud-edge collaboration as described in the first aspect when executing the instructions.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a program, and the program instructs a device to execute the incremental learning method for defect images under drone-cloud-edge collaboration as described in the first aspect. The present application provides a method and system for incremental learning of defect images under drone-cloud-edge collaboration, constructs a distributed cloud-edge collaborative transmission line defect recognition platform, deploys an inference and semi-automatic labeling system on the edge, and constructs a multi-model incremental training system on the cloud through digital twin technology; uses low-power wide-area Internet of Things technology through drones to realize real-time collection of line temperature, sag, and vibration state quantities, and uploads the collected data to the edge side; carries out inference and semi-automatic labeling system on the edge side to provide the required multiple pre-trained models, distributes the uploaded collected data in a first crowdsourcing manner, performs pre-labeling tasks, and summarizes samples with low confidence in automatic identification reasoning results, packages them and sends them to the central cloud; in the central cloud, samples are assisted in labeling by manual labor or Label Studio software, and then multiple incremental training models constructed by digital twin technology are retrained; the accuracy of the incrementally trained models is calculated respectively and compared with the accuracy of the original models; if the comparison result is higher than the threshold, the trained models are deployed to each edge node to update the original model.

[0064] Beneficial effects:

[0065] (1) The central cloud and edge nodes work together to improve model accuracy through incremental learning. After the model is deployed on the edge node and runs inference, samples with low confidence in the inference results are automatically identified and sent to the central cloud. Samples are annotated manually or with the assistance of other systems, and the model is then incrementally retrained. If the incrementally trained model has significantly improved accuracy compared to the original model, it can be deployed to the edge node to update the original model.

[0066] (2) Through edge model reasoning and semi-automatic labeling technology and equipment, labeling work is made more efficient, and samples are assisted by humans or other systems to make labeling more accurate.

[0067] (3) Through crowdsourcing, a mechanism is implemented for pipeline tasks, which divides the semi-automatic labeling task into multiple pipeline tasks. The pipeline tasks are executed separately through the combination of candidate models. The pipeline operation not only makes the labeling work more efficient, but also integrates the characteristics of different models to make the labeling results more accurate.

[0068] (4) It is easy to promote and apply, and the operation is relatively simple, making it suitable for promotion and application in more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flow chart of the incremental learning method for defect images under drone-cloud-edge collaboration provided in one embodiment of the present application.

[0070] Figure 2 A framework diagram of a distributed cloud-edge collaborative transmission line defect identification platform provided in one embodiment of the present application.

[0071] Figure 3 A diagram of the multi-level distributed management architecture of edge nodes provided in one embodiment of the present application.

[0072] Figure 4 This is a diagram of the dual Unet model structure architecture provided in one embodiment of the present application.

[0073] Figure 5 This is a structural diagram of the Logits distillation model provided in one embodiment of the present application.

[0074] Figure 6 Schematic diagram of the defect image incremental learning system module under drone-cloud-edge collaboration provided in one embodiment of the present application.

[0075] Figure 7 A schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0077] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art to which this application relates. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0078] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order. Features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0079] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] Example 1.

[0081] The present application provides a method and system for incremental learning of defect images under the collaboration of drones, cloud and edge, to build a distributed cloud-edge collaborative transmission line defect recognition platform, deploy an inference and semi-automatic annotation system on the edge, and build a multi-model incremental training system on the cloud through digital twin technology; the central cloud and the edge nodes cooperate with each other, and the annotation efficiency and model accuracy can be improved through incremental learning. At the same time, samples are assisted by manual or other systems, and then the model is incrementally trained again. If the incrementally trained model has a significant improvement in accuracy over the original model, it can be deployed to the edge node to update the original model. The mechanism for pipeline tasks is implemented through crowdsourcing, and the semi-automatic annotation tasks are divided into multiple small pipeline tasks. The pipeline operation not only makes the annotation work more efficient, but also integrates the characteristics of different models to make the obtained annotation results more accurate.

[0082] Figure 1 A flow chart of the incremental learning method for defect images under drone-cloud-edge collaboration provided in one embodiment of the present application.

[0083] like Figure 1 As shown in the figure, a method for incremental learning of defect images under UAV-cloud-edge collaboration includes:

[0084] S1: Build a distributed cloud-edge collaborative transmission line defect identification platform, deploy inference and semi-automatic labeling systems on the edge, and build a multi-model incremental training system on the cloud using digital twin technology.

[0085] It is understandable that in this embodiment, in step S1, a distributed cloud-edge collaborative transmission line defect identification platform is constructed, an inference and semi-automatic annotation system is deployed on the edge, and a multi-model incremental training system is constructed on the cloud using digital twin technology, including:

[0086] Figure 2 This is a framework diagram of a distributed cloud-edge collaborative transmission line defect identification platform provided in one embodiment of the present application. Figure 2 As shown, the distributed cloud-edge collaborative transmission line defect identification platform includes: a central cloud, an edge end, and a drone, wherein the edge end includes field equipment, an edge gateway, and an edge cloud. Figure 3 This is a diagram of the multi-level distributed management architecture of each edge node provided in one embodiment of the present application. Figure 3As shown in the figure, a multi-level distributed management architecture is used to achieve interconnection between edge nodes. Based on the conversion of the private communication protocols used by various field devices into the standardized OPC UA protocol, computing tasks are distributed to multiple nodes for parallel processing, enabling access to heterogeneous devices and data collection, transmission, and processing.

[0087] The reasoning and semi-automatic annotation system includes a data input module, a model annotation module, an RTMDet reasoning service module, and an annotation output module; the data input module is used to receive image data to be annotated; the model annotation module uses different pre-trained models to perform preliminary annotation on the image; the RTMDet reasoning service module optimizes and adjusts the preliminary annotation results; and the annotation output module saves and outputs the final annotation results.

[0088] The multi-model incremental training system includes a digital twin layer, a model management layer, and an incremental training engine; the digital twin layer is used to mirror the model physical structure according to the model of the model annotation module to obtain multiple mirror models; the model management layer is used to perform version control, performance monitoring and model dependency management on the multiple mirror models; the incremental training engine is used to adaptively adjust the learning rate through an online learning algorithm library and a distributed training framework.

[0089] Specifically, building a distributed cloud-edge collaborative global management platform needs to be based on resource management, with intelligent scheduling as the core and an open ecosystem as an extension. It should achieve agile application deployment through cloud-native technology, optimize collaborative strategies based on industry scenarios, and ultimately achieve the goal of "real-time response at the edge and global optimization in the cloud."

[0090] S2: Using low-power wide-area IoT technology through drones, real-time collection of line temperature, sag, and vibration status is achieved, and the collected data is uploaded to the edge side.

[0091] It is understood that in this embodiment, the drone can utilize LoRa-based LPWAN technology (such as LoRaWAN or LoPo-IoT). Its long-range transmission (up to 15 kilometers in rural areas) and low power consumption make it suitable for large-scale power transmission line inspections. LoRa modules (such as the RN2483) perform well in complex terrain and are adaptable to the dynamic movement of drones. Furthermore, the drone is equipped with temperature, sag, and vibration sensors (such as the temperature detection unit in the figure) to collect real-time data via LPWAN. For example, the LoRa module supports multi-node concurrent transmission, ensuring high-frequency data upload. Furthermore, the drone, acting as a mobile gateway, can optimize coverage blind spots and improve data collection efficiency.

[0092] Data collected by drones is uploaded to edge computing nodes (such as those deployed at substations or mobile edge servers) via LPWAN. Lightweight AI models are deployed on the edge to perform real-time fault feature extraction, sag anomaly detection (for example, through temperature-sag curve analysis), and vibration status assessment.

[0093] Cloud-side functionality: Deploys a multi-model incremental training system that supports dynamic model updates, algorithm library management, and large-scale data processing. Edge-side functionality: A lightweight inference system (such as TensorFlow Lite) enables real-time defect detection, and a semi-automatic annotation system uses an interactive interface to assist manual correction of low-confidence samples. Collaboration: MQTT / HTTP protocols are used for cloud-edge communication, with edge nodes responsible for data filtering and the cloud focusing on model optimization.

[0094] Specifically, the distributed cloud-edge collaborative transmission line defect identification platform includes: a central cloud, an edge terminal, and a drone. The edge terminal includes field equipment, an edge gateway, and an edge cloud, which is used to realize the access and data collection of heterogeneous devices based on the conversion of the private communication protocols adopted by various field equipment and devices into the standardized OPC UA protocol; through a lightweight computing framework and related algorithms, the data is cleaned and format converted and pre-processed on the edge side, and redundant data is effectively eliminated from the massive data generated on the edge side, reducing the amount of data uploaded to the cloud, and reducing the cloud platform load and transmission bandwidth pressure.

[0095] S3: The edge side is equipped with an inference and semi-automatic labeling system to provide the required multiple pre-trained models. It uses the first crowdsourcing method to distribute the uploaded collected data, perform pre-labeling tasks, and summarize samples with low confidence in the automatic identification reasoning results, package them and send them to the central cloud.

[0096] It is understandable that in this embodiment, the S3: carrying out the reasoning and semi-automatic annotation system on the edge side to provide the required multiple pre-trained models, distributing the uploaded collected data in a first crowdsourcing manner, performing the pre-annotation task, and summarizing the samples with low confidence in the automatic identification reasoning results, and packaging and sending them to the central cloud, includes:

[0097] The multiple pre-trained models include at least three models: a CNN model, an improved dual Unet model, and a Logits distillation model; the first crowdsourcing method is a pipeline task implementation mechanism;

[0098] The semi-automatic labeling task is divided into a plurality of pipeline tasks; a corresponding number of pre-trained models are selected from the plurality of pre-trained models according to the number of divided small tasks, and the pre-trained models are arranged and combined to obtain a plurality of candidate model combinations, and the pipeline tasks are executed respectively;

[0099] The optimal model combination is selected based on the confidence of the automatic recognition reasoning results, and the samples with low confidence are filtered out from the recognition results corresponding to the optimal model combination, summarized, packaged and sent to the central cloud.

[0100] Specifically, the method selects a corresponding number of pre-trained models from the plurality of pre-trained models according to the number of divided small tasks, performs permutations and combinations, obtains a plurality of candidate model combinations, and respectively executes the pipeline tasks; selects the optimal model combination according to the confidence of the automatic recognition reasoning result, and screens the samples with low confidence from the recognition results corresponding to the optimal model combination, summarizes them, and packages them and sends them to the central cloud, including:

[0101] The number of the divided small tasks is three;

[0102] Selecting, from the plurality of candidate model combinations, a first model in the combination to perform the task of labeling the type of the target object, a second model to perform the task of labeling the number and position of the target object, and a third model to perform the task of labeling the contour boundary of the target object;

[0103] The optimal candidate model combination is selected based on the weighted sum of the accuracy of the three execution results as the confidence level;

[0104] The optimal candidate model combination is used as the target model combination, and a dependency relationship is established between the models in the target model combination. The first model in the target model combination is used again to perform the task of labeling the type of target object, the second model performs the task of labeling the number and position of target objects, and the third model performs the task of labeling the contour boundary of the target object based on the execution result of labeling the type of target object combined with the execution result of labeling the number and position of target objects. The recognition results corresponding to the optimal model combination are filtered out and the samples with low confidence are summarized, packaged and sent to the central cloud.

[0105] Specifically, Figure 4 This is a diagram of the dual Unet model structure provided in one embodiment of the present application. Figure 4 As shown, the improved dual Unet model structure is an asymmetric Unet model, including UNet1 and UNet2, wherein the encoders of UNet1 and UNet2 extract low-level and deep-level semantic features respectively, and the low-level and deep-level semantic features are extracted in the decoding layer for interaction and fusion, and the prediction results are output. The loss function is the cross entropy loss function and Dice loss function The joint loss function composed of is:

[0106] ;

[0107] in, Representation sample The true distribution of Representation sample The predicted distribution of represents the total number of samples;

[0108] ;

[0109] in, Respectively The probability value of the class prediction and the true label value, is the number of categories, is the total number of categories, Indicates the Class weights;

[0110] ;

[0111] Figure 5 This is a diagram of the Logits distillation model structure provided in one embodiment of the present application. Figure 5 As shown in the figure, the Logits distillation model structure consists of three parallel network models, namely the standard teacher network model, the student network model, and the adversarial teacher network model. The standard teacher network model adopts the UNet network structure based on the attention mechanism as the backbone network, and embeds the attention mechanism into the decoding part of the UNet network. The student network model adopts the lightweight UNet network structure. The adversarial teacher network model adopts the U-CliqueNet model structure, and replaces the convolution layer of UNet with the Clique Block. The loss function consists of hard loss and soft loss. for:

[0112] ;

[0113] in, Representation sample The true distribution of Representation sample The predicted distribution of represents the total number of samples;

[0114] The soft loss function is :

[0115] ;

[0116] in, represents the temperature parameter, They represent the probability distributions of the standard teacher network model, the student network model, and the adversarial teacher network model after high temperature softening, is the total number of categories, is the divergence;

[0117] The total loss function is :

[0118] ;

[0119] in, represents the hyperparameters for adjusting the weights of hard loss and soft loss, .

[0120] Specifically, after using the CNN model to extract features from the input image, the category Logits is output through the fully connected layer, and the category probability distribution is generated by combining with Softmax. The high-confidence results can be directly used to label the target type; the dual Unet model is used to extract multi-scale features through the encoder, and the decoder generates the position heat map and quantity prediction of the target; the distillation model is used according to the labeled target type and the position heat map and quantity prediction results of the generated target, and the output of the Logits segmentation task of the teacher model is used as the supervision signal. The loss is used to optimize the sorting information of the boundary pixels to refine the contour annotation.

[0121] S4: Samples are labeled manually or with the assistance of Label Studio software in the central cloud, and then multiple incremental training models built using digital twin technology are retrained.

[0122] Specifically, MMDETECTION is an open-source object detection toolbox based on PyTorch, providing a rich set of pre-trained models and efficient training and inference interfaces. LABEL-STUDIO is a powerful annotation tool that supports a variety of annotation tasks. Combining these two tools enables semi-automated object detection and annotation. Integrating them into a distributed cloud-edge collaborative architecture effectively improves the annotation efficiency and accuracy of transmission line defect identification.

[0123] It is understood that in this embodiment, the step S4: labeling samples manually or with the assistance of Label Studio software in the central cloud, and then retraining multiple incremental training models built using digital twin technology, includes:

[0124] Collect an image dataset containing transmission line defects and divide it into training and test sets;

[0125] Preprocess the image, resize it according to different pre-trained models, and perform normalization operations to meet the input requirements of different models; at the same time, perform preliminary screening and compression on the collected images at the edge to reduce data transmission volume;

[0126] Select different pre-trained models and use the interfaces provided by MMDETECTION to perform preliminary image annotation, including:

[0127] Load the pre-trained model: Use the MMDETECTION configuration file and pre-trained weights to load the model; input the pre-processed image into the model for inference to obtain preliminary annotation boxes; manually review the inference results, manually drag the box, and correct the position of the box to obtain the corrected annotation results; save the annotation results;

[0128] Use labeled data to retrain multiple incremental training models built using digital twin technology.

[0129] S5: Calculate the accuracy of the incrementally trained models and compare them with the accuracy of the original model.

[0130] It is understood that, in this embodiment, respectively calculating the accuracy of the incrementally trained models and comparing them with the accuracy of the original models includes:

[0131] The accuracy of each model is evaluated by the average intersection-over-union ratio, and the formula is:

[0132] ;

[0133] in, represents the total number of categories, The true category is And the predicted category is The number of samples, The true category is And the predicted category is The number of samples.

[0134] S6: If the comparison result is higher than the threshold, the trained model is deployed to each edge node to update the original model.

[0135] Specifically, in the incremental learning scenario, when the accuracy of the incrementally trained model improves significantly, it is deployed to the edge node to update the original model. A specific threshold can be set, and the accuracy is compared with the threshold. The comparison between the accuracy and the threshold determines whether to participate in the global model update.

[0136] It is understood that in this embodiment, AI model training samples primarily come from data collected at the edge. The central cloud and edge nodes collaborate to improve model accuracy through incremental learning. After the model is deployed on the edge node and runs inference, samples with low confidence in the inference results are automatically identified and sent to the central cloud. Manual or other systems assist in labeling these samples, and the model is then incrementally retrained. If the incrementally trained model demonstrates significant improvement in accuracy over the original model, it can be deployed to the edge node to update the original model.

[0137] Example 2.

[0138] like Figure 6 As shown, the present application provides an incremental learning system for defect images under drone cloud-edge collaboration, which is applied to the incremental learning method for defect images under drone cloud-edge collaboration as described in Example 1, including: a cloud-edge collaboration platform construction module 11, a data acquisition module 12, a first labeling module 13, a second labeling module 14, an accuracy calculation module 15, and a model update module 16.

[0139] It can be understood that in this embodiment, the cloud-edge collaborative platform construction module 11 is used to build a distributed cloud-edge collaborative transmission line defect identification platform, deploy an inference and semi-automatic labeling system on the edge, and build a multi-model incremental training system on the cloud through digital twin technology.

[0140] It can be understood that in this embodiment, the data acquisition module 12 is used to realize real-time collection of line temperature, sag, and vibration status quantities through the drone using low-power wide-area Internet of Things technology, and upload the collected data to the edge side.

[0141] It can be understood that in this embodiment, the first labeling module 13 is used to carry out the reasoning and semi-automatic labeling system on the edge side to provide the required multiple pre-trained models, adopt the first crowdsourcing method to distribute the uploaded collected data, perform the pre-labeling task, and summarize the samples with low confidence in the automatic identification reasoning results, package them and send them to the central cloud.

[0142] It can be understood that in this embodiment, the second labeling module 14 is used to label samples manually or with the assistance of LabelStudio software in the central cloud, and then retrain multiple incremental training models constructed through digital twin technology.

[0143] It can be understood that, in this embodiment, the accuracy calculation module 15 is used to calculate the accuracy of the incrementally trained model and compare it with the accuracy of the original model.

[0144] It can be understood that, in this embodiment, the model updating module 16 is used to deploy the trained model to each edge node to update the original model if the comparison result is higher than the threshold.

[0145] The present application provides a method and system for incremental learning of defect images under drone-cloud-edge collaboration, constructs a distributed cloud-edge collaborative transmission line defect recognition platform, deploys an inference and semi-automatic labeling system on the edge, and constructs a multi-model incremental training system on the cloud through digital twin technology; uses low-power wide-area Internet of Things technology through drones to realize real-time collection of line temperature, sag, and vibration state quantities, and uploads the collected data to the edge side; carries out inference and semi-automatic labeling system on the edge side to provide the required multiple pre-trained models, distributes the uploaded collected data in a first crowdsourcing manner, performs pre-labeling tasks, and summarizes samples with low confidence in automatic identification reasoning results, packages them and sends them to the central cloud; in the central cloud, samples are assisted in labeling by manual labor or Label Studio software, and then multiple incremental training models constructed by digital twin technology are retrained; the accuracy of the incrementally trained models is calculated respectively and compared with the accuracy of the original models; if the comparison result is higher than the threshold, the trained models are deployed to each edge node to update the original model.

[0146] The main work to effectively improve the problems in drone image recognition is to complete the following two parts: first, to achieve incremental training of the transmission line defect recognition model under cloud-edge collaboration; second, to research edge-end model reasoning and semi-automatic labeling technology and equipment.

[0147] Through the collaboration between the central cloud and edge nodes, model accuracy can be improved through incremental learning. After the model is deployed on the edge node and runs inference, samples with low confidence in the inference results are automatically identified and sent to the central cloud. Manual or other systems assist in labeling these samples, and the model is then incrementally retrained. If the incrementally trained model demonstrates significant improvement in accuracy over the original model, it can be deployed to the edge node to update the original model.

[0148] Through edge model reasoning and semi-automatic labeling technology and equipment, labeling work is made more efficient, and samples are assisted by manual or other systems to make labeling more accurate.

[0149] Through crowdsourcing, a mechanism is implemented for pipeline tasks, which divides the semi-automatic labeling task into multiple small pipeline tasks. The pipeline tasks are executed separately through the combination of candidate models. The pipeline operation not only makes the labeling work more efficient, but also integrates the characteristics of different models to make the obtained labeling results more accurate.

[0150] It is easy to promote and apply, and the operation is relatively simple, making it suitable for promotion and application in more application scenarios.

[0151] To address the challenges of defect image recognition during drone inspections, we proposed an incremental learning method for defect images based on a cloud-edge collaborative architecture. This method uses edge computing nodes to achieve real-time defect detection and leverages the powerful computing power of the cloud for incremental model updates. We also designed an efficient data transmission and model update mechanism. Experimental results demonstrate that this method can effectively adapt to emerging defect types while ensuring real-time detection, significantly improving defect recognition accuracy and system adaptability.

[0152] Figure 7 This is an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device includes at least the following parts: a processor 101 and a memory 100 , a communication interface 103 , and a bus 102 .

[0153] In the embodiment of the present application, the memory 100 is used to store instructions executable by the processor 101. The processor 101 is configured to execute the instructions to implement the following Figure 6 The device module shown is for incremental learning of defect images under drone-cloud-edge collaboration.

[0154] In an embodiment of the present application, a computer-readable storage medium includes instructions, and the instructions instruct a device to execute the method of the first aspect. For example, the instructions instruct the device to execute Figure 1 The method is shown in the process steps.

[0155] The program running in the electronic device involved in one embodiment of the present application may be a program that controls a central processing unit (CPU) and the like to implement the functions of the above-mentioned embodiment involved in one embodiment of the present invention (a program that causes a computer to function). The information processed by these devices is temporarily stored in random access memory (RAM) during processing, and then stored in various ROMs such as read-only memory (Flash ROM) and hard disk drives (HDDs), where it is read, modified, and written as needed by the CPU.

[0156] It should be noted that a portion of the electronic device of the above embodiment may also be implemented by a computer. In this case, a program for implementing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer and executed.

[0157] It should be noted that the "computer" mentioned here refers to a computer built into an electronic device, employing hardware including an operating system (OS) and peripheral devices. Furthermore, "computer-readable recording medium" refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computers.

[0158] Furthermore, "computer-readable recording media" may include: media that dynamically store programs for a short period of time, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines; and media that store programs for a fixed period of time, such as volatile memory within computers acting as servers or clients in this context. Furthermore, the aforementioned program may be a program for implementing a portion of the aforementioned functions, or a program that can achieve the aforementioned functions by combining with a program already stored in a computer.

[0159] Furthermore, the electronic device in the above-described embodiments can also be implemented as a collection of multiple devices (a device group). Each device comprising the device group may include some or all of the functions or functional blocks of the electronic device in the above-described embodiments. A device group only needs to include all of the functions or functional blocks of the electronic device.

[0160] Those skilled in the art should recognize that the above embodiments are merely intended to illustrate the present application and are not intended to limit the present application. As long as they are within the spirit of the present application, appropriate changes and modifications to the above embodiments are within the scope of protection claimed in the present application.

Claims

1. A method for incremental learning of defect images under UAV-cloud-edge collaboration, characterized by: The method comprises: S1: Build a distributed cloud-edge collaborative transmission line defect identification platform, deploy inference and semi-automatic annotation systems on the edge, and build a multi-model incremental training system on the cloud using digital twin technology; S2: Using drones and low-power wide-area IoT technology, the system collects line temperature, sag, and vibration status in real time and uploads the collected data to the edge. S3: The edge-side inference and semi-automatic annotation system provides the required pre-trained models. It uses crowdsourcing to distribute the uploaded collected data, performs pre-annotation tasks, and aggregates samples with low confidence in the automatic identification and inference results, packaging them and sending them to the central cloud. S4: Samples are labeled manually or with the assistance of Label Studio software in the central cloud, and then multiple incremental training models built using digital twin technology are retrained; S5: Calculate the accuracy of the incrementally trained models and compare them with the accuracy of the original model; S6: If the comparison result is higher than the threshold, the trained model is deployed to each edge node to update the original model; In S1, a distributed cloud-edge collaborative transmission line defect identification platform is constructed, and a reasoning and semi-automatic annotation system is arranged on the edge end, including: the reasoning and semi-automatic annotation system includes a data input module, a model annotation module, an RTMDet reasoning service module and an annotation output module; the data input module is used to receive image data to be annotated; the model annotation module uses different pre-trained models to perform preliminary annotation on the image; the RTMDet reasoning service module optimizes and adjusts the preliminary annotation results; the annotation output module saves and outputs the final annotation results; The multi-model incremental training system includes a digital twin layer, a model management layer, and an incremental training engine; the digital twin layer is used to mirror the model physical structure according to the model of the model annotation module to obtain multiple mirror models; the model management layer is used to perform version control, performance monitoring, and model dependency management on the multiple mirror models; the incremental training engine is used to adaptively adjust the learning rate through an online learning algorithm library and a distributed training framework; S3: The edge side is equipped with an inference and semi-automatic annotation system to provide the required multiple pre-trained models, distribute the uploaded collected data in a crowdsourcing manner, perform pre-annotation tasks, and summarize samples with low confidence in the automatic identification reasoning results, package them and send them to the central cloud, including: The multiple pre-trained models include at least three models: a CNN model, an improved dual Unet model, and a Logits distillation model; the first crowdsourcing method is a pipeline task implementation mechanism; The semi-automatic labeling task is divided into a plurality of pipeline tasks; a corresponding number of pre-trained models are selected from the plurality of pre-trained models according to the number of divided small tasks, and the pre-trained models are arranged and combined to obtain a plurality of candidate model combinations, and the pipeline tasks are executed respectively; The optimal model combination is selected based on the confidence of the automatic recognition reasoning results, and the samples with low confidence are filtered out from the recognition results corresponding to the optimal model combination, summarized, packaged and sent to the central cloud.

2. The method for incremental defect image learning under UAV-cloud-edge collaboration according to claim 1 is characterized in that: In S1, a distributed cloud-edge collaborative transmission line defect identification platform is built, an inference and semi-automatic annotation system is deployed on the edge, and a multi-model incremental training system is built on the cloud using digital twin technology. This also includes: The distributed cloud-edge collaborative transmission line defect identification platform includes: a central cloud, an edge terminal, and a drone. Among them, the edge terminal includes field equipment, an edge gateway, and an edge cloud. A multi-level distributed management architecture is used to achieve interconnection between edge nodes. Based on the conversion of the private communication protocols used by various field devices into the standardized OPC UA protocol, computing tasks are distributed to multiple nodes for parallel processing, realizing the access and data collection, transmission, and processing of heterogeneous devices.

3. The method for incremental defect image learning under UAV-cloud-edge collaboration according to claim 2 is characterized in that: According to the number of divided small tasks, a corresponding number of pre-trained models are selected from the multiple pre-trained models for permutation and combination to obtain multiple candidate model combinations, and the pipeline tasks are executed respectively; The optimal model combination is selected based on the confidence level of the automatic recognition reasoning results. The samples with low confidence levels are filtered out from the recognition results corresponding to the optimal model combination, aggregated, and packaged and sent to the central cloud, including: The number of the divided small tasks is three; Selecting, from the plurality of candidate model combinations, a first model in the combination to perform the task of labeling the type of the target object, a second model to perform the task of labeling the number and position of the target object, and a third model to perform the task of labeling the contour boundary of the target object; The optimal candidate model combination is selected based on the weighted sum of the accuracy of the three execution results as the confidence level; The optimal candidate model combination is used as the target model combination, and a dependency relationship is established between the models in the target model combination. The first model in the target model combination is used again to perform the task of labeling the type of target object, the second model performs the task of labeling the number and position of target objects, and the third model performs the task of labeling the contour boundary of the target object based on the execution result of labeling the type of target object combined with the execution result of labeling the number and position of target objects. The recognition results corresponding to the optimal model combination are filtered out and the samples with low confidence are summarized, packaged and sent to the central cloud.

4. The method for incremental learning of defect images in UAV-cloud-edge collaboration according to claim 3 is characterized in that: S4: Samples are labeled manually or with the assistance of Label Studio software in the central cloud, and then multiple incremental training models built using digital twin technology are retrained, including: Collect an image dataset containing transmission line defects and divide it into training and test sets; Preprocess the image, resize it according to different pre-trained models, and perform normalization operations to meet the input requirements of different models; at the same time, perform preliminary screening and compression on the collected images at the edge to reduce data transmission volume; Select different pre-trained models and use the interfaces provided by MMDETECTION to perform preliminary image annotation, including: Load the pre-trained model: Use the MMDETECTION configuration file and pre-trained weights to load the model; input the pre-processed image into the model for inference to obtain preliminary annotation boxes; manually review the inference results, manually drag the box, and correct the position of the box to obtain the corrected annotation results; save the annotation results; Use labeled data to retrain multiple incremental training models built using digital twin technology.

5. The method for incremental learning of defect images in UAV-cloud-edge collaboration according to claim 1 is characterized in that: The improved dual Unet model structure is an asymmetric Unet model, including UNet1 and UNet2, wherein the encoders of UNet1 and UNet2 extract low-level and deep-level semantic features respectively, and the low-level and deep-level semantic features are extracted in the decoding layer for interaction and fusion, and the prediction results are output. The loss function is the cross entropy loss function. and Dice loss function The joint loss function composed of is: ； in, Representation sample The true distribution of Representation sample The predicted distribution of represents the total number of samples; ； in, Respectively The probability value of the class prediction and the true label value, is the number of categories, is the total number of categories, Indicates the Class weights; Then the joint loss function Expressed as: ； The Logits distillation model structure consists of three parallel network models, namely the standard teacher network model, the student network model, and the adversarial teacher network model. The standard teacher network model adopts the UNet network structure based on the attention mechanism as the backbone network, and embeds the attention mechanism into the decoding part of the UNet network. The student network model adopts the lightweight UNet network structure. The adversarial teacher network model adopts the U-CliqueNet model structure, and replaces the convolution layer of UNet with the Clique Block. The loss function consists of hard loss and soft loss, wherein the hard loss function is the cross entropy loss function, which is: ； in, Representation sample The true distribution of Representation sample The predicted distribution of represents the total number of samples; The soft loss function is : ； in, represents the temperature parameter, They represent the probability distributions of the standard teacher network model, the student network model, and the adversarial teacher network model after high temperature softening, is the total number of categories, is the divergence; The total loss function is : ； in, represents the hyperparameters for adjusting the weights of hard loss and soft loss, .

6. The method for incremental learning of defect images in UAV-cloud-edge collaboration according to claim 4 is characterized in that: The accuracy of each incrementally trained model is calculated and compared with the accuracy of the original model, including: The accuracy of each model is evaluated by the average intersection-over-union ratio, and the formula is: ， in, represents the total number of categories, The true category is And the predicted category is The number of samples, The true category is And the predicted category is The number of samples.

7. A system for incremental learning of defect images in a UAV-cloud-edge collaborative environment, applied to the method for incremental learning of defect images in a UAV-cloud-edge collaborative environment as claimed in any one of claims 1 to 6, characterized in that: include: Cloud-edge collaborative platform construction module: Build a distributed cloud-edge collaborative transmission line defect identification platform, deploy inference and semi-automatic annotation systems on the edge, and build a multi-model incremental training system on the cloud using digital twin technology; Data acquisition module: Using low-power wide-area IoT technology via drones, it can collect line temperature, sag, and vibration status in real time, and upload the collected data to the edge side. The first labeling module: The edge side is equipped with an inference and semi-automatic labeling system to provide the required multiple pre-trained models. It uses the first crowdsourcing method to distribute the uploaded collected data, perform pre-labeling tasks, and summarize samples with low confidence in the automatic identification reasoning results, package them and send them to the central cloud; Second labeling module: Samples are labeled manually or with the assistance of Label Studio software in the central cloud, and then multiple incremental training models built using digital twin technology are retrained; Accuracy calculation module: calculates the accuracy of the incrementally trained model and compares it with the accuracy of the original model; Model update module: If the comparison result is higher than the threshold, the trained model is deployed to each edge node to update the original model.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the incremental learning method for defect images under drone-cloud-edge collaboration as described in any one of claims 1 to 6 when executing the instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and the program instructs the device to execute the incremental learning method for defect images under drone-cloud-edge collaboration as described in any one of claims 1 to 6.

Images