End-cloud cooperative optimization method and device, computer device, and storage medium

Abstract

The application discloses an end-cloud cooperative optimization method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining original aerial photograph image data and preprocessing to obtain preprocessed data; a student model outputs inference information; the inference information is evaluated to obtain detection samples, and the detection samples are divided into normal samples or difficult example samples; encapsulated samples are obtained by encapsulation; review results are generated by review; cleaned samples are obtained by cleaning; knowledge distillation and model retraining are started to obtain an optimized student model; the optimized student model is cascaded and compressed to obtain a compressed model; an incremental update mechanism is used for the compressed model to generate a model differential patch package, and the differential patch package is transmitted to a UAV for model updating and deployment. The method of the application can solve the problems of real-time performance, detection accuracy, scene generalization ability and model updating efficiency, and realize comprehensive optimization and performance improvement of the UAV intelligent inspection system.

Description

Technical Field

[0001] This invention relates to the field of edge-cloud collaborative optimization technology, and more specifically to edge-cloud collaborative optimization methods, apparatus, computer equipment, and storage media. Background Technology

[0002] With the deep integration and rapid development of cutting-edge technologies such as aviation, artificial intelligence, and cloud computing, drones are increasingly widely used in fields such as power line inspection, traffic management, and emergency rescue. Equipped with various types of sensors, drones can quickly collect large-scale image or video data in complex environments, providing crucial technical support for intelligent perception and decision support. However, how to efficiently, accurately, and in real-time intelligently analyze inspection data remains one of the key technical challenges. Currently, intelligent processing of drone inspection data mainly employs two methods: First, offline processing, where the drone uploads the data to a high-performance cloud server for analysis after completing its inspection task. While this method can utilize the ample computing resources of the cloud to deploy high-precision deep learning models, significant latency exists in data processing, making it difficult to meet the time-sensitive requirements of applications such as emergency response and real-time early warning. Second, real-time image transmission processing, where the video stream collected by the drone is transmitted back to the ground station or cloud for online analysis via an image transmission link. This method improves response speed to some extent, but it places high demands on the bandwidth and stability of the communication link. In complex terrain, signal obstruction, or unstable network conditions, data packet loss, transmission delays, and even interruptions are likely to occur. To address these issues, edge computing solutions have been gradually introduced in recent years. These solutions deploy lightweight models directly on embedded devices on drones to reduce communication load and improve system response speed, enabling real-time front-end detection and analysis. However, limited by drone payload, power consumption, and onboard computing power, existing edge intelligent detection solutions still face the following key technical challenges: (1) The contradiction between limited airborne computing power and detection accuracy. The airborne computing platform of UAV has limited resources and can usually only deploy lightweight models that have been pruned and quantized. Compared with the large-scale deep learning models deployed on high-performance servers in the cloud, such lightweight models are significantly less accurate in scenarios with complex backgrounds and small targets, and are prone to high false negative and false positive rates; (2) Insufficient scene generalization ability. The environment of UAV inspection operations is complex and variable, and is affected by various factors such as lighting conditions, shooting angle and background changes. Existing models perform well on the validation set, but their performance often drops significantly in real new scenarios, making it difficult to meet the requirements for long-term, cross-scenario stable operation; (3) Lagging model updates and iterations. When the performance of the detection model does not meet expectations in practical applications, the existing update process usually requires steps such as data collection, manual annotation, model training and manual deployment. The process cycle is long and costly, making it difficult to achieve rapid model iteration and on-site adaptive optimization.

[0003] In summary, existing UAV intelligent inspection technologies struggle to achieve an effective balance between real-time performance, detection accuracy, and environmental adaptability. Therefore, there is an urgent need to develop an edge-cloud collaborative intelligent inspection method that leverages the low latency advantages of edge computing while integrating the powerful computing and modeling capabilities of the cloud, and supports automatic and continuous model optimization. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, computer equipment and storage medium for edge-cloud collaborative optimization.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: Edge-cloud collaborative optimization methods include: Acquire raw aerial image data and perform preprocessing operations on the image data to obtain preprocessed data; Preprocessed data is input into a student model deployed on a drone for real-time inference to output inference information; The credibility of the reasoning information is assessed to obtain the test samples, and the test samples are divided into normal samples or difficult samples. The difficult sample is encapsulated to obtain an encapsulated sample, and the encapsulated sample is sent back to the cloud. The teacher model is used in the cloud to review the encapsulated samples in order to generate review results; The verification results are cleaned to obtain a cleaned sample; When the number of cleaned samples accumulated in the cloud meets the preset threshold, knowledge distillation and model retraining are initiated to obtain the optimized student model. The optimized student model is then subjected to cascaded compression to obtain a compressed model; The compressed model is updated using an incremental update mechanism based on binary difference to generate a model difference patch package, which is then transmitted to the drone for model update and deployment.

[0006] The present invention also provides an edge-cloud collaborative optimization device, comprising: The preprocessing unit is used to acquire raw aerial image data and perform preprocessing operations on the image data to obtain preprocessed data. The input / output unit is used to input preprocessed data into the student model deployed on the drone for real-time inference, and output inference information. The evaluation and segmentation unit is used to evaluate the credibility of the reasoning information to obtain the test samples and classify the test samples into normal samples or difficult samples. The encapsulation and return unit is used to encapsulate difficult example samples to obtain encapsulated samples and return the encapsulated samples to the cloud. The verification unit is used in the cloud to verify the packaged samples using the teacher model in order to generate verification results; The cleaning unit is used to clean the data of the verification results to obtain a cleaned sample; The training unit is used to initiate knowledge distillation and model retraining when the number of cleaned samples accumulated in the cloud meets a preset threshold, so as to obtain an optimized student model. The compression unit is used to perform cascade compression on the optimized student model to obtain a compressed model; The update deployment unit is used to generate a model differential patch package by using an incremental update mechanism based on binary difference for the compressed model, and then transmits the differential patch package to the UAV for model update and deployment.

[0007] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0008] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0009] The beneficial effects of this invention compared with existing technologies are as follows: By combining real-time detection on the UAV end with high-precision verification in the cloud, and innovative technologies such as knowledge distillation and incremental updates, this invention effectively solves the shortcomings of existing UAV intelligent inspection technologies in terms of real-time performance, detection accuracy, scene generalization ability, and model update efficiency. It achieves comprehensive optimization and performance improvement of the UAV intelligent inspection system, providing strong technical support for the widespread application of UAVs in fields such as power line inspection, traffic management, and emergency rescue.

[0010] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a schematic diagram illustrating an application scenario of the edge-cloud collaborative optimization method provided in this embodiment of the invention. Figure 2 This is a flowchart illustrating the edge-cloud collaborative optimization method provided in an embodiment of the present invention. Figure 3 This is a schematic block diagram of the end-to-cloud collaborative optimization device provided in an embodiment of the present invention; Figure 4 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the edge-cloud collaborative optimization method provided in an embodiment of the present invention. Figure 2 This is a schematic flowchart illustrating the edge-cloud collaborative optimization method provided in this embodiment of the invention. This method is applied to a server that interacts with a terminal. Through the collaborative work of real-time detection by the UAV and high-precision verification in the cloud, combined with innovative technologies such as knowledge distillation and incremental updates, it effectively solves the shortcomings of existing UAV intelligent inspection technologies in terms of real-time performance, detection accuracy, scene generalization ability, and model update efficiency, achieving comprehensive optimization and performance improvement of the UAV intelligent inspection system.

[0017] Figure 2 This is a flowchart illustrating the edge-cloud collaborative optimization method provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S190.

[0018] S110. Acquire raw aerial image data and perform preprocessing operations on the image data to obtain preprocessed data; Specifically, when performing inspection missions, drones acquire raw aerial image data through various types of sensors (such as cameras). Preprocessing operations are then performed on the acquired image data, including but not limited to grayscale conversion, normalization, cropping, and resizing, to eliminate noise and standardize the data format, thereby obtaining preprocessed data suitable for subsequent processing.

[0019] In other words, preprocessing can improve the quality and consistency of image data, reduce interference factors in subsequent processing, and improve the accuracy and efficiency of model inference.

[0020] In one embodiment, the step of acquiring raw aerial image data and performing preprocessing operations on the image data to obtain preprocessed data includes: The drone continuously photographs the inspection area using its onboard visible light imaging equipment to obtain raw aerial image data; Specifically, the drone is equipped with an onboard visible light imaging device (such as a high-definition camera) to ensure sufficient resolution and frame rate to meet the needs of the inspection mission. Before takeoff, the flight path and shooting parameters (such as shooting frequency and focal length) are set so that the drone can continuously photograph the inspection area along the preset trajectory. During flight, the onboard visible light imaging device collects image data in real time, and this image data is stored in the drone's onboard storage device as raw aerial image data.

[0021] In other words, continuous shooting ensures that the drone acquires a complete image sequence during the inspection process, covering the entire inspection area and avoiding the omission of important information. At the same time, high-resolution and high-frame-rate imaging equipment can provide clear and detailed image data, providing a high-quality foundation for subsequent processing and analysis.

[0022] The acquired raw aerial image data is normalized in size, and the image is adjusted to a preset fixed size to obtain a normalized image. Specifically, the raw aerial image data is loaded into the image processing module of the UAV's onboard equipment or ground station. A preset fixed size (e.g., 640 pixels wide and 480 pixels high) is set as the standard size for subsequent processing. Each raw image undergoes size normalization processing. Specifically, if the original image size is larger than the preset size, it is reduced to the preset size using bilinear interpolation or other image scaling algorithms. If the original image size is smaller than the preset size, it is adjusted to the preset size using appropriate padding (such as zero padding or mirror padding) and scaling operations. The size-normalized image is saved as an intermediate processing result for subsequent steps.

[0023] In other words, size normalization ensures that all images have a uniform size format, which allows image data to be processed in a consistent manner, avoiding processing errors or inefficiencies caused by size differences. A uniform size format also facilitates the input requirements of subsequent models, improving the efficiency and accuracy of model inference.

[0024] The image after size normalization is converted to color space to a standard RGB three-channel format to obtain preprocessed data.

[0025] Specifically, in the image processing module, the size-normalized image data is loaded. The image's color space format is then checked; if the image is not in RGB format (e.g., it might be a grayscale image or another color space format), a color space conversion is performed. Using a color space conversion algorithm (such as an extension from grayscale to RGB, or a conversion from another color space to RGB), the image is converted to a standard RGB three-channel format. The converted RGB image is saved as the final preprocessed data for subsequent model inference or other processing steps.

[0026] In other words, the RGB three-channel format is the standard input format for most image processing and deep learning models. Converting images to this format ensures that the model can correctly parse and process image data. At the same time, a unified color space format helps improve the model's generalization ability and processing efficiency, and reduces errors caused by differences in color spaces.

[0027] More specifically, firstly, during the inspection mission, the UAV continuously captures aerial images of the inspection area using its onboard visible light imaging equipment, obtaining real-time aerial image data. The acquired images undergo preprocessing operations such as size normalization and color space conversion to ensure the input data format is consistent with the training and inference requirements of the edge-side student model. Secondly, the preprocessed images are input into the student model deployed on the UAV's edge computing unit for real-time inference. This student model is built on an improved YOLOv8n network, capable of rapidly detecting targets in the inspection images under limited computing power, and outputting corresponding target category information, bounding box coordinates, and detection confidence. Subsequently, the confidence of the student model's detection results is evaluated based on a pre-designed uncertainty assessment mechanism, dividing the detection samples into normal samples and difficult samples. Finally, the difficult samples are uniformly packaged and transmitted back to the cloud. The packaged content includes the original image data, the student model's inference results, and the detection confidence information.

[0028] Image acquisition and preprocessing include: During the inspection mission, the UAV continuously captures images of the inspection area using an onboard visible light imaging device, obtaining raw aerial image data. To ensure that the input image format remains consistent with the input format of the edge-end student model during training and inference, preprocessing is performed on the raw aerial images after image acquisition. The specific steps are as follows: Color space conversion: Converting aerial images captured by drones into standard RGB three-channel images, the calculation formula is as follows: ; in, This represents the original aerial image. Represents a three-channel RGB image. This represents a color space conversion operator used to eliminate color distribution differences introduced by different imaging devices or encoding methods.

[0029] Image size normalization: To adapt to the fixed input size of the student model at the edge, the RGB image is scaled proportionally. The calculation formula is as follows: ; ; Where H and W represent the height and width of the original aerial image, respectively. in W in Let represent the height and width of the input image required by the student-side model, respectively, and s represent the scaling factor. This represents the image scaling transformation function. This represents an image that has been scaled proportionally.

[0030] S120. Input the preprocessed data into the student model deployed on the drone for real-time reasoning, and output reasoning information; Specifically, an optimized, lightweight student model is deployed on an embedded device mounted on a drone. Preprocessed image data is input into the student model, which performs real-time inference based on its trained parameters and algorithms, outputting inference information such as object detection results and classification results.

[0031] In other words, real-time inference can respond quickly, meet the real-time requirements of UAVs in complex environments, and provide preliminary results for subsequent credibility assessment.

[0032] In one embodiment, the step of inputting preprocessed data into a student model deployed on a drone for real-time inference to output inference information includes: The preprocessed image data is input into a lightweight student model deployed on the edge computing unit of the drone. The student model performs real-time inference on the input image data and outputs the target category information, bounding box coordinates, and detection confidence.

[0033] Specifically, a lightweight student model is deployed on the edge computing unit of the drone (such as an onboard embedded computing device). Preprocessed image data (such as RGB images after size normalization and color space conversion) is then transferred from the storage unit to the edge computing unit. The preprocessed image data is then passed as input to the student model by calling its interface. This typically involves formatting the image data into the input format required by the model (such as a tensor format).

[0034] After receiving preprocessed image data, the student model performs forward propagation computation through its internal neural network structure. This process includes multiple convolutional layers, activation layers, pooling layers, and other operations to extract features from the image.

[0035] Object detection: The model uses features learned during training to identify target objects in an image and determine the category information, bounding box coordinates, and detection confidence of each target.

[0036] Category information: The model outputs the category to which each detected target belongs (such as "telephone pole", "vehicle", "pedestrian" etc.).

[0037] Bounding box coordinates: The model outputs the position of the target in the image, usually represented by a bounding box, including the coordinates of the top left and bottom right corners.

[0038] Detection confidence: The confidence score of each detection result output by the model represents the degree to which the model believes the detection result. It is usually a value between 0 and 1.

[0039] The category information, bounding box coordinates, and detection confidence obtained from inference are encapsulated into a structured data format (such as JSON) for easy subsequent processing and transmission. The encapsulated inference results are then stored in the UAV's onboard storage unit and transmitted to a ground station or other processing units for further analysis as needed.

[0040] More specifically, object detection is a core task in computer vision, aiming to automatically identify and locate specific targets in images or videos. Considering the combined requirements of real-time performance, lightweight design, and detection accuracy in UAV inspection applications, a single-stage object detection framework was chosen as the technical foundation for the edge (i.e., UAV) model. This framework exhibits significantly faster inference speed than two-stage detectors and is more suitable for edge devices with limited computing resources. Among various single-stage detectors, the YOLOv8 series models achieve a good balance between detection accuracy and inference efficiency. Based on this, its lightweight version, YOLOv8n, was selected as the foundation and further optimized as the edge student model to achieve efficient real-time inference under limited onboard computing power. The YOLOv8n network mainly consists of a backbone network, a neck network, and a head network. The backbone network is used to extract multi-level semantic features from the original input image; the neck network uses a feature pyramid and path aggregation mechanism to perform bidirectional fusion of feature maps from different levels of the backbone network from top to bottom and from bottom to top, so as to enhance the model's ability to perceive targets at different scales; the head network serves as the output layer of the detection task and is used to generate the bounding box coordinates, confidence scores, and category prediction results of the targets.

[0041] To further adapt to the stringent constraints of power consumption and computing power on UAV-borne embedded platforms, while also considering detection accuracy and model generalization ability, the YOLOv8n model has been optimized and improved as follows: A lightweight convolutional module is introduced to reconstruct the backbone feature extraction network; Considering the limited computing resources of edge devices, while the C2f module in the original YOLOv8n has strong feature fusion capabilities, its standard convolutional operations still result in a large number of parameters and computational overhead. Therefore, a lightweight C2f-DS module is constructed in the backbone network to replace the traditional C2f module. This significantly reduces the number of model parameters and computational complexity while maintaining effective feature extraction, thereby improving the inference speed of the edge model.

[0042] The C2f-DS module achieves a lightweight network structure design by replacing standard convolutional units with depthwise separable convolutional units. A depthwise separable convolutional unit consists of depthwise convolution and pointwise convolution operations. The depthwise convolution operation uses a 3×3 kernel with a stride of 1 to perform convolution operations on each channel of the input feature map, independently extracting spatial features from each channel without changing the number of channels. The pointwise convolution operation uses a 1×1 kernel with a stride of 1 to linearly combine features along the channel dimension, enabling information fusion between feature channels and dimensionality adjustments (upgrading or reducing the number of channels). To further demonstrate the lightweight computational advantages of the C2f-DS module, the following analysis is performed: Assume the input feature map size is H×W and the number of input channels is C.in The number of output channels is C out The kernel size is K×K (K=3). When using standard convolution, the computational cost F is... std It can be represented as: ; When using depthwise separable convolution, its computational cost F ds It can be represented as: ; The ratio of the computational complexity of the two can be expressed as: ;

[0043] When the number of output channels C out When the kernel size is large and K=3, the computational cost of depthwise separable convolution is about one-ninth that of standard convolution. This significantly reduces the computational complexity and computational power requirements of the model while ensuring feature representation capabilities, making it suitable for resource-constrained edge deployment scenarios.

[0044] Embedding multi-scale attention mechanisms enhances the ability to perceive small targets; To address the challenges of small targets being small in scale, having weak feature responses, and being easily overwhelmed by background information during multi-scale feature fusion from a UAV perspective, a Coordinate Attention (CA) mechanism is introduced in the neck feature fusion stage of the YOLOv8n network. This mechanism explicitly embeds spatial location information into the channel attention weights, enabling the model to focus more intently on the key region where the target is located during multi-scale feature fusion, effectively suppressing interference from complex backgrounds. This improves the detection accuracy of small and micro-targets without significantly increasing the model's computational complexity.

[0045] The specific implementation process of the Coordinate Attention (CA) module is as follows: Coordinate information embedding: Given an input feature tensor X, global average pooling encoding is performed on the input features along both the horizontal and vertical directions to extract direction-aware features containing location information. The aggregated feature of the c-th channel at height h along the horizontal direction is also considered. The calculation formula is as follows: ; Aggregation feature at width position w of the c-th channel along the vertical direction The calculation formula is as follows: ; Among them, H f W f These represent the height and width of the feature map, respectively. This represents the feature value of the c-th channel of the input feature tensor X at spatial location (h, i); This represents the feature value of the c-th channel of the input feature tensor X at spatial location (j, w).

[0046] Coordinate attention generation: This involves combining the two directional features obtained above. , The features are concatenated in the spatial dimension to form a joint feature representation, which is then input into a shared 1×1 convolutional transformation function. Channel dimensionality reduction is performed, followed by nonlinear activation function. The intermediate features are obtained: ;

[0047] Next, the intermediate feature f is split into f_1, f_2, f_3, f_4, f_5, f_6, f_7, f_8, f_9, f_1 ... h and f w The inputs are then fed into two independent 1×1 convolution transformation functions. and In the process, the number of channels is restored to match the input features, and then passed through a non-linear activation function. Generate attention weights along the height and width directions: ; Feature weighted fusion: The generated direction-aware attention weights are then applied to the original input feature maps to achieve adaptive feature recalibration. The formula for calculating the output features is as follows: ; in, and These represent the values ​​of the input and output features at spatial positions (i, j) in channel c, respectively. Through this method, the coordinate attention mechanism introduces precise spatial location information while modeling the channels, enabling the model to maintain a higher response intensity to small target regions during the multi-scale feature fusion stage, thereby effectively improving the accuracy of small target detection in UAV aerial photography scenarios.

[0048] Optimize the bounding box regression loss function to improve positioning accuracy; The YOLOv8 model uses Complete Intersection over Union (CIoU) as the boundary regression loss function. However, when the overlap between the predicted and ground truth bounding boxes is high, CIoU still has limitations in gradient distribution and convergence efficiency, which can easily lead to limited improvement in localization accuracy. Therefore, a Weighted Intersection over Union (WIoU) loss function is introduced to replace CIoU. By constructing a dynamic non-monotonic focusing mechanism, the gradient weights of samples are adaptively adjusted according to the anchor box quality, effectively reducing the interference of low-quality samples on the model training process, thereby improving the boundary box localization accuracy and generalization ability of the edge-end model in complex scenes. The implementation process of the WIoU loss function is as follows: Distance definition: Let the center of the predicted bounding box be (x, y), and the center of the ground truth bounding box be (x, y). gt y gt The width and height of the minimum bounding rectangles of the two are W and W respectively. g H g The distance attention factor R introduced in WIoU WIoU Defined as: ; By normalizing the center offset between the predicted bounding box and the ground truth bounding box, the model pays more attention to the localization error during the regression process.

[0049] Outlier definition: To measure the quality of anchor boxes, an outlier parameter β is introduced to characterize the degree of overlap between the predicted and ground truth boxes relative to the current training state. First, the intersection-union ratio (IoU) between the predicted and ground truth boxes is L. IoU For IoU loss, L IoU =1-IoU, then the outlier degree is defined as follows: ; in, This represents the momentum moving average of the IoU loss for all samples in the current batch. This represents the IoU loss value corresponding to the current predicted bounding box. A smaller β indicates a higher quality predicted bounding box; a larger β indicates a lower quality predicted bounding box.

[0050] Dynamic non-monotonic focusing coefficient: To avoid the model over-focusing on simple or extremely difficult samples, a non-monotonic focusing coefficient r is introduced to dynamically adjust the weights of different samples during training. Its expression is: ; Here, α and δ are hyperparameters used to control the shape and decay rate of the focusing function. This non-monotonic mechanism can simultaneously suppress the gradient contributions of both high-quality and low-quality samples, allowing the model to focus on learning medium-quality effective samples, thereby improving training stability and convergence efficiency. Specifically, α is set to 2.0 and δ to 1.5. Through this parameter combination, the focusing weight reaches its maximum value when β is close to δ, enabling the model to prioritize learning medium-quality samples during training, thus achieving more stable gradient update results in complex scenarios.

[0051] WIoU Loss Function Construction: The WIoU bounding box regression loss function is defined by integrating distance, outlier, and dynamic non-monotonic focusing coefficients; where L is the WIoU bounding box regression loss function. WIoU The final definition is: ; Through the above improvements, the WIoU loss function maintains regression stability while achieving adaptive optimization for samples of different quality, effectively improving the positioning accuracy and robustness of the edge student model in complex drone scenarios.

[0052] S130. Evaluate the credibility of the reasoning information to obtain the test samples, and classify the test samples into normal samples or difficult samples. Specifically, credibility assessment criteria are set, such as confidence thresholds and detection accuracy indicators. The inference information output by the student model is evaluated to determine whether its credibility meets the preset criteria. Based on the evaluation results, the inference information is divided into normal samples (high credibility) or difficult samples (low credibility).

[0053] In other words, credibility assessment can screen out difficult cases that require further processing, avoiding sending all data back to the cloud and reducing communication load and the waste of computing resources.

[0054] In one embodiment, the step of evaluating the credibility of the inference information to obtain detection samples, and classifying the detection samples into normal samples or difficult samples, includes: The credibility of reasoning information is evaluated based on a multidimensional uncertainty assessment mechanism, including the detection confidence index and the category prediction information entropy index, in order to obtain detection samples; Specifically, the detection confidence score output by the student model is used as one dimension of the evaluation. The confidence score is a value between 0 and 1; a higher value indicates a higher level of confidence in the model's detection results. Additionally, the information entropy of the probability distribution of the category predictions is calculated as another dimension of the evaluation. Information entropy measures the uncertainty of the prediction results; a higher information entropy indicates greater uncertainty in the prediction results.

[0055] For each detected target, its category prediction probability distribution is extracted. The information entropy of these probability distributions is calculated using a specific algorithm to quantify the uncertainty of the prediction results. Using the detection confidence and information entropy as input, the inference information is comprehensively evaluated using pre-defined evaluation rules or algorithms (such as threshold judgment, weighted summation, etc.). Based on the evaluation results, the confidence level of each detection result is determined, and the inference information is labeled as a detection sample.

[0056] In other words, by combining the two dimensions of detection confidence and information entropy, the credibility of inference information can be assessed more comprehensively, avoiding misjudgments that may arise from a single indicator. At the same time, the introduction of information entropy makes the quantification of prediction uncertainty more precise, helping to distinguish between detection results with high confidence but high uncertainty, and improving the accuracy of the assessment.

[0057] Based on the preset confidence threshold and information entropy threshold, the detected samples are divided into normal samples or difficult samples.

[0058] Specifically, preset detection confidence thresholds and information entropy thresholds are used. These thresholds can be adjusted based on actual application scenarios and experience. For example, the detection confidence threshold can be set to 0.8, and the information entropy threshold to 1.0.

[0059] For each detected sample, it is classified according to its detection confidence and information entropy: If the detection confidence is greater than or equal to the confidence threshold and the information entropy is less than or equal to the information entropy threshold, the sample is classified as a normal sample, indicating that the model has high confidence in the detection result and the prediction result is relatively certain. If the detection confidence is less than the confidence threshold, or the information entropy is greater than the information entropy threshold, the sample is classified as a hard sample, indicating that the model has low confidence in the detection result or the prediction result has high uncertainty, requiring further processing or verification.

[0060] In other words, by setting a preset threshold, the detected samples can be accurately divided into normal samples and difficult samples, providing clear guidance for subsequent processing. Normal samples can be used directly as reliable detection results, while difficult samples require further processing (such as being sent back to the cloud for verification). This classification method effectively reduces the amount of data that needs to be sent back to the cloud, optimizes the use of communication and computing resources, and improves the overall efficiency of the system.

[0061] More specifically, a strategy for mining difficult cases and edge-cloud collaboration based on multidimensional uncertainty assessment; While the lightweight edge-side student model possesses real-time inference capabilities, its feature representation and discrimination abilities remain limited in drone inspection scenarios such as drastic lighting changes, complex backgrounds, or target occlusion, making it prone to missed or false detections. To address this issue, a difficult example mining mechanism based on multidimensional uncertainty assessment is designed. This mechanism quantitatively evaluates the detection results of the edge-side student model and adaptively distributes the detection data accordingly. High-confidence detection results are directly output at the edge, while difficult examples with higher uncertainty are filtered and sent back to the cloud for in-depth analysis and processing. This effectively reduces communication load and improves overall detection performance.

[0062] Construction of multidimensional uncertainty measurement indicators; To objectively and quantitatively evaluate the detection results of the student model, we abandon the traditional judgment method that relies solely on a single confidence threshold and introduce a multi-dimensional uncertainty measurement system that combines the detection confidence index and the category prediction information entropy index, so as to more comprehensively characterize the reliability of the model's detection results.

[0063] Maximum confidence index: Obtain the probability distribution of the target within the detection box belonging to each preset category from the student model output layer, and take the maximum probability value as the maximum confidence index C. maxThis metric measures the model's direct confidence in the existence of a target and its category; a higher value indicates a more definitive judgment by the model regarding the current detection result.

[0064] Category prediction entropy metric: To capture the uncertainty state of the model in the class determination process, especially when the target presents an ambiguous judgment among multiple categories (such as the model simultaneously believing that the target may belong to category A or category B), the information theory-based category prediction entropy metric H(x) is introduced, and the calculation formula is as follows: ; Where N is the total number of categories in the detection task, p(x i H(x) represents the probability value that the student model predicts the target belongs to the i-th category. The higher the value of H(x), the more uniform the category probability distribution of the model output is, the more uncertain the model's judgment of the target category is, and the lower the reliability of the corresponding detection results.

[0065] Adaptive difficult example selection and hierarchical response mechanism; Based on the aforementioned multidimensional uncertainty metrics, an adaptive screening and tiered response mechanism for normal and difficult samples is constructed. This is achieved by pre-setting a confidence threshold T. low and the upper limit threshold T for category prediction entropy high The detection results of each frame at the edge are evaluated and split in real time, as specifically implemented as follows: Normal sample processing: When the test results simultaneously meet C max ≥T low And H(x) < T high At this point, the system determines that the detection result has a high degree of reliability. The edge device then only performs structured encapsulation of the detection result, generating a lightweight result data packet, and sends the result data packet back to the cloud server, thereby reducing the need for transmitting the original image data.

[0066] Difficult sample: When the detection result meets C max <T low Or H(x)≥T high When the system determines the current detection scene to be a difficult example scene, the edge processing unit marks the original image frame containing the corresponding target as a difficult example sample and sends it back to the cloud server via a high-priority queue. After receiving the difficult example sample, the cloud server uses a teacher model with stronger representation and generalization capabilities to perform secondary inference and verification on the image of the difficult example sample, corrects the prediction results of the student model, and generates high-precision pseudo-label data for subsequent model distillation and incremental training processes.

[0067] S140. The difficult sample is encapsulated to obtain an encapsulated sample, and the encapsulated sample is sent back to the cloud. Specifically, the selected difficult examples are encapsulated, including sample data, inference information, and confidence assessment results. Furthermore, the encapsulated samples are transmitted back to the cloud via a reliable communication link to ensure data integrity and transmission stability.

[0068] In other words, encapsulation processing can completely transmit key information to the cloud, providing sufficient data support for cloud review and subsequent processing.

[0069] In one embodiment, the step of encapsulating the difficult example samples to obtain encapsulated samples and then sending the encapsulated samples back to the cloud includes: Difficult sample examples are uniformly packaged, including the original image data, the inference results of the student model, and the detection confidence information. Then, the packaged difficult sample examples are sent back to the cloud via a communication link.

[0070] Specifically, the original image data corresponding to the difficult examples is extracted to ensure the integrity and originality of the image data. Simultaneously, the inference results of the student model on the difficult examples are extracted, including target category information and bounding box coordinates. Furthermore, the detection confidence of the student model on the difficult examples is recorded for subsequent analysis and reference. In addition, a unified data encapsulation format is designed, typically using structured data formats (such as JSON, XML, etc.) to ensure that the encapsulated data is easy to parse and transmit. Finally, the above three contents (original image data, inference results, and detection confidence information) are organized and encapsulated according to a preset format to form a complete encapsulated sample.

[0071] Based on the drone's communication capabilities and the actual application scenario, select a suitable communication link (such as 4G / 5G network, satellite communication, etc.) to ensure sufficient bandwidth and stability, guaranteeing complete and accurate data transmission to the cloud. Send the encapsulated sample to the cloud via the selected communication link. During transmission, employ a reliable data transmission protocol (such as TCP / IP) to ensure data integrity. Set up a receiving port and service in the cloud to receive and store the returned encapsulated sample. Monitor the data transmission status during transmission to ensure successful data arrival at the cloud. In case of transmission failure or data loss, promptly retransmit or take other remedial measures.

[0072] In other words, the encapsulated sample contains all the key information of the difficult sample, providing comprehensive data support for further analysis and processing in the cloud. Furthermore, the use of a unified encapsulation format ensures data consistency and standardization during transmission, facilitating cloud reception and parsing. Additionally, by selecting appropriate communication links and transmission protocols, the encapsulated sample is ensured to be transmitted to the cloud efficiently and accurately, reducing data transmission latency and errors.

[0073] S150. The cloud-based teacher model is used to review the encapsulated samples to generate review results; Specifically, a high-performance teacher model is deployed in the cloud, which typically has higher accuracy and stronger computing power. The teacher model verifies the returned encapsulated samples and generates verification results, including more accurate detection results, classification results, etc.

[0074] In other words, the review of the teacher model can correct the errors in the student model, improve the overall detection accuracy, and provide high-quality training data for subsequent model optimization.

[0075] In one embodiment, the cloud-based system uses a teacher model to review the encapsulated samples to generate review results, including: After receiving the encapsulated samples in the cloud, the teacher model is used to perform high-precision inference and verification on the encapsulated samples, and outputs high-precision detection results and pseudo-label data as verification results.

[0076] Specifically, a trained high-precision teacher model is deployed on a cloud server. Teacher models typically possess stronger computational power and higher accuracy, enabling them to handle complex detection tasks. Encapsulated samples are read from cloud storage, and the raw image data is extracted. This raw image data is then input into the teacher model for high-precision inference. The teacher model, through its complex neural network structure, performs detailed analysis of the images to identify target objects. The teacher model outputs high-precision detection results, including target category information and bounding box coordinates. Based on the teacher model's inference results, corresponding pseudo-label data is generated. Pseudo-label data serves as a marker for the teacher model's inference results and is used for subsequent model training and optimization. Pseudo-label data typically includes information such as target category, bounding box coordinates, and confidence level.

[0077] The high-precision detection results from the teacher model and the generated pseudo-label data are encapsulated to form a structured verification result. The encapsulation format typically uses structured data formats such as JSON and XML to facilitate subsequent processing and transmission.

[0078] In other words, the high-precision inference of the teacher model can correct the errors of the student model, providing more accurate detection results and improving overall detection accuracy. Simultaneously, the generated pseudo-label data can serve as high-quality training data for subsequent student model optimization, improving student model performance. Furthermore, the encapsulated verification results have a clear structure, facilitating parsing and processing by subsequent modules.

[0079] More specifically, to fully leverage the advantages of cloud computing resources and high-performance models, the system performs in-depth analysis and high-precision inference on the encapsulated samples transmitted from the edge, and continuously optimizes the edge-based student model through a knowledge distillation mechanism. First, the cloud-deployed teacher model utilizes ample computing resources and a large-parameter network structure to perform high-precision inference on the encapsulated samples uploaded from the edge, obtaining more accurate target detection results and providing high-quality reference information for subsequent model optimization. Second, for the inference output of the teacher model, the system uses data cleaning strategies such as confidence filtering, conflict resolution, and unified annotation information to screen and standardize the encapsulated samples, storing them incrementally in the cloud's difficult example database, thereby constructing high-quality, sustainably expandable training data resources. Finally, once the number of encapsulated samples meets preset conditions, the system automatically triggers a model retraining process based on knowledge distillation. By fusing new and old data with soft and hard label constraints, the system effectively transfers the discriminative capabilities of the teacher model in complex scenarios to the edge-based student model, achieving continuous evolution and performance iteration improvement of the detection model.

[0080] To address the insufficient generalization ability of edge-based models due to limitations in parameter count and computational resources, a large-scale object detection model with stronger representation and generalization capabilities is deployed in the cloud as a teacher model. The GroundingDINO model is selected as the cloud-based teacher model for high-precision inference and semantic augmentation annotation of encapsulated samples returned from the edge. GroundingDINO is an open-vocabulary object detection model based on the Transformer architecture. This model overcomes the limitation of traditional object detection models that can only identify predefined fixed categories by jointly modeling a visual feature extraction network and a text semantic encoding network, achieving a deep fusion of image features and linguistic semantic information. In this model, the object detection process not only relies on the image content itself but also incorporates semantic prior information carried by textual prompts, thereby improving the model's object understanding and localization capabilities in complex scenes.

[0081] Specifically, when the cloud receives hard example images (i.e., encapsulated samples) returned from the edge, the system constructs corresponding text prompts from the category name or description of the target to be detected, and inputs them along with the hard example images into the Grounding DINO model. This model establishes a correlation between the image feature map and the text embedding vector through self-attention and cross-modal attention mechanisms, using textual semantic information to guide and constrain visual features, thereby achieving more accurate target localization and category determination. Thanks to its large number of parameters and the advantage of pre-training on massive amounts of multi-source data, Grounding DINO maintains high detection accuracy even in scenarios with complex backgrounds, small target scales, drastic viewpoint changes, or ambiguous target semantics. Therefore, when Grounding DINO performs secondary inference on hard example images returned from the edge, it can effectively discover small targets missed by the improved YOLOv8n student model, or correct false detection results caused by background interference and semantic ambiguity in the student model, and generate highly reliable detection results and pseudo-annotation information, providing high-quality supervised data for subsequent knowledge distillation and model evolution.

[0082] S160. Perform data cleaning on the verification results to obtain a cleaned sample; Specifically, data cleaning rules are set, such as removing noisy data, correcting erroneous annotations, and standardizing data formats. Then, the review results generated by the teacher model are cleaned to obtain cleaned samples, ensuring data quality and consistency.

[0083] In other words, data cleaning can remove invalid or erroneous data, improve the quality of training data, and provide a reliable foundation for model optimization.

[0084] In one embodiment, the step of cleaning the review results to obtain a cleaned sample includes: The verification results are subjected to confidence filtering, conflict resolution, and unified labeling information to obtain cleaned samples.

[0085] Specifically, in the cloud-based data processing module, a preset confidence threshold is used to filter high-precision detection results. For example, setting the threshold to 0.75 means that only detection results with a confidence level higher than or equal to 0.75 will be retained. Simultaneously, each detection item in the review results is iterated over to check if its confidence level meets the preset threshold. If the confidence level of a detection item is lower than the threshold, it is marked as a low-confidence result and filtered out, not included in subsequent processing. Furthermore, detection results with a confidence level higher than or equal to the threshold are retained as the basis for subsequent data cleaning.

[0086] In the review results, multiple detections may point to the same target but have different categories or bounding box coordinates; this situation is called a conflict. For example, a target may be detected as both "vehicle" and "pedestrian," or the bounding box coordinates may differ significantly. In cases of conflict, the detection result with higher confidence is prioritized as the final result. If the confidence levels of two conflicting results are close, other metrics (such as bounding box size and shape) are further compared. For complex conflicts that are difficult to resolve through automatic rules, a manual review mechanism can be introduced, where professionals make the judgments and selections. Then, based on the conflict resolution strategy, the review results are updated to ensure that each target has only one accurate detection result.

[0087] The review results check whether the annotation information for each detection item conforms to the unified format requirements. Annotation information includes target category, bounding box coordinates, confidence level, etc. For annotation information that does not conform to the unified format, format conversion and adjustment are performed. For example, bounding box coordinates in different formats are unified to standard upper left and lower right corner coordinates. Simultaneously, the annotation information is standardized to ensure consistency in all data regarding category naming, coordinate representation, etc. For example, the case of category names is standardized, and the precision of bounding box coordinates is standardized. Finally, the data after confidence filtering, conflict resolution, and annotation information unification is stored as a cleaned sample for use by subsequent modules.

[0088] More specifically, the inference results of the cloud-based teacher model are not directly used for training. Instead, they need to undergo standardized cleaning, filtering, and formatting to reduce the adverse effects of noisy samples on the model evolution process and construct a high-quality incremental training dataset. Based on this, a cleaning and storage process for encapsulated samples is designed, specifically including the following steps: Pseudo-label generation and teacher confidence filtering: The system obtains the inference results of the cloud-based teacher model on difficult example images, including bounding box coordinates and class probabilities. To prevent misjudgments by the teacher model in extreme scenarios from negatively impacting student model training, a teacher confidence threshold T is set. teacher Only predicted bounding boxes with confidence scores above a threshold are retained as pseudo-labels for the difficult example image. The aforementioned teacher confidence threshold is used to measure the reliability of the teacher model's inference results, and a value in the range [0.6, 0.9] is considered robust. Specifically, T... teacher Set to 0.75. When a difficult example image fails to produce any valid detection results that meet the confidence requirements after inference by the teacher model, the system classifies the image as an invalid sample and discards it, thereby avoiding the erroneous introduction of pure background or noise information into the training dataset.

[0089] Sample conflict resolution and annotation information unification: In some cases, the difficult example images returned from the edge may contain both targets correctly detected by the student model and targets that were not correctly identified. To ensure the consistency and accuracy of the annotation results, the system performs intersection-union (IU) matching analysis on the pseudo-labels generated by the teacher model and the original detection results uploaded from the edge. For overlapping detection boxes with an IU greater than a preset threshold, the bounding box coordinates and category labels output by the teacher model are used first to correct the original results. For targets newly discovered by the teacher model that do not form a valid match with the original detection results, they are added as new annotation information to the current sample, thereby achieving a complete update of the annotation information for difficult example images.

[0090] Incremental Dataset Construction and Storage Management: After the above cleaning and conflict resolution processes, the difficult example images and their corresponding pseudo-label data are automatically serialized and stored in a cloud-based difficult example sample database. This database adopts an incremental management strategy, continuously accumulating and recording the version of newly added samples. When the accumulated number of cleaned samples reaches a preset retraining trigger threshold, the system automatically initiates subsequent model retraining or knowledge distillation tasks, thereby achieving continuous model evolution and performance improvement.

[0091] S170. When the number of cleaned samples accumulated in the cloud meets the preset threshold, knowledge distillation and model retraining are started to obtain the optimized student model. Specifically, when the number of cleaned samples accumulated in the cloud reaches a preset threshold, the knowledge distillation and model retraining process is triggered. Using the cleaned samples as training data, the knowledge of the teacher model is transferred to the student model through knowledge distillation technology. At the same time, the student model is retrained by combining its original knowledge to obtain an optimized student model.

[0092] In other words, knowledge distillation can transfer the high-precision knowledge of the teacher model to the student model, improve the detection accuracy of the student model, and at the same time maintain its lightweight characteristics to adapt to the onboard computing power limitations of drones.

[0093] In one embodiment, the step of initiating knowledge distillation and model retraining to obtain an optimized student model when the cumulative number of cleaned samples in the cloud meets a preset threshold includes: When the number of cleaned samples accumulated in the cloud-based difficult case database reaches a preset threshold, the knowledge distillation and model retraining process is initiated. A mixed training strategy of new and old data is used, combined with soft and hard label constraints, to transfer the high-precision detection capability of the teacher model to the student model, so as to obtain an optimized student model.

[0094] Specifically, the high-precision detection capabilities of the cloud-based teacher model on difficult samples are transferred to the lightweight student model at the edge, thereby improving the detection performance of the student model in complex scenarios. To avoid catastrophic forgetting when the student model is trained with new samples, a knowledge distillation retraining strategy based on mixed training of new and old samples and joint constraints of soft and hard labels is adopted to iteratively optimize the edge YOLOv8n student model.

[0095] Hybrid data sampling strategy: When starting the student model retraining task, in addition to loading the newly added cleaned sample dataset from the cloud-based difficult example database, the system also randomly samples a portion of historical samples from the original basic training dataset according to a preset ratio to form a hybrid training dataset. This hybrid sampling strategy enhances the student model's ability to learn from complex and rare scenarios while maintaining its memory of the feature distribution of common scenarios, thus ensuring the stability of the model's overall generalization performance.

[0096] Constructing a knowledge distillation loss function based on response; During the retraining of the student model, a composite optimization objective function consisting of hard-label supervision loss and soft-label distillation loss is constructed. This allows the student model to learn correct detection results while further approximating the output distribution of the teacher model. The loss function is minimized using a gradient descent-based optimization algorithm, continuously updating the network parameters of the YOLOv8n student model. This significantly improves detection accuracy and generalization ability in challenging scenarios while maintaining the model's lightweight characteristics.

[0097] Hard-label supervised loss: A hard-label supervised loss L is constructed by calculating the conventional object detection loss between the student model's predictions and the cleaned teacher pseudo-labels. hard The hard-labeled supervision loss includes target classification loss and bounding box regression loss, which are used to constrain the student model to learn the correct target category and spatial location relationship.

[0098] Soft-label distillation loss: To enable the student model to learn the discriminative relationships and uncertainty representation capabilities of the teacher model across different categories, a distillation loss function based on the response distribution is introduced. Let the logistic values ​​output by the student model and the teacher model be z, respectively. s and z t The probability distribution is softened by introducing the distillation temperature parameter T, and the corresponding formula for calculating the category probability is as follows: ; Among them, P i (z) represents the normalized probability output of the model (teacher or student) for the i-th category after introducing the temperature parameter T. iThis represents the model's original prediction score for the input sample belonging to the i-th class, and its magnitude reflects the model's relative preference for that class. j This represents the index of the logical values ​​for all categories. When T is close to 1, soft labels tend to become hard labels, and the student model mainly learns the information of the category with the highest prediction probability. When T is greater than 1, the probability scores tend to be smoother, allowing low-probability categories to also obtain effective gradients, which helps the student model learn the teacher model's ability to distinguish between background and similar non-target categories. Validation on the UAV inspection dataset shows that when T is 4, the model achieves the optimal balance between gradient stability and inter-class relationship information transmission, resulting in the highest detection accuracy.

[0099] Based on the softening probability distribution described above, the distillation loss L distill Defined as: ; By minimizing this distillation loss, the student model can effectively simulate the teacher model's discrimination behavior against fuzzy targets and complex categories during training.

[0100] Overall Optimization Objective Function: The final retraining optimization objective function for the student model is defined as the weighted sum of the hard-label supervision loss and the soft-label distillation loss. ; Here, α is a balance coefficient used to adjust the relative weights of hard-label supervision and soft-label distillation during training. When α is close to 1, the model mainly relies on hard-label training, which may lead to the student model learning incorrect supervision information. When α is close to 0, the model mainly imitates the teacher's output distribution. Although it can learn the teacher's generalization ability, it lacks strong constraints on the target location and category, which may lead to insufficient localization accuracy. Therefore, grid search experiments with a step size of 0.1 in the interval [0, 1] revealed that when α < 0.3, the student model tends to blindly follow the uncertain output of the teacher model, resulting in an overall low confidence level. When α > 0.8, the performance gain brought by distillation is significantly weakened. Considering the trade-off between localization accuracy and classification accuracy, α = 0.6 is determined to be the optimal value. Under this setting, hard labels provide basic localization and classification constraints, and soft labels effectively smooth out pseudo-label noise, significantly improving the robustness and generalization ability of the student model on difficult samples.

[0101] S180. Perform cascade compression on the optimized student model to obtain a compressed model; Specifically, cascaded compression technology is used to further compress the optimized student model, including operations such as parameter pruning and quantization, to reduce the size and computational complexity of the model, ensuring that the compressed model can run efficiently on UAV-borne equipment while maintaining high accuracy.

[0102] In other words, cascaded compression can further reduce the model's resource consumption, improve the model's operating efficiency on drones, and reduce communication load.

[0103] In one embodiment, the cascade compression of the optimized student model to obtain a compressed model includes: The optimized student model is then subjected to structured pruning to obtain the pruned model. Specifically, in a cloud-based or local model optimization environment, select a structured pruning strategy suitable for the student model. Structured pruning typically refers to removing redundant weights or neurons from the model according to certain rules (such as by channel, by layer, etc.). Prune the optimized student model according to the selected pruning strategy. For example, redundant channels in some convolutional layers can be removed, or unimportant neurons in some fully connected layers can be pruned. After pruning, validate the pruned model to ensure it still meets application requirements in key performance indicators (such as detection accuracy). If over-pruning is found to cause performance degradation, the pruning strategy can be adjusted appropriately and pruning can be repeated.

[0104] In other words, by removing redundant weights or neurons, the model size is significantly reduced, decreasing storage requirements. Simultaneously, the pruned model requires less computation during inference, thus improving its operational efficiency.

[0105] The pruned model is quantized to obtain the quantized model; Specifically, a suitable quantization scheme is selected based on the model's characteristics and application scenario. Common quantization schemes include quantizing floating-point weights to low-precision integers (e.g., from 32-bit floating-point to 8-bit integers). Quantization is then applied to the pruned model, converting the model's weights and activation functions from floating-point representation to low-precision integer representation. Quantization may introduce some precision loss, therefore, the quantized model needs to be calibrated to ensure its performance is close to the original model. Calibration is typically performed through fine-tuning with a small amount of training data.

[0106] In other words, quantization makes the model's weight representation more compact, further reducing the model's storage requirements. At the same time, low-precision integer operations are faster than floating-point operations, so the quantized model can achieve higher inference speeds, making it suitable for running on resource-constrained devices.

[0107] The quantized model is adapted and accelerated to obtain a compressed model.

[0108] Specifically, the quantized model is adapted to the characteristics of the target hardware platform (such as an embedded device on a drone). This may include adjusting the model's input / output format and optimizing memory access patterns. Simultaneously, specific acceleration techniques (such as using hardware acceleration libraries and optimizing algorithm implementations) are employed to further improve the model's running speed. For example, GPUs or dedicated hardware accelerators can be used to accelerate the model's inference process. The adapted and accelerated model is then tested on the target hardware platform to ensure it achieves the expected performance metrics, such as inference speed and power consumption, in a real-world operating environment.

[0109] In other words, the adapted and accelerated model can run more efficiently on the target hardware platform, further improving the model's inference speed. At the same time, by optimizing the model's operation, the adapted and accelerated model can reduce power consumption and extend the drone's flight time.

[0110] More specifically, the optimized student model is compressed using a cascaded compression strategy combining structured pruning and quantization to reduce the model parameter size and storage volume, meeting the constraints of edge devices in terms of computing power and communication bandwidth. Then, to avoid the bandwidth pressure and update delay caused by the traditional full update method, an incremental update mechanism based on binary difference is adopted to perform differentiated updates on the detection model deployed at the edge of the UAV. Finally, based on the designed dual-backup hot update deployment architecture, a smooth switch between the old and new models is achieved at the edge, so that the model update process can be completed without stopping the inspection task or affecting real-time inference, ensuring the continuity and stability of the UAV inspection system.

[0111] Model compression: To adapt to the dual constraints of storage space and communication bandwidth at the edge, the student model, after knowledge distillation and retraining in the cloud, needs further compression and format conversion before being distributed. This involves a cascaded compression strategy combining structured pruning and quantization, with optimizations tailored to the target edge hardware platform. Specifically, the steps include: Structured model pruning: For the retrained YOLOv8n student model, a sparse training and structured pruning method based on the batch normalization layer scaling factor is employed. During the model training phase, the scaling parameters in the batch normalization layer are adjusted. L1 regularization is introduced to automatically suppress channels that contribute little to feature extraction during backpropagation. After training, channels are pruned according to a preset threshold. Redundant channels with values ​​close to zero and their corresponding convolution kernels significantly reduce the number of model parameters and computational complexity without disrupting the model's backbone topology.

[0112] INT8 Post-Training Quantization: To further reduce the model's GPU memory usage and bandwidth requirements at the edge, Post-Training Quantization (PTQ) is employed. By performing inference on a calibration dataset, the distribution characteristics of weights and activations in each layer are statistically analyzed, and the optimal quantization scaling factor is calculated, mapping the original FP32 precision weights and activations to INT8 format. This quantization operation can compress the model size to approximately one-quarter of its original size and fully utilize the integer arithmetic units of edge AI acceleration chips, improving overall inference throughput.

[0113] TensorRT Hardware Acceleration: To fully leverage the parallel processing capabilities of edge computing hardware, the quantized model is built and deployed based on the TensorRT inference engine. Through TensorRT's operator fusion mechanism, operators such as convolution, bias, and activation are merged into a single computational kernel, reducing GPU memory access frequency. Simultaneously, the optimal kernel implementation is automatically selected based on the target hardware architecture, ultimately generating a serialized Engine file (i.e., a compressed model). This ensures that the model can still meet the real-time detection requirements of high frame rate and low latency under high-resolution input conditions.

[0114] S190. An incremental update mechanism based on binary difference is adopted for the compressed model to generate a model difference patch package, and the difference patch package is transmitted to the UAV for model update and deployment.

[0115] Specifically, the compressed model is updated using an incremental update mechanism based on binary difference to generate a model difference patch package. Simultaneously, the difference patch package is transmitted to the drone, which then updates the model and redeploys it by applying the patch package.

[0116] In other words, the incremental update mechanism can reduce the amount of data transmitted during model updates, improve update efficiency, reduce communication costs and time delays, and ensure that drones can deploy the latest optimized models in a timely manner.

[0117] Specifically, addressing the issue of unstable communication links in drone inspection scenarios, the traditional full-model download and update method is abandoned in favor of an incremental update mechanism based on binary differential to reduce network load during model updates. The specific process is as follows: Differential patch generation: The cloud server maintains the old version model file V currently running on the edge. old When the new version model V new After generation, the system uses the BSDiff algorithm to calculate the binary differences between the old and new model files. BSDiff identifies common subsequences between files based on a suffix sorting algorithm, extracts only the changed byte data and corresponding control instructions, and generates a differential patch package that is much smaller than the complete model file.

[0118] Transmission and Reconstruction: Edge devices only need to download the aforementioned differential patch package, which typically represents only 5% to 10% of the full model file. After downloading, the edge device performs a binary merging operation with the locally stored old version model file and the patch package, reconstructing the complete new version model locally. This update mechanism significantly reduces network traffic, shortens model upgrade time, and improves the success rate and reliability of model updates in weak network environments.

[0119] To avoid interrupting the real-time inspection task being performed by the UAV during model updates, a dual-backup hot update deployment architecture is designed. This architecture divides the model storage space of the edge device into two independent storage areas: a running slot and a backup slot. The model used by the current inspection task is loaded into the running slot, while newly generated model files are written to the backup slot in the background. The entire process does not consume the computing and GPU memory resources of the foreground inference process. The specific hot update process is as follows: Loading preheating: After the new model in the spare slot passes the MD5 integrity check, the system preloads the new model into the GPU memory and completes the inference context initialization during the idle period of the inference process or by utilizing the idle GPU memory resources.

[0120] Atomic switching: The detection program internally maintains an atomic pointer variable that points to the currently active model handle. Once the new model is loaded and available, the system only needs to atomically modify this pointer to point to the memory address corresponding to the new model.

[0121] Smooth transition: When the next frame image inference request arrives, the detection task will automatically switch to the new model for execution, and the old model will be safely released. The entire switching process is completed within milliseconds and is completely transparent to the upper-layer inspection business, realizing the ability to upgrade the model online while it is in operation, thereby ensuring the continuity, real-time performance, and system stability of the inspection task.

[0122] To facilitate understanding of this technical solution, the following specific embodiments are provided: To verify the effectiveness of the improved YOLOv8n model at the edge, ablation experiments were conducted using a dataset of smoke and fire images collected by DJI drones in inspection scenarios. While maintaining consistent training strategies, input resolutions, and testing environments, the original YOLOv8n model and models with progressively improved modules were compared and evaluated. First, by replacing the original C2f module with a C2f-DS module based on depthwise separable convolution, the model's parameter size was significantly reduced while simultaneously improving detection accuracy, demonstrating that the proposed lightweight feature extraction structure effectively improves the model's feature utilization efficiency. Second, further introducing the CoordAtt attention mechanism further improved the model's detection accuracy, indicating that this attention mechanism can guide the model to focus on key regions with clear spatial distribution characteristics, such as smoke and flames, during multi-scale feature fusion, thereby enhancing the model's target discrimination ability under complex background conditions. Finally, by introducing the WIoU loss function to optimize the bounding box regression process, the model's target localization accuracy was further improved, achieving optimal overall detection performance. Meanwhile, since the aforementioned improvements primarily focus on model structure optimization and loss function design during the training phase, without introducing additional computational overhead during the inference phase, the improved model still meets the application requirements for real-time edge detection in terms of parameter size and inference efficiency. In summary, each improved module achieves a good balance between improved detection accuracy and model lightweighting, fully validating the effectiveness of the proposed improvement strategy in edge UAV inspection and detection tasks.

[0123] To further verify the effectiveness of the proposed edge-cloud collaboration, based on the improved edge model described above, the cloud-based teacher model Grounding DINO is introduced, and the detection performance difference between relying solely on the edge student model and enabling the edge-cloud collaboration mechanism is compared and analyzed.

[0124] In complex background conditions, the edge-side student model misses some smoke targets; while the cloud-based teacher model's inference results can completely detect smoke targets in the scene, and its confidence in distinguishing smoke targets is significantly higher than that of the edge-side student model, demonstrating stronger feature discrimination ability. Furthermore, in small target detection scenarios, the edge-side student model misidentifies distant white buildings as smoke targets, exhibiting significant false detection; in contrast, the cloud-based teacher model can correctly distinguish between background interference and real smoke targets, and demonstrates more accurate localization ability for faint smoke. In summary, the edge-cloud collaborative detection mechanism can effectively compensate for the shortcomings of the lightweight edge-side model in feature representation and target discrimination in complex scenes. Through the high-precision inference of difficult examples by the cloud-based teacher model and the knowledge transfer and continuous feedback mechanism, the adaptive evolution of the edge-side detection model is achieved, thereby significantly improving the robustness, stability, and detection reliability of the overall detection system in real UAV inspection environments.

[0125] Figure 3 This is a schematic block diagram of an edge-cloud collaborative optimization device 300 provided in an embodiment of the present invention. Figure 3 As shown, corresponding to the above-described edge-cloud collaborative optimization method, the present invention also provides an edge-cloud collaborative optimization device 300. This edge-cloud collaborative optimization device 300 includes a unit for executing the above-described edge-cloud collaborative optimization method, and the device can be configured in a server. Specifically, please refer to... Figure 3 The edge-cloud collaborative optimization device 300 includes: The preprocessing unit 301 is used to acquire raw aerial image data and perform preprocessing operations on the image data to obtain preprocessed data. The input / output unit 302 is used to input preprocessed data into a student model deployed on a drone for real-time inference, and output inference information. Evaluation and segmentation unit 303 is used to evaluate the credibility of the reasoning information to obtain the detection sample and to classify the detection sample into normal sample or difficult sample. The encapsulation and return unit 304 is used to encapsulate the difficult sample to obtain the encapsulated sample and return the encapsulated sample to the cloud. The review unit 305 is used to review the encapsulated samples using the teacher model in the cloud to generate review results; The cleaning unit 306 is used to clean the verification results to obtain a cleaned sample; Training unit 307 is used to initiate knowledge distillation and model retraining when the number of cleaned samples accumulated in the cloud meets a preset threshold, so as to obtain an optimized student model. Compression unit 308 is used to perform cascade compression on the optimized student model to obtain a compressed model; The update deployment unit 309 is used to generate a model differential patch package by using an incremental update mechanism based on binary difference for the compressed model, and then transmits the differential patch package to the UAV for model update and deployment.

[0126] In one embodiment, the acquisition preprocessing unit 301 includes: The drone continuously captures images of the inspection area using its onboard visible light imaging equipment to obtain raw aerial image data. The acquired raw aerial image data is then normalized in size and adjusted to a preset fixed size to obtain a normalized image. The normalized image is then converted to a standard RGB three-channel format to obtain preprocessed data.

[0127] In one embodiment, the evaluation and division unit 303 includes: The credibility of inference information is assessed based on a multidimensional uncertainty assessment mechanism, including a detection confidence index and a category prediction information entropy index, to obtain detection samples. Based on the preset confidence threshold and information entropy threshold, the detection samples are divided into normal samples or difficult samples.

[0128] In one embodiment, the review unit 305 includes: After receiving the encapsulated samples in the cloud, the teacher model is used to perform high-precision inference and verification on the encapsulated samples, and outputs high-precision detection results and pseudo-label data as verification results.

[0129] In one embodiment, the cleaning unit 306 includes: The verification results are subjected to confidence filtering, conflict resolution, and unified labeling information to obtain cleaned samples.

[0130] In one embodiment, the training unit 307 includes: When the number of cleaned samples accumulated in the cloud-based difficult case database reaches a preset threshold, the knowledge distillation and model retraining process is initiated. A mixed training strategy of new and old data is used, combined with soft and hard label constraints, to transfer the high-precision detection capability of the teacher model to the student model, so as to obtain an optimized student model.

[0131] In one embodiment, the compression unit 308 includes: The optimized student model is structured and pruned to obtain the pruned model; the pruned model is then quantized to obtain the quantized model; and the quantized model is then adapted and accelerated to obtain the compressed model.

[0132] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the aforementioned end-to-cloud collaborative optimization device 300 and its various units can be found in the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0133] The aforementioned edge-cloud collaborative optimization device 300 can be implemented as a computer program, which can, for example... Figure 4 It runs on the computer device shown.

[0134] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0135] See Figure 4The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0136] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform an edge-cloud collaborative optimization method.

[0137] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0138] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an edge-cloud collaborative optimization method.

[0139] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: The system acquires raw aerial image data and preprocesses it to obtain preprocessed data. This preprocessed data is then input into a student model deployed on a drone for real-time inference, outputting inference information. The credibility of the inference information is assessed to obtain detection samples, which are then categorized as normal or difficult samples. Difficult samples are encapsulated to obtain encapsulated samples, which are then transmitted back to the cloud. The cloud uses a teacher model to review the encapsulated samples, generating review results. The review results are then cleaned to obtain cleaned samples. When the cumulative number of cleaned samples in the cloud meets a preset threshold, knowledge distillation and model retraining are initiated to obtain an optimized student model. The optimized student model is then cascaded and compressed to obtain a compressed model. Finally, an incremental update mechanism based on binary difference is used to generate a model difference patch package, which is transmitted to the drone for model update and deployment.

[0141] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0142] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0143] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps: The system acquires raw aerial image data and preprocesses it to obtain preprocessed data. This preprocessed data is then input into a student model deployed on a drone for real-time inference, outputting inference information. The credibility of the inference information is assessed to obtain detection samples, which are then categorized as normal or difficult samples. Difficult samples are encapsulated to obtain encapsulated samples, which are then transmitted back to the cloud. The cloud uses a teacher model to review the encapsulated samples, generating review results. The review results are then cleaned to obtain cleaned samples. When the cumulative number of cleaned samples in the cloud meets a preset threshold, knowledge distillation and model retraining are initiated to obtain an optimized student model. The optimized student model is then cascaded and compressed to obtain a compressed model. Finally, an incremental update mechanism based on binary difference is used to generate a model difference patch package, which is transmitted to the drone for model update and deployment.

[0144] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0146] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0147] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0149] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An edge-cloud collaborative optimization method, characterized in that, include: Acquire raw aerial image data and perform preprocessing operations on the image data to obtain preprocessed data; Preprocessed data is input into a student model deployed on a drone for real-time inference to output inference information; The credibility of the reasoning information is assessed to obtain the test samples, and the test samples are divided into normal samples or difficult samples. The difficult sample is encapsulated to obtain an encapsulated sample, and the encapsulated sample is sent back to the cloud. The teacher model is used in the cloud to review the encapsulated samples in order to generate review results; The verification results are cleaned to obtain a cleaned sample; When the number of cleaned samples accumulated in the cloud meets the preset threshold, knowledge distillation and model retraining are initiated to obtain the optimized student model. The optimized student model is then subjected to cascaded compression to obtain a compressed model; The compressed model is updated using an incremental update mechanism based on binary difference to generate a model difference patch package, which is then transmitted to the drone for model update and deployment.

2. The edge-cloud collaborative optimization method according to claim 1, characterized in that, The process of acquiring raw aerial image data and performing preprocessing operations on the image data to obtain preprocessed data includes: The drone continuously photographs the inspection area using its onboard visible light imaging equipment to obtain raw aerial image data; The acquired raw aerial image data is normalized in size, and the image is adjusted to a preset fixed size to obtain a normalized image. The image after size normalization is converted to color space to a standard RGB three-channel format to obtain preprocessed data.

3. The edge-cloud collaborative optimization method according to claim 1, characterized in that, The process of evaluating the credibility of the reasoning information to obtain test samples, and classifying the test samples into normal samples or difficult samples, includes: The credibility of reasoning information is evaluated based on a multidimensional uncertainty assessment mechanism, including the detection confidence index and the category prediction information entropy index, in order to obtain detection samples; Based on the preset confidence threshold and information entropy threshold, the detected samples are divided into normal samples or difficult samples.

4. The edge-cloud collaborative optimization method according to claim 1, characterized in that, The cloud-based system uses a teacher model to review the encapsulated samples and generate review results, including: After receiving the encapsulated samples in the cloud, the teacher model is used to perform high-precision inference and verification on the encapsulated samples, and outputs high-precision detection results and pseudo-label data as verification results.

5. The edge-cloud collaborative optimization method according to claim 1, characterized in that, The process of cleaning the verification results to obtain a cleaned sample includes: The verification results are subjected to confidence filtering, conflict resolution, and unified labeling information to obtain cleaned samples.

6. The edge-cloud collaborative optimization method according to claim 1, characterized in that, When the cumulative number of cleaned samples in the cloud meets a preset threshold, knowledge distillation and model retraining are initiated to obtain an optimized student model, including: When the number of cleaned samples accumulated in the cloud-based difficult case database reaches a preset threshold, the knowledge distillation and model retraining process is initiated. A mixed training strategy of new and old data is used, combined with soft and hard label constraints, to transfer the high-precision detection capability of the teacher model to the student model, so as to obtain an optimized student model.

7. The edge-cloud collaborative optimization method according to claim 1, characterized in that, The process of cascading and compressing the optimized student model to obtain a compressed model includes: The optimized student model is then subjected to structured pruning to obtain the pruned model. The pruned model is quantized to obtain the quantized model; The quantized model is adapted and accelerated to obtain a compressed model.

8. An edge-cloud collaborative optimization device, characterized in that, include: The preprocessing unit is used to acquire raw aerial image data and perform preprocessing operations on the image data to obtain preprocessed data. The input / output unit is used to input preprocessed data into the student model deployed on the drone for real-time inference, and output inference information. The evaluation and segmentation unit is used to evaluate the credibility of the reasoning information to obtain the test samples and classify the test samples into normal samples or difficult samples. The encapsulation and return unit is used to encapsulate difficult example samples to obtain encapsulated samples and return the encapsulated samples to the cloud. The verification unit is used in the cloud to verify the packaged samples using the teacher model in order to generate verification results; The cleaning unit is used to clean the data of the verification results to obtain a cleaned sample; The training unit is used to initiate knowledge distillation and model retraining when the number of cleaned samples accumulated in the cloud meets a preset threshold, so as to obtain an optimized student model. The compression unit is used to perform cascade compression on the optimized student model to obtain a compressed model; The update deployment unit is used to generate a model differential patch package by using an incremental update mechanism based on binary difference for the compressed model, and then transmits the differential patch package to the UAV for model update and deployment.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images