Unmanned aerial vehicle city inspection image lightweight cleaning method based on data engine

Abstract

This invention discloses a lightweight image cleaning method for urban inspection images using a data engine, belonging to the field of UAV data processing technology. The invention acquires UAV inspection images and synchronous telemetry pose data and inputs them into a lightweight data engine. The first stage of the engine uses a low-computing-power image quality assessment algorithm to quickly remove low-quality frames with motion blur and abnormal exposure. The second stage combines spatiotemporal pose information to construct a projection grid, calculates the line-of-sight angle and physical overlap rate of adjacent frames, and adaptively discards highly redundant images using a non-pixel matching method. The third stage calls a lightweight perception model to perform target-level validity verification on the retained images, dynamically updates the cleaning rules, and outputs a high-quality image set. This invention reduces computational power consumption and solves the problems of massive amounts of urban inspection UAV image data, high redundancy, and high cleaning costs.

Description

Technical Field

[0001] This invention belongs to the field of UAV data processing technology, and specifically relates to a lightweight cleaning method for UAV urban inspection images based on a data engine. Background Technology

[0002] With the rapid development of the low-altitude economy, drones are widely used in urban inspections (such as cracking down on illegal constructions, inspecting exterior walls, and conducting traffic patrols). When drones perform high-frequency inspection tasks, their onboard high-definition visual sensors generate massive amounts of video and image data, which places extremely high demands on data processing, transmission, and storage.

[0003] Current technologies for processing drone inspection images typically employ two methods: one is to transmit all raw data back to a cloud server via an image transmission link for cleaning and analysis; the other is to deploy heavily weighted deep learning models (such as traditional object detection or image quality assessment networks) on the drone's edge for real-time filtering. However, full transmission consumes a significant amount of communication bandwidth and incurs high data transmission latency and expensive storage costs; while edge-side heavy AI models have a large number of parameters, which quickly depletes the drone's limited edge computing power and battery power. Furthermore, traditional deduplication algorithms often rely on visual feature point matching such as SURF / ORB, which is prone to failure when dealing with weak texture scenes such as large areas of solid-color walls or reflective glass curtain walls in urban areas, and the computation time is extremely long.

[0004] Therefore, there is still a need for a lightweight image cleaning method that can run on edge devices with limited computing power, has low latency and environmental adaptability, in order to solve the technical problems of large bandwidth consumption of massive inspection data, high computing power consumption on the edge side, and easy failure of cleaning in complex scenarios. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the problems in related technologies, this invention provides a lightweight cleaning method for UAV urban inspection images based on a data engine, thereby overcoming the aforementioned technical problems existing in the prior art.

[0007] (II) Technical Solution

[0008] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0009] S1. Extract the pixel matrix features of the original image sequence collected during the UAV urban inspection process, as well as the spatiotemporal pose features in the telemetry log.

[0010] S2. Perform parameterless low-level image quality assessment on pixel matrix features. By calculating the high-frequency texture gradient variance and gray-level histogram distribution characteristics of the image, remove motion-blurred frames with a sharpness score less than the blur threshold and poor-quality frames with abnormal exposure to obtain effective quality images.

[0011] S3. Extract the spatiotemporal pose state features corresponding to the effective quality images, and calculate the physical projection polygons of adjacent images on the real urban ground plane based on the camera pinhole imaging model and coordinate system mapping matrix.

[0012] The physical overlap rate of adjacent images is quantified by calculating the intersection-union ratio of the physical projection polygons. Valid quality images with a physical overlap rate greater than the overlap threshold are eliminated to obtain low-redundancy candidate images.

[0013] S4. Input the low-redundancy candidate images into the lightweight target detection model, calculate the probability of the presence of effective inspection business targets in the images and the semantic bounding box, remove invalid images with pure background, and obtain high-value images.

[0014] S5. Based on the characteristics of the current flight status of the UAV and the real-time data rejection rate of each stage (S2, S3, S4), construct an environmental feedback adjustment closed loop, and dynamically calculate and update the fuzzy threshold and physical overlap rate threshold.

[0015] High-value images and their corresponding spatiotemporal metadata are structured, encapsulated, and transmitted back to output a high-quality urban inspection image set.

[0016] Preferably, step S1 includes the following steps:

[0017] S11. Access the visual sensor and flight control system via the UAV's onboard bus to acquire video stream data and telemetry data from the Mavlink protocol, and obtain the original image sequence;

[0018] S12. Extract the RGB pixel matrix of the original image sequence as pixel matrix features; extract the GPS latitude and longitude, relative takeoff point altitude, three-axis accelerometer data, and the pitch angle, yaw angle, and roll angle of the gimbal from the telemetry data as spatiotemporal pose features.

[0019] S13. Using the frame timestamp of the visual video stream as a reference, a linear interpolation algorithm is used to perform hard synchronization alignment of timestamps on telemetry data with inconsistent frequencies, generating image-pose data pairs with pose labels.

[0020] Preferably, step S2 includes the following steps:

[0021] S21. Convert the original pixel matrix features into a single-channel grayscale matrix;

[0022] S22. The Laplacian operator is used to perform a two-dimensional spatial convolution operation on the single-channel gray-level matrix to extract the second derivative response value of the image; the global variance of all second derivative response values ​​is calculated and defined as the image sharpness score; when the sharpness score is less than the blur threshold, the image is judged as a blurry and poor quality frame and is removed.

[0023] S23. Calculate the pixel brightness histogram of the grayscale matrix and extract the mean and skewness features of the histogram. When the mean is greater than the overexposure threshold and the skewness is less than the lower limit of skewness, or the mean is less than the underexposure threshold and the skewness is greater than the upper limit of skewness, it is judged as an abnormally exposed poor frame and removed.

[0024] Preferably, step S3 includes the following steps:

[0025] S31. Establish a four-level spatial transformation link that includes the world coordinate system, the UAV body coordinate system, the camera gimbal coordinate system, and the image pixel coordinate system.

[0026] S32. Read the spatiotemporal pose state features corresponding to the current image and the pre-calibrated camera intrinsic parameter matrix; multiply the pixel coordinates of the four vertices of the image by the inverse of the camera intrinsic parameter matrix, the gimbal rotation matrix and the body translation matrix in sequence, and calculate the intersection of the ray and the urban ground plane elevation model through the ray projection algorithm to obtain the set of projection polygon coordinates of the current image in the real physical space.

[0027] S33. Read the set of coordinates of the projected polygons of the previous frame retained image, and use the polygon clipping algorithm to calculate the intersection area of ​​the current polygon and the polygon of the previous frame.

[0028] S34. Divide the intersection area by the union area of ​​the two polygons to obtain the physical overlap rate; when the physical overlap rate is greater than the dynamic overlap threshold, the current image is determined to be a highly redundant image and is removed.

[0029] Preferably, in step S32, when calculating the intersection of the ray and the urban ground plane elevation model, if high-precision digital elevation model data cannot be obtained, the ground plane is assumed to be an absolute horizontal plane with a relative take-off point height of 0, and the Z-axis coordinate of the ray endpoint is forcibly set to 0, and the physical coordinates of the X-axis and Y-axis are solved in reverse.

[0030] Preferably, step S4 includes the following steps:

[0031] S41. Construct a network architecture for a lightweight object detection model, wherein the architecture includes depthwise separable convolutional layers and global average pooling layers;

[0032] Collect historical data; the historical data includes various real business image samples collected in urban inspection scenarios, whose labels include the category of the target (including cracks in building exterior walls, illegal construction, illegally parked vehicles, and damaged power facilities) and the precise bounding box coordinates of the target in the image;

[0033] S42. Using the high-precision, large-parameter model in the cloud as the teacher network and the lightweight target detection model as the student network, the high-precision, large-parameter model in the cloud is trained using historical data. The hard-label cross-entropy loss and the soft-label KL divergence loss are jointly optimized, and the optimal weights are obtained by finding the feature distribution of the student network that approximates the teacher network through an optimization algorithm. The optimal weights are then applied to the lightweight target detection model to obtain the optimized lightweight target detection model.

[0034] S43. Scale the low-redundancy candidate images retained in S3 to a fixed resolution and input them into the optimized lightweight object detection model. The model outputs a prediction tensor containing the object category, confidence score, and bounding box coordinates.

[0035] S44. Traverse the prediction tensor. If the highest confidence of all prediction boxes in the image is lower than the set validity perception threshold, or the area of ​​the prediction box is less than the proportion of the area of ​​the whole image to the small target threshold, then the image is determined not to contain a valid inspection target and is removed as an invalid background image.

[0036] Preferably, in step S42, the optimal weights are obtained by jointly optimizing the hard-label cross-entropy loss and the soft-label KL divergence loss, and by using an optimization algorithm to find the feature distribution of the student network that approximates the teacher network. This includes the following steps:

[0037] S421. Define the distillation loss function;

[0038] S422. Construct a chromosome population and set its size; use each chromosome in the chromosome population as a random weight for a lightweight target detection model in the search space; set the maximum number of optimization iterations.

[0039] S423. Begin iteration. In each iteration, calculate the loss value of each chromosome in the chromosome population based on the distillation loss function to obtain a set of chromosome loss values. Select chromosomes in the chromosome population based on the chromosome loss values ​​in the set of chromosome loss values ​​to obtain a selected chromosome population. Perform crossover and mutation operations on the chromosomes in the selected chromosome population to obtain an operated chromosome population. In each iteration, select the current optimal weight based on the chromosome performance values ​​in the set of chromosome performance values.

[0040] S424. Repeat the iteration. When the maximum number of optimization iterations is reached, stop the iteration and take the current optimal weight as the optimal weight.

[0041] Preferably, step S5 includes the following steps:

[0042] S51. Define the status monitoring time window; within each time window, count the number of images removed due to blurring, abnormal exposure, high redundancy, and no target, and calculate the real-time removal rate for each stage of S2, S3, and S4.

[0043] S52. Extract the three-axis horizontal velocity and vertical climb rate from the current flight status features of the UAV, and calculate the current spatial composite scalar velocity of the UAV.

[0044] S53. Construct an environmental feedback adjustment closed-loop function; input the real-time rejection rate and spatial synthesis scalar velocity as independent variables into the closed-loop function, dynamically calculate and output the dynamic fuzziness threshold and dynamic overlap threshold of the next time sliding window;

[0045] S54. Compress the retained high-quality images in JPEG format, and write the corresponding GPS latitude and longitude, timestamp, and detected target category into the image file header in EXIF ​​metadata format, and transmit them back to the ground station or cloud server through the drone wireless image transmission link.

[0046] Preferably, the update rules for the dynamic blur threshold and dynamic overlap threshold in S53 are as follows:

[0047] The update rule for the dynamic overlap threshold is as follows: establish a baseline overlap threshold; calculate the ratio of the UAV's spatial synthetic scalar velocity to the standard inspection speed; when the ratio is less than 1, it indicates that the UAV is in a slow or hovering state, and the dynamic overlap threshold is equal to the baseline overlap threshold multiplied by the ratio, thereby increasing the redundancy removal force by lowering the threshold; when the ratio is greater than or equal to 1, the dynamic overlap threshold is equal to the baseline overlap threshold.

[0048] The update rule for the dynamic blur threshold is as follows: establish a baseline blur threshold; if the blur rejection rate is greater than the set upper limit of gust interference within the current time window, it indicates that the ambient light is generally dim or the drone is in a state of continuous high-frequency vibration. In order to ensure that enough data enters the subsequent process, the dynamic blur threshold is set to the baseline blur threshold minus the penalty step size; otherwise, it is restored to the baseline blur threshold.

[0049] Preferably, a lightweight cleaning system for UAV urban inspection images based on a data engine is provided to implement the lightweight cleaning method for UAV urban inspection images based on a data engine. The system includes:

[0050] Multi-source data synchronization access module: used to read camera video streams and flight control telemetry logs in real time through hardware interfaces, and to complete the spatiotemporal alignment of heterogeneous data through timestamp interpolation algorithms to generate image-pose data pairs;

[0051] Low-level quality rapid assessment module: used to receive image-pose data pairs, and use the Laplacian variance operator and grayscale histogram statistical algorithm to filter out blurry and abnormally exposed poor-quality images through matrix operations;

[0052] Spatiotemporal geometric redundancy filtering module: used to receive qualified images and pose data, map the image frustum to the physical ground plane using the camera pinhole model, and intercept highly redundant images in a purely geometric way by calculating the intersection-union ratio of adjacent projected polygons.

[0053] Lightweight semantic perception module: It is used to call the built-in lightweight target detection model that has undergone knowledge distillation, perform fast forward reasoning on low-redundancy images, determine whether there are valid inspection targets, and complete semantic-level image purification.

[0054] Adaptive closed-loop scheduling module: Used to monitor the real-time rejection rate and drone flight speed of each module, dynamically adjust the filtering thresholds at each level according to the preset mathematical feedback formula, and package and push the finally cleaned images and metadata to the remote receiving node.

[0055] (III) Beneficial Effects

[0056] The present invention has the following beneficial effects:

[0057] This invention abandons the traditional visual feature point matching deduplication method and introduces spatiotemporal pose state features. It uses a camera pinhole model to map the image frustum to the physical ground plane and calculates the intersection-union ratio of adjacent projected polygons to quantify the image overlap rate in a geometric way. This method has a shorter single-frame calculation time and is not affected by weak texture interference in the image, thus reducing the computational cost of deduplication.

[0058] This invention employs a lightweight target detection model based on joint optimization of genetic algorithm and knowledge distillation for semantic perception. While ensuring high-precision identification of inspection targets, the model size and inference resource consumption are smaller. It can eliminate pure background images with no business value and only transmit useful structured image sets, saving image transmission bandwidth and cloud storage space.

[0059] This invention constructs a mathematical feedback closed-loop function based on the spatial synthesis scalar velocity of UAVs and the real-time rejection rate at each stage; the data engine can dynamically calculate and update the fuzzy threshold and overlap threshold according to complex urban conditions such as UAV hovering, acceleration, encountering gusts of wind, or sudden changes in lighting, breaking the rigid logic of fixed thresholds and avoiding data deletion or omission under extreme conditions, thus ensuring the integrity and availability of inspection data.

[0060] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of the invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, the drawings can be obtained from these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating the lightweight cleaning method for urban patrol images using drones based on a data engine, as described in this invention.

[0063] Figure 2 This invention presents a schematic diagram of a lightweight cleaning system for drone-based urban inspection images, based on a data engine. Detailed Implementation

[0064] The technical solutions of the embodiments of the invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the invention, and not all embodiments. Based on the embodiments of the invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the invention.

[0065] Furthermore, to address the technical problems raised in the background section, please refer to... Figure 1 This invention provides a lightweight image cleaning method for urban patrol images using drones based on a data engine, comprising:

[0066] Example 1:

[0067] S1. Extract the pixel matrix features of the original image sequence collected during the UAV urban inspection process, as well as the spatiotemporal pose features in the telemetry log.

[0068] S2. Perform parameterless low-level image quality assessment on pixel matrix features. By calculating the high-frequency texture gradient variance and gray-level histogram distribution characteristics of the image, remove motion-blurred frames with a sharpness score less than the blur threshold and poor-quality frames with abnormal exposure to obtain effective quality images.

[0069] S3. Extract the spatiotemporal pose state features corresponding to the effective quality images, and calculate the physical projection polygons of adjacent images on the real urban ground plane based on the camera pinhole imaging model and coordinate system mapping matrix.

[0070] The physical overlap rate of adjacent images is quantified by calculating the intersection-union ratio of the physical projection polygons. Valid quality images with a physical overlap rate greater than the overlap threshold are eliminated to obtain low-redundancy candidate images.

[0071] S4. Input the low-redundancy candidate images into the lightweight target detection model, calculate the probability of the presence of effective inspection business targets in the images and the semantic bounding box, remove invalid images with pure background, and obtain high-value images.

[0072] S5. Based on the characteristics of the current flight status of the UAV and the real-time data rejection rate of each stage (S2, S3, S4), construct an environmental feedback adjustment closed loop, and dynamically calculate and update the fuzzy threshold and physical overlap rate threshold.

[0073] High-value images and their corresponding spatiotemporal metadata are structured, encapsulated, and transmitted back to output a high-quality urban inspection image set.

[0074] The above embodiments implement a layer-by-layer cleaning method scheduled by a lightweight data engine through multi-source data extraction, low-level quality assessment, spatiotemporal geometric deduplication, lightweight semantic perception, and closed-loop adaptive adjustment. By converting the complex image cleaning task into mathematical statistics and geometric calculations, and only calling the lightweight target detection model in the final stage, the computing power consumption of the UAV is reduced. At the same time, the constructed dynamic feedback closed loop can automatically adjust the cleaning rules according to the flight status and environmental changes, ensuring high quality and low redundancy of the output images while saving image transmission bandwidth and cloud storage costs.

[0075] The above embodiment S1 includes the following steps:

[0076] S11. Access the visual sensor and flight control system via the UAV's onboard bus to acquire video stream data and telemetry data from the Mavlink protocol, and obtain the original image sequence;

[0077] In specific implementation, the above embodiment S11 is as follows: the vision sensor outputs a 4K video stream with a resolution of 3840×2160 through an onboard USB 3.0 or CSI interface, with a frame rate of 30fps (i.e., one frame of image is generated every 33.3 milliseconds); the flight control system continuously outputs Mavlink protocol data packets at a baud rate of 115200 through an onboard UART serial port, which contains the global position and attitude information of the UAV, and the output frequency is configured to 50Hz (i.e., one log is generated every 20 milliseconds).

[0078] S12. Extract the RGB pixel matrix of the original image sequence as pixel matrix features; extract the GPS latitude and longitude, relative takeoff point altitude, three-axis accelerometer data, and the pitch angle, yaw angle, and roll angle of the gimbal from the telemetry data as spatiotemporal pose features.

[0079] In specific implementation, the above embodiment S12 specifically involves: for the i-th frame of the video stream, extracting its RGB pixel matrix I with a size of 3840×2160×3. i And record the system arrival timestamp Timg(i) Simultaneously, the Mavlink data packets are parsed to extract the UAV's longitude L. on Latitude L at Relative height (Alt) and gimbal camera pitch angle (Pitch) θ Yaw angle ψ Roll angle ϕ And record the flight control system timestamp T fc(j) ;

[0080] S13. Using the frame timestamp of the visual video stream as a reference, a linear interpolation algorithm is used to perform hard synchronization alignment of timestamps on telemetry data with inconsistent frequencies, generating image-pose data pairs with pose labels.

[0081] In specific implementation, the above embodiment S13 is as follows: due to the image generation time T img(i) It is usually not exactly equal to any flight control system timestamp T. fc(j) The system searches for a match T in the flight control log cache queue. fc(j) ≤T img(i) <T fc(j+1) The interpolation formula for aligning two adjacent telemetry records, taking the relative height Alt as an example, is as follows: Among them, Alt i T represents the relative altitude of the drone after interpolation, corresponding to the i-th frame of the image; img(i) This represents the arrival timestamp of the i-th frame image; j and j+1 represent the sequence numbers of the telemetry data packets output by the flight control system, where j represents the j-th record and j+1 represents the j-th record immediately following it; T fc(j) T fc(j+1) These represent the timestamps of the j-th and (j+1)-th flight control data entries, respectively; Alt j Alt j+1 The timestamps of the j-th and j+1-th flight control data records represent the actual relative altitudes of the UAV records.

[0082] Furthermore, considering that simple linear interpolation cannot guarantee sub-pixel level alignment accuracy when the UAV is flying at high speeds (>5m / s) or making sharp turns, as an optional embodiment, this invention introduces a kinematic compensation mechanism based on IMU high-frequency data: when acquiring two adjacent flight control timestamps T... fc(j) With T fc(j+1) When processing low-frequency pose data, simultaneously extract IMU triaxial angular velocity and triaxial acceleration data at a frequency of 200Hz within the same time interval; use the IMU pre-integration algorithm to calculate the values ​​within this time difference (T). img(i) -T fc(j)The nonlinear displacement increment and attitude rotation increment within the range are superimposed on the linear interpolation result to obtain the compensated high-precision pose state characteristics; through the above, the spatiotemporal synchronization error under high-speed maneuvering state is reduced from the average of 0.5 meters to less than 0.08 meters, ensuring the accuracy of subsequent physical projection;

[0083] The above embodiments overcome the problem of mismatch between the sampling rates of video streams and heterogeneous flight control and telemetry data by defining a timestamp linear interpolation alignment mechanism; by hard binding the originally isolated pixel matrix with spatial geometric coordinates with high precision at the underlying timestamp, each frame of the image has physical spatial attributes, laying the data foundation for subsequent use of low-computing-power geometric calculations to replace high-computing-power visual matching for deduplication.

[0084] The above embodiment S2 includes the following steps:

[0085] S21. Convert the original pixel matrix features into a single-channel grayscale matrix;

[0086] In specific implementation, the above embodiment S21 is as follows: In specific implementation, the standard psychological grayscale formula is adopted: Gray i (x,y)=0.299×R i (x,y)+0.587×G i (x,y)+0.114×B i (x,y); where Gray i (x,y) represents the grayscale value of the i-th frame image at pixel coordinates (x,y) after conversion; R i (x,y), G i (x,y), B i (x, y) represent the pixel values ​​of the i-th frame image in the red, green, and blue channels, respectively; the 3840×2160×3 tensor is compressed into a 3840×2160×1 single-channel matrix to reduce the amount of subsequent computation;

[0087] S22. The Laplacian operator is used to perform a two-dimensional spatial convolution operation on the single-channel gray-level matrix to extract the second derivative response value of the image; the global variance of all second derivative response values ​​is calculated and defined as the image sharpness score; when the sharpness score is less than the blur threshold, the image is judged as a blurry and poor quality frame and is removed.

[0088] In specific implementation, the above embodiment S22 specifically involves: defining the Laplacian convolution kernel. ; K lap With the grayscale matrix Gray i Perform a convolution with stride 1 on (x,y) to obtain the edge gradient matrix L. i ; Calculate L i Variance of all pixel values: Where M and N are the width and height of the image, and μ L For matrix L i The mean of the image; the blurrier the image, the fewer the high-frequency edges, and the greater the variance Var(L). i The smaller;

[0089] By extracting 10,000 clear image samples from stable drone inspections under normal weather conditions, the statistical distribution of their Laplace variance was calculated. The lower quartile was taken as the initial threshold to objectively encompass minor jitter during normal drone flight. In this embodiment, the initial dynamic blur threshold Th was calculated. b =85.0; This embodiment calculates the current frame Var(L) i If )=42.1<85.0, it is determined that the frame is severely motion-blurred due to the shaking of the drone gimbal, and the drop operation is performed directly;

[0090] Furthermore, to ensure the statistical significance and robustness of the fuzzy determination under different lighting conditions (sunny days, cloudy days) and different flight speeds, the system does not adopt an absolutely fixed 85.0 determination criterion. Instead, as an optional implementation, a local window adaptive mechanism is introduced: the system maintains a Laplacian variance queue of historical normal images with a length of 50 frames in real time, and calculates the mean μ of this queue. hist With standard deviation σ hist ; if and only if the variance Var(L) of the current frame i Simultaneously satisfying the condition of being less than the initial threshold of 85.0 and less than μ hist −2σ hist Only when the image is outside the 95% confidence interval is it determined to be severe motion blur caused by sudden gimbal jitter; this effectively avoids the accidental removal of images when the drone enters a shadow area, causing a reasonable decrease in overall image contrast.

[0091] S23. Calculate the pixel brightness histogram of the grayscale matrix and extract the mean and skewness features of the histogram. When the mean is greater than the overexposure threshold and the skewness is less than the lower limit of skewness, or the mean is less than the underexposure threshold and the skewness is greater than the upper limit of skewness, it is judged as an abnormally exposed poor frame and removed.

[0092] In specific implementation, the above embodiment S23 specifically involves: counting the number of pixels at each gray level (0-255) in the gray-scale matrix, generating a probability histogram P(k); and calculating the average brightness. Where k represents the gray level index; P(k) represents the probability of gray level k, which is equal to the number of pixels with gray value k in the image divided by the total number of pixels in the image (3840×2160), reflecting the relative frequency of each brightness level in the image;

[0093] Further calculation of the third-order central moment yields the skewness: ; where σ exp The standard deviation of brightness is represented by the following: A standard image set is collected under complex urban lighting conditions (including sunny midday, cloudy evening, and backlighting from glass curtain walls), and its grayscale mean and skewness distribution are statistically analyzed. The 95% confidence interval of the features of images artificially labeled as normally exposed is calculated, and the upper and lower bounds of this confidence interval are set as corresponding evaluation thresholds. The above statistical methods ensure that the system can accurately intercept extreme abnormal exposures without mistakenly deleting normal highlight or shadow images. The overexposure threshold is set to 210, and the underexposure threshold is set to 40. The lower limit of skewness is set to -1.5 (representing a long tail on the left and a large accumulation of pixels in the bright area on the right), and the upper limit of skewness is set to 1.5 (representing a long tail on the right and a accumulation of pixels in the dark area on the left). When the drone's camera suddenly points at the reflective glass curtain wall, μ is calculated. exp =235>210, and Skew=−2.1<−1.5, the frame is determined to be severely overexposed and is directly rejected;

[0094] The above embodiments eliminate the need for traditional deep learning image evaluation networks (such as IQA models) by deploying Laplacian variance and histogram skewness calculations at the mathematical and statistical level. On an airborne ARM architecture CPU, the evaluation time for a single frame is only about 4.2 milliseconds. This parameterless computation mechanism in the data cleaning method intercepts about 25% of the garbage data generated by environmental physical interference at a low computing cost (utilization rate <5%), reducing the computational burden of subsequent stages.

[0095] The above embodiment S3 includes the following steps:

[0096] S31. Establish a four-level spatial transformation link that includes the world coordinate system, the UAV body coordinate system, the camera gimbal coordinate system, and the image pixel coordinate system.

[0097] S32. Read the spatiotemporal pose state features corresponding to the current image and the pre-calibrated camera intrinsic parameter matrix; multiply the pixel coordinates of the four vertices of the image by the inverse of the camera intrinsic parameter matrix, the gimbal rotation matrix and the body translation matrix in sequence, and calculate the intersection of the ray and the urban ground plane elevation model through the ray projection algorithm to obtain the set of projection polygon coordinates of the current image in the real physical space.

[0098] In the above embodiment S32, when calculating the intersection of the ray and the urban ground plane elevation model, if high-precision digital elevation model data cannot be obtained, the ground plane is assumed to be an absolute horizontal plane with a relative take-off point height of 0, and the Z-axis coordinate of the ray endpoint is forcibly set to 0, and the physical coordinates of the X-axis and Y-axis are solved in reverse.

[0099] In specific implementation, the above embodiment S32 specifically involves: pre-calibrating the camera intrinsic parameter matrix. ; where f x f y c is the focal length in pixels.x c y The coordinates of the optical center are (1920, 1080 in this example); extract the homogeneous coordinates of the four corner points of the image in the pixel coordinate system, such as the top left corner p1=[0,0,1]. T Where T represents transpose;

[0100] Furthermore, to address the issue of decreased pose accuracy due to GPS signal obstruction by tall buildings in urban canyons, as an optional implementation, before extracting image pixel coordinates, the Brown-Conrady distortion model is used to correct radial and tangential distortion of the image, removing the polygon edge distortion error caused by barrel distortion from wide-angle lenses; the system automatically calls the local odometry data output by the airborne visual SLAM (VINS-Mono in this embodiment) and fuses it with the degraded GPS data using Kalman filtering to provide pose input after accuracy compensation; starting from the camera optical center using the intrinsic parameter inverse matrix, the pointing pixel p1 is converted into a normalized direction vector v in the camera coordinate system. c =K −1 p1;

[0101] Based on the pose characteristics aligned with S13, a 3×3 gimbal rotation matrix R is constructed using the gimbal's pitch angle θ, yaw angle ψ, and roll angle ϕ. gimbal Transform the direction vector to the UAV body coordinate system, and then to the world coordinate system (the ENU local coordinate system with the takeoff point as the origin) to obtain the world ray vector V. w =R drone *R gimbal *v c Among them, R drone This represents the rotation matrix of the UAV body;

[0102] In this embodiment, the drone's current altitude Alt i The distance is 100 meters; the ray equation is P. hit =P drone +t*V w Among them, P hit P represents the position of the intersection of the ray and the ground plane in the world coordinate system. drone V represents the three-dimensional position of the UAV in the world coordinate system; t represents the ray parameter, i.e., the distance (scalar) from the UAV to the intersection point on the ground; V w This represents the ray direction vector from the UAV's position to the ground projection point in the world coordinate system;

[0103] Since the default urban ground level height is 0 (i.e., Z=0), that is, 0=Alt i +t*V w(z) Among them, V w(z) V represents the ray direction vector in the world coordinate system. wZ-axis component; solve for parameters Substituting t into the ray equation, we obtain the physical coordinates (X1, Y1, 0) of the top-left pixel on the real ground; similarly, we find the four corner points to form the ground projection quadrilateral Poly for the current frame. i ={(X1,Y1),(X2,Y2),(X3,Y3),(X4,Y4)}, where the unit is meters;

[0104] Furthermore, in scenarios where drones perform vertical facade inspections close to the walls of tall buildings, the horizontal ground plane projection model will become invalid. In this case, the system adaptively switches based on the gimbal pitch angle θ: when the angle between the camera's line of sight and the horizontal plane is less than 45 degrees, the system switches the projection bearing surface from the horizontal plane with Z=0 to a vertical facade model parallel to the drone's flight path (i.e., setting Y=fixed distance and inversely solving the X and Z coordinates), thereby accurately calculating the physical coverage polygon during facade inspection.

[0105] S33. Read the set of coordinates of the projected polygons of the previous frame retained image, and use the polygon clipping algorithm to calculate the intersection area of ​​the current polygon and the polygon of the previous frame.

[0106] In specific implementation, the above embodiment S33 specifically involves: reading the projected quadrilateral Poly corresponding to the image successfully retained from the previous frame from memory. last The Sutherland-Hodgman polygon clipping algorithm is used to calculate the Poly... i With Poly last Geometric intersection polygons Poly intersect The actual physical area of ​​each polygon is calculated using the cross product formula, denoted as Area(Poly). i Area (Poly) last ) and Area (Poly intersect );

[0107] S34. Divide the intersection area by the union area of ​​the two polygons to obtain the physical overlap rate; when the physical overlap rate is greater than the dynamic overlap threshold, the current image is determined to be a highly redundant image and is removed.

[0108] In specific implementation, the above embodiment S34 specifically involves: calculating the physical overlap rate IoU: ;

[0109] Based on the minimum requirements for ensuring 3D reconstruction or continuous target tracking in photogrammetry (typically 60%-70%), and combining the theoretical expected value of physical overlap between adjacent frames calculated using the UAV's standard cruise speed and the camera's field of view, an upper limit of 80% of this expected value is taken as the initial threshold to retain sufficient safety redundancy. Therefore, this embodiment sets the initial dynamic overlap threshold Th.iou =0.80 (i.e., 80% overlap); if the drone hovers in the air while waiting for flight path instructions, and the calculated IoU between the current frame and the previous retained frame is 0.95 > 0.80, the system determines that the current frame has altitude spatial redundancy, directly discards the frame, and does not update Poly. last If, after the drone has flown a certain distance, a frame is calculated to have an IoU of 0.75 ≤ 0.80, then that frame is retained, and the quadrilateral of that frame is updated with a new Poly. last ;

[0110] Furthermore, due to the perspective effect, the ground resolution of pixels at the image edges is much lower than that of the central region. A Gaussian distance weight mask is introduced when calculating the intersection area: the physical center region of the polygon is given a weight of 1.0, which is then decayed to 0.5 towards the edges using a Gaussian function. The weighted effective intersection-union ratio is then calculated. Through the above operations, the high redundancy caused by the overlap of low-resolution edge regions is prevented, thus protecting the high-value information at the center from being mistakenly removed.

[0111] The above embodiments no longer use traditional methods; traditional methods for judging image redundancy require extracting and matching SURF or ORB visual feature points, which often takes more than 200 milliseconds per frame and is prone to failure when facing large areas of solid-color walls or reflective glass; this embodiment reduces the pixel matching problem to a 3D geometric projection problem based on flight control logs; by matrix multiplication and calculating the intersection area, the time for judging redundancy per frame is reduced, the judgment speed is improved, and it is not affected by the lack of texture in the image content, adapting to the limited computing power of the UAV edge, and can intercept more than 60% of redundant and unusable images;

[0112] The above embodiment S4 includes the following steps:

[0113] S41. Construct a network architecture for a lightweight object detection model, wherein the architecture includes depthwise separable convolutional layers and global average pooling layers to reduce the number of model parameters;

[0114] Collect historical data; the historical data includes various real business image samples collected in urban inspection scenarios, whose labels include the category of the target (such as cracks in building exterior walls, illegal construction, illegally parked vehicles, and damaged power facilities) and the precise bounding box coordinates of the target in the image;

[0115] In specific implementation, the above embodiment S41 is as follows: This embodiment uses the YOLOv8x architecture as the basis, replaces the traditional standard convolution with depthwise separable convolution, and splits the 3×3 convolution into 3×3 single-channel spatial convolution and 1×1 cross-channel point convolution, which reduces the number of parameters in a single layer by about 88%; the overall number of model parameters is controlled within 1.5M, and the model size after INT8 quantization is only 1.8MB; through the above operations, the YOLOv8x model, which requires a lot of computing power, can be applied to drones with limited computing power;

[0116] The lightweight target detection model has a limited input resolution of 416×416. The network layer design adopts the inverted residual structure of MobileNetV3 combined with a single-stage detection head, and the computational cost (FLOPS) is strictly controlled within 1.2G. The INT8 quantized model has a single forward inference latency of about 15 milliseconds on an edge NPU (such as Jetson Xavier NX) to meet real-time requirements.

[0117] S42. Using the high-precision, large-parameter model in the cloud as the teacher network and the lightweight target detection model as the student network, the high-precision, large-parameter model in the cloud is trained using historical data. The hard-label cross-entropy loss and the soft-label KL divergence loss are jointly optimized, and the optimal weights are obtained by finding the feature distribution of the student network that approximates the teacher network through an optimization algorithm. The optimal weights are then applied to the lightweight target detection model to obtain the optimized lightweight target detection model.

[0118] In the above embodiment S42, the optimal weights are obtained by jointly optimizing the hard-label cross-entropy loss and the soft-label KL divergence loss, and by using an optimization algorithm to find the feature distribution of the student network that approximates the teacher network. This includes the following steps:

[0119] S421. Define the distillation loss function;

[0120] S422. Construct a chromosome population, setting the size of the chromosome population to q. In this embodiment, q is set to 50. The chromosome population is then represented as follows: h i Represents the 1st chromosome in the population. A chromosome; each chromosome in the chromosome population is used as a random weight for a lightweight target detection model within the search space; a maximum number of optimization iterations is set;

[0121] S423. Begin iteration. In each iteration, calculate the loss value of each chromosome in the chromosome population based on the distillation loss function to obtain a set of chromosome loss values. Select chromosomes from the chromosome population based on the chromosome loss values ​​in the set of chromosome loss values ​​to obtain a selected chromosome population. Perform crossover and mutation operations on the chromosomes in the selected chromosome population (the crossover is calculated by the mean of the parent values, and the mutation is Gaussian mutation) to obtain the chromosome population after the operation. In each iteration, select the current optimal weight based on the chromosome performance values ​​in the set of chromosome performance values.

[0122] S424. Repeat the iteration. When the maximum number of optimization iterations is reached, stop the iteration and take the current optimal weight as the optimal weight.

[0123] In specific implementation, the above embodiment S42 specifically involves: pre-performing distillation training on a cloud GPU cluster; the training using historical data, genetic algorithm optimization, and knowledge distillation process described in this embodiment S42 are all limited to preparation steps executed offline on a cloud server cluster; in specific implementation, pre-performing distillation training is performed on a cloud GPU cluster; the genetic algorithm uses 50 chromosome populations to undergo multiple rounds of crossover mutation, taking several hours in the cloud to find the optimal weight distribution; before the drone takes off, only the optimal lightweight model weights extracted offline from the cloud and reduced to 1.8MB are distributed and solidified into the onboard NPU; during the online inference phase of flight inspection, the drone performs forward inference operations S43 and S44 without performing any model training or weight optimization, thereby minimizing the edge computing power consumption during operation;

[0124] The teacher network uses a large YOLOv8x model with 68M parameters; the distillation loss function is defined as: ; where y true Represents the true label in historical data; L CE P represents the cross-entropy loss for the true label, i.e., the difference between the class probabilities predicted by the student network and the true class labels; student D represents the probability distribution of the student network output categories; KL The KL divergence is used to bring the student network output logic value z closer together. student With teacher network logical value z teacher The probability distribution is soft; σ represents the softmax function; T is the distillation temperature parameter; α, β, and γ are the weights for hard label loss, soft label divergence, and bounding box regression, respectively. These values ​​were obtained through testing on the validation set using a grid search method. T=3.0 effectively softens the probability distribution, while the combination of α=0.4, β=0.5, and γ=0.1 achieves the best balance between classification accuracy and localization precision. Therefore, T=3.0, α=0.4, β=0.5, and γ=0.1 are set.

[0125] Construct a chromosome population of size q=50, where each chromosome represents a set of hyperparameters or weight initialization vectors for the student network. Before reaching the maximum number of iterations, calculate the fitness of each chromosome in each round according to the distillation loss function. Through crossover (calculation of the mean of parent values) and Gaussian mutation operations, continuously breed the chromosomes and finally select the current optimal weight that minimizes the distillation loss. Then apply the weights to the lightweight target detection model.

[0126] S43. Scale the low-redundancy candidate images retained in S3 to a fixed resolution and input them into the optimized lightweight object detection model. The model outputs a prediction tensor containing the object category, confidence score, and bounding box coordinates.

[0127] In practice, the few remaining images after S2 quality filtering and S3 redundancy filtering (at which point the frame rate has typically dropped to 2-3 fps) are scaled to 416×416 resolution and input into the onboard NPU (Neural Processing Unit); the model's forward inference output is a tensor of dimension D×(4+C), where D is the number of predicted boxes and 4 represents [x center ,y center [,width,height], where x center y center represents the X and Y coordinates of the center point of the predicted bounding box in the image coordinate system, respectively; width and height represent the pixel width and pixel height of the predicted bounding box, respectively; C represents the probability distribution of the business category (such as cracks, illegal buildings, and illegally parked vehicles).

[0128] S44. Traverse the prediction tensor. If the highest confidence of all prediction boxes in the image is lower than the set validity perception threshold, or the area of ​​the prediction box is less than the proportion of the area of ​​the whole image to the small target threshold, then the image is determined not to contain a valid inspection target and is removed as an invalid background image.

[0129] In specific implementation, the above embodiment S44 is as follows: Based on the validation set PR (precision-recall) curve, the recall inflection point value of 0.45, which can filter 99% of false positives, is taken, and therefore the validity perception threshold Th is set. conf =0.45; The threshold for small targets, Th, is derived by inversely calculating the ratio of the theoretical physical projection area of ​​the smallest target of concern (e.g., a 5cm wide crack) in a 4K image to the drone's normal flight altitude (e.g., 100 meters). area =0.005 (meaning the target area accounts for less than 0.5%); after inference, the system scans the tensor; if it finds that the maximum classification confidence of all predicted boxes in the image is only 0.21 (below 0.45), it means that the picture may just be a blue sky or ordinary grass, without containing the target of business interest, and the system directly discards the frame.

[0130] If a predicted bounding box exists with a confidence level of 0.88, but its calculated area is 10 × 10 = 100 square pixels, then the total area of ​​the image is approximately 3840 × 2160 ≈ 8.29 × 10⁻⁶. 6 If the area ratio is approximately 0.000012 < 0.005, the system determines that the target is too far away and has no value for inspection and analysis, and discards the frame. Only images containing a confidence level > 0.45 and an area ratio > 0.005 are retained as the final high-value keyframes.

[0131] The above embodiments introduce offline knowledge distillation technology to compress the massive semantic understanding model that originally needed to run in the cloud into a miniature model that can be deployed at the edge of the drone. Since the number of images reaching the S4 stage has been greatly reduced after the initial screening in the first two rounds (S2 and S3), even if AI model inference is called, it will not cause overload of computing power on the edge. This step filters out the garbage data that is clear and non-repetitive but has no business content, thereby improving the value of the data.

[0132] The above embodiment S5 includes the following steps:

[0133] S51. Define the status monitoring time window; within each time window, count the number of images removed due to blurring, abnormal exposure, high redundancy, and no target, and calculate the real-time removal rate for each stage of S2, S3, and S4.

[0134] In specific implementation, the above embodiment S51 is as follows: set the time sliding window W=5 seconds, and within 5 seconds (a total of 150 original images are generated), through statistics, 45 frames are removed due to blur (removal rate 30%), 10 frames are removed due to overexposure, and 80 frames are removed due to redundancy.

[0135] S52. Extract the three-axis horizontal velocity and vertical climb rate from the current flight status features of the UAV, and calculate the current spatial composite scalar velocity of the UAV.

[0136] In specific implementation, the above embodiment S52 is specifically as follows: In specific implementation, through the V of the flight control log... x V y V z Components, calculating the spatial composition velocity: ;where V x V y V z These represent the velocity components of the UAV in the x, y, and z coordinate systems, respectively; in this embodiment, the spatial composite velocity V is calculated. drone It is 1.2 m / s;

[0137] S53. Construct an environmental feedback regulation closed-loop function; input the real-time rejection rate and the spatial synthesis scalar velocity as independent variables into the closed-loop function, and dynamically calculate and output the dynamic fuzzy threshold and the dynamic overlap threshold for the next time window;

[0138] The update rules for the dynamic fuzzy threshold and the dynamic overlap threshold in the above embodiment S53 are as follows:

[0139] The update rule for the dynamic overlap threshold is: establish a reference overlap threshold; calculate the ratio of the drone's spatial synthesis scalar velocity to the inspection standard operation speed; when the ratio is less than 1, it indicates that the drone is in a slow speed or hovering state, and the dynamic overlap threshold is equal to the reference overlap threshold multiplied by this ratio, increasing the redundant rejection intensity by reducing the threshold; when the ratio is greater than or equal to 1, the dynamic overlap threshold is equal to the reference overlap threshold;

[0140] The update rule for the dynamic fuzzy threshold is: establish a reference fuzzy threshold; if within the current time window, the fuzzy rejection rate is greater than the set upper limit ratio of gust interference, it indicates that the external environmental light is generally dim or the drone is in a continuous high-frequency vibration state. To ensure that enough data enters the subsequent link, set the dynamic fuzzy threshold to the reference fuzzy threshold minus the penalty step; otherwise, restore it to the reference fuzzy threshold;

[0141] In specific implementation, the above embodiment S53 is specifically: based on the standard course overlap rate specification (usually required to be 70%-80%) when the drone is flying in a straight line at a constant speed, add a 5% tolerance margin to prevent missed frames, so as to objectively set the dynamic overlap threshold Th iou to be 0.85; set the inspection standard operation speed V std = 5.0 m / s; construct an update rule formula: RA v = V drone / V std = 1.20 / 5.0 = 0.24; Since RA v < 1, it indicates that the drone is flying extremely slowly or in a hovering and focusing state, and the picture change is extremely small; at this time, it is necessary to enhance the redundant rejection intensity (that is, lower the threshold for judging redundancy); calculate the new threshold Th iou new = Th iou base *RA v = 0.85 * 0.24 = 0.20;

[0142] Furthermore, the threshold being reduced to 0.20 means that as long as the picture has 20% overlap, it is regarded as redundant and discarded, forcing the system to only retain very few frames when hovering, preventing the data volume from being too large; if V drone ≥ 5.0, then keep 0.85 unchanged;

[0143] For dynamic fuzzy thresholding, the baseline fuzzy threshold Th is determined based on the quartiles of the normal sample variance. blur base =85.0; Based on the mathematical balance calculation between the single-battery endurance of the drone and the minimum effective image coverage required for a single inspection mission, if the rejection rate exceeds 25%, the amount of effective data retained will be insufficient to support the complete mosaic or detection of the business objective. Therefore, an upper limit R for gust interference is set. max =25%; According to S51 statistics, the current blur removal rate is 30% > 25%; This indicates that the drone may have entered a canyon wind zone or the light is generally dim (causing a slower shutter speed); If the threshold of 85.0 is maintained, almost all images will be deleted, and downstream businesses will have no images available; The average decrease in image Laplacian variance when drones encountered 4-5 level gusts in historical logs was statistically analyzed; With a penalty step size of 15.0, it can be ensured that after lowering the threshold, approximately 50% of usable images with edge sharpness but still business readability can be recovered; Construct the update rule formula: Th blur new =Th blur base -15.0 = 70.0; By proactively lowering the requirements, slightly blurred but valuable images are allowed to pass through, ensuring the integrity of the data chain;

[0144] To prevent drastic fluctuations in the rejection rate under extreme environments (such as sudden strong winds or rapid changes in light intensity) that could cause threshold jumps, an exponential moving average (EMA) smoothing filter and an absolute boundary protection mechanism are introduced as an optional implementation. The actual update formula is: the final dynamic overlap threshold Th iou =(Th iou new +Th iou base ) / 2; Final dynamic overlap threshold Th blur final =(Th blur new +Th blur base ) / 2; and the absolute lower limit of the overlap threshold is mandatory to be no less than 0.10, and the absolute lower limit of the fuzzy threshold is no less than 40.0;

[0145] Furthermore, the system sets a safety degradation bottom line strategy for data retention: regardless of how the threshold is calculated, the system enforces the spatial sampling protection rule: that is, based on GPS coordinates, for every 2 meters of physical distance the drone moves, it is forced to unconditionally retain 1 frame of image to avoid the risk of a data vacuum in a certain area due to algorithm misjudgment;

[0146] S54. Compress the retained high-quality images in JPEG format, and write the corresponding GPS latitude and longitude, timestamp, and detected target category into the image file header in EXIF ​​metadata format, and transmit them back to the ground station or cloud server through the drone wireless image transmission link.

[0147] Furthermore, the system introduces a dynamic adaptive strategy for the communication link: real-time reading of the packet loss rate and signal strength (RSSI) of the image transmission link. When in a weak signal area (RSSI < -85dBm), the system automatically lowers the overlap threshold by 15% based on the S53 calculation, further compressing the amount of data transmitted back to ensure the delivery rate of the core frame.

[0148] Actual measurements show that the processing latency of the first two stages (S2 and S3) of this invention is approximately 5 milliseconds per frame, intercepting about 60%-70% of the data. However, it is precisely this rapid interception in the first two stages that reduces the original 30fps video stream to approximately 10fps before inputting it into step S4. The lightweight model in step S4 has a single-frame inference latency of 15 milliseconds. The total end-to-end processing latency is strictly controlled within 25 milliseconds. For a drone with a flight speed of 10m / s, the spatial displacement deviation caused by 25 milliseconds is only 0.25 meters, which meets the requirements for real-time mapping and inspection.

[0149] Compared to full data transmission solutions, this invention achieves a data compression rate of 85% while maintaining 98% information integrity in weak texture scenes (such as large-area glass curtain walls, where traditional feature matching accuracy is less than 40%, while the accuracy of this invention's geometric projection method reaches over 95%). Compared to pure AI model frame-by-frame filtering solutions, this invention, due to its pre-processing of extremely low-computing-power mathematical and geometric filtering, reduces the overall resource utilization of the onboard CPU / NPU from a continuous 90% full load to an average of 22%, extending the effective cruise time of a single drone battery by approximately 18%.

[0150] The above embodiments, by constructing a mathematical feedback formula based on the physical motion state of the UAV (synthetic velocity) and the statistical distribution of the data stream (rejection rate), give the data engine the ability to adapt to the environment; break the rigid logic of fixed threshold in traditional image processing algorithms, and ensure that data quality and data retention rate can always be dynamically balanced under various extreme and complex urban conditions such as UAV hovering, acceleration, encountering gusts of wind or sudden changes in lighting, thus achieving all-weather intelligent lightweight cleaning.

[0151] Example 2:

[0152] Please see Figure 2 A lightweight cleaning system for drone urban inspection images based on a data engine, implementing the aforementioned lightweight cleaning method for drone urban inspection images based on a data engine, the system comprising:

[0153] Multi-source data synchronization access module: used to read camera video streams and flight control telemetry logs in real time through hardware interfaces, and to complete the spatiotemporal alignment of heterogeneous data through timestamp interpolation algorithms to generate image-pose data pairs;

[0154] Low-level quality rapid assessment module: used to receive image-pose data pairs, and use the Laplacian variance operator and grayscale histogram statistical algorithm to filter out blurry and abnormally exposed poor-quality images through matrix operations;

[0155] Spatiotemporal geometric redundancy filtering module: used to receive qualified images and pose data, map the image frustum to the physical ground plane using the camera pinhole model, and intercept highly redundant images in a purely geometric way by calculating the intersection-union ratio of adjacent projected polygons.

[0156] Lightweight semantic perception module: It is used to call the built-in lightweight target detection model that has undergone knowledge distillation, perform fast forward reasoning on low-redundancy images, determine whether there are valid inspection targets, and complete semantic-level image purification.

[0157] Adaptive closed-loop scheduling module: Used to monitor the real-time rejection rate and drone flight speed of each module, dynamically adjust the filtering thresholds at each level according to the preset mathematical feedback formula, and package and push the finally cleaned images and metadata to the remote receiving node.

[0158] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0159] The preferred embodiments of the invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A lightweight image cleaning method for urban inspection images using drones based on a data engine, characterized in that, Includes the following steps: S1. Extract the pixel matrix features of the original image sequence collected during the UAV urban inspection process, as well as the spatiotemporal pose features in the telemetry log. S2. Perform parameterless low-level image quality assessment on pixel matrix features. By calculating the high-frequency texture gradient variance and gray-level histogram distribution characteristics of the image, remove motion-blurred frames with a sharpness score less than the blur threshold and poor-quality frames with abnormal exposure to obtain effective quality images. S3. Extract the spatiotemporal pose state features corresponding to the effective quality images, and calculate the physical projection polygons of adjacent images on the real urban ground plane based on the camera pinhole imaging model and coordinate system mapping matrix. The physical overlap rate of adjacent images is quantified by calculating the intersection-union ratio of the physical projection polygons. Valid quality images with a physical overlap rate greater than the overlap threshold are eliminated to obtain low-redundancy candidate images. S4. Input the low-redundancy candidate images into the lightweight target detection model, calculate the probability of the presence of effective inspection business targets in the images and the semantic bounding box, remove invalid images with pure background, and obtain high-value images. S5. Based on the characteristics of the current flight status of the UAV and the real-time data rejection rate of each stage (S2, S3, S4), construct an environmental feedback adjustment closed loop, and dynamically calculate and update the fuzzy threshold and physical overlap rate threshold. High-value images and their corresponding spatiotemporal metadata are structured, encapsulated, and transmitted back to output a high-quality urban inspection image set.

2. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 1, characterized in that, S1 includes the following steps: S11. Access the visual sensor and flight control system via the UAV's onboard bus to acquire video stream data and telemetry data from the Mavlink protocol, and obtain the original image sequence; S12. Extract the RGB pixel matrix of the original image sequence as pixel matrix features; extract the GPS latitude and longitude, relative takeoff point altitude, three-axis accelerometer data, and the pitch angle, yaw angle, and roll angle of the gimbal from the telemetry data as spatiotemporal pose features. S13. Using the frame timestamps of the visual video stream as a reference, a linear interpolation algorithm is used to perform hard synchronization alignment of the timestamps of telemetry data with inconsistent frequencies, generating image-pose data pairs with pose labels.

3. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 1, characterized in that, S2 includes the following steps: S21. Convert the original pixel matrix features into a single-channel grayscale matrix; S22. The Laplacian operator is used to perform a two-dimensional spatial convolution operation on the single-channel gray-level matrix to extract the second derivative response value of the image; the global variance of all second derivative response values ​​is calculated and defined as the image sharpness score; when the sharpness score is less than the blur threshold, the image is judged as a blurry and poor quality frame and is removed. S23. Calculate the pixel brightness histogram of the grayscale matrix and extract the mean and skewness features of the histogram. When the mean is greater than the overexposure threshold and the skewness is less than the lower limit of skewness, or the mean is less than the underexposure threshold and the skewness is greater than the upper limit of skewness, it is judged as an abnormally exposed poor frame and removed.

4. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 1, characterized in that, S3 includes the following steps: S31. Establish a four-level spatial transformation link that includes the world coordinate system, the UAV body coordinate system, the camera gimbal coordinate system, and the image pixel coordinate system. S32. Read the spatiotemporal pose state features corresponding to the current image and the pre-calibrated camera intrinsic parameter matrix; multiply the pixel coordinates of the four vertices of the image by the inverse of the camera intrinsic parameter matrix, the gimbal rotation matrix and the body translation matrix in sequence, and calculate the intersection of the ray and the urban ground plane elevation model through the ray projection algorithm to obtain the set of projection polygon coordinates of the current image in the real physical space. S33. Read the set of coordinates of the projected polygons of the previous frame retained image, and use the polygon clipping algorithm to calculate the intersection area of ​​the current polygon and the polygon of the previous frame. S34. Divide the intersection area by the union area of ​​the two polygons to obtain the physical overlap rate; when the physical overlap rate is greater than the dynamic overlap threshold, determine that the current image is a highly redundant image and remove it.

5. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 4, characterized in that, In S32, when calculating the intersection of the ray and the urban ground plane elevation model, if high-precision digital elevation model data cannot be obtained, the ground plane is assumed to be an absolute horizontal plane with a relative take-off point height of 0. The Z-axis coordinate of the ray endpoint is forcibly set to 0, and the physical coordinates of the X-axis and Y-axis are solved in reverse.

6. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 1, characterized in that, S4 includes the following steps: S41. Construct a network architecture for a lightweight object detection model, wherein the architecture includes depthwise separable convolutional layers and global average pooling layers; Collect historical data; the historical data includes various real business image samples collected in urban inspection scenarios; S42. Using the high-precision, large-parameter model in the cloud as the teacher network and the lightweight target detection model as the student network, the high-precision, large-parameter model in the cloud is trained using historical data. The hard-label cross-entropy loss and the soft-label KL divergence loss are jointly optimized, and the optimal weights are obtained by finding the feature distribution of the student network that approximates the teacher network through an optimization algorithm. The optimal weights are then applied to the lightweight target detection model to obtain the optimized lightweight target detection model. S43. Scale the low-redundancy candidate images retained in S3 to a fixed resolution and input them into the optimized lightweight object detection model. The model outputs a prediction tensor containing the object category, confidence score, and bounding box coordinates. S44. Traverse the prediction tensor. If the highest confidence of all predicted boxes in the image is lower than the set validity perception threshold, or the area of ​​the predicted box is less than the proportion of the area of ​​the whole image to the small target threshold, then the image is determined not to contain a valid inspection target and is removed as an invalid background image.

7. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 6, characterized in that, In step S42, the optimal weights are obtained by jointly optimizing the hard-label cross-entropy loss and the soft-label KL divergence loss, and by using an optimization algorithm to find the feature distribution of the student network that approximates the teacher network. This includes the following steps: S421. Define the distillation loss function; S422. Construct a chromosome population and set the size of the chromosome population; use each chromosome in the chromosome population as a random weight for a lightweight target detection model in the search space; Set the maximum number of optimization iterations; S423. Begin iteration. In each iteration, calculate the loss value of each chromosome in the chromosome population based on the distillation loss function to obtain a set of chromosome loss values. Select chromosomes in the chromosome population based on the chromosome loss values ​​in the set of chromosome loss values ​​to obtain a selected chromosome population. Perform crossover and mutation operations on the chromosomes in the selected chromosome population to obtain an operated chromosome population. In each iteration, select the current optimal weight based on the chromosome performance values ​​in the set of chromosome performance values. S424. Repeat the iteration. When the maximum number of optimization iterations is reached, stop the iteration and take the current optimal weight as the optimal weight.

8. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 1, characterized in that, S5 includes the following steps: S51. Define the status monitoring time window; within each time window, count the number of images removed due to blurring, abnormal exposure, high redundancy, and no target, and calculate the real-time removal rate for each stage of S2, S3, and S4. S52. Extract the three-axis horizontal velocity and vertical climb rate from the current flight status features of the UAV, and calculate the current spatial composite scalar velocity of the UAV. S53. Construct an environmental feedback adjustment closed-loop function; input the real-time rejection rate and spatial synthesis scalar velocity as independent variables into the closed-loop function, dynamically calculate and output the dynamic fuzziness threshold and dynamic overlap threshold of the next time sliding window; S54. Compress the retained high-quality images in JPEG format, and write the corresponding GPS latitude and longitude, timestamp, and detected target category into the image file header in EXIF ​​metadata format, and transmit them back to the ground station or cloud server via the drone wireless image transmission link.

9. The lightweight cleaning method for UAV urban inspection images based on a data engine according to claim 8, characterized in that, The update rules for the dynamic blur threshold and dynamic overlap threshold in S53 are as follows: The update rule for the dynamic overlap threshold is as follows: establish a baseline overlap threshold; calculate the ratio of the UAV spatial composite scalar velocity to the standard inspection operation velocity; when the ratio is less than 1, the dynamic overlap threshold is equal to the baseline overlap threshold multiplied by the ratio; when the ratio is greater than or equal to 1, the dynamic overlap threshold is equal to the baseline overlap threshold. The update rule for the dynamic fuzzy threshold is: establish a baseline fuzzy threshold; If, within the current time window, the blur rejection rate exceeds the set upper limit for gust interference, the dynamic blur threshold is set to the baseline blur threshold minus the penalty step size; otherwise, it reverts to the baseline blur threshold.

10. A lightweight image cleaning system for UAV urban inspection based on a data engine, characterized in that: The system implementing the lightweight cleaning method for UAV urban inspection images based on a data engine as described in any one of claims 1-9 includes: Multi-source data synchronization access module: used to read camera video streams and flight control telemetry logs in real time through hardware interfaces, and to complete the spatiotemporal alignment of heterogeneous data through timestamp interpolation algorithms to generate image-pose data pairs; Low-level quality rapid assessment module: used to receive image-pose data pairs, and use the Laplacian variance operator and grayscale histogram statistical algorithm to filter out blurry and abnormally exposed poor-quality images through matrix operations; Spatiotemporal geometric redundancy filtering module: used to receive qualified images and pose data, map the image frustum to the physical ground plane using the camera pinhole model, and intercept highly redundant images in a purely geometric way by calculating the intersection-union ratio of adjacent projected polygons. Lightweight semantic perception module: It is used to call the built-in lightweight target detection model that has undergone knowledge distillation, perform fast forward reasoning on low-redundancy images, determine whether there are valid inspection targets, and complete semantic-level image purification. Adaptive closed-loop scheduling module: Used to monitor the real-time rejection rate and drone flight speed of each module, dynamically adjust the filtering thresholds at each level according to the preset mathematical feedback formula, and package and push the finally cleaned images and metadata to the remote receiving node.

Images