Adaptive optimization method and system for multi-modal images of unmanned aerial vehicle inspection

Abstract

The application discloses a multi-modal image adaptive optimization method and system suitable for unmanned aerial vehicle inspection, relates to the technical field of unmanned aerial vehicle inspection, and comprises the following steps: S1, target recognition is performed on real-time video streams obtained by unmanned aerial vehicle inspection to obtain an inspection target and position deviation information of the inspection target in image coordinates; S2, a posture adjustment instruction is generated based on the position deviation information and a double closed-loop collaborative control strategy to enable the inspection target to be adjusted to a central region of an image; S3, image quality assessment is performed on the inspection target in the central region, exposure parameters and focusing parameters of an image acquisition device are adaptively adjusted according to an image quality assessment result to generate a target image; and S4, it is judged whether the target image is qualified, if yes, image shooting is performed, and if not, S2 is performed. Through the construction of a double closed-loop collaborative control and a multi-dimensional image quality assessment system, the imaging quality, operation efficiency and automation level of unmanned aerial vehicle inspection in a complex field environment are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection technology, specifically to a multimodal image adaptive optimization method and system adapted for UAV inspection. Background Technology

[0002] Currently, power distribution network drone inspections primarily rely on manual pilot control and experience-based judgment, resulting in limited intelligence and significant technical bottlenecks in practical applications. In target recognition, existing solutions often employ unoptimized general detection models, which lack sufficient accuracy in identifying small, easily obscured critical components such as insulator strings, clamps, and pins, leading to high rates of missed detections and false alarms. Furthermore, the large number of model parameters makes efficient deployment in airborne edge computing units difficult. In image acquisition, camera exposure and focus parameters depend on real-time manual adjustments, which can easily lead to overexposure, underexposure, or image shift blurring under complex lighting conditions such as backlighting and shadows. This results in low single-shot image pass rates, requiring repeated flights and severely hindering inspection efficiency. In target tracking and alignment, existing technologies often use single gimbal control or simple proportional feedback, which struggles to quickly and accurately stabilize the inspection target in the center of the frame under dynamic disturbances such as wind and drone vibration, causing field-of-view shifts and reduced image usability. Therefore, there is an urgent need for an image acquisition method that can achieve full-process closed-loop adaptive optimization to improve the imaging quality, operational efficiency, and automation level of drone inspections in complex environments.

[0003] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to propose a multimodal image adaptive optimization method and system adapted to UAV inspection. By constructing a closed-loop control process that integrates lightweight real-time target recognition, dual closed-loop collaborative attitude and position control, and multi-dimensional image quality assessment and parameter adaptive adjustment, the imaging quality, operational efficiency and automation level of UAV inspection in complex field environments can be improved.

[0005] In a first aspect, one technical solution provided in this embodiment of the invention is: a multimodal image adaptive optimization method adapted to UAV inspection, comprising: S1. Perform target recognition on the real-time video stream acquired by the UAV inspection to obtain the inspection target and its positional deviation information in the image coordinates; S2. Based on the position deviation information and combined with the dual closed-loop collaborative control strategy, generate attitude adjustment instructions to adjust the inspection target to the center region of the image; S3. Perform image quality assessment on the inspection target located in the central area, and adaptively adjust the exposure parameters and focus parameters of the image acquisition device to generate the target image based on the image quality assessment results. S4. Determine whether the target image meets the image quality standards. If it does, perform image capture; otherwise, perform S2.

[0006] As a preferred embodiment, the steps of performing target recognition on the real-time video stream acquired by the UAV inspection and obtaining the inspection target and its positional deviation information in the image coordinates are as follows: S11. Input the fused image formed by frame extraction and preprocessing of the real-time video stream into the target detection model for inference, and output the target category, confidence level and center pixel coordinates in the image coordinate system. S12. The position deviation information is calculated based on the center pixel coordinates and the preset image center coordinates.

[0007] As a preferred embodiment, the target detection model is trained from a lightweight improved YOLO model using test data, specifically including: YOLOv8n was selected as the baseline model. The number of network layers and network channels of the backbone network of the baseline model were compressed. At the same time, the network structure of the Neck part of the baseline model was enhanced and the EMA attention mechanism module was introduced to obtain a lightweight improved YOLO model. A training dataset consisting of inspection images of power distribution towers with target annotations is constructed; data augmentation strategies including Mosaic, MixUp, and HSV spatial perturbations are used to augment the training dataset; The lightweight improved YOLO model is trained on the enhanced training dataset to obtain the object detection model.

[0008] As a preferred embodiment, the step of generating attitude adjustment commands based on the position deviation information and a dual-closed-loop collaborative control strategy to adjust the inspection target to the center region of the image is as follows: S21. Determine whether the position deviation information exceeds the first trigger threshold; if so, start the inner gimbal closed-loop control to adjust the gimbal attitude so that the inspection target moves towards the center of the image. S22. Continuously monitor the position deviation information. If it fails to converge to within the second trigger threshold within a first preset time, or if the position deviation information instantaneously exceeds the third trigger threshold, then synchronously start the outer layer UAV position closed-loop control to adjust the UAV position. S23. When the position deviation information continues to converge to within the second trigger threshold within a second preset time, exit the outer layer UAV position closed-loop control and maintain the inner layer gimbal closed-loop control until the target is stable in the center area of ​​the image.

[0009] As a preferred embodiment, the step of activating the inner gimbal closed-loop control to adjust the gimbal attitude and move the inspection target toward the image center is as follows: S211. Based on the pixel deviation in the position deviation information, calculate the desired angular velocity adjustment amount in the gimbal coordinate system used to eliminate the pixel deviation; S212. The desired angular velocity adjustment amount is calculated and smoothed by an event-triggered PID controller to generate a gimbal angular velocity control command. S213. After the gimbal angular velocity control command is limited to a preset maximum angular velocity value, it is sent to the gimbal actuator to adjust the pitch and yaw angles at the specified angular velocity to move the inspection target toward the image center.

[0010] As a preferred embodiment, the steps for initiating closed-loop control of the outer UAV position to adjust the UAV position are as follows: S221. Based on the position deviation information, the flight status parameters of the UAV, and the parameters of the image acquisition device, the UAV position fine-tuning amount in the computer body coordinate system; S222. The fine-tuning amount is converted into a smooth speed command, and the smooth speed command is subjected to speed constraint processing and then sent to the UAV flight control system to adjust the UAV position.

[0011] As a preferred embodiment, the steps of performing image quality assessment on the inspection target located in the central region, and adaptively adjusting the exposure parameters and focus parameters of the image acquisition device to generate the target image based on the image quality assessment results are as follows: S31. Perform a multi-dimensional illumination status assessment on the image region containing the inspection target, extract illumination features and calculate a comprehensive illumination evaluation index. S32. Input the comprehensive illumination evaluation index into the preset exposure decision model, obtain and execute the corresponding exposure parameter adjustment action, the exposure parameter adjustment action includes at least adjusting the ISO value, exposure compensation value or shutter speed; S33. After the lighting conditions meet the preset conditions, the image clarity of the inspection target area is evaluated to obtain the corresponding clarity evaluation value. S34. If the sharpness evaluation value is lower than the preset sharpness threshold, adjust the optical focal length or perform digital image sharpening processing. S35. Repeat S31~S34 until both the comprehensive illumination evaluation index and the sharpness evaluation value meet the preset imaging quality standard to obtain the target image.

[0012] Secondly, an embodiment of the present invention also provides a technical solution: a multimodal image adaptive optimization system, applicable to the multimodal image adaptive optimization method for UAV inspection described in the first aspect, comprising: The edge computing module performs target recognition on the real-time video stream acquired by the UAV inspection, and obtains the inspection target and its positional deviation information in the image coordinates; The dual-closed-loop control module generates attitude adjustment commands based on the position deviation information and the dual-closed-loop collaborative control strategy to adjust the inspection target to the center region of the image. The quality assessment module performs image quality assessment on the inspection targets located in the central area, and adaptively adjusts the exposure parameters and focus parameters of the image acquisition device to generate target images based on the image quality assessment results. The decision trigger module executes image capture when the target image meets the image quality standards.

[0013] Thirdly, one technical solution provided in this embodiment of the invention is: an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the multimodal image adaptive optimization method for adapting to UAV inspection as described in the first aspect.

[0014] Fourthly, one technical solution provided in this embodiment of the invention is: a storage medium storing computer-executable instructions, wherein when the computer-executable instructions are loaded and executed by a processor, the steps of the multimodal image adaptive optimization method for adapting to UAV inspection as described in the first aspect are implemented.

[0015] The present invention has at least the following substantial beneficial effects: (1) In view of the problem that the target recognition, attitude control and parameter optimization links in the prior art operate in isolation and lack coordination, resulting in low overall operation efficiency and unstable imaging quality, this application uses the image recognition result as the input of the control system, and uses the control effect and image quality evaluation result together as the decision basis for parameter adjustment and photo triggering. It also forms an iterative optimization loop through post-photo verification, thereby integrating the originally discrete operation steps into an organic and coordinated overall control mechanism, which significantly improves the repeatability, measurability and overall operation efficiency of the inspection process, and ensures the consistency and high availability of imaging results.

[0016] (2) To address the problem that a single UAV gimbal control in complex environments (such as wind, changes in lighting, and small target size) has slow response and poor anti-disturbance capability, making it difficult to quickly and accurately stabilize the inspected target in the center of the image, this application introduces a dual-loop collaborative control strategy consisting of an inner gimbal closed loop and an outer UAV position closed loop. It also designs an intelligent triggering and exit mechanism based on deviation thresholds and time criteria, achieving collaborative operation of viewpoint correction and position compensation. Specifically, the inner-loop gimbal control provides rapid response and fine-tuning based on pixel deviation, while the outer-loop UAV position control provides macroscopic position compensation when deviation is continuous or sudden. The two work collaboratively through priority logic. This collaborative mechanism effectively overcomes the physical limitations and response delays of a single actuator, significantly improving the system's target tracking stability and centering convergence speed under dynamic disturbances, thereby ensuring the viewpoint foundation for subsequent high-quality image acquisition.

[0017] (3) To address the problem that camera exposure, focus, and other parameters rely on manual adjustment under complex lighting conditions and zoom scenarios, leading to frequent issues such as overexposure, underexposure, and blurring, and a low first-pass yield, this application integrates a multi-dimensional lighting evaluation model with a sharpness discrimination model based on the Laplacian operator. The system can analyze the lighting distribution and local sharpness of the image in real time, intelligently make decisions, and drive the actuators to adjust ISO, exposure compensation, shutter speed, and optical zoom focal length until the image quality indicators meet the preset stringent standards. This ensures that high-quality images with balanced brightness and clear details can be automatically output even in variable external environments, greatly reducing the need for re-flying due to image quality problems.

[0018] The above description of the invention is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0019] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0020] Figure 1 The flowchart illustrates the multimodal image adaptive optimization method for drone inspection according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of a multimodal image adaptive optimization system according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only one preferred embodiment of this invention and are only used to explain this invention. They do not limit the scope of protection of this invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations (or steps) can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures; the process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0024] Example 1: As Figure 1 As shown, one technical solution provided in this embodiment of the invention is: a multimodal image adaptive optimization method for UAV inspection, comprising: S1. Perform target recognition on the real-time video stream acquired by the UAV inspection to obtain the inspection target and its positional deviation information in the image coordinates.

[0025] As an optional implementation, the steps of performing target recognition on the real-time video stream acquired by the UAV inspection and obtaining the inspection target and its positional deviation information in the image coordinates are shown in S11~S12.

[0026] S11. Input the fused image formed by frame extraction and preprocessing of the real-time video stream into the target detection model for inference, and output the target category, confidence level and center pixel coordinates in the image coordinate system.

[0027] As an optional implementation, the target detection model is obtained by training a lightweight improved YOLO model with test data, specifically including: YOLOv8n was selected as the baseline model. The number of network layers and network channels of the backbone network of the baseline model were compressed. At the same time, the network structure of the Neck part of the baseline model was enhanced and the EMA attention mechanism module was introduced to obtain a lightweight improved YOLO model. A training dataset consisting of inspection images of power distribution towers with target annotations is constructed; data augmentation strategies including Mosaic, MixUp, and HSV spatial perturbations are used to augment the training dataset; The lightweight improved YOLO model is trained on the enhanced training dataset to obtain the object detection model.

[0028] Understandably, in power distribution network inspections, direct use of real-time video streams for target detection is susceptible to interference from environmental occlusion and lighting changes, and insufficient feature extraction leads to missed detection of small targets and high recognition latency. This embodiment addresses these issues by first extracting frames (e.g., extracting single-frame images at a fixed 50ms frame rate to match subsequent control cycles) and preprocessing the real-time video streams simultaneously acquired by the visible light and infrared cameras mounted on the UAV. Preprocessing employs letterbox adaptive resizing to uniformly adjust frame images of different resolutions to 640×640 pixels, while linearly normalizing pixel values ​​from 0-255 to 0-1 to eliminate the influence of brightness amplitude differences. Then, a multi-view image fusion network integrating an adaptive guidance module and a ResNeXt feature extraction structure deeply couples the texture details (visible light) and environmental adaptability (infrared) features of the dual-modal frame images, generating a highly robust fused image. This fused image is then input into a specially optimized lightweight YOLO model for real-time inference. The model is based on YOLOv8n, with C2f modules pruned. The number of Bottleneck layers was reduced from 3 to 2, and the number of dimensionality reduction channels was reduced from [64, 128, 256, 512] to [48, 96, 192, 384] to achieve lightweighting. Simultaneously, a double upsampling branch was added from the P3 / 8 feature layer to the P2 / 4 feature layer in the Neck part, and an EMA attention module was introduced to enhance feature extraction for small targets ≤10cm (such as pins). During inference, a sliding window is used to traverse and fuse images, and the output includes the inspection target category (e.g., insulator strings, wire clamps, etc., 8 categories), and the placement... The results of confidence (e.g., outputting confidence of 0.92 when a surge arrester is detected) and the center pixel coordinates in the image coordinate system (with the image center (320, 320) as the origin) (e.g., outputting the center coordinates of the clamp (318, 325)) enable accurate identification and real-time positioning of tower components in complex inspection scenarios. This not only offsets environmental interference through multimodal fusion but also controls inference latency to the millisecond level through model lightweighting. At the same time, the mAP of small target detection is significantly improved, providing accurate and real-time coordinate input basis for the subsequent dual-loop alignment module.

[0029] For example, in the above embodiments, the input of the lightweight improved YOLO model can be set as: a 640×640×3 RGB image, with preprocessing using letterbox adaptive resizing + normalization (pixel values ​​0-255 normalized to 0-1); the output is the detection box coordinates (x, y, w, h) + confidence score + class probability, where the classes are subdivided into 8 categories (insulator string, clamp, surge arrester, hardware, pin, grading ring, tower head, tower body); the dataset size is set as follows: a certain... Images of 5,000 poles in the region were used, and 120,000 samples were labeled, with 80% used for training, 10% for validation, and 10% for testing. Furthermore, the data augmentation strategies were: Mosaic probability 0.5, MixUp probability 0.3, HSV spatial random perturbation ±20%, and horizontal flip probability 0.5. Furthermore, the label format was COCO format, which included the target center point coordinates, width and height (normalized to the range of 0-1), and occlusion label (0=no occlusion, 1=partial occlusion, 2=severe occlusion).

[0030] S12. The position deviation information is calculated based on the center pixel coordinates and the preset image center coordinates.

[0031] It is understandable that the center pixel coordinates of the inspected target output by the target detection need to be converted into quantized deviation information that can directly drive alignment control. Otherwise, the subsequent position adjustment of the gimbal and the UAV will lack a clear basis and cannot achieve precise alignment. Therefore, this embodiment first presets the center coordinates of the image coordinate system (based on the pre-processed 640×640 image resolution, such as defining the image center pixel coordinates as (320, 320), which is the origin of the pixel coordinate system, with the u-axis horizontally to the right and the v-axis vertically downward). Then, based on the center pixel coordinates of the inspected target output by the target detection (u_obj, v_obj), the difference calculation formulas Δu=u_obj–320 and Δv=v_obj–320 are used. 20. Calculate the position deviation information, and set the trigger threshold to |Δu|>30 or |Δv|>30 pixels; for example, when the center pixel coordinates of the detected wire clamp target are (356, 338), the corresponding calculations are (Δu=356-320=36) and Δv=338-320=18). In this embodiment, the offset between the target center and the image center is quantified by the coordinate difference in the pixel coordinate system. This realizes the transformation of the target position information into standardized and quantifiable deviation parameters, providing a precise trigger threshold judgment basis and control quantity calculation input for subsequent dual closed-loop dynamic deviation compensation control, ensuring that the alignment adjustment can respond to the target offset in a targeted manner, and improving the accuracy and real-time performance of the alignment control.

[0032] S2. Based on the position deviation information and combined with the dual closed-loop collaborative control strategy, an attitude adjustment command is generated to adjust the inspection target to the center area of ​​the image.

[0033] In this embodiment, a dual-closed-loop collaborative control strategy is used to convert position deviation information into attitude adjustment commands, enabling the inspection target to be quickly and accurately centered and resistant to disturbances such as wind and shaking, ensuring that the target remains stably in the center area of ​​the image.

[0034] As an optional implementation, the step of generating attitude adjustment commands based on the position deviation information and the dual closed-loop collaborative control strategy to adjust the inspection target to the center region of the image is shown in S21~S23.

[0035] S21. Determine whether the position deviation information exceeds the first trigger threshold; if so, start the inner gimbal closed-loop control to adjust the gimbal attitude so that the inspection target moves towards the center of the image. S22. Continuously monitor the position deviation information. If it fails to converge to within the second trigger threshold within a first preset time, or if the position deviation information instantaneously exceeds the third trigger threshold, then synchronously start the outer layer UAV position closed-loop control to adjust the UAV position. S23. When the position deviation information continues to converge to within the second trigger threshold within a second preset time, exit the outer layer UAV position closed-loop control and maintain the inner layer gimbal closed-loop control until the target is stable in the center area of ​​the image.

[0036] It should be noted that the first trigger threshold is |Δu|>30 or |Δv|>30 pixels, the second trigger threshold is |Δu|<10 and |Δv|<10 pixels, the third trigger threshold is |Δu|>80 or |Δv|>80 pixels, the first preset time is 15 seconds, and the second preset time is 3 seconds.

[0037] Understandably, a single closed-loop control alone cannot simultaneously ensure both rapid response to target centering and stability against disturbances such as wind and vibration. It either introduces adjustment delays or makes the target prone to shifting due to external interference. Therefore, this embodiment first presets three types of trigger thresholds and two types of preset times; then, it executes dual closed-loop control according to steps S21-S23. Through the collaborative logic of "inner layer rapid response + outer layer fallback correction," it prioritizes the low-latency characteristics of the gimbal for rapid alignment, and then uses UAV position adjustments to offset stubborn deviations and strong disturbances. This achieves initial centering within milliseconds and stable convergence within seconds, ensuring both alignment speed and improved anti-disturbance capability, keeping the target consistently stable in the center of the image, providing a precise target position basis for subsequent image acquisition.

[0038] As an optional implementation, steps S211 to S213 are shown in the step of activating the inner gimbal closed-loop control to adjust the gimbal attitude so that the inspection target moves toward the center of the image.

[0039] S211. Based on the pixel deviation in the position deviation information, calculate the desired angular velocity adjustment amount in the gimbal coordinate system used to eliminate the pixel deviation; S212. The desired angular velocity adjustment amount is calculated and smoothed by an event-triggered PID controller to generate a gimbal angular velocity control command. S213. After the gimbal angular velocity control command is limited to a preset maximum angular velocity value, it is sent to the gimbal actuator to adjust the pitch and yaw angles at the specified angular velocity to move the inspection target toward the image center.

[0040] It is understandable that pixel deviation information cannot directly drive the gimbal actuator, and simply outputting angular velocity is prone to overshoot and jitter, resulting in insufficient target alignment accuracy. This embodiment first calculates the desired angular velocity adjustment amount in the gimbal coordinate system to eliminate deviation based on the mapping relationship between pixel deviation (Δu, Δv) in the position deviation and the gimbal-image coordinate system; then, the desired angular velocity adjustment amount is calculated by an event-triggered PID controller, and instantaneous fluctuations are eliminated by a smoothing filtering algorithm to generate a stable gimbal angular velocity control command; finally, the control command is limited to ensure that it does not exceed the preset maximum angular velocity value of 30° / s, and then sent to the gimbal actuator to drive it to adjust the pitch and yaw angles at the specified angular velocities. This embodiment achieves accurate mapping from pixel deviation to gimbal action, avoiding overshoot and jitter while ensuring the rapid response of gimbal adjustment.

[0041] For example, an event-triggered PID controller is used with the following parameters: Kp=0.02, Ki=0.001, Kd=0.01, sampling period T_s=50ms, and the formula for calculating the desired angular velocity of the gimbal is: ω=Kp·Δu+Ki·∫Δu·dt+Kd·(Δu-Δu_prev) / T_s, with the maximum angular velocity limited to 30° / s, where Δu_prev is the pixel coordinate deviation value at the last sampling time.

[0042] As an optional implementation, the step of initiating the outer UAV position closed-loop control to adjust the UAV position is shown in S221~S222.

[0043] S221. Based on the position deviation information, the flight status parameters of the UAV, and the parameters of the image acquisition device, the UAV position fine-tuning amount in the computer body coordinate system; S222. The fine-tuning amount is converted into a smooth speed command, and the smooth speed command is subjected to speed constraint processing and then sent to the UAV flight control system to adjust the UAV position.

[0044] It should be noted that when the inner gimbal closed loop fails to stably center the target within the set time, the system automatically initiates the outer UAV position closed loop compensation; the specific triggering conditions are: continuous excessive deviation: pixel deviation fails to converge for 5 seconds (|Δu|>30 pixels or |Δv|>30 pixels); transient deviation exceeds the limit: single frame pixel deviation exceeds the threshold (|Δu|>80 pixels or |Δv|>80 pixels).

[0045] Understandably, when the inner gimbal closed loop cannot eliminate continuous deviations or transient large deviations, it is necessary to use UAV position fine-tuning correction to break through the gimbal adjustment limit and avoid target offset affecting image acquisition quality. Therefore, this embodiment first clarifies the core parameters (position deviation information (Δu, Δv), flight status parameters including UAV flight altitude, image acquisition device parameters are camera equivalent focal length f≈1500 pixels, empirical proportional coefficient kx=0.12, ky=0.13), and then executes S221~S222 to achieve fine-tuning of UAV position. This avoids body shaking caused by step commands and accurately offsets stubborn deviations and wind disturbances, ensuring that the inspection target is stably converged to the center area of ​​the image.

[0046] Furthermore, the position fine-tuning amount of the UAV is calculated using the displacement mapping formula: ; Where Δu and Δv are the pixel deviations (in pixels) between the center of the inspected target and the center of the image in the image coordinate system; h is the flight altitude of the UAV relative to the target (in meters), obtained through RTK / barometer fusion positioning, with a typical value of 8-15m; f is the equivalent focal length of the camera (in pixels), which can be calculated through the camera intrinsic parameter matrix: f = fmm × sensor width; actual sensor width f = actual sensor width fmm × sensor width. For a typical 4mm lens, f ≈ 1500 pixels; kx and ky are empirical scaling coefficients, set to 0.1-0.15 according to actual calibration, used to compensate for model errors and environmental factors such as wind disturbance.

[0047] Furthermore, to avoid drone jitter caused by a step response, the displacement increment is converted into a smooth velocity command, as shown in the following formula: ; Where Tresponse=0.5 s, Tresponse=0.5s is the system response time constant; the maximum horizontal velocity constraint is: vmax=1.0 m / s; the maximum acceleration constraint is: amax=0.5m / s^2; to prevent the UAV from getting too close to the wire or obstacle, a hard constraint for position adjustment is set, with a maximum horizontal adjustment range of ±0.5 m (with the initial hovering point as the origin). When the cumulative adjustment is close to the boundary, the system automatically decays the speed command and stops in advance.

[0048] Furthermore, the system automatically exits the outer UAV position closed-loop control when any of the following conditions are met: Manual Exit: The system receives an exit command from the remote control. Deviation convergence: for 3 consecutive seconds, |Δu| < 15s and |Δv| < 15s; Adjustment timeout: A single position adjustment lasts for more than 10 seconds; Boundary reached: The cumulative adjustment amount reaches the safety boundary (±0.5m); Upon exiting, the system smoothly returns to zero speed (gradually changing over 1 second), restores pure PTZ closed-loop control mode, and records the adjusted parameters for pre-compensation in similar scenarios in the future.

[0049] S3. Perform image quality assessment on the inspection target located in the central area, and adaptively adjust the exposure parameters and focus parameters of the image acquisition device to generate the target image based on the image quality assessment results.

[0050] As an optional implementation, the steps of performing image quality assessment on the inspection target located in the central area and adaptively adjusting the exposure parameters and focus parameters of the image acquisition device to generate the target image based on the image quality assessment results are shown in S31~S35.

[0051] S31. Perform multi-dimensional illumination status assessment on the image area containing the inspection target, extract illumination features, and calculate a comprehensive illumination evaluation index.

[0052] Understandably, single-dimensional lighting assessment cannot fully reflect the lighting conditions in complex scenarios of power distribution network inspection (such as direct sunlight or tree shade), which can easily lead to overexposure or underexposure due to subsequent dimming deviations. Therefore, this embodiment first locks the ROI region containing the inspection target (expanding the target box by 20%), and then extracts three types of core lighting features (RGB mean (Lr, Lg, Lb), contrast ratio C, and highlight ratio S). h By using a fusion formula to calculate a comprehensive illumination evaluation index, this embodiment covers illumination dimensions such as brightness, detail level, and strong light distribution through complementary multi-dimensional features, thereby achieving a quantitative assessment of illumination status and avoiding the one-sidedness of single-dimensional assessment.

[0053] It should be noted that the RGB mean values ​​(Lr, Lg, Lb) are obtained by averaging the gray values ​​of the R, G, and B channels of all pixels within the ROI region.

[0054] It should be noted that, secondly, the contrast C uses the Tenengrad gradient function, C=Σ|∇I|² / (M·N), where M=640, N=640, and the threshold range C∈[500, 3000] is normal.

[0055] It should be noted that the highlight ratio S h The overexposure threshold S is obtained by comparing the number of pixels with a brightness value >250 within the ROI region to the total number of pixels. h >0.15.

[0056] It should be noted that the comprehensive illumination evaluation index is calculated using the following fusion formula: L_index=α·(mean(L_r,L_g,L_b)-128) / 128+β·(C-1750) / 1750+γ·S h ; The weighting parameters are set as follows: α=0.5, β=0.3, γ=0.2.

[0057] S32. Input the comprehensive illumination evaluation index into the preset exposure decision model, obtain and execute the corresponding exposure parameter adjustment action, the exposure parameter adjustment action includes at least adjusting the ISO value, exposure compensation value or shutter speed.

[0058] It should be noted that before implementing this embodiment, an exposure decision model needs to be constructed. This model is trained on a large number of scene-action pairs (e.g., 1 million), contains two fully connected feature extraction layers (dimensions 64 and 32), uses ReLU6 as the activation function, and is optimized by INT8 quantization.

[0059] It should be noted that 1 million data points were collected in both the simulation environment and the real-world scenario to construct scene-action pairs. Each scene-action pair contains the following data: ; Reward function design: ; Brightness Bonus: ; Contrast Bonus: ; Highlight penalty: ; Shadow Punishment: ; Action change penalty: ; Weighting coefficients: w1=0.4, w2=0.3, w3=0.2, w4=0.1, λ=0.05; s t To reinforce the system state at time t during learning, information such as the illumination characteristics (RGB mean, contrast, etc.), sharpness characteristics, and target position deviation of the target image at that time is required; at To reinforce the exposure adjustment action selected at time t during learning; r t To enhance the reward value at time t during learning, it is calculated jointly by factors such as brightness, contrast, highlights, shadows, and motion changes; s t+1 To reinforce the system state at time (t+1) during learning, i.e., to execute action s t The updated image feature state; L global C is the global brightness normalized value of the target area; S is the contrast value of the target area; h It is the highlight ratio of the target area, S s △Param is the shadow percentage of the target area, and △Param is the parameter change of the current action.

[0060] It is understandable that the comprehensive illumination evaluation index can only quantify the illumination state. It needs to be transformed into targeted exposure parameter adjustment actions to improve image illumination quality. Moreover, traditional fixed-rule dimming cannot adapt to complex dynamic changes in illumination. Therefore, this embodiment first constructs a preset exposure decision model, and then inputs the comprehensive illumination evaluation index L_index into the model. The model outputs a 7-dimensional Q-value vector corresponding to 7 types of discrete exposure parameter adjustment actions (ISO+50, ISO-50, EV+0.3, EV-0.3, shutter speed ×1.1, shutter speed ×0.9, hold). The "ISO-50" with the largest Q value is selected as the optimal adjustment action. Then, this action is executed to adjust the current ISO value from 400 to 350. This embodiment learns the mapping relationship between illumination state and exposure parameters through a reinforcement learning model, and uses Q-value decision to select the optimal action, realizing dynamic adaptive adjustment of exposure parameters under complex illumination scenes. This allows the comprehensive illumination evaluation index to quickly converge to a suitable range of |L_index|<0.1, effectively avoiding overexposure and underexposure problems.

[0061] S33. After the lighting conditions meet the preset conditions, the image clarity of the inspection target area is evaluated to obtain the corresponding clarity evaluation value.

[0062] It is understandable that meeting the lighting requirements only ensures suitable image brightness. Insufficient zoom, wind disturbance, and other factors may still cause blurry details in the inspection target area. Therefore, it is necessary to quantify the sharpness to determine whether the defect identification requirements are met. Thus, this embodiment first clarifies that the preset condition for meeting the lighting requirements is a comprehensive lighting evaluation index |L_index| < 0.1, and then locks the ROI region of the inspection target (specifically, a 3×3 Laplacian convolution kernel is used to perform a convolution operation on this ROI region, V = Σ|∇²I(x, y)| / N_roi). Blur rating is calculated as follows: ∇²I(x, y) is the output of a 3×3 Laplacian operator convolution, with a value range of [-4, 4], and N_roi is the number of pixels in the ROI region. After normalization: V_norm = 20V / Vmax, the sharpness evaluation value is obtained, where Vmax is the theoretical maximum blur value. After normalization, V_norm is mapped to the [0, 20] interval. Threshold settings are: sharp: V_norm > 15.0, slightly blurry: V_norm > 15.0, severely blurry: V_norm ≤ 10.0). This embodiment utilizes the sensitivity of the Laplacian operator to image edge details, quantifies the richness of details through convolution operations, and achieves an objective quantitative evaluation of sharpness. This avoids the bias of subjective judgment and accurately identifies subtle blur problems.

[0063] S34. If the sharpness evaluation value is lower than the preset sharpness threshold, adjust the optical focal length or perform digital image sharpening processing.

[0064] It is understandable that even with suitable lighting conditions, the target area may still suffer from insufficient clarity due to factors such as mismatched focal length with target distance or slight shaking caused by wind disturbance. Without targeted optimization, this can affect the accuracy of subsequent defect identification. Therefore, this embodiment first defines preset clarity thresholds (clear: V_norm>15.0, slightly blurry: V_norm>15.0, severely blurry: V_norm≤10.0). When the clarity evaluation value is lower than the threshold, optical focal length adjustment is initiated first: if the target box area occupies less than 15% of the image, the focal length is gradually adjusted from 24mm to 48mm (the upper limit of 2x zoom) in steps of 1mm / frame, while simultaneously employing a "pull-push-search" algorithm (coarse adjustment step size 10). With a focus unit and a fine-tuning step size of 2 units, and a search range of 0-1023, the search searches for the sharpness peak. If the sharpness evaluation value still does not meet the standard after the optical zoom reaches its upper limit, digital image sharpening is performed. The USM sharpening algorithm is used, with a sharpening radius r=2, intensity s=150%, and threshold t=5. Targeted sharpening is performed on the 50% area in the center of the ROI. This embodiment achieves a precise improvement in sharpness through a hierarchical optimization strategy of "optical zoom to improve physical resolution + digital sharpening to compensate for detail contrast". This allows the sharpness evaluation value to quickly converge to the sharpness standard of V_norm>15.0, ensuring that the detail resolution of the inspection target meets the input requirements of the defect recognition model and avoiding defect omissions due to image blur.

[0065] S35. Repeat S31~S34 until both the comprehensive illumination evaluation index and the sharpness evaluation value meet the preset imaging quality standard to obtain the target image.

[0066] This embodiment uses iterative feedback to dynamically offset environmental disturbances and adjustment deviations, achieving dual stability in both lighting suitability and image clarity. This ensures that the output target image meets the input requirements of the defect recognition model in terms of brightness distribution and detail sharpness, avoiding invalid acquisitions and repeated re-flights due to insufficient single adjustments, and improving the first-pass yield of inspection imaging.

[0067] S4. Determine whether the target image meets the image quality standards. If it does, perform image capture; otherwise, perform S2.

[0068] It is understandable that even after closed-loop optimization of lighting and sharpness, the target image may still deviate from the standard due to sudden environmental disturbances (such as instantaneous strong winds or sudden changes in lighting) or cumulative deviations in the adjustments of various modules. This could result in any of the following indicators deviating from the standard: target centering, lighting, or sharpness. Directly capturing such an image would produce an invalid image. Therefore, this embodiment first clarifies that the image quality standard is that the target is centered (|Δu|<10 and |Δv|<10), the lighting is suitable (|L_index|<0.1), and the sharpness meets the standard (V_norm>15.0) simultaneously. Then, the optimized target image is synchronously verified: for example, if the target image detection result is Δu=8, Δv=7, and L_index|<10, then the image quality standard is 10. If x=0.07 and V_norm=16.2 both meet the standards, the camera will automatically take a picture. If Δu=13 (target offset), L_index=0.13 (lighting exceeds the standard), or V_norm=13.8 (insufficient clarity) is detected, and any of the indicators does not meet the standard, the system will return to S2 to restart the dual closed-loop collaborative control. This will synchronously link the subsequent lighting adjustment and clarity optimization processes. Through a full-dimensional quality verification and process backtracking mechanism, a full-process closed loop of alignment-optimization-verification-backtracking is constructed to ensure that the captured images meet the input requirements for distribution network inspection defect identification, eliminate invalid shooting and repeated re-flying, and greatly improve the efficiency and data reliability of inspection operations.

[0069] Example 2: One technical solution provided in this embodiment of the invention is a multimodal image adaptive optimization system, applicable to the multimodal image adaptive optimization method for UAV inspection described in Example 1, such as... Figure 2 As shown, it includes: The edge computing module 101 performs target recognition on the real-time video stream acquired by the UAV inspection, and obtains the inspection target and its position deviation information in the image coordinates; The dual-closed-loop control module 102 generates attitude adjustment commands based on the position deviation information and the dual-closed-loop collaborative control strategy to adjust the inspection target to the center area of ​​the image. The quality assessment module 103 performs image quality assessment on the inspection target located in the central area, and adaptively adjusts the exposure parameters and focus parameters of the image acquisition device to generate the target image based on the image quality assessment results. The decision trigger module 104 performs image capture when the target image meets the image quality standards.

[0070] In this embodiment, the edge computing module receives a multimodal real-time video stream from a UAV. After frame extraction, preprocessing, and multi-view image fusion, it inputs a lightweight improved YOLO model for inference and outputs the inspection target (such as an insulator string) and its position deviation information (Δu, Δv). Based on this deviation information, the dual closed-loop control module generates attitude adjustment commands through a collaborative strategy of rapid centering using the inner gimbal PID and fine-tuning of the UAV position to stabilize the target in the center region of the image (|Δu|<10 and |Δv|<10). For the target in the center region, the quality assessment module first extracts the RGB mean, contrast, and other illumination features to calculate the comprehensive illumination evaluation index L_index, and adaptively adjusts I_index. Parameters such as SO and exposure compensation are set so that |L_index| < 0.1. Then, the sharpness evaluation value V_norm is calculated using the Laplacian operator. The optical focal length is optimized or USM sharpening is performed to make V_norm > 15.0. The decision triggering module simultaneously verifies the three standards of target centering, suitable lighting, and sharpness compliance. It determines that the image quality meets the standards and triggers shooting. Through the data flow linkage and collaborative control of each module, a closed-loop optimization mechanism is constructed to realize the automation and precision of the inspection image from target alignment to quality compliance and shooting triggering. This greatly improves the first-shot pass rate of images and the efficiency of inspection operations, and provides high-quality and standardized image data support for power distribution network defect identification.

[0071] Example 3: One technical solution provided in this embodiment of the invention is: an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the multimodal image adaptive optimization method for UAV inspection as described in Example 1.

[0072] Example 4: One technical solution provided in this embodiment of the invention is: a storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the multimodal image adaptive optimization method for UAV inspection as described in Example 1 above.

[0073] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

[0074] In the embodiments provided in this application, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another structure, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between structures or units, and may be electrical, mechanical, or other forms.

[0075] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, in the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The specific embodiments described above are preferred embodiments of the multimodal image adaptive optimization method and system adapted to UAV inspection of the present invention, and are not intended to limit the specific scope of the present invention. The scope of the present invention includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. A multimodal image adaptive optimization method adapted to UAV inspection, characterized by: include: S1. Perform target recognition on the real-time video stream acquired by the UAV inspection to obtain the inspection target and its positional deviation information in the image coordinates; S2. Based on the position deviation information and combined with the dual closed-loop collaborative control strategy, generate attitude adjustment instructions to adjust the inspection target to the center region of the image; S3. Perform image quality assessment on the inspection target located in the central area, and adaptively adjust the exposure parameters and focus parameters of the image acquisition device to generate the target image based on the image quality assessment results. S4. Determine if the target image matches; if it does, perform image capture. If it does not meet the requirements, proceed to step S2.

2. The multimodal image adaptive optimization method for UAV inspection according to claim 1, characterized in that: The steps for performing target recognition on the real-time video stream acquired by the UAV inspection and obtaining the inspection target and its positional deviation information in the image coordinates are as follows: S11. Input the fused image formed by frame extraction and preprocessing of the real-time video stream into the target detection model for inference, and output the target category, confidence level and center pixel coordinates in the image coordinate system. S12. The position deviation information is calculated based on the center pixel coordinates and the preset image center coordinates.

3. The multimodal image adaptive optimization method for UAV inspection according to claim 2, characterized in that: The target detection model is obtained by training a lightweight improved YOLO model on test data, and specifically includes: YOLOv8n was selected as the baseline model. The number of network layers and network channels of the backbone network of the baseline model were compressed. At the same time, the network structure of the Neck part of the baseline model was enhanced and the EMA attention mechanism module was introduced to obtain a lightweight improved YOLO model. A training dataset consisting of inspection images of power distribution towers with target annotations is constructed; data augmentation strategies including Mosaic, MixUp, and HSV spatial perturbations are used to augment the training dataset; The lightweight improved YOLO model is trained on the enhanced training dataset to obtain the object detection model.

4. The multimodal image adaptive optimization method for UAV inspection according to claim 1, characterized in that: The steps for generating attitude adjustment commands based on the position deviation information and a dual-closed-loop collaborative control strategy to adjust the inspection target to the center region of the image are as follows: S21. Determine whether the position deviation information exceeds the first trigger threshold; if so, start the inner gimbal closed-loop control to adjust the gimbal attitude so that the inspection target moves towards the center of the image. S22. Continuously monitor the position deviation information. If it fails to converge to within the second trigger threshold within a first preset time, or if the position deviation information instantaneously exceeds the third trigger threshold, then synchronously start the outer layer UAV position closed-loop control to adjust the UAV position. S23. When the position deviation information continues to converge to within the second trigger threshold within a second preset time, exit the outer layer UAV position closed-loop control and maintain the inner layer gimbal closed-loop control until the target is stable in the center area of ​​the image.

5. The multimodal image adaptive optimization method for UAV inspection according to claim 4, characterized in that: The steps for activating the inner gimbal closed-loop control to adjust the gimbal attitude and move the inspection target toward the image center are as follows: S211. Based on the pixel deviation in the position deviation information, calculate the desired angular velocity adjustment amount in the gimbal coordinate system used to eliminate the pixel deviation; S212. The desired angular velocity adjustment amount is calculated and smoothed by an event-triggered PID controller to generate a gimbal angular velocity control command. S213. After the gimbal angular velocity control command is limited to a preset maximum angular velocity value, it is sent to the gimbal actuator to adjust the pitch and yaw angles at the specified angular velocity to move the inspection target toward the image center.

6. The multimodal image adaptive optimization method for UAV inspection according to claim 4, characterized in that: The steps for initiating closed-loop control of the outer drone position to adjust the drone's position are as follows: S221. Based on the position deviation information, the flight status parameters of the UAV, and the parameters of the image acquisition device, the UAV position fine-tuning amount in the computer body coordinate system; S222. The fine-tuning amount is converted into a smooth speed command, and the smooth speed command is subjected to speed constraint processing and then sent to the UAV flight control system to adjust the UAV position.

7. The multimodal image adaptive optimization method for UAV inspection according to claim 4, characterized in that: The steps for performing image quality assessment on the inspection target located in the central region, and adaptively adjusting the exposure and focus parameters of the image acquisition device to generate the target image based on the image quality assessment results are as follows: S31. Perform a multi-dimensional illumination status assessment on the image region containing the inspection target, extract illumination features and calculate a comprehensive illumination evaluation index. S32. Input the comprehensive illumination evaluation index into the preset exposure decision model, obtain and execute the corresponding exposure parameter adjustment action, the exposure parameter adjustment action includes at least adjusting the ISO value, exposure compensation value or shutter speed; S33. After the lighting conditions meet the preset conditions, the image clarity of the area where the inspection target is located is evaluated to obtain the corresponding clarity evaluation value. S34. If the sharpness evaluation value is lower than the preset sharpness threshold, adjust the optical focal length or perform digital image sharpening processing. S35. Repeat S31~S34 until the comprehensive illumination evaluation index and the sharpness evaluation value both meet the preset imaging quality standard to obtain the target image.

8. A multimodal image adaptive optimization system, applicable to the multimodal image adaptive optimization method for UAV inspection as described in any one of claims 1 to 7, characterized in that: include: The edge computing module performs target recognition on the real-time video stream acquired by the UAV inspection, and obtains the inspection target and its positional deviation information in the image coordinates; The dual-closed-loop control module generates attitude adjustment commands based on the position deviation information and the dual-closed-loop collaborative control strategy to adjust the inspection target to the center region of the image. The quality assessment module performs image quality assessment on the inspection targets located in the central area, and adaptively adjusts the exposure parameters and focus parameters of the image acquisition device to generate target images based on the image quality assessment results. The decision trigger module executes image capture when the target image meets the image quality standards.

9. An electronic device, characterized in that: The system includes a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the multimodal image adaptive optimization method for UAV inspection as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the multimodal image adaptive optimization method for UAV inspection as described in any one of claims 1 to 7.

Images