Unmanned aerial vehicle inspection method and system

Abstract

The unmanned aerial vehicle inspection method and system provided in the application include at least a ground station and an airborne edge computing platform deployed on an unmanned aerial vehicle. The airborne edge computing platform determines a defect candidate region based on a target region image collected during inspection, and generates an ROI spatial mask image according to a multi-modal spatial registration residual error, a motion blur estimation length, and a distance adaptive correction parameter. The defect semantic joint value is determined according to a real-time communication state, a defect confidence, and a defect severity level, and the defect region and the background region are subjected to differential image processing based on the joint value to generate region encoding data and send the region encoding data to the ground station. The ground station reconstructs an image according to the region encoding data, and generates a re-inspection control instruction when the detection result meets the re-inspection condition and sends the re-inspection control instruction to the airborne edge computing platform to control the unmanned aerial vehicle to perform target region re-inspection. In this way, the transmission effectiveness of the inspection detection information is improved, and the intelligent feedback loop of the inspection result is realized.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV inspection method and system. Background Technology

[0002] With the development of low-altitude economy and intelligent operation and maintenance technology, drone inspection is gradually being applied to scenarios such as power facilities, industrial equipment, and transportation infrastructure. Drone inspection can acquire information about target areas by carrying image acquisition equipment and combine it with ground analysis systems to achieve remote detection, which improves inspection efficiency and operational safety compared to traditional manual inspection methods.

[0003] In current drone inspection processes, it is typically necessary to transmit collected image or video data to a ground-based system for analysis and processing. Due to the large volume of inspection data and the limited communication resources between the drone and the ground station, practical applications are susceptible to factors such as network status changes and transmission distance, leading to ineffective data transmission and making it difficult to ensure that the ground station can obtain crucial information in a timely manner to meet inspection requirements. Furthermore, existing inspection systems usually require manual intervention to determine subsequent processing methods after data analysis.

[0004] Therefore, how to improve the effectiveness of key detection information transmission under limited communication resources during drone inspections and achieve intelligent feedback loops for inspection results has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the technical deficiencies in the prior art regarding how to improve the effectiveness of key detection information transmission and achieve intelligent feedback loops for inspection results under limited communication resources during UAV inspection processes.

[0006] In a first aspect, this application provides a UAV inspection method applied to a UAV inspection system, the system including at least a ground workstation and an airborne edge computing platform deployed on the UAV, the method including: The airborne edge computing platform determines the candidate defect region based on the target area image collected during UAV inspection, and expands the candidate defect region according to the multimodal spatial registration residual, motion blur estimation length and object distance adaptive correction parameters to obtain the ROI spatial mask image. The joint semantic value of the defect is determined based on the real-time communication status and the defect confidence and severity level of the defect candidate region. Based on the joint semantic value of defects, image data processing is performed on the defect region and background region in the ROI spatial mask image to generate region coding data, and the region coding data is sent to the ground workstation. The ground workstation reconstructs the image based on the regional coding data, and when the detection result of the reconstructed image meets the re-inspection conditions, it generates a re-inspection control command and sends it to the airborne edge computing platform to control the UAV to perform target area re-inspection.

[0007] Optionally, based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters, the defect candidate region is expanded to obtain a ROI spatial mask image, including: Based on the pixel offset error in the multimodal image registration process corresponding to the target region image, determine the multimodal spatial registration residual; The motion blur estimation length is determined based on the drone's speed, the exposure time of the image acquisition device on the drone, and the pixel resolution. The adaptive correction parameters for object distance are determined based on the focal length of the image acquisition device, the physical size of the sensor, the target object distance, and the single pixel size. The number of expanded pixels of the ROI is calculated based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters. Based on the number of pixels expanded in the ROI, the boundary of the defect candidate region is expanded to obtain the ROI spatial mask image.

[0008] Optionally, the number of ROI extended pixels is calculated based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters, including: Calculate the number of expanded pixels of the ROI using the following formula:

[0009] in, Increase the number of pixels for the ROI. Based on a preset minimum pixel threshold, For multimodal space registration residuals, For length estimation in motion fuzzy conditions, For adaptive correction parameters of object distance, The focal length of the image acquisition device. For the physical dimensions of the sensor, For the target object distance, For a single pixel size, , and This is the weighted adjustment coefficient.

[0010] Optionally, the joint semantic value of a defect is determined based on the real-time available bandwidth and the defect confidence and severity level of the defect candidate region, including:

[0011] in, For defect semantic joint value, For the defect confidence level, The severity level of the defect. This represents the maximum severity level of the defect. For real-time available bandwidth, The rated standard bandwidth required for full-frame lossless transmission and These are the weighting coefficients, and .

[0012] Optionally, based on the joint semantic value of defects, image data processing is performed on the defect region and background region in the ROI spatial mask image to generate region-coded data, including: Based on the joint semantic value of defects, the coding control information corresponding to the defect region and the background region in the ROI spatial mask image is determined respectively. Based on the encoding control information, corresponding image data processing is performed on the defect area and the background area respectively to generate corresponding region encoding data.

[0013] Optionally, based on the joint semantic value of the defect, the encoded control information corresponding to the defect region and the background region in the ROI spatial mask image is determined, including: When the joint semantic cost of the defect is less than the first preset threshold, the encoding control information corresponding to the defect region in the ROI spatial mask image is configured to be encoded using the first quantization parameter, and the encoding control information corresponding to the background region in the ROI spatial mask image is configured to be encoded using the second quantization parameter, wherein the first quantization parameter is less than the second quantization parameter. When the joint cost of the defect semantics is not less than the first preset threshold and less than the second preset threshold, the encoding control information corresponding to the defect region is configured to be encoded using the third quantization parameter, and the encoding control information corresponding to the background region is configured to be encoded using the fourth quantization parameter. The second quantization parameter is not greater than the third quantization parameter, and the third quantization parameter is less than the fourth quantization parameter. When the joint semantic cost of the defect is greater than or equal to the second preset threshold, the encoding control information corresponding to the defect area is configured to be encoded using a lossless encoding method, and the encoding control information corresponding to the background area is configured to stop encoding and transmission.

[0014] Optionally, image reconstruction is performed based on region-coded data, including: If the region coding data includes the coding data of the defect region and the coding data of the background region, then the coding data of the background region is decoded and its size is restored to obtain the background restored image; Based on the location coordinates of the defect area, the decoded defect recovery image is mapped to the corresponding position in the background recovery image; Based on the boundary between the defect recovery image and the background recovery image, a preset pixel width is extended to both sides of the boundary to obtain the transition region; Based on the gradient weight values ​​corresponding to each pixel in the transition region, the defect restoration image and the background restoration image are weighted and fused to obtain the reconstructed image.

[0015] Optionally, when the detection results of the reconstructed image meet the re-inspection conditions, a re-inspection control command is generated and sent to the airborne edge computing platform, including: The reconstructed image is subjected to target detection, defect classification, and confidence evaluation to obtain the defect classification results, the classification probability of the defect classification results, and the confidence evaluation results of the defect targets. Based on the infrared and visible light data corresponding to the reconstructed image, infrared interpretation and visible light interpretation are performed respectively to obtain infrared interpretation results and visible light interpretation results. If the confidence assessment result is lower than the preset safety threshold, or the defect classification result of the defect target is a suspected high-risk defect and the corresponding classification probability is lower than the preset probability threshold, or there is a semantic conflict between the infrared interpretation result and the visible light interpretation result, then the detection result of the reconstructed image is determined to meet the re-inspection conditions, and the location of the re-inspection target is determined according to the location information corresponding to the defect target. Based on the target location and data collection requirements for re-inspection, a re-inspection control command is generated and sent to the airborne edge computing platform.

[0016] Optionally, a re-inspection control command is generated based on the target location and re-inspection data collection requirements, and sent to the airborne edge computing platform, including: Based on the spatial pose information of the target area image collected by the UAV, the location of the re-inspection target is spatially mapped to determine the three-dimensional spatial coordinates corresponding to the location of the re-inspection target. The flight path is planned based on the three-dimensional spatial coordinates to determine the waypoints for re-inspection, and the corresponding flight altitude, gimbal attitude and image acquisition parameters are determined according to the re-inspection and data acquisition requirements. Based on the re-inspection waypoints, flight altitude, gimbal attitude, and image acquisition parameters, a re-inspection control command is generated.

[0017] Secondly, this application provides a drone inspection system, comprising: the system includes at least a ground workstation and an airborne edge computing platform deployed on the drone, the system being used to perform the steps of any of the drone inspection methods described in the above embodiments.

[0018] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The UAV inspection method and system provided in this application determine candidate defect regions based on target area images acquired by the UAV inspection using an airborne edge computing platform. These candidate regions are then expanded using multimodal spatial registration residuals, motion blur estimation lengths, and adaptive distance correction parameters to obtain a ROI spatial mask image that characterizes the spatial extent of the defect region. Further, based on real-time communication status and the defect confidence and severity levels corresponding to the candidate defect regions, a joint semantic value for the defect is determined. Image data processing is then performed on both the defect region and the background region based on this value to generate region coding data. This allows different regions in the inspection image to be differentiated based on their defect detection value, prioritizing the transmission of key image information related to defect detection under limited communication resources, thus improving the effectiveness of inspection information transmission. Simultaneously, a ground workstation reconstructs the image based on the region coding data and determines whether to generate a re-inspection control command based on the detection results of the reconstructed image. When the re-inspection conditions are met, the airborne edge computing platform is controlled to perform a re-inspection of the target region, enabling ground analysis results to be fed back to the UAV inspection process. This forms a closed-loop inspection feedback system driven by detection results, improving the information interaction efficiency of the UAV inspection process. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the unmanned aerial vehicle (UAV) inspection method provided in this application embodiment; Figure 2 A flowchart illustrating the process of determining encoding control information provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an unmanned aerial vehicle (UAV) inspection system provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application provides a UAV inspection method applied to a UAV inspection system. The system includes at least a ground workstation and an airborne edge computing platform deployed on the UAV. The UAV inspection system is used to realize defect detection, data transmission, and re-inspection control during the UAV inspection process. The airborne edge computing platform is used to process target area images acquired by the UAV in real time, and the ground workstation is used to receive inspection data, perform detection and analysis, and generate re-inspection control commands based on the detection results. Figure 1 As shown, the method may include the following steps: S110: The airborne edge computing platform determines the candidate defect region based on the target area image collected during UAV inspection, and expands the candidate defect region according to the multimodal spatial registration residual, motion blur estimation length and object distance adaptive correction parameters to obtain the ROI spatial mask image.

[0023] In this step, the airborne edge computing platform first determines the candidate defect regions in the target area based on the target area images collected during UAV inspection. Then, it combines the multimodal registration error during image acquisition, the UAV's motion state, and the shooting distance to correct and expand the range of the candidate defect regions, thus obtaining the ROI spatial mask image for subsequent data processing.

[0024] Understandably, a defect candidate region refers to an area suspected of having a defect, identified through image analysis. This could be an image region corresponding to surface damage, abnormal heating, structural changes, etc. Multimodal spatial registration residuals are used to represent positional matching deviations between different types of images, such as the spatial offset between infrared and visible light images. Motion blur estimation length is used to represent the range of image blur caused by UAV movement. Object distance adaptive correction parameters are used to adjust the expansion range of the defect region based on the distance between the UAV and the target device. ROI spatial mask images are used to identify the areas in the target region image that require focused processing.

[0025] Specifically, during the drone's flight along the pre-set inspection route, the onboard edge computing platform simultaneously acquires image data of the target area using its infrared and visible light cameras. The infrared camera performs temperature calibration on the acquired infrared images, extracting information on temperature anomalies and hotspot distribution in the target area; the visible light camera preprocesses the acquired visible light images, completing equipment structure identification, component location, and defect appearance feature extraction. Subsequently, the onboard edge computing platform performs spatial registration and feature fusion on the thermal features from the infrared images and the visual semantic features from the visible light images, determining high-confidence defect candidate regions based on the fusion results.

[0026] After determining the candidate defect region, the airborne edge computing platform performs position compensation on the candidate defect region based on the spatial registration residual between the infrared image and the visible light image; determines the motion blur estimation length based on the UAV's flight speed, attitude changes, or image edge diffusion, and expands the defect edge region that may be affected by motion; determines the object distance adaptive correction parameter based on the shooting distance between the UAV and the target device, and adjusts the defect region range at different distances, thereby generating a ROI spatial mask image.

[0027] In one implementation, when a drone inspects insulators of a transmission line, the airborne edge computing platform can use infrared images to identify areas of abnormal temperature on the insulators and visible light images to identify damaged areas on the surface of the insulators. It can determine candidate defect areas through multimodal fusion. Furthermore, by combining the registration residuals of infrared and visible light images, the motion blur effect during the drone's flight, and the distance between the drone and the insulator, the candidate areas are expanded to obtain a spatial mask image of the ROI covering the defect area.

[0028] In another implementation, when a drone inspects transformer bushings or other industrial equipment, the airborne edge computing platform can determine candidate defect regions based on features such as texture changes and edge anomalies in visible light images, or it can determine candidate defect regions by combining temperature rise anomaly information in infrared images; then, the range of candidate regions is corrected according to the current shooting distance and the drone's motion state to generate the corresponding ROI spatial mask image.

[0029] By expanding the candidate defect region based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters, the ROI spatial mask image can more accurately cover the actual defect region, reducing the omission of defect information caused by image acquisition deviation, and providing an accurate data foundation for subsequent image processing and transmission of the defect region.

[0030] S120: Determine the joint semantic value of the defect based on the real-time communication status and the defect confidence and severity level of the defect candidate region.

[0031] In this step, based on the real-time communication status and the defect confidence and severity level of the defect candidate region, the joint semantic value of the defect is determined. This value is used to comprehensively evaluate the data transmission priority of different defect regions under the current communication conditions, providing a basis for subsequent image data processing.

[0032] Understandably, real-time communication status indicates the data transmission conditions between the airborne edge computing platform and the ground workstation, and may include information such as current communication bandwidth, network latency, signal strength, or link stability. Defect confidence indicates the likelihood that a candidate defect region will be identified as a real defect, and can be determined based on the output of the defect detection model. Defect severity level indicates the degree of impact of the defect on the equipment's operating status, and can be divided into different levels based on defect type, defect range, or historical fault data. Defect semantic joint cost value comprehensively characterizes the detection and transmission value of a defect region; the higher the value, the more priority the corresponding defect region needs to be retained and transmitted.

[0033] Specifically, after identifying candidate defect regions, the airborne edge computing platform acquires the real-time communication status between the UAV and the ground workstation, and reads the defect confidence level and severity level corresponding to each candidate defect region. Subsequently, based on the real-time communication status, it determines the currently available data transmission capacity, and combines the defect detection results of the candidate defect regions to perform a joint evaluation of each candidate defect region, obtaining the corresponding joint semantic value of the defects.

[0034] Airborne edge computing platforms can determine the amount of data that can be transmitted based on real-time communication bandwidth. When communication bandwidth is low, the joint semantic value of defects corresponding to high-confidence and high-severity defect areas can be increased, allowing limited communication resources to be prioritized for transmitting critical defect information. When communication conditions are good, the transmission priority differences between different defect areas can be reduced to transmit more inspection image information. For example, for multiple defect candidate areas identified within the same inspection area, they can be sorted according to the defect confidence and severity level of each area, and the sorting results can be adjusted based on the current communication status to determine the joint semantic value of defects corresponding to each defect area.

[0035] The method for determining the joint cost value of defect semantics can be adjusted according to the actual inspection scenario. It can be obtained by weighting the real-time communication status, defect confidence, and defect severity level according to preset weights, or it can be based on a pre-trained evaluation model that outputs the corresponding joint cost value based on the input communication status parameters and defect feature parameters. For example, in a transmission line inspection scenario, for defect areas with significant temperature anomalies that may affect the safe operation of the line, a higher defect severity level can be set, prioritizing the allocation of higher data transmission value even if current communication bandwidth is limited. In a scenario involving the detection of minor scratches on equipment surfaces, the joint cost value of the corresponding area can be reduced based on a lower defect severity level.

[0036] By combining real-time communication status, defect confidence level, and defect severity level to determine the joint semantic value of defects, the airborne edge computing platform can dynamically evaluate the data value of different areas based on current communication conditions and defect detection needs, providing a basis for subsequent differentiated image processing and key detection information transmission.

[0037] S130: Based on the joint semantic value of defects, image data processing is performed on the defect region and background region in the ROI spatial mask image to generate region coding data, and the region coding data is sent to the ground workstation.

[0038] In this step, based on the joint semantic value of defects, differentiated image data processing is performed on the defect region and background region in the ROI spatial mask image to generate region-coded data suitable for transmission, and the region-coded data is sent to the ground workstation. By employing different data processing strategies for different value regions, limited communication resources are prioritized to carry key defect information.

[0039] It is understandable that the defect region refers to the part of the ROI spatial mask image corresponding to the defect candidate region, used to preserve defect-related image information; the background region refers to other image regions besides the defect region. Region coding data refers to the data obtained after encoding the ROI spatial mask image, which may include defect region coding data, background region coding data, and corresponding region location information, used for image reconstruction on the ground workstation side.

[0040] Specifically, after obtaining the joint semantic value of defects, the airborne edge computing platform determines the image data processing strategy based on the value corresponding to different regions. Specifically, for defect regions with high joint semantic value, higher-precision data processing methods can be used, such as lossless coding, low-loss compression, or high-bitrate coding, to preserve key features such as defect edges, textures, and temperature changes. For background regions with low joint semantic value, data processing methods with higher compression ratios can be used, such as lossy coding, low-bitrate coding, or regional downsampling, to reduce data transmission volume.

[0041] The airborne edge computing platform can dynamically adjust the data processing parameters corresponding to defect areas and background areas based on real-time communication status. For example, when the communication bandwidth between the UAV and the ground workstation is low, the encoding quality of high-defect semantic joint value areas can be improved, while the data volume of background areas can be reduced. When the communication bandwidth is high, the encoding accuracy of background areas can be appropriately increased to transmit more complete inspection image information. After processing, the airborne edge computing platform encapsulates the encoded data corresponding to different areas, generates regional encoded data, and sends it to the ground workstation via the communication link.

[0042] Furthermore, after completing the encoding processing of the defect area and background area, the airborne edge computing platform synchronously encapsulates the region encoding data corresponding to the defect area and the structured metadata containing information such as defect coordinates, timestamps, and UAV GPS location, enabling different types of data to be correlated and parsed on the ground workstation side. Specifically, the airborne edge computing platform can use the defect area encoding data as high-fidelity defect slice data and the background area encoding data as background video stream data. During the synchronous encapsulation process, unified time identification information and data identification information are added, such as generating a globally unique identifier (UUID) and a GPS timestamp. The UUID is written into the defect slice file name and structured metadata, and the timestamp or UUID information is written into the enhanced information field corresponding to the video stream, so that the ground workstation can synchronously match data from different channels based on data identification and time information. The encapsulated data can be sent to the ground workstation using different transmission channels according to the data type. For example, background area encoding data can be pushed in real time as a low bitrate background video stream via the SRT protocol, defect area encoding data can be uploaded as a high-fidelity defect slice image via HTTP POST, and structured metadata can be published to the ground MQTT Broker via the MQTT protocol. The airborne edge computing platform can also adjust its data transmission strategy based on the real-time communication link status. When communication bandwidth is sufficient, it can simultaneously transmit the complete background video stream, high-quality encoded data of the defect area, and complete structured metadata. When communication bandwidth is limited, it can prioritize the transmission of encoded data of the defect area and defect-related feature information, and reduce the transmission quality of the encoded data of the background area, such as reducing the frame rate of the background video stream or further performing background area downsampling. When the communication link is subjected to strong interference or is in a low-bandwidth state, it can activate the network outage protection mode, store the high-quality defect area data and background area data in the local storage medium of the airborne edge computing platform, and send simplified alarm information containing defect codes and GPS three-dimensional coordinates to the ground workstation through a narrowband communication link.

[0043] In one implementation, when a drone inspects insulators on a power transmission line, the onboard edge computing platform determines a ROI spatial mask image based on temperature anomaly areas in the infrared image and surface defect areas in the visible light image. It then prioritizes data transmission for different areas based on the joint semantic value of the defects. For defect areas with the risk of abnormal heating or breakage, a low-loss encoding method can be used to preserve defect details; for the background area surrounding the insulator, a high compression ratio encoding method can be used to reduce the data volume. Ultimately, this generates region-coded data containing high-quality coded data for the defect areas and compressed coded data for the background areas.

[0044] In another implementation, when a drone inspects industrial equipment such as transformer bushings and switchgear, the airborne edge computing platform can determine the corresponding semantic joint value of defects based on different defect types such as surface cracks, corrosion, and abnormal temperature rises. For example, for severely defective areas that may affect the safe operation of the equipment, a high-precision coding method can be used; for normal or low-risk areas, a lower-precision coding method can be used, thereby generating regional coding data suitable for the current communication conditions and sending it to the ground workstation.

[0045] By processing image data of defect areas and background areas separately based on the joint semantic value of defects, the airborne edge computing platform can reduce the amount of non-critical image data transmission under limited communication resources, and prioritize the transmission of defect information with detection value, thereby improving the effectiveness of inspection data transmission.

[0046] S140: The ground workstation reconstructs the image based on the regional coding data, and when the detection result of the reconstructed image meets the re-inspection conditions, it generates a re-inspection control command and sends it to the airborne edge computing platform to control the UAV to perform target area re-inspection.

[0047] In this step, the ground workstation performs image restoration based on the area coding data sent by the airborne edge computing platform, and performs defect detection analysis on the restored image. When the detection result meets the preset re-inspection conditions, a re-inspection control command is generated to control the UAV to return to the target area to perform further inspection, thereby realizing feedback control of the UAV behavior based on the inspection results.

[0048] It is understandable that image reconstruction refers to the process of restoring the target region image based on region coding data. This can be done by decoding and restoring based on coding parameters, or by fusing and reconstructing information from the defect region and the background region. Re-inspection conditions are used to determine whether the current detection results require the UAV to further collect information about the target region. These conditions can be determined based on parameters such as defect detection confidence, defect severity, or consistency of multiple detection results.

[0049] Specifically, after receiving the region-coded data sent by the airborne edge computing platform, the ground workstation parses the region-coded data to obtain the high-precision coded data corresponding to the defect region and the compressed coded data corresponding to the background region. Based on the coded information of each region, it performs image decoding and fusion processing to obtain a reconstructed image. Subsequently, the ground workstation can use a defect detection algorithm to perform a secondary inspection on the reconstructed image to determine the location, type, and corresponding detection confidence level of defects in the target region, and determine whether the conditions for re-inspection are met based on the detection results.

[0050] The ground workstation can determine whether the confidence level of defect detection in the reconstructed image meets the requirements for re-inspection based on a preset threshold. When the defect confidence level is higher than the preset threshold, it is determined that further re-inspection is required. It can also determine whether to trigger re-inspection based on the severity level of the defect. For example, for high-level defects that may affect the safe operation of equipment, a re-inspection request can be generated even if the current detection result has a certain degree of uncertainty. It can also make a comprehensive judgment by combining the results of multiple inspections. When abnormal features are detected continuously in the same area, it is determined that the conditions for re-inspection are met.

[0051] When the detection results meet the re-inspection conditions, the ground workstation generates a re-inspection control command based on the location of the defective area and the current UAV status, and sends the re-inspection control command to the airborne edge computing platform. The re-inspection control command may include the target re-inspection position, flight adjustment parameters, and image acquisition control parameters, which are used to control the UAV to adjust its flight path or gimbal attitude to re-acquire images of the target area.

[0052] In one implementation, when a UAV inspects insulators of a transmission line, the ground station reconstructs an image containing information on the surface condition and temperature distribution of the insulator based on the area coding data. Through secondary detection, it determines that a certain insulator area has a high temperature anomaly and a high confidence level of defect. At this time, a re-inspection control command containing the location of the target insulator and the shooting parameters is generated to control the UAV to approach the area and re-shoot.

[0053] In another implementation, when a drone inspects transformer bushings or industrial equipment, the ground workstation can detect surface cracks, corrosion, or abnormal heating of the equipment based on the reconstructed image. When the detection results show that the defect area is large or the defect level is high, the workstation can trigger a re-inspection condition and send a re-inspection control command to the drone, causing the drone to adjust the inspection route or change the shooting angle to obtain more accurate defect information.

[0054] By using a ground workstation to reconstruct images and detect defects in the regional coded data, and generating re-inspection control commands based on the detection results, the inspection data analysis results can be fed back to the UAV to execute subsequent inspection actions, realizing closed-loop control from data acquisition and defect analysis to autonomous re-inspection, and improving the automation level of the UAV inspection process.

[0055] In the above embodiments, the airborne edge computing platform determines candidate defect regions based on the target area images collected by the UAV inspection. It then expands these candidate regions by combining multimodal spatial registration residuals, motion blur estimation length, and adaptive object distance correction parameters to obtain a ROI spatial mask image that characterizes the spatial range of the defect region. Furthermore, based on the real-time communication status and the defect confidence and severity levels corresponding to the candidate defect regions, a joint semantic value for the defect is determined. Based on this value, image data processing is performed on both the defect region and the background region to generate region coding data. This allows different regions in the inspection image to be differentiated based on their defect detection value, thereby prioritizing the transmission of key image information related to defect detection under limited communication resources and improving the effectiveness of inspection information transmission. Simultaneously, the ground workstation reconstructs the image based on the region coding data and determines whether to generate a re-inspection control command based on the detection results of the reconstructed image. When the re-inspection conditions are met, the airborne edge computing platform is controlled to execute a re-inspection of the target region, enabling the ground analysis results to be fed back to the UAV inspection process. This forms a closed-loop inspection feedback system driven by the detection results, improving the information interaction efficiency of the UAV inspection process.

[0056] In one embodiment, the defect candidate region is expanded based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters to obtain a ROI spatial mask image, including: Based on the pixel offset error in the multimodal image registration process corresponding to the target region image, determine the multimodal spatial registration residual; The motion blur estimation length is determined based on the drone's speed, the exposure time of the image acquisition device on the drone, and the pixel resolution. The adaptive correction parameters for object distance are determined based on the focal length of the image acquisition device, the physical size of the sensor, the target object distance, and the single pixel size. The number of expanded pixels of the ROI is calculated based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters. Based on the number of pixels expanded in the ROI, the boundary of the defect candidate region is expanded to obtain the ROI spatial mask image.

[0057] In this embodiment, by determining the multimodal image registration error, the image blurring effect caused by the drone's movement, and the image scale change caused by the shooting distance, and comprehensively calculating the expansion range of the defect candidate region, a ROI spatial mask image covering complete defect information is generated.

[0058] Understandably, the multimodal spatial registration residual is used to characterize the pixel offset error that still exists after spatial registration of infrared and visible light images, and can be determined through the root mean square error in dual-light camera calibration experiments. The motion blur estimation length is used to characterize the range of pixel dragging in the target area caused by exposure of the image acquisition device during UAV movement, and can be determined based on the UAV's speed, exposure time, and image pixel resolution. The object distance adaptive correction parameter is used to characterize the impact of image scale changes on the expansion range of the defect area at different shooting distances, and can be determined based on the focal length of the image acquisition device, the sensor's physical size, the target object distance, and the single pixel size.

[0059] Specifically, after acquiring the target area image, the airborne edge computing platform first determines the offset error between corresponding pixel positions in the two images based on the multimodal image registration results between the infrared and visible light images, and uses this offset error as the multimodal spatial registration residual. For example, by matching hotspot areas in the infrared image with device structural features in the visible light image, the pixel offset between matching points can be calculated, and the corresponding multimodal spatial registration residual can be determined based on the root mean square error of multiple matching points.

[0060] The airborne edge computing platform determines the motion blur estimation length based on the current inspection status of the UAV. Specifically, it acquires the UAV's current flight speed, the exposure time of the image acquisition device, and the corresponding pixel resolution of the image. Based on these parameters, it calculates the pixel offset of the target region in the image during the UAV's movement, and uses this pixel offset as the motion blur estimation length. It is understandable that when the UAV's flight speed is high or the exposure time is long, the range of motion influence in the image increases, and the corresponding expansion range of the defect candidate region also needs to be increased accordingly.

[0061] The airborne edge computing platform determines adaptive distance correction parameters based on image acquisition device parameters and target object distance. Specifically, it acquires the focal length, sensor physical size, and single-pixel size of the image acquisition device, and calculates the actual projection scale of the target area in the image based on the distance between the UAV and the target device. The distance correction range for candidate defect areas is then determined based on this projection scale. For example, when the UAV approaches power transmission equipment for close-range inspection, the increased proportion of the target area in the image leads to more significant image distortion and spatial mapping errors affecting defect boundaries; in this case, the adaptive distance correction parameters can be increased. Conversely, during long-distance inspection, the corresponding correction range can be decreased.

[0062] After determining the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters, the airborne edge computing platform calculates the number of expanded pixels for the Region of Interest (ROI) based on these parameters, and expands the boundary of the defect candidate region according to the number of expanded pixels for the ROI, thus obtaining the ROI spatial mask image. Specifically, the expansion can be calculated by fusing the pixel influence range corresponding to the three parameters, for example, by weighting the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters to obtain the final number of expanded pixels; alternatively, expansion rules can be preset according to different inspection scenarios, and the expansion range can be determined based on the current parameter values.

[0063] By jointly determining the number of ROI extended pixels based on multimodal spatial registration residuals, motion blur estimation length, and object distance adaptive correction parameters, the generated ROI spatial mask image can adapt to different image acquisition errors and inspection environment changes, improving the accuracy of defect area coverage and providing a reliable foundation for subsequent defect information extraction and data transmission.

[0064] In one embodiment, the number of expanded pixels of the ROI is calculated based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters, including: Calculate the number of expanded pixels of the ROI using the following formula:

[0065] in, Increase the number of pixels for the ROI. Based on a preset minimum pixel threshold, For multimodal space registration residuals, For length estimation in motion fuzzy conditions, For adaptive correction parameters of object distance, The focal length of the image acquisition device. For the physical dimensions of the sensor, For the target object distance, For a single pixel size, , and This is the weighted adjustment coefficient.

[0066] In this embodiment, by constructing a formula for calculating the number of pixels to be expanded in the ROI, the multimodal image registration error, the image blurring effect caused by the drone's movement, and the image scale change caused by the change in shooting distance are comprehensively calculated to determine the range of pixels that the defect candidate region needs to be expanded.

[0067] Understandably, this formula converts each influencing factor into a uniform pixel expansion amount by setting corresponding weighted adjustment coefficients for different influencing factors, and constrains the calculation results by using a basic preset minimum pixel threshold to ensure that the number of expanded pixels of ROI is not less than the preset range, thereby avoiding the omission of defect boundary information due to an excessively small expansion range.

[0068] After acquiring the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters, the airborne edge computing platform inputs these parameters into the formula for calculating the number of expanded pixels of the ROI. It then calculates the influence of different factors on the boundary offset of the defect area and obtains the final number of expanded pixels of the ROI through weighted fusion. Specifically, the corresponding weighting coefficients can be adjusted according to different inspection scenarios to ensure that different factors have an adaptive impact on the expansion results. For example, in scenarios where there is a large spatial offset between infrared and visible light images, the weight corresponding to the multimodal spatial registration residual can be increased to expand the coverage of the defect area; during high-speed drone inspections, the weight corresponding to the motion blur estimation length can be increased to compensate for image edge offset caused by motion; and when capturing device details at close range, the weight corresponding to the object distance adaptive correction parameter can be increased to adapt to changes in the target area scale.

[0069] After calculating the number of expanded pixels for the Region of Interest (ROI), the airborne edge computing platform expands the boundary of the defect candidate region based on this number of expanded pixels, generating a corresponding ROI spatial mask image. The expansion process can be carried out by expanding the defect candidate region in all directions with the same number of pixels, or different expansion ranges can be set for different directions based on the image location of the defect candidate region, device structural features, or direction of motion, thereby improving the adaptability of the region expansion.

[0070] For example, in the scenario of inspecting insulators on power transmission lines, due to potential positional discrepancies between infrared hotspots and visible light structural areas, and the attitude changes during drone flight, the airborne edge computing platform can expand the defect candidate region based on the calculated number of expanded pixels of the ROI, ensuring that both abnormally heated areas and appearance defect areas are included in the ROI spatial mask image. In the scenario of inspecting transformer bushings, the expansion range can be dynamically adjusted based on the size of the surface defect and the shooting distance, enabling the coverage of finer defect boundaries during close-up shooting and reducing invalid background areas during long-distance shooting.

[0071] By comprehensively calculating the number of expanded pixels of ROI using multimodal spatial registration error, motion state, and shooting distance, the defect candidate region can be dynamically adjusted according to the actual inspection environment, improving the regional coverage accuracy of the ROI spatial mask image and providing a reliable foundation for subsequent defect region data processing and transmission.

[0072] In one embodiment, the joint semantic value of a defect is determined based on the real-time available bandwidth and the defect confidence and severity level of the defect candidate region, including:

[0073] in, For defect semantic joint value, For the defect confidence level, The severity level of the defect. This represents the maximum severity level of the defect. For real-time available bandwidth, The rated standard bandwidth required for full-frame lossless transmission and These are the weighting coefficients, and .

[0074] In this embodiment, by constructing a joint cost calculation formula for defect semantics, the defect detection results, defect severity, and current communication resource status are jointly evaluated to determine the data transmission value of different defect candidate regions.

[0075] Understandably, this formula comprehensively considers defect semantic information and communication status information. Defect confidence and severity levels reflect the importance of the defective area, while the relationship between real-time available bandwidth and the rated standard bandwidth required for full-frame lossless transmission reflects current communication conditions. By fusing these factors, the joint semantic cost of defects can simultaneously characterize both defect detection value and communication resource consumption requirements.

[0076] The airborne edge computing platform acquires the defect confidence level, defect severity level, and current real-time available bandwidth corresponding to each defect candidate region. These parameters are then input into the defect semantic joint cost value calculation formula to obtain the corresponding defect semantic joint cost value. Specifically, the weighting coefficients can be adjusted according to the inspection scenario, allowing the importance of defect information and communication status to have different impacts on the calculation results. For example, in a fine-grained equipment inspection scenario, the influence weights corresponding to defect confidence level and defect severity can be increased, prioritizing transmission of high-risk defect regions. In scenarios with limited communication resources, a non-linear adjustment term corresponding to bandwidth changes can be used to dynamically adjust the joint cost value according to changes in communication status.

[0077] When the real-time available bandwidth is close to the bandwidth required for full-frame lossless transmission, the bandwidth factor has a relatively small impact on the joint generation value. The airborne edge computing platform can primarily determine the region processing priority based on the defect detection results. When the real-time available bandwidth decreases, the bandwidth adjustment term increases, thereby increasing the joint generation value corresponding to high-value defect regions and adjusting subsequent data processing strategies. Data processing strategies can include improving the encoding quality of defect regions, reducing the amount of data in background regions, or caching data during communication anomalies.

[0078] For example, in the scenario of power transmission line inspection, when an abnormal heating area with high confidence and high severity is detected in an insulator, even if the communication bandwidth is insufficient, the area can still obtain a high joint semantic value of the defect, so as to prioritize the retention of key defect information; in the scenario of ordinary equipment inspection, for low confidence defect areas, their joint semantic value can be reduced, and non-critical data transmission can be reduced.

[0079] By calculating the joint semantic value of defects based on defect detection information and communication status, the airborne edge computing platform can dynamically evaluate the data transmission priority of different areas, thereby improving the effectiveness of transmitting key defect information under limited communication resources.

[0080] In one embodiment, based on the joint semantic value of defects, image data processing is performed on the defect region and background region in the ROI spatial mask image to generate region-coded data, including: Based on the joint semantic value of defects, the coding control information corresponding to the defect region and the background region in the ROI spatial mask image is determined respectively. Based on the encoding control information, corresponding image data processing is performed on the defect area and the background area respectively to generate corresponding region encoding data.

[0081] In this embodiment, different regions in the ROI spatial mask image are differentially encoded based on the joint semantic value of defects, so that the defect region and the background region adopt image processing methods that match their data values ​​to generate corresponding region encoded data.

[0082] Understandably, encoding control information is used to control parameter configuration during image data processing, and may include information such as encoding method, compression ratio, encoding precision, or data transmission priority. Defective areas correspond to higher data value, so it is necessary to preserve defect feature information as much as possible; background areas correspond to relatively lower data value, so a higher compression ratio can be used to reduce the overall amount of data transmitted.

[0083] After obtaining the joint semantic value of defects for each region, the airborne edge computing platform determines the coding control information corresponding to the defect region and the background region based on the joint semantic value. Specifically, for regions with high joint semantic value of defects, a higher coding precision or a lower compression ratio can be set to preserve key features such as defect edges, textures, and temperature changes; for regions with low joint semantic value of defects, a lower coding precision or a higher compression ratio can be set to reduce the communication resources occupied by background data.

[0084] The airborne edge computing platform processes image data for different regions within the ROI spatial mask image based on determined coding control information. Image data processing can employ methods such as lossless coding, low-loss coding, lossy compression, or resolution adjustment. For example, lossless or low-loss coding can be used to generate defect region coded data, while high-compression coding can be used to generate background region coded data. The coding results for different regions are then encapsulated to obtain region coded data.

[0085] In one implementation, when a drone inspects insulators on a power transmission line, the onboard edge computing platform determines the coded control information corresponding to the abnormally heated areas of the insulators based on the joint semantic value of defects. For areas with obvious temperature anomalies and high defect levels, a high-quality coding method can be used to preserve infrared temperature distribution and visible light texture information; for surrounding environmental areas, a high compression ratio coding method can be used to reduce the amount of data, ultimately generating regional coded data containing key defect information.

[0086] In another implementation, when a drone inspects industrial equipment such as transformer bushings and switchgear, the airborne edge computing platform can determine the coding strategy based on the combined cost of defective areas such as surface cracks, corrosion, or structural anomalies. For example, higher-precision coding can be used for defective areas that may affect the safe operation of the equipment, while lower-precision coding can be used for normal equipment areas, thereby reducing invalid data transmission while ensuring the integrity of defect information.

[0087] By determining the coding control information for different regions based on the joint semantic value of defects, the airborne edge computing platform can adopt different data processing strategies for defect regions and background regions. This reduces the overall amount of transmitted data while improving the retention of key defect information, thus supporting inspection data transmission under conditions of limited communication resources.

[0088] like Figure 2 As shown, in one embodiment, based on the joint semantic value of the defect, the encoded control information corresponding to the defect region and the background region in the ROI spatial mask image is determined, including: S210: When the joint cost of defect semantics is less than the first preset threshold, the encoding control information corresponding to the defect region in the ROI spatial mask image is configured to be encoded using the first quantization parameter, and the encoding control information corresponding to the background region in the ROI spatial mask image is configured to be encoded using the second quantization parameter, wherein the first quantization parameter is less than the second quantization parameter. S220: When the joint cost of the defect semantics is not less than the first preset threshold and less than the second preset threshold, the encoding control information corresponding to the defect area is configured to be encoded using the third quantization parameter, and the encoding control information corresponding to the background area is configured to be encoded using the fourth quantization parameter, wherein the second quantization parameter is not greater than the third quantization parameter and the third quantization parameter is less than the fourth quantization parameter. S230: When the joint cost of the defect semantics is greater than or equal to the second preset threshold, the encoding control information corresponding to the defect area is configured to be encoded using lossless encoding, and the encoding control information corresponding to the background area is configured to stop encoding and transmission.

[0089] In this embodiment, based on the different intervals where the semantic joint value of the defect is located, the encoding control information corresponding to the defect area and the background area in the ROI spatial mask image is determined, so that the airborne edge computing platform can dynamically adjust the data processing method of different areas according to the degree of defect risk and the current communication status, so as to realize the priority transmission of key defect information.

[0090] Understandably, quantization parameters are used to control the degree of compression during image encoding. A smaller quantization parameter indicates higher encoding accuracy and better preservation of image details; a larger quantization parameter indicates a higher degree of compression and a smaller amount of generated data. Lossless encoding is used to completely preserve pixel information in defective areas, avoiding image feature loss during the encoding process. The first and second preset thresholds are used to distinguish different encoding control states and can be set according to the actual inspection task and communication conditions; for example, the first preset threshold could be 1.2 and the second preset threshold could be 2.5.

[0091] After obtaining the joint semantic value of defects, the airborne edge computing platform compares it with a first preset threshold and a second preset threshold. Based on the comparison results, it determines the current system state and configures corresponding encoding control information. Specifically, when the joint semantic value of defects is less than the first preset threshold, it can be considered that the system is currently in full-screen high-definition mode, indicating that no obvious defects have been detected or the communication bandwidth is sufficient. The airborne edge computing platform can configure the encoding control information corresponding to the defect area to be encoded using a first quantization parameter, and configure the encoding control information corresponding to the background area to be encoded using a second quantization parameter, with the first quantization parameter being less than the second quantization parameter. The defect area can use an extremely high-quality encoding method to preserve the defect texture, edges, and detailed features as much as possible; the background area can be encoded and transmitted without spatial downsampling, maintaining its original physical resolution, with a moderate compression ratio, enabling the ground workstation to acquire complete environmental detail information.

[0092] When the joint semantic cost of defects is not less than a first preset threshold and less than a second preset threshold, it can be considered that the current state is in semantic adaptive compression mode. This indicates that a suspected defect area has been found and the communication bandwidth is limited. The airborne edge computing platform can configure the encoding control information corresponding to the defect area to be encoded using a third quantization parameter, and configure the encoding control information corresponding to the background area to be encoded using a fourth quantization parameter, ensuring that the second quantization parameter is not greater than the third quantization parameter and the third quantization parameter is less than the fourth quantization parameter. Specifically, an asymmetric encoding strategy for defect areas and background areas can be initiated. Adaptive lightweight low-loss encoding is used for defect areas, and the quantization parameters are dynamically adjusted according to the joint semantic cost of defects, so that the encoding quality of defect areas improves as the risk increases. For background areas, spatial downsampling processing can be performed, for example, by reducing the spatial resolution of the background area by a preset ratio, such as half, and increasing the quantization parameters to reduce the amount of background data, thereby prioritizing the release of communication resources for transmitting high-definition image data of defect areas.

[0093] When the joint cost of the defect semantics is greater than or equal to the second preset threshold, it can be considered that the current state is in an extreme network outage or core preservation mode, indicating that there is a sudden critical defect or the communication link is close to failure. The airborne edge computing platform can configure the encoding control information corresponding to the defect area to be encoded using a lossless encoding method, and configure the encoding control information corresponding to the background area to stop encoding and transmission. Specifically, the airborne edge computing platform can stop the transmission of the background area data stream, retain only the high-definition data stream corresponding to the defect area, and switch the defect area to a pixel-level absolute lossless compression method to preserve the original image matrix information. At the same time, when the communication link cannot meet the data transmission requirements, the lossless encoded data of the defect area can be written to the local high-speed storage medium of the airborne terminal, the network outage data protection mechanism can be activated, and a simplified structured alarm message can be sent to the ground workstation through the degraded communication link. The alarm message can include target area location information, defect type information, defect severity information, and time information.

[0094] In one implementation, when a drone performs an inspection of insulators on a power transmission line, if an abnormal insulator temperature is detected and the current communication bandwidth is sufficient, and the semantic joint cost of the defect is low, the airborne edge computing platform can adopt a full-screen high-definition mode, using high-quality encoding for the abnormal area and maintaining high-resolution transmission for the background area of ​​the line. If a suspected discharge defect is detected and the communication bandwidth decreases, a semantic adaptive compression mode is entered, using low-loss encoding for the defect area and downsampling compression for the surrounding environment. If a severe overheating defect is detected and the communication link is abnormal, an extreme network outage or core preservation mode is entered, where only the defect area information is saved and transmitted.

[0095] By dynamically switching different encoding control modes according to the range of the joint value of defect semantics, the airborne edge computing platform can match the corresponding data processing strategies under different states such as full-screen high definition, semantic adaptive compression, and extreme network outage to ensure core integrity. This reduces the amount of non-critical data transmission while ensuring the integrity of critical defect information, and improves the effectiveness and reliability of inspection information transmission under limited communication resources.

[0096] In one embodiment, image reconstruction based on region-coded data includes: If the region coding data includes the coding data of the defect region and the coding data of the background region, then the coding data of the background region is decoded and its size is restored to obtain the background restored image; Based on the location coordinates of the defect area, the decoded defect recovery image is mapped to the corresponding position in the background recovery image; Based on the boundary between the defect recovery image and the background recovery image, a preset pixel width is extended to both sides of the boundary to obtain the transition region; Based on the gradient weight values ​​corresponding to each pixel in the transition region, the defect restoration image and the background restoration image are weighted and fused to obtain the reconstructed image.

[0097] In this embodiment, based on the defect region coding data and background region coding data in the region coding data, images of different regions are decoded and restored, and a complete reconstructed image is generated through boundary fusion to restore the inspection image information on the ground workstation side.

[0098] It is understandable that a background restoration image refers to an image obtained by decoding and restoring the encoded data of the background region, while a defect restoration image refers to a high-precision image obtained by decoding the encoded data of the defect region. The transition region is used to connect the boundary regions between the defect restoration image and the background restoration image, reducing boundary differences caused by different encoding methods through pixel fusion.

[0099] After receiving the region-coded data, the ground station parses it to obtain the defect region-coded data, background region-coded data, and the corresponding location coordinates of the defect region. If the region-coded data includes both defect region-coded data and background region-coded data, the background region-coded data is first decoded and its size restored to obtain a restored background image that matches the size of the original image. Subsequently, based on the location coordinates of the defect region, the defect restored image obtained by decoding the defect region-coded data is mapped to the corresponding position in the restored background image, achieving spatial alignment between the defect region and the background region.

[0100] To eliminate the boundary abruptness between the defect recovery image and the background recovery image caused by differences in coding precision, the ground workstation constructs a transition region based on the boundary between the two. Specifically, a preset pixel width can be extended from the defect region boundary to both the defect region and the background region, for example, by 5 to 10 pixels on each side of the boundary, to form the transition region.

[0101] The ground workstation performs weighted fusion of the defect restoration image and the background restoration image based on the gradient weight values ​​corresponding to each pixel within the transition area. The gradient weight values ​​can be determined based on the position of the pixel from the defect boundary. For example, a smooth curve with cosine modulation can be used to generate the weights, so that the weights on the side closer to the background gradually approach 0, and the weights on the side closer to the defect gradually approach 1, thereby achieving a smooth transition between the two regions.

[0102] In one implementation, when a UAV inspects insulators on a power transmission line, the ground station reconstructs the background area image based on the area coding data and maps the high-resolution reconstructed image of the abnormally heated area to the corresponding location based on the coordinates of the defect area. Transition area fusion is then used to eliminate the boundary differences between the abnormal area and the surrounding environment, resulting in a complete inspection image. In another implementation, when a UAV inspects transformer bushings or industrial equipment, the ground station can reconstruct images of cracked, corroded, or abnormally heated areas based on the defect area coding data and embed them into the background image based on the area coordinates. Pixel gradient fusion is then used to reduce image discontinuities caused by different compression ratios.

[0103] By restoring the background region, accurately mapping the defect region, and performing pixel fusion based on the transition region, the differentially encoded defect region and background region can form a continuous and complete reconstructed image, improving the image reconstruction quality and providing reliable data for subsequent defect detection.

[0104] In one embodiment, when the detection result of the reconstructed image meets the re-inspection conditions, a re-inspection control command is generated and sent to the airborne edge computing platform, including: The reconstructed image is subjected to target detection, defect classification, and confidence evaluation to obtain the defect classification results, the classification probability of the defect classification results, and the confidence evaluation results of the defect targets. Based on the infrared and visible light data corresponding to the reconstructed image, infrared interpretation and visible light interpretation are performed respectively to obtain infrared interpretation results and visible light interpretation results. If the confidence assessment result is lower than the preset safety threshold, or the defect classification result of the defect target is a suspected high-risk defect and the corresponding classification probability is lower than the preset probability threshold, or there is a semantic conflict between the infrared interpretation result and the visible light interpretation result, then the detection result of the reconstructed image is determined to meet the re-inspection conditions, and the location of the re-inspection target is determined according to the location information corresponding to the defect target. Based on the target location and data collection requirements for re-inspection, a re-inspection control command is generated and sent to the airborne edge computing platform.

[0105] In this embodiment, the ground workstation determines whether a re-inspection needs to be performed based on the detection and analysis results of the reconstructed image, and generates a corresponding re-inspection control command when the re-inspection conditions are met, so as to control the UAV to re-collect target area information.

[0106] Understandably, re-inspection conditions are used to determine whether the current defect detection results need further confirmation, and can be determined based on detection confidence, suspected high-risk defect classification results, and consistency between infrared and visible light detection. Re-inspection acquisition requirements describe the acquisition parameters needed to re-acquire information about the target area, which may include shooting position, shooting angle, and image acquisition mode.

[0107] After acquiring the reconstructed image, the ground workstation performs object detection, defect classification, and confidence assessment on the reconstructed image. Specifically, the ground workstation can utilize deep learning methods such as deep cross-modal attention networks to extract multi-scale features from the defect regions in the reconstructed image. Combining the texture, structural, and temperature features of the defect regions, it outputs the defect classification result, classification probability, and confidence assessment result corresponding to the defect target. The confidence assessment result characterizes the reliability of the current defect detection result.

[0108] The ground workstation performs infrared and visible light interpretations based on the infrared and visible light data corresponding to the reconstructed image, respectively, and determines whether there is a semantic conflict between the two interpretation results. Specifically, the ground workstation can spatially correlate the structural defect level output by the visible light branch with the temperature anomaly level output by the infrared branch using a multimodal dual-light semantic decision-making method. When the defect severity level output by either mode reaches a preset danger threshold within the same spatial coordinate region, while the other mode determines it to be normal or without anomalies, a semantic conflict can be determined between the infrared and visible light interpretation results.

[0109] The ground workstation determines whether the conditions for re-inspection are met based on the above detection results. Specifically, when the confidence assessment result is lower than the preset safety threshold, it indicates that the reliability of the current defect detection result is insufficient, and a re-inspection can be triggered; when a suspected high-risk defect is detected but the corresponding classification probability is lower than the preset probability threshold, it indicates that the high-frequency features of the defect area edge are insufficient due to factors such as environmental noise, making accurate judgment impossible, and a re-inspection can be triggered; when there is a semantic conflict between the infrared interpretation result and the visible light interpretation result, a re-inspection can also be triggered.

[0110] After confirming that the conditions for re-inspection are met, the ground workstation determines the re-inspection target location based on the location of the defective target, and generates re-inspection control commands based on the re-inspection data acquisition requirements. These commands are then sent to the airborne edge computing platform to control the UAV to re-execute target area data acquisition. The re-inspection control commands may include target location coordinates, flight adjustment parameters, gimbal attitude parameters, and image acquisition parameters.

[0111] In one implementation, when a drone inspects insulators on a transmission line, the ground station detects a suspected high-temperature defect in a certain insulator area through secondary detection. However, due to insufficient classification probability caused by environmental noise, the defect type cannot be determined. In this case, a re-inspection control command can be generated based on the location of the insulator, controlling the drone to adjust the shooting angle for re-collection. In another implementation, when a drone inspects transformer bushings, if visible light data indicates a structural crack in a certain area, while infrared data indicates no corresponding temperature anomaly, the ground station can determine that there is a semantic conflict between the two modes and control the drone to perform a re-inspection based on the location of the area to obtain more accurate defect information.

[0112] By performing secondary detection, defect classification, confidence assessment, and multimodal consistency judgment on the reconstructed image, the ground workstation can automatically trigger re-inspection and generate control commands based on the detection results, realizing a closed loop of feedback from the inspection results to the UAV side and improving the reliability of defect detection.

[0113] In one embodiment, a re-inspection control command is generated based on the re-inspection target location and re-inspection data collection requirements, and sent to the airborne edge computing platform, including: Based on the spatial pose information of the target area image collected by the UAV, the location of the re-inspection target is spatially mapped to determine the three-dimensional spatial coordinates corresponding to the location of the re-inspection target. The flight path is planned based on the three-dimensional spatial coordinates to determine the waypoints for re-inspection, and the corresponding flight altitude, gimbal attitude and image acquisition parameters are determined according to the re-inspection and data acquisition requirements. Based on the re-inspection waypoints, flight altitude, gimbal attitude, and image acquisition parameters, a re-inspection control command is generated.

[0114] In this embodiment, a re-inspection control command is generated based on the re-inspection target location and re-inspection data collection requirements. By determining the spatial location corresponding to the target area, planning the re-inspection flight path, and configuring the data collection parameters, the UAV can autonomously return to the target area to perform a refined re-inspection.

[0115] Understandably, the re-inspection waypoint indicates the spatial location that the UAV needs to reach when performing a re-inspection task. Flight altitude, gimbal attitude, and image acquisition parameters are used to control the UAV's shooting state, and may include gimbal pitch angle, yaw angle, lens focal length, and infrared acquisition parameters. The re-inspection control command is used to send the flight path adjustment and acquisition control parameters to the onboard edge computing platform to control the UAV to complete the re-acquisition of the target area.

[0116] The ground station uses the spatial pose information recorded when the UAV acquires images of the target area to spatially map the location of the target to be re-inspected, determining the corresponding three-dimensional spatial coordinates of the target area. Specifically, the spatial pose information can include the three-dimensional positioning coordinates of the UAV during acquisition and the gimbal attitude information. The ground station can calculate the position coordinates of the defective target in actual space through reverse geometric projection based on the UAV's position, gimbal pitch angle, and yaw angle, and use these position coordinates as the location of the target to be re-inspected. For example, based on the UAV's GPS position and gimbal attitude information recorded during the initial inspection, the defective area in the image coordinates can be mapped to actual spatial coordinates to obtain the corresponding three-dimensional defect location.

[0117] The ground station performs trajectory planning based on the determined three-dimensional spatial coordinates, identifies the re-inspection waypoints corresponding to the UAV's re-inspection mission, and determines the flight altitude, gimbal attitude, and image acquisition parameters according to the re-inspection data acquisition requirements. Specifically, trajectory planning can employ a particle swarm optimization algorithm to parameterize candidate trajectories in three-dimensional space into multi-dimensional vectors. Through iterative optimization, a smooth and energy-efficient re-inspection trajectory is determined while satisfying flight constraints such as the UAV's maximum climb angle and maximum turning radius. Furthermore, the ground station can combine the inspection safety altitude, the sensor's minimum focusing distance, and the target defect detection requirements to determine the target flight altitude and gimbal attitude during UAV re-inspection, and calculate the corresponding lens focal length parameters based on the required image resolution.

[0118] When the ground workstation needs to acquire clearer defect details, it can calculate the lens focal length and infrared acquisition parameters in reverse, based on the pixel resolution requirements for secondary inspection, such as the target pixel accuracy for crack detection, and the actual distance between the UAV and the target area. It can also adjust the gimbal pitch and yaw angles according to the defect type to make the acquisition equipment face the optimal observation direction of the defect area.

[0119] The ground workstation generates re-inspection control commands based on the re-inspection waypoints, flight altitude, gimbal attitude, and image acquisition parameters, and sends them to the airborne edge computing platform. Specifically, the aforementioned spatial position parameters and acquisition control parameters can be encapsulated into data commands corresponding to the UAV communication protocol. For example, these can be encapsulated into multi-dimensional control commands that include trajectory adjustment, flight altitude adjustment, gimbal angle adjustment, and lens parameter control, and sent to the UAV via the communication link, causing the UAV to interrupt its current inspection route and perform a re-inspection task of the target area.

[0120] By automatically performing spatial mapping, trajectory planning, and parameter configuration based on the location of the re-inspection target, the ground workstation can generate multi-dimensional re-inspection control commands adapted to the target defects, enabling UAVs to perform autonomous and refined re-inspection, reducing manual intervention, and improving the closed-loop control capability of the inspection.

[0121] In one embodiment, this application provides a drone inspection system, comprising: the system including at least a ground workstation and an airborne edge computing platform deployed on the drone, the system being used to perform the steps of any of the drone inspection methods described in the above embodiments.

[0122] Indicatively, such as Figure 3 As shown, the UAV inspection system includes an airborne multimodal sensing unit 310, an airborne edge computing platform 320, a 5G private network communication module 330, a ground workstation 340, and an autonomous re-inspection flight control interface 350.

[0123] The airborne multimodal sensing unit, deployed on the UAV gimbal, includes an infrared camera and a visible light camera, used to collaboratively acquire image data of the target area. The infrared camera is used to acquire information on temperature anomalies and hotspot distribution, while the visible light camera is used to acquire information on equipment structure, component location, and defect appearance features. The two image data streams are registered and synchronized before being sent to the airborne edge computing platform.

[0124] The airborne edge computing platform is deployed inside the UAV and is used for defect detection, ROI spatial mask generation, region differential encoding, and data synchronization and encapsulation of acquired images. The airborne edge computing platform can also be configured with local storage units to store inspection data in the event of communication failures.

[0125] The 5G private network communication module is used to establish a data communication link between the airborne edge computing platform and the ground workstation, and supports various data transmission methods such as video streaming, file uploading, and message publishing to send encoded data and structured metadata to the ground workstation.

[0126] The ground workstation is used to receive and process data sent by the airborne edge computing platform. It may include a data receiving unit, an image reconstruction unit, a defect detection unit, a confidence assessment unit, a multimodal result analysis unit, an autonomous trajectory planning unit, and a re-inspection command generation unit. It is used to complete image reconstruction, secondary defect detection, detection result evaluation, and re-inspection decision generation.

[0127] The autonomous re-inspection flight control interface is located between the UAV flight control system and the gimbal control system, and is used to receive re-inspection control commands sent by the ground workstation. The re-inspection control commands can include the re-inspection target position, flight altitude, gimbal attitude, and image acquisition parameters, so as to control the UAV to perform trajectory adjustments, attitude adjustments, and re-acquisition of the target area.

[0128] Furthermore, the infrared and visible light image data collected by the airborne multimodal sensing unit are sent to the airborne edge computing platform. After the airborne edge computing platform completes defect detection, ROI spatial mask generation, region encoding, and synchronous encapsulation, it sends the data to the ground workstation via the 5G private network communication module. After the ground workstation completes data reception, image reconstruction, and defect analysis, when the detection results meet the re-inspection conditions, it sends a re-inspection control command to the UAV through the autonomous re-inspection flight control interface to control the UAV to perform re-inspection of the target area, realizing data acquisition, defect analysis, and closed-loop feedback during the UAV inspection process.

[0129] Those skilled in the art will understand that the above-described UAV inspection system structure is only a partial structural illustration related to the solution of this application and does not constitute a limitation on the system applied to the solution of this application. In actual applications, the system may include more or fewer functional units, or the above functional units may be combined or adjusted.

[0130] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0131] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0132] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for unmanned aerial vehicle (UAV) inspection, characterized in that, An application to a drone inspection system, the system comprising at least a ground workstation and an airborne edge computing platform deployed on the drone, the method comprising: The airborne edge computing platform determines the defect candidate region based on the target area image collected by the UAV during inspection, and expands the defect candidate region according to the multimodal spatial registration residual, motion blur estimation length and object distance adaptive correction parameters to obtain the ROI spatial mask image. The joint semantic value of the defect is determined based on the real-time communication status and the defect confidence and severity level of the defect candidate region. Based on the joint semantic value of the defects, image data processing is performed on the defect region and background region in the ROI spatial mask image to generate region coding data, and the region coding data is sent to the ground workstation. The ground workstation reconstructs the image based on the regional coding data, and when the detection result of the reconstructed image meets the re-inspection conditions, it generates a re-inspection control command and sends it to the airborne edge computing platform to control the UAV to perform target area re-inspection.

2. The method according to claim 1, characterized in that, The step of expanding the defect candidate region based on the multimodal spatial registration residual, motion blur estimation length, and object distance adaptive correction parameters to obtain a ROI spatial mask image includes: Based on the pixel offset error in the multimodal image registration process corresponding to the target region image, the multimodal spatial registration residual is determined; The motion blur estimation length is determined based on the drone's speed, the exposure time of the image acquisition device carried by the drone, and the pixel resolution. Based on the focal length of the image acquisition device, the physical size of the sensor, the target distance, and the single pixel size, determine the adaptive correction parameters for the object distance; The number of ROI extended pixels is calculated based on the multimodal spatial registration residual, the motion blur estimation length, and the object distance adaptive correction parameters. Based on the number of pixels expanded in the ROI, the boundary of the defect candidate region is expanded to obtain the ROI spatial mask image.

3. The method according to claim 2, characterized in that, The step of calculating the number of expanded pixels of the ROI based on the multimodal spatial registration residual, the motion blur estimation length, and the object distance adaptive correction parameter includes: The number of expanded pixels of the ROI is calculated using the following formula: in, The number of pixels to expand the ROI. Based on a preset minimum pixel threshold, Register residuals for the multimodal space. The length of the motion fuzzy estimation, The object distance adaptive correction parameter is... The focal length of the image acquisition device. The physical dimensions of the sensor are... The target object distance, The single pixel size, , and This is the weighted adjustment coefficient.

4. The method according to claim 1, characterized in that, The step of determining the joint semantic value of a defect based on real-time available bandwidth and the defect confidence and severity level of the defect candidate region includes: in, The joint semantic value of the defect is given. Let the confidence level of the defect be [value]. The severity level of the defect. This represents the maximum severity level of the defect. For real-time available bandwidth, The rated standard bandwidth required for full-frame lossless transmission and These are the weighting coefficients, and .

5. The method according to claim 1, characterized in that, Based on the joint semantic value of the defects, image data processing is performed on the defect region and background region in the ROI spatial mask image to generate region-coded data, including: Based on the joint semantic value of the defect, the coding control information corresponding to the defect region and the background region in the ROI spatial mask image is determined respectively; Based on the encoding control information, corresponding image data processing is performed on the defective region and the background region respectively to generate corresponding region encoding data.

6. The method according to claim 5, characterized in that, The step of determining the coding control information corresponding to the defect region and background region in the ROI spatial mask image based on the joint semantic value of the defect includes: When the joint semantic cost of the defect is less than the first preset threshold, the encoding control information corresponding to the defect region in the ROI spatial mask image is configured to be encoded using the first quantization parameter, and the encoding control information corresponding to the background region in the ROI spatial mask image is configured to be encoded using the second quantization parameter, wherein the first quantization parameter is less than the second quantization parameter. When the joint cost of the defect semantics is not less than the first preset threshold and less than the second preset threshold, the encoding control information corresponding to the defect region is configured to be encoded using the third quantization parameter, and the encoding control information corresponding to the background region is configured to be encoded using the fourth quantization parameter, wherein the second quantization parameter is not greater than the third quantization parameter, and the third quantization parameter is less than the fourth quantization parameter. When the joint cost of the defect semantics is greater than or equal to the second preset threshold, the encoding control information corresponding to the defect region is configured to be encoded using a lossless encoding method, and the encoding control information corresponding to the background region is configured to stop encoding and transmission.

7. The method according to claim 1, characterized in that, The image reconstruction based on the region-coded data includes: If the region encoding data includes the encoding data of the defect region and the encoding data of the background region, then the encoding data of the background region is decoded and its size is restored to obtain a background restored image; Based on the location coordinates corresponding to the defect area, the decoded defect recovery image of the defect area is mapped to the corresponding position in the background recovery image; Based on the boundary between the defect recovery image and the background recovery image, a preset pixel width is extended to both sides of the boundary to obtain a transition region; Based on the gradient weight values ​​corresponding to each pixel within the transition region, the defect restoration image and the background restoration image are weighted and fused to obtain a reconstructed image.

8. The method according to claim 1, characterized in that, When the detection result of the reconstructed image meets the re-inspection conditions, a re-inspection control command is generated and sent to the airborne edge computing platform, including: The reconstructed image is subjected to target detection, defect classification, and confidence evaluation to obtain the defect classification result of the defect target, the classification probability of the defect classification result, and the confidence evaluation result of the defect target. Based on the infrared and visible light data corresponding to the reconstructed image, infrared interpretation and visible light interpretation are performed respectively to obtain infrared interpretation results and visible light interpretation results; If the confidence assessment result is lower than the preset safety threshold, or the defect classification result of the defect target is a suspected high-risk defect and the corresponding classification probability is lower than the preset probability threshold, or the infrared interpretation result and the visible light interpretation result have a semantic conflict, then the detection result of the reconstructed image is determined to meet the re-inspection conditions, and the re-inspection target position is determined according to the position information corresponding to the defect target. Based on the target location and data collection requirements for re-inspection, a re-inspection control command is generated and sent to the airborne edge computing platform.

9. The method according to claim 8, characterized in that, The step of generating a re-inspection control command based on the re-inspection target location and re-inspection data collection requirements, and sending it to the airborne edge computing platform, includes: Based on the spatial pose information of the target area image collected by the UAV, the location of the re-inspection target is spatially mapped to determine the three-dimensional spatial coordinates corresponding to the location of the re-inspection target. Based on the three-dimensional spatial coordinates, a flight path is planned to determine the waypoints for re-inspection, and the corresponding flight altitude, gimbal attitude, and image acquisition parameters are determined according to the re-inspection and data acquisition requirements. A re-inspection control command is generated based on the re-inspection waypoint, the flight altitude, the gimbal attitude, and the image acquisition parameters.

10. A drone inspection system, characterized in that, include: The system includes at least a ground workstation and an airborne edge computing platform deployed on the UAV, and the system is used to perform the steps of the UAV inspection method as described in any one of claims 1 to 9.

Images