Portable unmanned aerial vehicle inspection image real-time analysis method

Abstract

The invention relates to the field of image processing, in particular to a portable unmanned aerial vehicle inspection image real-time analysis method, which comprises the following steps of: acquiring a plurality of original inspection images and metadata of each original inspection image, and labeling a defect state label of each original inspection image; standardizing the original inspection image to obtain a standard inspection image; a trained lightweight convolutional neural network is obtained; obtaining a trained analysis model; and acquiring a new original inspection image, obtaining a new standard inspection image based on the new original inspection image, then obtaining defect category prediction probability distribution of the new original inspection image, and taking the category with the highest probability as a real-time analysis result. An existing unmanned aerial vehicle inspection image analysis method has the problem of poor defect recognition accuracy. The portable unmanned aerial vehicle inspection image real-time analysis method provided by the invention is high in defect recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a method for real-time analysis of portable unmanned aerial vehicle (UAV) inspection images. Background Technology

[0002] Power lines are the lines that transmit electrical energy between power plants, substations, and electricity users. Regular inspections of power lines are crucial for ensuring the stable and safe operation of the power grid. Traditional inspection methods mainly involve manual ground inspections or manual helicopter inspections, which are inefficient and pose significant safety risks. Therefore, with the widespread adoption of drones, drone inspection has become a widely used method for power line inspection.

[0003] Drone inspections typically involve drones carrying high-definition visible light cameras flying along a predetermined power line corridor. Operators manually control the drones or use automated flight paths to capture multiple raw inspection images of key power equipment components from various angles and distances. High-definition visible light cameras specifically refer to those with a resolution of 4K or higher. Key power equipment components include poles, conductors, insulators, and hardware. The raw inspection images are then analyzed manually or using established analytical models to determine the presence and specific type of defects. The analytical model can be deployed in a portable drone inspection image analysis device, which is typically a ruggedized laptop or portable computing device. This portable computing device usually includes a shockproof, waterproof, and dustproof enclosure, a motherboard within the enclosure providing hardware support for image data reception and processing, a high-capacity rechargeable battery powering all electrical components, a display screen, and an integrated touchpad. The motherboard integrates a CPU, GPU, memory, a USB port for data import, an HDMI port for external display, an Ethernet port, and a wireless communication module. The high-capacity rechargeable battery features circuitry for charging and discharging via a USB interface. The enclosure includes a top cover and a bottom cover; the display screen is mounted inside the top cover, and the integrated touchpad is mounted inside the bottom cover. Both the display screen and the integrated touchpad are connected to corresponding interfaces on the motherboard via data cables.

[0004] However, existing drone inspection image analysis methods have the following problems: (1) During the acquisition process, UAV inspection images are affected by airflow disturbance and mechanical vibration, which often introduces composite degradation such as non-rigid deformation and random blur. Conventional geometric correction methods based on fixed analysis models are difficult to effectively handle such spatiotemporally continuous composite jitter patterns, which can easily lead to residual distortion in the corrected inspection images, thereby affecting the accuracy of subsequent defect detection. (2) In UAV inspection images, defects in power equipment often manifest as abrupt changes in local high-frequency textures, while the background area is mostly low-frequency or regular textures. Conventional random space enhancement methods lack specificity and may destroy or cover up key defect features, making it difficult for the analysis model to learn subtle defect patterns. (3) When faced with inspection images with residual jitter and complex backgrounds, conventional convolutional neural networks will blur the subtle defect features due to their fixed receptive field and pooling operation, and the simple splicing of features at different levels cannot make full use of multi-scale information, resulting in insufficient sensitivity of the analysis model to jitter blur and defects of different sizes. Therefore, existing UAV inspection image analysis methods suffer from poor defect identification accuracy. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a portable UAV inspection image real-time analysis method with high defect identification accuracy.

[0006] To address the aforementioned technical problems, this invention provides a portable unmanned aerial vehicle (UAV) inspection image real-time analysis method, comprising: S1. Collect multiple original inspection images and metadata of each original inspection image, and label the defect status of each original inspection image. S2. Standardize the original inspection image to obtain a standard inspection image. Use the clear standard inspection image and its corresponding jitter simulation image to train a lightweight convolutional neural network for jitter correction of the standard inspection image, and obtain a trained lightweight convolutional neural network. Construct and train a deep learning-based analysis model to obtain a trained analysis model. S201. Calculate a standard inspection image based on the original inspection image, select multiple clear standard inspection images to synthesize their corresponding jitter simulation images, and use the clear standard inspection images and their corresponding jitter simulation images to train a lightweight convolutional neural network for jitter correction of the standard inspection images, thus obtaining the trained lightweight convolutional neural network. S202. Generate a frequency domain saliency map based on the calibrated inspection image, obtain a normalized saliency map after normalization, generate a protection mask and a drop mask to guide data augmentation, and enhance the calibrated inspection image based on the protection mask and the drop mask; S203. Construct an analysis model with ResNet-50 as the backbone network, embedding a dynamic receptive field module and a hierarchical feature adaptive fusion module; S204. Calculate the defect category prediction probability distribution based on the conditional modulation feature vector, and calculate the total loss function; S205. Iteratively train and update the parameters of the analysis model to obtain the trained analysis model; S3. Acquire new original inspection images. After obtaining new standard inspection images based on the new original inspection images, input the new standard inspection images into the trained lightweight convolutional neural network to generate corrected inspection images. Input the corrected inspection images and their metadata into the trained analysis model to obtain the defect category prediction probability distribution of the new original inspection images. Use the category with the highest probability as the real-time analysis result.

[0007] As a further improvement of the present invention, step S1 includes: S101. Control a drone equipped with a high-definition visible light camera to conduct a flight inspection along a predetermined power line corridor, take pictures of key power equipment components from multiple angles and different distances, and obtain multiple original inspection images and metadata of each original inspection image. Metadata includes shooting time and lighting conditions; S102. Power sector technicians use image annotation tools to label the defect status of each original inspection image.

[0008] Preferably, the lighting conditions in S101 include cloudy days, sunny days, dusk, and backlighting.

[0009] As a further improvement of the present invention, step S201 includes: S2011. Calculate standard inspection images based on original inspection images; S2012. Generate the composite jitter field at each pixel; S2013. Select multiple clear standard inspection images, apply the composite jitter field at each pixel to the clear standard inspection images, and synthesize jitter simulation images with simulated jitter degradation effects corresponding to each clear standard inspection image; S2014. A lightweight convolutional neural network for jitter correction of standard inspection images is trained using clear standard inspection images and their corresponding jitter simulation images, resulting in a trained lightweight convolutional neural network.

[0010] As a further improvement of the present invention, step S202 includes: S2021. Perform two-dimensional discrete wavelet transform on the corrected inspection image to convert the corrected inspection image from the spatial domain to the frequency domain and separate the frequency components of different directions and scales to obtain low-frequency approximate sub-bands, horizontal high-frequency detail sub-bands, vertical high-frequency detail sub-bands and diagonal high-frequency detail sub-bands. S2022. The energy information and local contrast information of the high-frequency detail subbands in the horizontal direction, the high-frequency detail subbands in the vertical direction, and the high-frequency detail subbands in the diagonal direction are fused to generate a frequency domain saliency map. S2023. Normalize the frequency domain saliency map and generate a guard mask and a drop mask based on the guard threshold and the drop threshold to guide data augmentation, thus obtaining a normalized saliency map; S2024. Perform random rectangular cropping constrained by a protection mask and random region occlusion constrained by a discard mask on the calibrated inspection image; S2025. Select high-frequency texture regions from other calibrated inspection images of the same category and seamlessly integrate them into the low saliency regions of the current image.

[0011] Preferably, step S2025 includes: S20251. Randomly select texture region blocks from other calibration inspection images that belong to the same category as the current calibration inspection image; S20252. Select a center position according to the masking rules to obtain the selected target position, and define the transplant target region around the selected target position for performing texture fusion; S20253. Using the Poisson image editing method, seamlessly blend texture region blocks into the selected target location of the current calibration inspection image, perform Poisson image editing and fusion, and generate a texture transfer enhancement image.

[0012] As a further improvement of the present invention, step S203 includes: S2031. Replace the standard convolution in the first layer of the backbone network with a multi-branch differential convolution module to construct the input layer of the analysis model, and calculate the output feature map of the multi-branch differential convolution module based on texture transplantation to enhance the image; S2032. Insert a dynamic receptive field module after each residual block of the backbone network and calculate the output feature map of the dynamic receptive field module; S2033. A hierarchical feature adaptive fusion module is used to calculate hierarchical fusion features.

[0013] As a further improvement of the present invention, step S204 includes: S2041. Calculate conditional modulation feature vectors based on hierarchical fusion features; S2042. Calculate the total loss function consisting of the standard cross-entropy classification loss and the image correction loss.

[0014] Preferably, step S2041 includes: S20411. Input the metadata encoding vector into two independent small multilayer perceptrons to generate conditional scaling vector and conditional bias vector respectively, and obtain the conditional modulation feature vector based on the conditional scaling vector and conditional bias vector. S20412. Input the conditional modulation feature vector into the classifier to obtain the predicted probability distribution of the defect category.

[0015] As a further improvement of the present invention, the defect status label in S1 includes no obvious defects found, insulator damage, vibration damper slippage, foreign objects in the conductor, and hardware corrosion.

[0016] The beneficial effects of the present invention are as follows: The portable UAV inspection image real-time analysis method provided by the present invention has high accuracy in defect identification.

[0017] First, the S201's lightweight convolutional neural network (CNN) acquires its geometric correction capability by learning the mapping from jittered simulated images to their corresponding inverse displacement fields. The CNN is trained using a large number of "degraded (jittered simulated images) - sharp (sharp standard inspection images)" images, with the sharp standard inspection images serving as the sharpness image supervision signal. The trainable parameters of the CNN are adjusted by minimizing the constrained image correction loss, enabling the CNN to learn to predict the inverse jitter field that can correct jittered simulated images back to standard inspection images. The three-layer convolutional encoder, due to its hierarchical feature extraction capabilities, can understand image distortion patterns from local to global perspectives and has the potential to predict complex inverse displacement fields. The trained CNN can effectively recover the geometric structure of standard inspection images, achieving adaptive correction of non-rigid deformations.

[0018] Secondly, S202 generates a guiding mask through frequency domain saliency analysis and adopts a texture transfer strategy to protect potential defect features and enrich the texture context during the enhancement process, so as to protect defect features in a targeted manner when enhancing data.

[0019] Furthermore, S203 uses ResNet-50 as the backbone network and embeds a dynamic receptive field module and a hierarchical feature adaptive fusion module to enhance sensitivity to jittery edges, adaptively adjust the receptive field, and effectively fuse multi-scale features.

[0020] Finally, this method encodes metadata such as shooting time and illumination into conditional vectors to dynamically modulate deep features, thereby mitigating intra-class differences and improving the discriminative power of the analysis model in changing environments. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the principle of the present invention; Figure 2 The robustness analysis diagram of different methods under sunny lighting conditions is shown. Figure 3 The robustness analysis of different methods under cloudy lighting conditions is shown in the figure. Figure 4 The robustness analysis diagrams for different methods under twilight conditions are shown. Figure 5 The figure shows the robustness analysis of different methods under backlight conditions. Detailed Implementation

[0022] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, this invention provides a portable UAV inspection image real-time analysis method: S1. Collect multiple original inspection images and metadata of each original inspection image, and label the defect status of each original inspection image. S101. Control a drone equipped with a high-definition visible light camera to conduct a flight inspection along a predetermined power line corridor, take pictures of key power equipment components from multiple angles and different distances, and obtain multiple original inspection images and metadata of each original inspection image. The drone is a multi-rotor drone; The original inspection image is of dimension 1. A three-dimensional tensor containing RGB color channels, wherein... The height of the image, i.e., the number of pixel rows. The width of the image, i.e., the number of pixel columns; Metadata includes shooting time, lighting conditions, and may also include the approximate flight altitude of the drone. Lighting conditions include cloudy, sunny, dusk, and backlighting. Shooting time is derived from the EXIF ​​information of the original inspection images. Lighting conditions are recorded by the operator simultaneously during shooting or afterward in conjunction with the weather records corresponding to the shooting time. The approximate flight altitude of the drone is derived from the drone's flight log. S102. Power industry technicians use image annotation tools to label the defect status of each original inspection image; Power industry technicians use image annotation tools to mark defects in each original inspection image, including: (1) Identify and locate equipment defects in the image, and use bounding boxes to accurately mark the area where the defects are located; (2) Match the corresponding defect status label according to the morphological characteristics of the defect; when training the analysis model, the bounding box annotation and the category label together serve as the supervision signal for the original inspection image with defects. Defect status labels include no obvious defects found, insulator damage, vibration damper slippage, foreign objects in conductors, and hardware corrosion.

[0024] S2. Standardize the original inspection image to obtain a standard inspection image. Use the clear standard inspection image and its corresponding jitter simulation image to train a lightweight convolutional neural network for jitter correction of the standard inspection image, and obtain a trained lightweight convolutional neural network. Construct and train a deep learning-based analysis model to obtain a trained analysis model. S201. Calculate a standard inspection image based on the original inspection image, select multiple clear standard inspection images to synthesize their corresponding jitter simulation images, and use the clear standard inspection images and their corresponding jitter simulation images to train a lightweight convolutional neural network for jitter correction of the standard inspection images, thus obtaining the trained lightweight convolutional neural network. Since the original inspection images are basically all jittery, the clear standard inspection image here refers to the standard inspection image with a relatively small degree of jitter. S2011. Calculate standard inspection images based on original inspection images; The original inspection image is converted from the integer pixel format of an RGB image to a floating-point format through normalization processing to obtain a standard inspection image. Each pixel value in the standard inspection image is mapped to a range... , dimension The pixel values ​​of the standard inspection image represent relative brightness information. A value of 0 represents the darkest point in the standard inspection image, and a value of 1 represents the brightest point in the standard inspection image, in order to eliminate the influence of units. Standard inspection images The calculation formula is: , In the formula, Original inspection image; The minimum value operator; This is the operator for finding the maximum value.

[0025] S2012. Generate the composite jitter field at each pixel; In pixels The generated composite jitter field is a two-dimensional vector containing horizontal and vertical displacements. The vector at each location represents the horizontal and vertical displacements of that pixel due to simulated jitter. For pixel index, For the horizontal index of the image, Index the image vertically; In pixels The generated composite jitter field The calculation formula is: , In the formula, These are global affine transformation component functions used to simulate rigid image deformation caused by changes in the overall attitude of the UAV, such as rotation, scaling, and translation. The set of parameters for controlling the affine transformation includes rotation angle parameters, scaling factor parameters, and translation parameters. These parameters are randomly sampled within a certain range to simulate different posture changes. is the elastic deformation intensity coefficient, an adjustable scalar hyperparameter used to control the contribution intensity of local elastic deformation in the entire composite dithering field, and can be set to 0.3; This is a local elastic deformation component function used to simulate non-rigid detail deformation caused by high-frequency mechanical vibration; The set of parameters for controlling elastic deformation specifically includes parameters such as mesh size and the range of random displacement.

[0026] Global affine transformation component functions By random generation Affine transformation matrix To achieve this, for homogeneous coordinates in standard inspection images Its transformed coordinates are ,in, This is a transpose operation.

[0027] Local elastic deformation component function Simulation can be performed by superimposing smooth random displacement fields onto an image grid. Random vectors can be generated first on a lower-resolution grid, such as... A high-resolution grid is upsampled to the full resolution of the image using bilinear or bicubic interpolation, resulting in a smooth deformation field.

[0028] S2013. Select multiple clear standard inspection images, apply the composite jitter field at each pixel to the clear standard inspection images, and synthesize jitter simulation images with simulated jitter degradation effects corresponding to each clear standard inspection image through differentiable image sampling operations. , The dimension is ; For coordinates The pixel values ​​of the synthesized jitter simulation image are obtained by displacing and sampling the standard inspection image according to the composite jitter field at each pixel. In coordinates Pixel values ​​of the synthesized jitter simulation image The calculation formula is: , In the formula, It is a differentiable image sampling function used to interpolate pixel values ​​from a standard inspection image based on provided floating-point coordinates; The term represents the sampling coordinate field, i.e., "original coordinates + simulated jitter displacement", in pixels. The generated composite jitter field It is a two-dimensional displacement vector. The coordinates (x,y) are added to this displacement vector to obtain the new sampling coordinates.

[0029] Differentiable image sampling functions are implemented using bilinear interpolation, given a floating-point coordinate. At that time, the differentiable image sampling function first finds the standard inspection image. The four nearest-neighbor integer pixels enclosing the coordinate are used. Weights are calculated based on the horizontal and vertical distances between the floating-point coordinate and these four pixels. Finally, the pixel value at the sampling coordinate is obtained by weighted summing of the pixel values ​​of these four points. .

[0030] To simulate the complex jitter patterns during UAV flight, S2012 integrates global attitude changes and local high-frequency vibrations, and generates a two-dimensional composite jitter field by superimposing a random affine transformation field and a local elastic deformation field. S2013 generates a large number of spatiotemporally continuous jitter simulation images that closely resemble the jitter of real UAV flight, obtaining "degraded (jitter simulation image) - clear (clear standard inspection image)" image pairs, providing sufficient supervision data for training lightweight convolutional neural networks. S2014. A lightweight convolutional neural network for jitter correction of standard inspection images is trained using clear standard inspection images and their corresponding jitter simulation images, resulting in a trained lightweight convolutional neural network. The lightweight convolutional neural network is defined as a three-layer convolutional encoder with trainable parameters as follows: The input to the lightweight convolutional neural network is a jittered simulated image. The output is an inverse dithering field. The inverse jitter field is a two-dimensional displacement field with the same dimensions as the input image space, and each location contains an inverse displacement vector used to cancel out the original composite jitter field. Lightweight convolutional neural networks can perform geometric correction on jitter-simulated images.

[0031] By combining the inverse jitter field with resampling of the jitter simulation image, a corrected inspection image is obtained to counteract or reduce the effect of simulated jitter, thereby recovering a corrected result whose geometry is closer to that of a clear standard inspection image. The training objective of the lightweight convolutional neural network is to minimize the corrected inspection image. With clear standard inspection images The structural differences between them.

[0032] The dimension of the inspection image is corrected. This represents the image whose geometric distortion has been repaired after neural network correction; the image is then inspected. The calculation formula is: , In the formula, For in pixels At this point, a lightweight convolutional neural network predicts the inverse displacement vector used to counteract the original composite jitter field.

[0033] Image correction loss is a loss function used to constrain lightweight convolutional neural networks. The calculation formula is: , In the formula, The structural similarity index measures the visual similarity between two images by comprehensively comparing the mean brightness, standard deviation of contrast, and covariance structure of corresponding local regions.

[0034] S202. Generate a frequency domain saliency map based on the calibrated inspection image, obtain a normalized saliency map after normalization, generate a protection mask and a drop mask to guide data augmentation, and enhance the calibrated inspection image based on the protection mask and the drop mask; S2021. Perform two-dimensional discrete wavelet transform on the corrected inspection image to convert the corrected inspection image from the spatial domain to the frequency domain and separate the frequency components of different directions and scales to obtain low-frequency approximate sub-bands, horizontal high-frequency detail sub-bands, vertical high-frequency detail sub-bands and diagonal high-frequency detail sub-bands, so as to separate the low-frequency background information and high-frequency detail texture information in the corrected inspection image for targeted analysis. The low-frequency approximate subband is represented as It mainly includes the main outline of the inspection image and smooth background information, such as smoothed areas; The high-frequency detail subband in the horizontal direction is represented as It mainly contains vertical edge and texture information, such as horizontal edges; The high-frequency detail subband in the vertical direction is represented as It mainly contains horizontal edge and texture information, such as vertical edges; The high-frequency detail subband in the diagonal direction is represented as It mainly contains texture information in the diagonal direction, such as areas containing a lot of diagonal details or noise.

[0035] S2022. The energy information and local contrast information of the horizontal, vertical and diagonal high-frequency detail subbands are fused and spatial continuity is obtained by Gaussian smoothing to generate a frequency domain saliency map for identifying complex texture regions. The numerical value at each position in the frequency domain saliency map reflects the complexity of the texture of the corresponding region in the original image and the possibility of abrupt changes; a higher value indicates that the region is more likely to be a key feature region. The calculation formula is: , In the formula, This is the index for the high-frequency subband, and it's a variable used to iterate through the set; To traverse the set; The standard deviation is A two-dimensional Gaussian kernel function is used to smooth the fused energy-contrast map, eliminating isolated high-response points caused by noise. The standard deviation hyperparameter is used to control the smoothness. This is a two-dimensional convolution operation; For the first The absolute value of the coefficients of each high-frequency subband; The contrast enhancement coefficient is a scalar hyperparameter greater than zero, used to adjust the weight of local contrast in saliency calculation. It can be set to 0.5. This is a local standard deviation function used to calculate the standard deviation of the first standard deviation. Each pixel in the sub-band is centered at a radius of [missing information]. The standard deviation within a local circular window characterizes the texture contrast of that local region; The radius of the circular window used in the calculation of local standard deviation is a positive integer hyperparameter, which can be set to 3.

[0036] S2023. Normalize the frequency domain saliency map and generate a guard mask and a drop mask based on the guard threshold and the drop threshold to guide data augmentation, so as to obtain a normalized saliency map to map the numerical range of the saliency map to the normalized interval. The normalized significance plot is normalized to The frequency domain saliency plot of the interval indicates that the larger the value, the more complex the texture of the corresponding region in the original image, and the more likely it is to be a key feature region; normalized saliency plot. The calculation formula is: ; The normalized saliency information is transformed into binary spatial constraints, protecting critical areas and perturbing non-critical areas during the data augmentation process; definition A protection mask is used to identify highly salient regions that need protection, where points with a value of 1 indicate that they need protection. When the value at a certain point is greater than the protection threshold, The value at that point is 1, otherwise it is 0; definition The drop mask is used to identify low-salience regions that can be occluded, where points with a value of 1 indicate that they can be dropped or occluded. When the value at a certain point is less than the discard threshold, The value at that point is 1, otherwise it is 0; Protection threshold It is a scalar hyperparameter between 0 and 1, used to determine highly significant regions, and can be set to 0.7; Discard threshold It is a scalar hyperparameter between 0 and 1, and satisfies This is used to identify regions of low significance and can be set to 0.3.

[0037] S2024. Perform random rectangular cropping constrained by a protection mask and random region occlusion constrained by a discard mask on the calibration inspection image to complete spatial enhancement and avoid damaging potential defect areas in the calibration inspection image. Random rectangle cropping refers to a random cropping operation performed on an image under the constraint that the cropping rectangle must completely contain all highly saliency connected regions within the protection mask. Random rectangular cropping constrained by the protection mask refers to the constraint that when performing random rectangular cropping on the correction inspection image, the generated cropping rectangle must completely contain the smallest bounding rectangle of all connected regions with a value of 1 in the protection mask, ensuring that no region marked as highly salient will be cropped. Random region occlusion refers to the occlusion operation that generates and covers random rectangular blocks only within the low salience region identified by the drop mask; Random region occlusion constrained by the discard mask refers to the occlusion of random regions in a corrected inspection image only when the discard mask is used. An occlusion rectangle is generated inside the area marked as 1 to avoid direct occlusion of key texture areas.

[0038] S2025. Select high-frequency texture regions from other calibrated inspection images of the same category and seamlessly integrate them into the low-salience regions of the current image to enhance the robustness of the analysis model to defect textures; Other corrected inspection images of the same category are other images in the training dataset that have the same defect category label as the corrected inspection image currently undergoing enhancement processing. These images come from the same batch of training sets and have already undergone adaptive jitter correction preprocessing.

[0039] S20251. Randomly select texture region blocks from other calibration inspection images belonging to the same category as the current calibration inspection image. ; The selection criteria are based on the high high-frequency energy within the texture region block. The high high-frequency energy refers to the sum of the squares of all high-frequency sub-band coefficients after wavelet decomposition. A high high-frequency energy indicates that the region contains rich edge, texture, or detail information, making it more suitable as an enhanced texture source.

[0040] S20252. Select a center position according to the masking rules to obtain the selected target position, and define the transplant target region around the selected target position for performing texture fusion; Central position of the transplant target area The discard mask must satisfy the corresponding downsampling coordinates. The value at point is 1 (i.e., it is located in a low significance region), where, The horizontal coordinates of the center of the target region for transplantation. The vertical coordinates of the center of the target region for transplantation. This is the floor operator.

[0041] Select a target location as a coordinate point (center point); The target region for transplantation refers to the area centered on the selected target location, encompassing texture regions. An image region of a specified size; S20253. Using the Poisson image editing method, seamlessly blend texture region blocks into the selected target location of the current calibration inspection image, perform Poisson image editing and fusion, and generate a texture transfer enhancement image. Texture transfer enhances the dimension of the image. This is done to maintain a smooth transition at the edges of the target area for transplantation, allowing the transplanted texture to blend naturally with the surrounding background.

[0042] S203. Construct an analysis model with ResNet-50 as the backbone network, embedding a dynamic receptive field module and a hierarchical feature adaptive fusion module; The backbone network includes characteristic stages such as the first stage, the second stage, the third stage, the fourth stage, and the fifth stage; S2031. Replace the standard convolution in the first layer of the backbone network with a multi-branch differential convolution module to construct the input layer of the analysis model. Calculate the output feature map of the multi-branch differential convolution module based on texture transfer to enhance the image. Enhance the network's initial extraction capability for image edges and texture features through differential operations, especially making it more sensitive to blurred edges caused by jitter. Output feature map of multi-branch differential convolution module The calculation formula is: , In the formula, This is the branch index, used to traverse from the 1st to the 2nd. One branch; The total number of branches is a preset positive integer hyperparameter, which can be set to 4. For the first The learnable scalar weights of each branch are used to balance the contributions of each branch, and the initial value is usually set to 1. For the first The first of the branches Convolution kernel weight tensor; For the first The second of the branches Convolution kernel weight tensor; Enhance images for texture transfer; Item and The differences between terms form a difference filter, used to enhance the response to feature changes. The structure of a multi-branch difference convolution includes... The module has several parallel branches. Each branch performs two convolution operations independently and then calculates their difference. The difference outputs of all branches are multiplied by their respective learnable scalar weights and then summed to obtain the final output of the module.

[0043] and All dimensions are ,in, This is the number of output channels for each branch, which can be set. , This is the total number of output channels planned for the multi-branch differential convolutional layer, preferably set to 64; This is the input number of channels, the default setting. , and The two processes perform two different 3×3 convolutions on the input, respectively. The term representing the difference between the two forms an equivalent high-pass filter (achieving edge detection capability). For the th term... Each branch has two convolutional kernels. Specifically, an equivalent differential filter is defined by the difference between the weights of the two convolutional kernels. Compared with a single fixed convolutional kernel, it can capture the pattern of feature changes more directly and flexibly, and is especially stronger in responding to blurred edges caused by jitter.

[0044] S2032. Insert a dynamic receptive field module after each residual block of the backbone network and calculate the output feature map of the dynamic receptive field module. The dynamic receptive field module dynamically fuses multiple dilated convolutional branches with different dilation rates through a content-adaptive attention mechanism to flexibly adjust the receptive field size and enhance the analysis model's ability to perceive local details and global structure. Let the input feature map of the dynamic receptive field module be... Output feature map of the dynamic receptive field module The calculation formula is: , In the formula, The void ratio is an element in the void ratio set. For the set of void ratios, we can set ; These are scalar attention weights, representing the weights applied to the current input feature. In this case, the void ratio is The importance of the branches; The corresponding void ratio is The learnable convolutional kernel weights are trainable parameters, with a dilatation rate. Convolve the input features to extract features at the corresponding scale; The void ratio is The dilated convolution operation; The scalar attention weights are dynamically calculated by a lightweight weight generation subnetwork based on the content of the input feature map of the dynamic receptive field module. This includes: performing global average pooling on the input feature map of the dynamic receptive field module to obtain a channel vector, and mapping this vector through two fully connected layers to... Each scalar is then normalized using the Softmax function to obtain scalar attention weights, which satisfy the following conditions: .

[0045] The term represents dynamically weighted multi-scale contextual features, and the original features are fused with the dynamically weighted multi-scale contextual features through residual connections.

[0046] S2033. A hierarchical feature adaptive fusion module is adopted, which takes the output feature maps of the four feature stages of the backbone network as input, and integrates feature information of different spatial resolutions and semantic levels through cross-attention mechanism and gating fusion to calculate hierarchical fusion features. Hierarchical fusion feature representation is an enhanced feature that integrates multi-scale and multi-level semantic information, which is beneficial for detecting defects of different sizes and complexities; hierarchical fusion features The calculation formula is: , In the formula, This is the feature stage index, with values ​​of 2, 3, 4, and 5. Corresponding to the second stage of the backbone network, Corresponding to the third stage of the backbone network, Corresponding to the fourth stage of the backbone network, This corresponds to the fifth stage of the backbone network; Main backbone network The feature map output by each stage is an enhanced feature generated by fusing feature information from other stages through the cross-attention and gating mechanism within the hierarchical feature adaptive fusion module.

[0047] The feature maps in the first stage of the backbone network have the highest spatial resolution, but contain the shallowest semantic information, mainly low-level edges and textures, and require a large amount of computation. In order to balance computational efficiency and feature effectiveness, fusion begins from the second stage, which has richer semantic information.

[0048] S204. Calculate the defect category prediction probability distribution based on the conditional modulation feature vector, and calculate the total loss function; S2041. Calculate conditional modulation feature vectors based on hierarchical fusion features; S20411. Input the metadata encoding vector into two independent small multilayer perceptrons to generate conditional scaling vector and conditional bias vector respectively. Based on the conditional scaling vector and conditional bias vector, obtain the conditional modulation feature vector of fused metadata to realize dynamic calibration of feature expression according to external shooting conditions, so as to alleviate intra-class differences caused by changes in shooting angle and illumination. The dimension of the conditional modulation feature vector is The final classification features characterize the fused metadata, where, The dimension of the hierarchical fusion feature vector determines the input size of the subsequent conditional modulation module and classifier, which can be set to 1024. Conditional modulation feature vector The calculation formula is: , In the formula, It is the first independent small multilayer sensor; The metadata encoding vector represents the contextual information of the external shooting conditions during image acquisition, and is composed of a temporal encoding vector and an illumination condition embedding vector. This is element-wise multiplication; This is a global average pooling operation; It is a second independent small multilayer perceptron with a structure consisting of two fully connected layers; Because the hourly portion of the shooting time is periodic, and there are 24 hours in a day, the hourly portion is converted into a two-dimensional vector to encode the periodicity of time. The conversion formula is as follows: ; Since there are four types of lighting conditions, a learnable embedding layer is used to convert the lighting conditions into low-dimensional vectors, such as representing cloudy days as a one-dimensional vector. Sunny day is represented as a one-dimensional vector. Dusk is represented as a one-dimensional vector. Backlighting is represented as a one-dimensional vector. ; When the hour portion is 12, its time encoding vector is represented as follows: When the weather is sunny, the embedding vector of its illumination condition is represented as follows: After concatenation, the resulting metadata encoding vector is: .

[0049] The first independent small multilayer perceptron has a structure of two fully connected layers. The first layer is used to map the input dimension to the intermediate hidden dimension, and the second layer is used to map the hidden dimension to the same dimension as the classification feature dimension. The item is a dimension learned from the input metadata encoding vector. The conditional scaling vector is used to perform channel-level adaptive scaling on the features after global pooling, thereby emphasizing or suppressing certain feature channels in response to different external shooting conditions. The second independent small multilayer perceptron also has a structure of two fully connected layers. The item is a dimension learned from the input metadata encoding vector. The conditional bias vector is used to add channel-level adaptive biases to the scaled features, further fine-tuning the feature distribution to compensate for feature shifts caused by external conditions; S20412. The conditionally modulated feature vector is input into the classifier to obtain the defect category prediction probability distribution; the classifier uses a standard fully connected layer with a weight matrix dimension of . , will input The linear transformation of the dimensional conditional modulation eigenvector is as follows: The original score vector of the dimension is input into the Softmax function for normalization to obtain the defect category prediction probability distribution; The probability distribution for defect category prediction is represented as follows: , In the formula, The analysis model predicts that the sample belongs to the first... The probability of each category; The analysis model predicts that the sample belongs to the first... The probability of each category; The analysis model predicts that the sample belongs to the first... The probability of each category; The total number of categories includes: 1-No obvious defects found, 2-Insulator damage, 3-Slippage of vibration damper, 4-Foreign objects on conductors, 5-Corrosion of hardware; S2042. Calculate the total loss function consisting of the standard cross-entropy classification loss and the image correction loss; The cross-entropy classification loss is calculated based on the predicted probability distribution of defect categories and the defect state label of the sample; The image correction loss is calculated based on the corrected inspection image and the original inspection image; Total loss function The calculation formula is: , In the formula, The standard cross-entropy loss is used to measure the probability distribution of defect category predictions by the analysis model. The difference between the one-hot encoding and the one-hot encoding of the sample's true label (i.e., the defect status label in one-hot encoding format); This is the loss balancing parameter, a hyperparameter used to balance the weights of the two loss parameters, and can be set to a value of 0.3. Image correction loss; To verify the performance stability of the method of the present invention under different lighting conditions, experiments were conducted to compare the accuracy of the method of the present invention with that of three conventional techniques under different lighting conditions, and the results were obtained. Figures 2-5 The accuracy is the distribution of accuracy across thirty repeated experiments under each lighting condition, presented as a box plot showing statistical characteristics, including the median, interquartile range, and discrete points. For example... Figures 2-5 As shown in the figures, the box distribution clearly demonstrates that the method of this invention achieves the highest accuracy under all lighting conditions, with the lowest box height and fewest discrete points, exhibiting extremely high stability and consistency. Traditional CNNs (Convolutional Neural Networks) perform reasonably well under sunny and cloudy conditions, but under twilight and backlighting conditions, the boxes shift significantly downwards and increase in height, indicating a greater susceptibility to lighting changes. Methods based on SIFT (Scale Invariant Feature Transform) and simple threshold segmentation show significant performance fluctuations and low accuracy under all lighting conditions, with a loose box distribution and numerous discrete points. This clearly verifies the effectiveness of the method of this invention (Conditional Aware Adaptive Classification Mechanism). In UAV inspection image analysis tasks, there are problems such as intra-class differences caused by changes in shooting angle and lighting, and blurred feature boundaries due to residual jitter. Conventional image classification methods typically directly input deep features into fully connected layers for supervised classification, which can easily lead to insufficient generalization ability of the analysis model in complex environments and decreased discrimination against subtle defects. S204 dynamically modulates deep features by encoding metadata such as shooting time and lighting conditions into conditional vectors, effectively mitigating intra-class feature differences caused by lighting variations, thereby significantly improving the robustness of the analysis model in real-world complex inspection environments. It employs a joint loss function, combining cross-entropy classification loss and image correction loss, to optimize classification probabilities and constrain the analysis model to learn clear feature class boundaries, thereby enhancing the discriminative power and robustness of the analysis model.

[0050] S205. Iteratively train and update the parameters of the analysis model to obtain the trained analysis model; use the training dataset constructed above to perform end-to-end supervised learning on the analysis model; Each sample in the training dataset includes a corrected inspection image obtained through adaptive correction preprocessing and a texture-transplanted enhanced image obtained after data augmentation, along with their corresponding ground truth class labels, bounding box annotations, and metadata information. S2051. Randomly select batch samples from the dataset, input the data contained in each sample into the constructed anti-jitter improved analysis model, perform the forward propagation process, and obtain the defect category prediction probability distribution through condition-aware adaptive classification; generate a corrected inspection image during the forward propagation process; S2052. Calculate the total loss function based on the analysis model output and the real supervision signal. The total loss is obtained by weighted summation of cross-entropy classification loss and image correction loss. S2053. After calculating the total loss, an optimization algorithm based on gradient descent is used to calculate the gradient of the total loss relative to all trainable parameters of the analysis model through backpropagation. S2054. The optimizer uses these gradients to update all trainable parameters, enabling the analysis model to produce more accurate predictions and generate clearer calibrated inspection images in the next iteration.

[0051] S2055. The process of "forward propagation - loss calculation - back propagation - parameter update" is repeated cyclically, which is one iteration. The training process will continue for multiple rounds until the stopping iteration condition is met, which indicates that the performance of the analysis model tends to stabilize. When the stopping condition is met, the parameters of the analysis model with the best performance at this time are saved to obtain the trained analysis model. The stopping condition is when the classification accuracy of the analysis model on an independent validation dataset no longer improves for several consecutive rounds or reaches the preset maximum number of training rounds.

[0052] S3. New raw inspection images are acquired using a drone and a high-definition visible light camera. After obtaining new standard inspection images based on these raw images, the new standard inspection images are input into a trained lightweight convolutional neural network for adaptive jitter correction preprocessing. This quickly generates corrected inspection images with good geometric structure recovery to eliminate the negative impact of flight jitter. The corrected inspection images and their metadata are input into a trained analysis model to obtain the defect category prediction probability distribution of the image. The category with the highest probability is used as the real-time analysis result.

[0053] A trained lightweight convolutional neural network and a trained analysis model can be deployed in a portable drone inspection image analysis device, enabling real-time image analysis and preliminary defect identification at the drone inspection site without relying on high-speed network transmission and cloud computing resources, greatly improving the efficiency and timeliness of inspection operations.

Claims

1. A method for real-time analysis of portable unmanned aerial vehicle (UAV) inspection images, characterized in that, include: S1. Collect multiple original inspection images and metadata of each original inspection image, and label the defect status of each original inspection image. S2. Standardize the original inspection image to obtain a standard inspection image. Use the clear standard inspection image and its corresponding jitter simulation image to train a lightweight convolutional neural network for jitter correction of the standard inspection image, and obtain a trained lightweight convolutional neural network. Construct and train a deep learning-based analysis model to obtain a trained analysis model. S201. Calculate a standard inspection image based on the original inspection image, select multiple clear standard inspection images to synthesize their corresponding jitter simulation images, and use the clear standard inspection images and their corresponding jitter simulation images to train a lightweight convolutional neural network for jitter correction of the standard inspection images, thus obtaining the trained lightweight convolutional neural network. S202. Generate a frequency domain saliency map based on the calibrated inspection image, obtain a normalized saliency map after normalization, generate a protection mask and a drop mask to guide data augmentation, and enhance the calibrated inspection image based on the protection mask and the drop mask; S203. Construct an analysis model with ResNet-50 as the backbone network, embedding a dynamic receptive field module and a hierarchical feature adaptive fusion module; S204. Calculate the defect category prediction probability distribution based on the conditional modulation feature vector, and calculate the total loss function; S205. Iteratively train and update the parameters of the analysis model to obtain the trained analysis model; S3. Acquire new original inspection images, obtain new standard inspection images based on the new original inspection images, and input the new standard inspection images into the trained lightweight convolutional neural network to generate corrected inspection images. The corrected inspection image and its metadata are input into the trained analysis model to obtain the defect category prediction probability distribution of the new original inspection image, and the category with the highest probability is used as the real-time analysis result.

2. The portable UAV inspection image real-time analysis method according to claim 1, characterized in that, The steps in S1 include: S101. Control a drone equipped with a high-definition visible light camera to conduct a flight inspection along a predetermined power line corridor, take pictures of key power equipment components from multiple angles and different distances, and obtain multiple original inspection images and metadata of each original inspection image. Metadata includes shooting time and lighting conditions; S102. Power sector technicians use image annotation tools to label the defect status of each original inspection image.

3. The portable UAV inspection image real-time analysis method according to claim 2, characterized in that, The lighting conditions in S101 include cloudy days, sunny days, dusk, and backlighting.

4. The portable UAV inspection image real-time analysis method according to claim 1, characterized in that, The steps in S201 include: S2011. Calculate standard inspection images based on original inspection images; S2012. Generate the composite jitter field at each pixel; S2013. Select multiple clear standard inspection images, apply the composite jitter field at each pixel to the clear standard inspection images, and synthesize jitter simulation images with simulated jitter degradation effects corresponding to each clear standard inspection image; S2014. A lightweight convolutional neural network for jitter correction of standard inspection images is trained using clear standard inspection images and their corresponding jitter simulation images, resulting in a trained lightweight convolutional neural network.

5. The portable UAV inspection image real-time analysis method according to claim 1, characterized in that, The steps in S202 include: S2021. Perform two-dimensional discrete wavelet transform on the corrected inspection image to convert the corrected inspection image from the spatial domain to the frequency domain and separate the frequency components of different directions and scales to obtain low-frequency approximate sub-bands, horizontal high-frequency detail sub-bands, vertical high-frequency detail sub-bands and diagonal high-frequency detail sub-bands. S2022. The energy information and local contrast information of the high-frequency detail subbands in the horizontal direction, the high-frequency detail subbands in the vertical direction, and the high-frequency detail subbands in the diagonal direction are fused to generate a frequency domain saliency map. S2023. Normalize the frequency domain saliency map and generate a guard mask and a drop mask based on the guard threshold and the drop threshold to guide data augmentation, thus obtaining a normalized saliency map; S2024. Perform random rectangular cropping constrained by a protection mask and random region occlusion constrained by a discard mask on the calibrated inspection image; S2025. Select high-frequency texture regions from other calibrated inspection images of the same category and seamlessly integrate them into the low saliency regions of the current image.

6. The portable UAV inspection image real-time analysis method according to claim 5, characterized in that, The steps in S2025 include: S20251. Randomly select texture region blocks from other calibration inspection images that belong to the same category as the current calibration inspection image; S20252. Select a center position according to the masking rules to obtain the selected target position, and define the transplant target region around the selected target position for performing texture fusion; S20253. Using the Poisson image editing method, seamlessly blend texture region blocks into the selected target location of the current calibration inspection image, perform Poisson image editing and fusion, and generate a texture transfer enhancement image.

7. The portable UAV inspection image real-time analysis method according to claim 1, characterized in that, The steps in S203 include: S2031. Replace the standard convolution in the first layer of the backbone network with a multi-branch differential convolution module to construct the input layer of the analysis model, and calculate the output feature map of the multi-branch differential convolution module based on texture transplantation to enhance the image; S2032. Insert a dynamic receptive field module after each residual block of the backbone network and calculate the output feature map of the dynamic receptive field module; S2033. A hierarchical feature adaptive fusion module is used to calculate hierarchical fusion features.

8. The portable UAV inspection image real-time analysis method according to claim 1, characterized in that, The steps in S204 include: S2041. Calculate conditional modulation feature vectors based on hierarchical fusion features; S2042. Calculate the total loss function consisting of the standard cross-entropy classification loss and the image correction loss.

9. A portable UAV inspection image real-time analysis method according to claim 8, characterized in that, The steps in S2041 include: S20411. Input the metadata encoding vector into two independent small multilayer perceptrons to generate conditional scaling vector and conditional bias vector respectively, and obtain the conditional modulation feature vector based on the conditional scaling vector and conditional bias vector. S20412. Input the conditional modulation feature vector into the classifier to obtain the predicted probability distribution of the defect category.

10. A method for real-time analysis of portable UAV inspection images according to claim 1, characterized in that, The defect status labels in S1 include no obvious defects found, insulator damage, vibration damper slippage, foreign objects in the conductor, and hardware corrosion.

Images