A method for unmanned aerial vehicle inspection image recognition and abnormality evaluation

Abstract

The application relates to the technical field of image recognition, and specifically discloses a method for image recognition and abnormality evaluation in unmanned aerial vehicle inspection, which effectively eliminates environmental light interference by means of active multi-angle light compensation and differential imaging mechanism, combines a light-invariant feature expression generated by maximum value fusion of each channel differential feature pixel by pixel by a light-weight network on a machine carrier, inputs a light-invariant feature map as a condition to a ground terminal, decouples normal background and abnormal features by a conditional variational autoencoder, reconstructs a non-abnormal background image by decoding only the normal background code, generates an abnormal segmentation map by residual analysis, converts discrete single detection results into continuous, structured and quantifiable abnormal evolution data flow through a cross-time skeleton graph structure construction and graph matching mechanism, and finally outputs a joint risk index through fusion analysis of length, width, acceleration and node degree change, so as to provide an active decision basis with high timeliness and strong robustness for structural safety maintenance.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections. Background Technology

[0002] As an efficient and flexible remote inspection method, drone inspection has been widely used in the field of safety monitoring of various infrastructures. By carrying visible light cameras and other visual sensors, drones can collect images of the surface condition of structures such as bridges, tunnels, building exteriors, and civil defense projects, and use image recognition technology to detect anomalies. This is an important way to ensure structural safety and reduce the risks of manual inspection.

[0003] Existing anomaly recognition methods based on UAV inspection images typically employ deep learning models to analyze and process the acquired images. These methods mostly follow a supervised learning paradigm, heavily relying on a large number of pixel-level manually annotated anomaly samples to train the model, enabling it to learn the ability to distinguish abnormal targets from normal backgrounds. However, in engineering practice, structural surface anomalies are diverse and occur infrequently, such as cracks in concrete surfaces, coating peeling, water seepage, and corrosion of metal components. The visual characteristics of various anomalies differ significantly, making it extremely time-consuming and labor-intensive to collect and accurately annotate a training dataset that covers all anomaly types. Furthermore, it is difficult to exhaustively represent all possible anomaly forms in real-world scenarios. When anomaly samples are insufficient or imbalanced, the model's generalization ability and robustness in recognizing rare anomalies are severely limited.

[0004] Furthermore, many early anomalies are extremely subtle in images. Due to limitations such as shooting angle, lighting changes, shadow occlusion, and interference from background textures, traditional image recognition models often struggle to reliably capture these subtle features. To improve the ability to represent subtle anomalies, some methods employ deeper network structures. However, this leads to a sharp increase in computational load and model parameters, making it difficult to achieve real-time processing with the limited computing resources on the UAV's onboard device. If all inspection images are transmitted back to a ground server for centralized processing, the massive amount of image data generated by each inspection will place a heavy burden on data storage, transmission bandwidth, and subsequent management.

[0005] More importantly, most current anomaly detection methods based on UAV inspection images can only analyze a single acquired image independently, outputting static information such as whether an anomaly exists at the current moment, and the location and size of the anomaly. In actual structural maintenance work, in addition to knowing the current state of the anomaly, understanding its changes over time is crucial for assessing structural safety. However, the existing single-inspection independent mode severs the state connection of the same structure across multiple inspections, failing to reflect the development process of the anomaly over time. For structural maintenance decisions, distinguishing between a long-term stable existing defect and a rapidly developing active anomaly is far more valuable than simply knowing that an anomaly exists. However, existing technologies typically only provide isolated attribute information of the anomaly target at the current inspection moment, making it difficult to further analyze and comprehensively judge the development trend of the anomaly based on this information, resulting in limited support for maintenance decisions from the inspection results. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method for image recognition and anomaly assessment during unmanned aerial vehicle (UAV) inspections, which can effectively solve the problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections, comprising: Step 1, controlling the UAV at preset acquisition points on the surface of a target structure, projecting supplementary light onto the surface of the target structure at at least two different illumination angles, and simultaneously acquiring multi-channel differential images corresponding to different illumination angles; Step 2, inputting the multi-channel differential images into a lightweight initial screening network deployed on the UAV's onboard unit, outputting candidate image patches through the lightweight initial screening network, and transmitting the candidate image patches back to the ground station; Step 3, inputting the candidate image patches into a preset... The constructed anomaly feature decoupling model decouples the anomaly features of the candidate image patch from the normal background features, and generates a segmentation result map containing the anomaly target based on the decoupled features; Step four: Based on the segmentation result map, extract the attribute information of the anomaly target identified in the current inspection, and compare the attribute information with the historical attribute information of the same target structure across time series to obtain the attribute change amount of the anomaly target; Step five: Based on the attribute change amount, evaluate the evolution trend of the anomaly target, and when the evolution trend reaches the preset risk condition, output the corresponding risk warning information.

[0008] As a further method, supplementary light is projected onto the surface of the target structure at at least two different illumination angles, and multi-channel differential images corresponding to the different illumination angles are acquired simultaneously. The specific process is as follows: Using the supplementary lighting device mounted on the UAV, a first supplementary light is projected onto the surface of the target structure at a first incident angle at the preset acquisition point, and a first inspection image is acquired simultaneously. A second supplementary light is projected onto the surface of the target structure at a second incident angle, and a second inspection image is acquired simultaneously. Pixel-level difference operations are performed on the first inspection image and the second inspection image to generate a multi-channel difference image. Each channel of the multi-channel difference image corresponds to the difference result under different combinations of illumination angles.

[0009] As a further method, the multi-channel differential image is input to a lightweight initial screening network deployed on the UAV's onboard unit, and candidate image patches are output through the lightweight initial screening network, specifically as follows: The multi-channel difference image is input into the lightweight initial screening network, which includes multiple parallel coding branches. These branches process each channel of the multi-channel difference image to extract the difference feature map corresponding to each channel. The difference feature map corresponding to each channel is then input into a feature fusion layer, which performs a pixel-by-pixel maximum value operation on the difference feature map corresponding to each channel to finally generate an illumination-invariant feature map. The lightweight initial screening network determines whether there is a suspected anomaly region based on the illumination-invariant feature map. If so, it outputs a candidate image patch containing the suspected anomaly region.

[0010] As a further method, the candidate map tiles are input into a pre-constructed anomaly feature decoupling model at the ground station. The anomaly feature decoupling model decouples the anomaly features of the candidate map tiles from the normal background features. The specific decoupling process is as follows: The anomalous feature decoupling model is specifically a conditional variational autoencoder. The encoder receives the candidate map patches and uses the illumination invariant feature maps corresponding to the candidate map patches as conditional inputs to encode a global latent vector. The encoder has a vector splitting module that splits the global latent vector into normal background latent codes and anomalous latent codes. The vector splitting module is connected to an orthogonal projection layer, which applies orthogonality constraints to the normal background latent codes and the anomalous latent codes.

[0011] As a further method, the anomaly feature decoupling model generates a segmentation result map containing the anomaly target based on the decoupled features. The specific generation process is as follows: The decoder of the conditional variational autoencoder only receives the normal background latent code processed by the orthogonal projection layer, and uses the illumination invariant feature map as auxiliary input to decode and reconstruct a background reconstruction image containing only normal background structure; calculates the residual image between the candidate image patch and the background reconstruction image; performs adaptive threshold segmentation on the residual image, and finally obtains the segmentation result image containing the abnormal target.

[0012] As a further method, based on the segmentation result map, the attribute information of the abnormal targets identified in the current inspection is extracted, and the attribute information is compared with the historical attribute information of the same target structure across time series to obtain the attribute change of the abnormal targets, specifically: The skeleton of the abnormal target region in the segmentation result graph is extracted to generate an abnormal target graph structure composed of nodes and edges. The nodes represent the endpoints or bifurcation points of the skeleton, and the edges represent the skeleton line segments connecting adjacent nodes. The historical abnormal target graph structure of the same abnormal target is retrieved by the position labels corresponding to the candidate graph blocks. The abnormal target graph structure is matched with the historical abnormal target graph structure, and the absolute change in length, absolute change in width, and degree of each edge in the matched abnormal target graph structure are calculated as the attribute change of the abnormal target.

[0013] As a further method, the evolution trend of the abnormal target is evaluated based on the change in the attribute, and when the evolution trend reaches a preset risk condition, corresponding risk warning information is output. The specific process is as follows: The acceleration of the abnormal target is calculated by obtaining the absolute changes in length and width through multiple consecutive inspections. When the acceleration exceeds a preset acceleration threshold, the abnormal target is determined to be in an accelerating expansion trend. When the degree change of the node indicates the addition of a new branch point in the abnormal target's graph structure, the abnormal target is determined to be in a morphological differentiation trend. When the acceleration does not exceed the preset acceleration threshold and no new branch point is added, and the absolute changes in length and width remain within the change range for a preset number of consecutive inspections, the abnormal target is determined to be in a steady-state expansion trend. Corresponding risk weights are assigned to the accelerating expansion trend, the morphological differentiation trend, and the steady-state expansion trend, and a joint risk index is calculated. When the joint risk index exceeds a preset comprehensive risk threshold, corresponding risk warning information is generated and output.

[0014] As a further method, the risk warning information includes at least the location label of the abnormal target, the abnormal target graph structure, the historical abnormal target graph structure, the absolute change in length, the absolute change in width, the degree change of the node, the determination result of the evolution trend, and the corresponding risk level identifier.

[0015] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects: (1) This invention provides a method for image recognition and anomaly assessment of UAV inspections. Through a closed-loop design, it realizes a complete intelligent inspection link, from the acquisition and rapid screening of multi-angle illumination difference images on the airborne end, to the decoupling and segmentation of features on the ground end, and then to the cross-temporal graph structure evolution analysis and fusion early warning. It effectively eliminates ambient light interference by using active multi-angle supplementary lighting and differential imaging mechanisms. Combined with the illumination invariant feature expression generated by the pixel-by-pixel maximum value fusion of differential features of each channel by the lightweight network on the airborne end, it can robustly and quickly lock small anomaly candidate areas and only send back key image blocks under the limited computing power of UAVs, which significantly reduces the data transmission bandwidth and storage burden. The ground end uses a conditional variational autoencoder with illumination invariant feature map as conditional input. Orthogonal decoupling is applied to normal background and anomalous features, and an image without anomalies is reconstructed by decoding only the normal background code. Then, an accurate anomaly segmentation map is generated by residual analysis. This enables the accurate capture of rare anomalies with diverse shapes without the need for pre-labeling of all types of anomaly samples. The cross-temporal skeleton graph structure construction and graph matching mechanism transform discrete single detection results into a continuous, structured, and quantifiable anomaly evolution data stream. Finally, through the fusion analysis of changes in multi-dimensional attributes such as length, width, acceleration, and node degree, a trend assessment and joint risk index containing judgments of accelerated expansion, morphological differentiation, or steady-state expansion are output. This provides a proactive decision-making basis for structural safety maintenance that is both timely, robust, and insightful in its evolution.

[0016] (2) The present invention generates an illumination-invariant feature map by independently encoding each channel of the multi-channel differential image and fusing the maximum value of each pixel. This feature fusion strategy enables the lightweight primary screening network to naturally have the ability to remove complex illumination interference from the physical imaging level in airborne processing. This allows the limited computing power hardware mounted on the compact platform of the UAV to directly utilize the illumination-insensitive features after differential enhancement to stably highlight potential abnormal targets with extremely weak contrast without relying on the computationally expensive illumination normalization algorithm or deep denoising module. Thus, it can achieve rapid screening of small-class abnormal suspicious areas with high recall and low false alarm on embedded processors with strictly limited power consumption, and significantly extend the effective working time of a single inspection.

[0017] (3) This invention extracts the skeleton from the segmentation results and generates a graph structure composed of nodes and edges, transforming the morphological evolution of abnormal targets into a graph data structure that can be accurately matched. This makes each inspection no longer an isolated pixel-level comparison, but rather relies on the strict alignment of the graph structure in terms of topological and geometric attributes to trace the evolution of the same abnormality. This graph-level cross-time sequence comparison method provides a highly consistent and spatiotemporally continuous measurement basis for subsequent multidimensional evolution trend quantitative analysis, enabling the discrete inspection data collected by UAVs to truly serve the accurate characterization and reliable prediction of the development law of surface defects of structures. Attached Figure Description

[0018] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] Reference Figure 1 The diagram shows a flowchart of the method of the present invention. The present invention provides a method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections, including: Step 1: The ground station control system controls the UAV to project supplementary light onto the target structure surface at preset acquisition points with at least two different illumination angles, and simultaneously acquires multi-channel differential images corresponding to different illumination angles; Step 2: The multi-channel differential images are input into a lightweight initial screening network deployed on the UAV's onboard terminal, and candidate image patches are output through the lightweight initial screening network and transmitted back to the ground station; Step 3: The candidate image patches are input into a pre-constructed anomaly detection system at the ground station. The normal feature decoupling model decouples the abnormal features of the candidate map patch from the normal background features in the latent space, and generates a segmentation result map containing the abnormal target based on the decoupled features; Step four: Based on the segmentation result map, extract the attribute information of the abnormal target identified in the current inspection, and compare the attribute information with the historical attribute information recorded in the historical inspection of the same target structure across time series to obtain the attribute change amount of the abnormal target; Step five: Based on the attribute change amount, evaluate the evolution trend of the abnormal target, and when the evolution trend reaches the preset risk condition, output the corresponding risk warning information.

[0022] Preferably, the target structure is an artificial or natural structure that requires regular inspection to identify surface anomalies, such as a dam, bridge, wind turbine blade, or building exterior wall; the target structure surface refers to the outer surface area of ​​the target structure that is directly exposed to the external environment and can be photographed by the UAV; the preset collection points are stored in the UAV's onboard memory or the inspection task database of the ground station control system. The preset collection points are generated by importing the three-dimensional model of the target structure and performing coverage sampling, or by recording hovering points along the planned route during the first teaching flight. The three-dimensional model refers to the digital three-dimensional representation of the target structure, specifically obtained through lidar scanning, oblique photogrammetry, or building information modeling. The model is a point cloud model or mesh model constructed using Building Information Modeling (BIM) technology, containing precise geometric dimensions and spatial coordinate information. The input is a 3D model file of the target structure (e.g., .ply, .obj, .ifc, etc.) and parameters such as camera field of view and safety distance. The output is a list of preset acquisition points containing a series of discrete spatial coordinate points and the beam emission direction of the corresponding supplementary lighting device or the attitude of the UAV / gimbal. This list is stored as part of the flight path mission file in the onboard memory of the UAV or the inspection mission database of the ground station control system. The preset acquisition points are located in spatial positions that can clearly capture the target area on the surface of the target structure, and the points maintain a fixed degree of overlap and safety distance.

[0023] Specifically, the process of projecting supplementary light onto the target structure surface at at least two different illumination angles and simultaneously acquiring multi-channel differential images corresponding to different illumination angles is as follows: Using the supplementary lighting device mounted on the UAV, at the preset collection point, a first supplementary light is projected onto the surface of the target structure at a first incident angle and a first inspection image is collected simultaneously. A second supplementary light is projected onto the surface of the target structure at a second incident angle and a second inspection image is collected simultaneously. The first incident angle and the second incident angle are different from each other. Specifically, the first incident angle is a grazing angle with an angle of 15 to 30 degrees relative to the normal of the target structure surface, and the second incident angle is an oblique angle with an angle of 45 to 60 degrees relative to the normal of the target structure surface.

[0024] It should be added that the supplementary lighting device is a high-brightness LED array or LED floodlight module mounted on the UAV gimbal or fuselage, which has the function of controllable beam direction. It adjusts the beam direction to project supplementary light by receiving instructions from the ground station control system or airborne control unit. The first supplementary light is a directional beam projected by the supplementary lighting device at the first incident angle to enhance the surface texture and shadows. The second supplementary light is a directional beam projected by the supplementary lighting device at the second incident angle to provide relatively uniform illumination. The first inspection image is an image synchronously acquired by a visible light or infrared inspection camera mounted on the UAV at the moment the first supplementary light is projected. The second inspection image is an image synchronously acquired by the same inspection camera at the moment the second supplementary light is projected.

[0025] Pixel-level difference operations are performed on the first inspection image and the second inspection image to generate a multi-channel difference image. Each channel of the multi-channel difference image corresponds to the difference result under different combinations of illumination angles.

[0026] It also needs to be explained that the pixel-level difference operation to generate a multi-channel difference image specifically involves: after strictly registering the first inspection image and the second inspection image, for each pixel position in the registered two images, subtracting the corresponding pixel value of the second inspection image from the pixel value of the first inspection image to obtain a difference image as the first channel; simultaneously, calculating the average image of the registered first and second inspection images, and calculating the difference image between the average image and the first inspection image as the second channel, and calculating the difference image between the average image and the second inspection image as the third channel; these three difference images together form a three-channel multi-channel difference image. Each independent channel of the multi-channel difference image encodes pixel-level difference information calculated from a specific combination of illumination angles (e.g., the difference between image A and image BB, or the difference between image A and the average image). This difference information can highlight surface anomalies exhibiting different shadow or reflection characteristics under different illumination directions. The difference result specifically refers to the numerical matrix obtained by difference operation, which characterizes the difference in pixel intensity between images under different lighting conditions. Specifically, it is represented as a two-dimensional array with the same size as the input inspection image (height H × width W), or as a three-dimensional tensor of H × W × CC when there are multiple difference channels, where CC is the number of difference channels.

[0027] Furthermore, the multi-channel differential image is input into a lightweight initial screening network deployed on the UAV's onboard unit, and candidate image patches are output through the lightweight initial screening network, specifically as follows: First, the UAV's onboard terminal is located on the UAV's embedded computing platform, which includes a GPU or NPU for accelerating neural network inference. The lightweight initial screening network is constructed as follows: a lightweight convolutional neural network architecture is designed, with the multi-channel difference image as input and bounding box candidate patches with anomaly confidence scores as output. This network uses depthwise separable convolutions as its backbone to reduce the number of parameters and computational cost. Using a UAV inspection image dataset containing various structural surface defects and normal backgrounds, the network is trained in a supervised learning manner to distinguish between abnormal regions and normal backgrounds, and to generate bounding box regression parameters for candidate patches. The loss function used during training is a weighted sum of classification loss (i.e., binary cross-entropy loss) and bounding box regression loss (i.e., intersection-union loss). The bounding box regression parameters refer to a set of values ​​predicted by the network for each preset anchor box or feature map spatial location to adjust the position and size of the bounding box. The network weights are iteratively updated through backpropagation until the model converges.

[0028] The multi-channel difference image is input into the lightweight primary screening network, which includes multiple parallel coding branches. Each of the multiple parallel coding branches processes each channel of the multi-channel difference image to extract the difference feature map corresponding to each channel.

[0029] It should be noted that the function of the multi-path parallel coding branch is to provide a dedicated shallow feature extraction path for each independent channel of the multi-channel difference image, so as to capture the unique texture anomaly responses under different illumination difference combinations. The multi-path parallel coding branch processes each channel of the multi-channel difference image to extract the difference feature map corresponding to each channel. The implementation method is as follows: the channel dimension of the multi-channel difference image is split, and each single-channel image is fed as input into a corresponding coding branch composed of several depthwise separable convolutional layers. This branch independently performs convolution, activation and pooling operations, and finally outputs a single-channel or multi-channel feature map that reflects the difference characteristics of that channel, i.e., the difference feature map.

[0030] The differential feature maps corresponding to each channel are input into a feature fusion layer. The feature fusion layer performs a pixel-by-pixel maximum value operation on the differential feature maps corresponding to each channel, and finally generates an illumination-invariant feature map.

[0031] It should be further noted that the feature fusion layer is located inside the lightweight initial screening network. By integrating features from different illumination difference branches, it suppresses false positive texture responses caused by a single specific illumination angle and enhances true anomalous features that stably appear under various illumination combinations. The difference feature maps corresponding to all channels are stacked in the channel dimension, and then a pixel-by-pixel Max operation is performed on the stacked feature map in the channel dimension. That is, for a pixel at spatial position (i, j), i represents the row index, with a value range from 0 to the height of the feature map minus 1 (H-1); j represents the column index, with a value range from 0 to the width of the feature map minus 1 (W-1). The maximum pixel value of all channels at this position is taken as the feature value of the fused feature map at that position. The resulting single-channel or dimensionality-reduced feature map is the illumination-invariant feature map, which represents a stable anomalous response that is insensitive to changes in illumination angle.

[0032] The lightweight initial screening network determines whether there are any suspicious regions of anomaly based on the illumination invariance feature map. If so, it outputs candidate map patches containing the suspicious regions of anomaly.

[0033] The lightweight initial screening network determines the existence of suspected anomaly-like regions as follows: The illumination-invariant feature map is input into a detection head consisting of one or more convolutional layers. This detection head predicts a set of bounding box coordinate offsets and an anomaly-like confidence score for each spatial location in the feature map. The bounding box coordinate offsets are obtained by convolving the input illumination-invariant feature map with convolutional layers in the detection head. The anomaly-like confidence score is a raw score output from another channel of the same dataset, which is then compressed to the range of 0 to 1 after processing with a sigmoid activation function. When the anomaly-like confidence score of a predicted bounding box exceeds a preset initial screening threshold (e.g., 0.5), the region contained in that bounding box is determined to be a suspected anomaly-like region. The preset initial screening threshold is used as a binary decision threshold to determine whether a predicted region is a candidate anomaly. This threshold is stored in the lightweight initial screening network configuration file on the UAV's onboard device. The detection head of the lightweight initial screening network outputs the filtered bounding box coordinates and crops the corresponding image region, i.e., the candidate image patch, from the original inspection image. The candidate patch is represented as a three-dimensional tensor of a rectangular image sub-region containing potential anomalous targets, cropped from the original inspection image.

[0034] Specifically, at the ground station, the candidate image patches are input into a pre-constructed anomaly feature decoupling model. The anomaly feature decoupling model decouples the anomaly features of the candidate image patches from the normal background features. The specific decoupling process is as follows: First, the ground station is a high-performance computer or server system located on the ground that maintains point-to-point or relay communication with the UAV. It receives the candidate map tiles and associated telemetry data (including at least the real-time spatial position of the UAV at the moment the candidate map tile is acquired, its three-axis attitude angles (pitch, roll, yaw), timestamp, and camera parameters) transmitted back by the UAV via a wireless data link. The ground station runs image processing and analysis software, which interacts with subsequent algorithm modules such as the anomaly feature decoupling model. The construction process of the anomaly feature decoupling model is as follows: The main body adopts a two-stage training strategy. First, a pre-training stage is performed, using UAV inspection image patches containing only normal background structures to train a conditional variational autoencoder. At this stage, only reconstruction loss and KL divergence loss are used. Then, a second-stage fine-tuning stage is performed, introducing training samples containing anomalous targets and unfreezing or enabling the network part after the orthogonal projection layer. The total loss function is a weighted sum of reconstruction loss, KL divergence loss, and orthogonality loss. All losses are jointly optimized through backpropagation, ultimately enabling the model to separate anomalous features from normal background features during the encoding stage. The reconstruction loss measures the difference between the reconstructed background image and the input original candidate image patch at the pixel level, usually using mean squared error or mean absolute error. The KL divergence loss measures the difference between the distribution of the latent vectors predicted by the encoder and the standard normal distribution. It is the core constraint term of the variational autoencoder, used to make the latent space continuous and regular. The orthogonality loss measures the absolute value of the cosine of the angle between the normal background latent code and the anomalous latent code. Minimizing this loss forces the two latent vectors to be orthogonal to each other in the latent space.

[0035] The anomaly feature decoupling model is specifically a conditional variational autoencoder. The encoder receives the candidate map patch and uses the illumination invariant feature map corresponding to the candidate map patch as a conditional input to encode a global latent vector.

[0036] A conditional variational autoencoder (CAE) is a generative model that learns a low-dimensional latent representation of input data and generates new data from this representation under given conditional constraints. In this scheme, it utilizes conditional information (light-invariant feature maps) to guide the decoupling process, making the encoding of the latent space more structured. The encoder concatenates the input candidate map patches with the conditional light-invariant feature maps along the channel dimension to form a combined tensor. This combined tensor is then passed through a series of convolutional and downsampling layers to progressively compress the spatial dimension and increase the number of channels. Finally, two fully connected layers predict the mean vector and log-variance vector of the latent distribution, respectively. A vector, namely the global latent vector, is sampled from this distribution. This global latent vector is represented by a multi-dimensional vector (e.g., 256-dimensional or 512-dimensional).

[0037] The encoder has a vector splitting module inside, which splits the global latent vector into a normal background latent code and an abnormal latent code.

[0038] The vector splitting module is a parameterless operation layer located at the encoder output. Its function is to forcibly separate the encoded unified global representation into two independent information carriers. The splitting method is as follows: half of the global latent vector dimension is pre-defined as the dimension of the normal background latent code, and the other half as the dimension of the abnormal latent code. The splitting module simply assigns the first half of the global latent vector to the normal background latent code and the second half to the abnormal latent code. The aforementioned abnormal features and normal background features correspond to and are encoded in the abnormal latent code and the normal background latent code, respectively. Specifically, the normal background latent code is a low-dimensional latent variable carrying background feature information such as normal surface texture, structure, and geometry, while the abnormal latent code is a low-dimensional latent variable carrying abnormal target feature information such as cracks, peeling, and corrosion. The two are compressed representations of feature information in the latent space; that is, features and latent codes are related as information content and encoding carriers.

[0039] The vector splitting module is connected to an orthogonal projection layer, which imposes orthogonality constraints on the normal background latent code and the abnormal latent code. In other words, it forces the two to be orthogonal to each other in the latent space.

[0040] By forcing the two latent codes to be orthogonal to each other in a high-dimensional latent space, the information coupling and mutual interference between the normal background representation and the anomalous feature representation are minimized. This forces the model to learn to separate the two fundamentally different types of information and store them in independent subspaces during the encoding stage, thereby ensuring that a pure background image without any anomalous traces can be reconstructed using only the normal background latent code.

[0041] Furthermore, the abnormal feature decoupling model generates a segmentation result map containing the abnormal target based on the decoupled features. The specific generation process is as follows: The decoder of the conditional variational autoencoder only receives the normal background latent code after it has been processed by the orthogonal projection layer, and uses the illumination invariance feature map as an auxiliary input to decode and reconstruct a background reconstruction image that contains only the normal background structure.

[0042] The decoder is a network component in a conditional variational autoencoder responsible for reconstructing latent vectors back into image space. Its function is to generate a reconstructed image with the same spatial dimensions as the original input image, based on the input latent vectors and conditions. The decoding and reconstruction process involves concatenating the normal background latent code with a feature map extracted from an illumination-invariant feature map, or fusing them through a spatial adaptive normalization layer. Then, through a series of upsampling and convolutional layers, the spatial resolution is gradually restored and the number of channels is reduced, ultimately outputting a three-channel color image, i.e., the reconstructed background image. The reconstructed background image is represented by a three-channel tensor with the same height and width as the original candidate image patch. The normal background structure refers to the regular, non-abnormal visual texture and geometric shape formed on the surface of the target structure during design, construction, or long-term use, such as the uniform pitting of a concrete surface, the coating texture of a metal surface, or a complete and defect-free structural edge.

[0043] The residual image between the candidate image patch and the background reconstructed image is calculated. Specifically, after converting the candidate image patch and the background reconstructed image into grayscale images, the absolute value of the difference between grayscale values ​​is calculated point by point at the corresponding pixel position. The formula is: RR(i,j)=|C(i,j)-B(i,j)|, where RR is the residual image, C is the grayscale image of the candidate image patch, and B is the grayscale image of the background reconstructed image. The final residual image is represented by a single-channel grayscale tensor of the same size as the input image. The size of its pixel value reflects the probability and intensity of an anomaly at that position.

[0044] Adaptive thresholding is performed on the residual image. Specifically, the mean μ and standard deviation σ of all pixel values ​​in the residual image are calculated, and a segmentation threshold T = μ + k is set. σ, where k is an empirical coefficient (e.g., 3), and then all pixels in the residual image with pixel values ​​greater than T are marked as foreground (abnormal region) with a pixel value of 1; the remaining pixels are marked as background with a pixel value of 0, and finally the segmentation result image containing the abnormal target is obtained.

[0045] The anomalous target refers to a localized area appearing on the surface of the target structure that differs visually from the normal background structure and poses a potential threat to the structural integrity, safety, or function. This includes, but is not limited to, cracks, coating peeling, water seepage, and corrosion of metal components. The segmentation result image is represented as a binary image with the same spatial dimensions as the original candidate patches, where regions with a pixel value of 1 represent the segmented anomalous target regions.

[0046] Specifically, based on the segmentation result image, the attribute information of the abnormal targets identified in the current inspection is extracted, and the attribute information is compared with the historical attribute information of the same target structure across time series to obtain the attribute change of the abnormal targets. It needs to be explained that extracting the attribute information of abnormal targets through the segmentation result map specifically involves: firstly, performing connected component analysis on the binary segmentation map to identify each independent abnormal target region; then, calculating the area, perimeter, minimum bounding rectangle size, and spatial coordinates of the center point for each abnormal target region. These quantified descriptive quantities collectively constitute the attribute information of the abnormal target. The attribute information includes geometric morphological attributes and spatial location attributes. The geometric morphological attributes are represented by the graph structure obtained from the subsequent skeleton extraction, and the spatial location attributes are the location labels of the abnormal targets. Historical attribute information of the same target structure is extracted from the inspection history database of the ground station. This database uses the location label as the primary key and associates and stores the segmentation result map, graph structure, attribute information, and timestamp of the abnormal targets obtained from each inspection. The historical attribute information includes the historical abnormal target graph structure and its associated geometric metrics. The cross-time series comparison method involves performing graph matching between the currently extracted abnormal target graph structure and the retrieved historical abnormal target graph structures. For the corresponding nodes and edges in the matching, indices such as the absolute change in length, the absolute change in width, and the degree change of the nodes are calculated one by one.

[0047] The skeleton of the abnormal target region in the segmentation result graph is extracted to generate an abnormal target graph structure composed of nodes and edges. The nodes represent the endpoints or bifurcation points of the skeleton, and the edges represent the skeleton line segments connecting adjacent nodes.

[0048] The abnormal target region is represented by a connected region with a pixel value of 1 in the segmentation result image. An abnormal target graph structure composed of nodes and edges is generated. The specific process is as follows: For each independent binary region of the abnormal target, a mathematical morphology skeletonization algorithm (e.g., the Zhang-Suen algorithm or a distance-transform-based skeleton extraction method) is applied to obtain a centerline skeleton with a single pixel width; all endpoints (with only one skeleton neighbor pixel) and bifurcation points (with three or more skeleton neighbor pixels) on the skeleton are identified and defined as nodes of the graph; the skeleton pixel chains connecting adjacent nodes are defined as edges of the graph, and each edge stores the normal direction width information at each position on its corresponding skeleton line segment, thus forming an attribute graph structure with topological and geometric attributes for the nodes and edges.

[0049] By using the location tags corresponding to the candidate map tiles, the historical abnormal target map structure generated in the historical inspections for the same abnormal target can be retrieved. For example, a candidate map tile currently being processed, located at 185 meters above sea level in section 3 of the dam, has the location tag Dam3-185m. The ground station software queries the inspection history database using this tag, retrieves all inspection records with the location tag Dam3-185m in the historical records, and extracts the abnormal target map structure generated in the most recent inspection or on a specified date as the historical abnormal target map structure, thereby realizing the retrieval of historical data.

[0050] The abnormal target graph structure is matched with the historical abnormal target graph structure. Specifically, graph matching refers to using graph isomorphism or subgraph matching algorithms to establish a spatial and topological correspondence between the nodes and edges in the current graph structure and the nodes and edges in the historical graph structure. The matching process first performs rigid or non-rigid registration based on the spatial coordinates of the nodes to align the two graphs. Then, it determines the matching pairs between nodes by maximizing topological similarity and spatial proximity. The absolute change in length, absolute change in width, and degree change of each edge in the matched abnormal target graph structure are calculated. The absolute change in length refers to the absolute value of the difference between the length of the current skeleton line segment on the matched edge and the length of the skeleton line segment at the same historical position. The absolute change in width refers to the absolute value of the difference between the current average normal width on the matched edge and the historical average normal width. The degree change of a node refers to the difference between the degree of the current node (i.e., the number of edges connected to the node) and the degree of the historical matched node. These together serve as the attribute changes of the abnormal target.

[0051] Furthermore, the process of assessing the evolution trend of the abnormal target based on the attribute change and outputting corresponding risk warning information when the evolution trend reaches a preset risk condition is as follows: The evolutionary trend describes the dynamic behavioral pattern of the geometric attributes of an abnormal target changing over time. It is not a curve-like trend, but rather a numerical state label for classification and judgment, specifically including three mutually exclusive or coexisting discrete states: accelerated expansion trend, morphological differentiation trend, and steady-state expansion trend. The preset risk conditions refer to a series of threshold rules and logical combinations used to determine whether the evolutionary trend constitutes a structural security threat, including but not limited to acceleration thresholds, new bifurcation point events, and determination of the duration of change.

[0052] The acceleration of the size change of the abnormal target is calculated by obtaining the absolute change in length and the absolute change in width from multiple consecutive inspections. Specifically, the inspection timestamps and corresponding size values ​​(length or width) of the same abnormal target from the most recent three consecutive inspections are extracted. The size value is used as a function of time, and the second derivative is obtained by using the second-order central difference method or fitting a quadratic function to calculate the size change acceleration value at the current moment.

[0053] When the acceleration exceeds a preset acceleration threshold, the abnormal target is determined to be in an accelerating expansion trend. The preset acceleration threshold represents the maximum acceptable dimensional change acceleration threshold determined by structural safety assessment specifications or historical data statistics, extracted from the threshold table of the safety parameter file or database configured by the ground station system.

[0054] When the degree change of a node indicates the addition of a new branch point in the abnormal target graph structure, the abnormal target is determined to be in a morphological differentiation trend. Specifically, the degree change of a node is positive and the degree of historical matching nodes is 1 or 2 (i.e., endpoints and intermediate connection points), while the degree of the current node becomes 3 or above. This increase in degree directly indicates that the node has changed from a simple connection point to a branch point, that is, a new branch point has been added to the skeleton topology. The relationship between the new branch point and the degree change of a node is that the new branch point is a topological event directly characterized by the specific metric change of the node's degree change from less than 3 to greater than or equal to 3.

[0055] When the acceleration does not exceed the preset acceleration threshold and no new bifurcation point is added, and the absolute change in length and the absolute change in width both remain within a non-zero small change range during a preset number of continuous inspections, the abnormal target is determined to be in a steady-state expansion trend.

[0056] It should be explained that the preset number of times refers to the minimum number of consecutive inspections required to determine that the change mode is continuous, set by the user or by the system default, such as the most recent four consecutive inspections. This parameter is extracted from the configuration file of the ground station system. The small change range refers to the absolute value of the change being greater than 0 but less than a percentage threshold that defines a small increase. The lower limit of this threshold is a small positive value that excludes measurement noise (e.g., 0.1%), and the upper limit is the dividing line that defines accelerated expansion (e.g., 1%). The specific values ​​of the upper and lower limits are stored as system parameters in the configuration file of the ground station, thereby ensuring that the extremely slow but stable creep expansion is captured.

[0057] Risk weights are assigned to the accelerating expansion trend, the morphological differentiation trend, and the steady-state expansion trend, and a joint risk index is calculated. It should be emphasized that the accelerating expansion trend, the morphological differentiation trend, and the steady-state expansion trend are three numerical judgment states representing different sources of risk, rather than curve shapes. The joint risk index R is calculated using the formula R = w1 × α × Sacc + w2 × β × Sdiff + w3 × γ × Sstable, where Sacc, Sdiff, and Sstable are the Boolean judgment values ​​(0 or 1) for the corresponding trends. α, β, and γ are preset basic risk scores for each trend, which are the inherent danger scores assigned to the three evolutionary trends of accelerated expansion, morphological differentiation, and steady-state expansion based on structural safety assessment standards or historical case statistics. For example, they can be preset to 0.8, 0.6, and 0.3 respectively, representing the basic risk equivalent of the trend type, and are stored in the configuration file of the ground station risk assessment and early warning module. w1, w2, and w3 are dynamic adjustment weights calculated based on the current acceleration value, the number of newly added bifurcation points, and the duration of continuous stable expansion, with values ​​ranging from 0 to 1. The duration of continuous stable expansion refers to the cumulative time during which the size change of the abnormal target remains within a small range in the most recent preset number of inspections. w1 is a monotonically increasing function of acceleration value; the greater the acceleration, the higher w1 is. w2 is an increasing function of the number of new bifurcation points. w3 is an increasing function of the duration of continuous and stable expansion. The specific function mapping relationship and parameters are defined by domain experts and stored in the ground station configuration.

[0058] When the joint risk index exceeds the preset comprehensive risk threshold, the corresponding risk warning information is generated and output.

[0059] The preset comprehensive risk threshold represents the minimum joint risk index value required to trigger a risk warning. This threshold is a decision boundary defined in business operations based on historical cases and expert experience, extracted from the configuration parameters or experience rule base of the ground station's risk assessment module. The risk assessment and early warning module on the ground station is responsible for generating and outputting risk warning information.

[0060] Specifically, the risk warning information includes at least the location label of the abnormal target, the abnormal target graph structure, the historical abnormal target graph structure, the absolute change in length, the absolute change in width, the degree change of the node, the determination result of the evolution trend, and the corresponding risk level identifier.

[0061] The location tag of the anomalous target refers to a unique code or coordinate string that identifies the spatial location of the anomalous target on the target structure, such as Dam Section 3 - Elevation 185m - Longitude XX - Latitude YY. The determination result of the evolution trend refers to the final trend status text output after logical rule determination based on the calculated attribute change amount, for example: determined as: accelerated expansion trend. The risk level identifier refers to a standardized risk severity label obtained by mapping the interval of the joint risk index value, such as high risk, medium risk, or low risk; specifically, two preset thresholds are extracted from the configuration file of the ground station risk assessment module, including a high joint risk threshold and a low joint risk threshold. When R is greater than or equal to the high joint risk threshold, it is determined to be high risk; when R is less than the high joint risk threshold and R is greater than or equal to the low joint risk threshold, it is determined to be medium risk; when R is less than the low joint risk threshold, it is determined to be low risk.

[0062] The parameters in the aforementioned early warning information provide accurate anomaly snapshots and multi-dimensional quantitative basis for subsequent structural maintenance decisions, maintenance resource scheduling, and updates to the full life cycle structural health record. This supports maintenance personnel in quickly locating, characterizing, and classifying risks, enabling predictive maintenance.

[0063] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

Claims

1. A method for image recognition and anomaly assessment during unmanned aerial vehicle (UAV) inspections, characterized in that, Includes the following steps: Step 1: Control the drone to the preset acquisition points on the surface of the target structure, project supplementary light onto the surface of the target structure at at least two different illumination angles, and simultaneously acquire multi-channel differential images under the corresponding different illumination angles. Step 2: Input the multi-channel differential image into a lightweight initial screening network deployed on the UAV's onboard terminal, output candidate image patches through the lightweight initial screening network, and transmit the candidate image patches back to the ground station; Step 3: Input the candidate map tiles into the pre-constructed anomaly feature decoupling model at the ground station. The anomaly feature decoupling model decouples the anomaly features of the candidate map tiles from the normal background features, and generates a segmentation result map containing the anomaly target based on the decoupled features. Step 4: Based on the segmentation result map, extract the attribute information of the abnormal targets identified in the current inspection, and compare the attribute information with the historical attribute information of the same target structure across time series to obtain the attribute change amount of the abnormal targets; Step 5: Based on the change in the attribute, assess the evolution trend of the abnormal target, and when the evolution trend reaches the preset risk condition, output the corresponding risk warning information.

2. The method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections according to claim 1, characterized in that: The process of projecting supplementary light onto the surface of the target structure at at least two different illumination angles and simultaneously acquiring multi-channel differential images corresponding to different illumination angles is as follows: Using the supplementary lighting device mounted on the UAV, at the preset collection point, a first supplementary light is projected onto the surface of the target structure at a first incident angle and a first inspection image is collected simultaneously; a second supplementary light is projected onto the surface of the target structure at a second incident angle and a second inspection image is collected simultaneously. Pixel-level difference operations are performed on the first inspection image and the second inspection image to generate a multi-channel difference image. Each channel of the multi-channel difference image corresponds to the difference result under different combinations of illumination angles.

3. The method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections according to claim 2, characterized in that: The multi-channel differential image is input into a lightweight initial screening network deployed on the UAV's onboard unit, and candidate image patches are output through the lightweight initial screening network, specifically as follows: The multi-channel difference image is input into the lightweight primary screening network, which includes multiple parallel coding branches. Each of the multiple parallel coding branches processes each channel of the multi-channel difference image to extract the difference feature map corresponding to each channel. The differential feature maps corresponding to each channel are input into the feature fusion layer, which then performs a pixel-by-pixel maximum value operation on the differential feature maps corresponding to each channel to finally generate an illumination invariant feature map. The lightweight initial screening network determines whether there are any suspicious regions of anomaly based on the illumination invariance feature map. If so, it outputs candidate map patches containing the suspicious regions of anomaly.

4. The method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections according to claim 3, characterized in that: At the ground station, the candidate map tiles are input into a pre-built anomaly feature decoupling model. The anomaly feature decoupling model decouples the anomaly features of the candidate map tiles from the normal background features. The specific decoupling process is as follows: The anomaly feature decoupling model is specifically a conditional variational autoencoder. The encoder receives the candidate map patch and uses the illumination invariant feature map corresponding to the candidate map patch as a conditional input to encode it into a global latent vector. The encoder is equipped with a vector splitting module, which splits the global latent vector into normal background latent codes and abnormal latent codes. The vector splitting module is connected to an orthogonal projection layer, which applies orthogonality constraints to the normal background latent code and the abnormal latent code.

5. The method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections according to claim 4, characterized in that: The abnormal feature decoupling model generates a segmentation result map containing the abnormal target based on the decoupled features. The specific generation process is as follows: The decoder of the conditional variational autoencoder only receives the normal background latent code after it has been processed by the orthogonal projection layer, and uses the illumination invariance feature map as an auxiliary input to decode and reconstruct a background reconstruction image that contains only the normal background structure. Calculate the residual image between the candidate patch and the reconstructed background image; Adaptive thresholding is performed on the residual image to obtain the final segmentation result image containing the abnormal target.

6. The method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections according to claim 5, characterized in that: Based on the segmentation result map, the attribute information of the abnormal targets identified in the current inspection is extracted, and the attribute information is compared with the historical attribute information of the same target structure across time series to obtain the attribute change of the abnormal targets. Specifically: The skeleton of the abnormal target region in the segmentation result graph is extracted to generate an abnormal target graph structure composed of nodes and edges. The nodes represent the endpoints or bifurcation points of the skeleton, and the edges represent the skeleton line segments connecting adjacent nodes. By using the location labels corresponding to the candidate map tiles, retrieve the historical abnormal target map structure of the same abnormal target; The abnormal target graph structure is matched with the historical abnormal target graph structure. The absolute change in length, absolute change in width, and degree of each edge in the matched abnormal target graph structure are calculated and used as the attribute change of the abnormal target.

7. The method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections according to claim 6, characterized in that: The process of assessing the evolution trend of the abnormal target based on the attribute change and outputting corresponding risk warning information when the evolution trend reaches a preset risk condition is as follows: The acceleration of the abnormal target is calculated by obtaining the absolute changes in length and width through multiple consecutive inspections. When the acceleration exceeds a preset acceleration threshold, the abnormal target is determined to be in an accelerating expansion trend; When the degree change of the node indicates that a new branching point has been added in the abnormal target graph structure, it is determined that the abnormal target is in a morphological differentiation trend. When the acceleration does not exceed the preset acceleration threshold and no new bifurcation point is added, and the absolute change in length and the absolute change in width remain within the change range during a preset number of continuous inspections, the abnormal target is determined to be in a steady-state expansion trend. Risk weights are assigned to the accelerated expansion trend, the morphological differentiation trend, and the steady-state expansion trend, and a joint risk index is calculated. When the joint risk index exceeds the preset comprehensive risk threshold, the corresponding risk warning information is generated and output.

8. The method for image recognition and anomaly assessment of unmanned aerial vehicle (UAV) inspections according to claim 7, characterized in that: The risk warning information includes at least the location label of the abnormal target, the abnormal target graph structure, the historical abnormal target graph structure, the absolute change in length, the absolute change in width, the degree change of the node, the determination result of the evolution trend, and the corresponding risk level identifier.

Images