Unmanned aerial vehicle adaptive inspection method based on multi-layer representation learning

Abstract

The application discloses a kind of unmanned aerial vehicle self-adapting inspection methods based on multi-layer representation learning, including two stages of automatic rough inspection and self-adapting fine inspection, and the spatial structure of bridge, component characteristics and disease characteristics are learned layer by layer.In the automatic rough inspection stage, the instance attribute of bridge component is quickly learned by combining spatial clustering post-processing method and point cloud semantic segmentation network.Then, the point cloud space is compressed and converted using simulation field model and dimension reduction technology, to guide the unmanned aerial vehicle to carry out efficient and comprehensive spatial inspection in plane geometry.In the self-adapting fine inspection stage, the disease characteristics in the automatic rough inspection stage are learned and extracted as prior information using the hybrid convolution structure Light-PVIT optimized in spatial-channel two dimensions.These information guide the unmanned aerial vehicle to carry out fine detection on the disease area under small field of view.The unmanned aerial vehicle self-adapting inspection method of the application significantly improves the efficiency and accuracy of bridge inspection, and provides reliable technical support for bridge maintenance work.

Description

Technical Field

[0001] The present invention belongs to the field of structural health detection in civil engineering, and specifically relates to an unmanned aerial vehicle (UAV) adaptive inspection method based on multi-layer representation learning. Background Art

[0002] As a critical component of transportation networks, bridges play a vital role in ensuring public safety and economic smooth operation. However, due to multiple factors, including the natural environment, vehicle loads, and construction, bridge structures are susceptible to various defects, such as cracks, spalling, and corrosion. These defects can not only compromise the structural integrity and load-bearing capacity of bridges but also lead to serious accidents. Therefore, regular bridge inspections are crucial to ensuring the stability and safety of bridge structures. Bridge inspections cover multiple aspects, including the deck, superstructure, and substructure. Especially for the superstructure, most of its components are located at high altitudes. Traditional inspection methods rely primarily on technicians using inspection vehicles or other aerial work platforms. However, these traditional methods have numerous limitations: First, they typically require expensive equipment and specialized personnel, resulting in high costs and time-consuming inspections. Second, due to the limitations of the inspection platform, a thorough inspection of the entire bridge is difficult, potentially leading to the omission of important defects or hidden dangers. Furthermore, inspection results are often influenced by the experience and subjective judgment of engineers, lacking objectivity and accuracy. Therefore, the development of efficient and objective new technologies for regular bridge inspections, particularly for applications at high altitudes, has become an urgent challenge for the engineering community.

[0003] To address this challenge, researchers are working to explore and develop advanced intelligent detection equipment, using advanced machine vision and sensor technology to improve the accuracy and efficiency of bridge field detection operations. For the detection of bridge bottom structures, mobile robot detection systems, camera-equipped cable structures, and spherical drones are used to quickly detect diseases in the bridge bottom. For the detection needs of high-altitude suspension structures, innovative climbing robot structures and tank robots equipped with multiple types of sensors are used to collect detailed data on the surface defects of the steel cable. To address the detection challenges of high-pier and high-tower facade structures, new sensors and stereo vision inertial fusion drone systems and ring climbing robots are used to assess the crack situation of high-rise piers. The challenge of bridge data analysis is the massiveness and complexity of the data, which cannot provide effective information for daily detection and maintenance tasks. Manual processing methods not only have low efficiency, but also have differences in analysis results due to individual subjectivity. Most data is saved in image form, and traditional image analysis techniques still rely heavily on manual labor. In response to this challenge, academia has gradually shifted its focus to deep learning technology. Deep learning network structures can automatically learn and extract useful feature information from large-scale image data, enabling automatic identification of bridge data and significantly improving data processing efficiency and accuracy. Currently, disease detection research and practice mainly focus on three core levels of tasks: classification, detection, and segmentation. The goal of the classification task is to assign input images to pre-defined categories, with each category representing a specific disease state. Compared to classification learning, the detection task not only determines the overall category of the image, but also needs to predict the location of the disease in the image by predicting the bounding box. The segmentation task involves classifying each pixel in the image into different categories, accurately determining the outline and location of the disease.

[0004] In the current field of mobile detection platform development, the mainstream method still relies on manual operation. This not only limits the comprehensiveness and efficiency of detection, but also makes it difficult to focus on disease areas. In addition, high-precision visual analysis techniques in data processing tasks, while improving detection accuracy, also limit their widespread application due to high demand for computing resources and strict requirements for hardware equipment.

[0005] Traditional manual flight drone detection methods for bridges have significant limitations and risk factors, mainly due to operational dependence and field of view limitations. These challenges highlight the need for more efficient and safe drone path planning methods.

[0006] In addition, traditional point cloud segmentation methods mainly include attribute method, region segmentation, model fitting, etc. These methods usually rely on predefined rules or models, and are difficult to adapt to complex or irregular point cloud data. For large-scale point cloud data, a long processing time is required. In recent years, in order to improve the recognition effect of the network, many researches focus on designing complex local feature extraction structure and constructing larger model. This method often consumes a lot of resources when processing large scene models. SUMMARY

[0007] In view of the above technical problems, the present application provides a new strategy for unmanned aerial vehicle adaptive patrol based on multi-layer representation learning. The strategy innovatively designs a two-stage patrol process: the first stage is automatic rough patrol based on bridge space understanding. In this stage, the unmanned aerial vehicle is equipped with a large field of view camera to quickly obtain the geometric information of the bridge, and combined with the motion recovery structure algorithm, a three-dimensional model of the bridge is quickly constructed. Further, the attribute information of the bridge component level is analyzed, and a point cloud instance segmentation framework based on an optimization clustering algorithm is proposed to realize the end-to-end comprehensive extraction of the attribute information of each component of the bridge. Through the simulation of the field of view model and the geometric space dimension reduction projection technology, the point cloud space is compressed and converted, and an efficient space patrol path is automatically planned for each independent component of the bridge in the plane geometry, ensuring that the unmanned aerial vehicle can conduct comprehensive coverage patrol on the bridge components. The second stage is adaptive detailed patrol based on disease information feedback. In order to improve the identification speed and accuracy of the disease, the self-attention mechanism is reorganized from the space and channel two key dimensions, and then a lightweight model named Light-PVIT is constructed. Combined with the identified damage area and the established space conversion criteria, the patrol path is adaptively adjusted to guide the unmanned aerial vehicle to detect the damage area in detail with a small field of view.

[0008] To achieve the above purpose, the technical scheme of the present application is as follows:

[0009] The unmanned aerial vehicle adaptive patrol method based on multi-layer representation learning is applied to the detection of bridge bottom structure, comprising the following steps:

[0010] S1, automatic rough patrol based on bridge space understanding;

[0011] S11, using the large field of view camera carried by the unmanned aerial vehicle to quickly obtain the geometric information of the bridge, and combining with the motion recovery structure algorithm, a three-dimensional model of the bridge is quickly constructed;

[0012] S12, analyzing the attribute information of the bridge component level, and proposing a point cloud instance segmentation framework based on an optimization clustering algorithm to realize the end-to-end comprehensive extraction of the attribute information of each component of the bridge;

[0013] S13, the point cloud space is compressed and converted by the simulated field of view model and the geometric space dimension reduction projection method, and an efficient space inspection path is automatically planned for each independent component of the bridge in plane geometry, so as to ensure that the unmanned aerial vehicle can comprehensively cover the bridge component for inspection;

[0014] S2: adaptive detailed inspection based on disease information feedback;

[0015] S21, reorganize the self-attention mechanism from the two key dimensions of space and channel, and construct a lightweight model named Light-PVIT to identify the damage area of the bridge component;

[0016] S22, combining the damage area of the bridge component identified in step S21 and the geometric space dimension reduction projection method, the inspection path planned in step S13 is adaptively adjusted to guide the unmanned aerial vehicle to detect the damage area in detail.

[0017] Step S11 specifically includes the following sub-steps:

[0018] S111, the satellite map and the prior information of the bridge are used to plan the path of the unmanned aerial vehicle;

[0019] S112, the multi-view three-dimensional reconstruction data of the bridge is collected, and then the three-dimensional model of the whole scene is constructed.

[0020] Step S12 specifically includes the following sub-steps:

[0021] S121, the PointNeXt network is used as a tool to learn the representation of bridge component information, and the design concept of encoder and decoder is used to process point cloud data and finely segment the structure; in this process, the encoder captures the high-level semantic features in the point cloud, and the decoder remaps these high-level semantic features to construct the fine segmentation structure of the point cloud;

[0022] S122, further intra-class division is performed based on semantic segmentation, and a strategy combining the density-based spatial clustering post-processing method and the semantic segmentation network is used until all points in the point cloud model are accessed.

[0023] Step S13 specifically includes the following sub-steps:

[0024] S131, according to the simulated field of view model, a rough inspection route is planned for each component in the bridge point cloud space, the bridge component form belongs to a regular spatial geometric body, and the compression and transformation of the point cloud space are performed by combining these characteristics and using the dimension reduction idea, and the spatial relationship is described in plane geometry;

[0025] S132, three non-collinear points P 1,P1, P2, P3, constitute the initial point set, as the basis for subsequent model fitting; calculate the normal vector of the initial point set and construct an initial spatial plane reference model accordingly; traverse each remaining point in the component point cloud data to perform distance measurement operations, and any point with a distance from the plane model below the tolerance threshold is identified as an inner point t;

[0026] S133, repeat the above iteration process, select the model with the largest consensus set as the representative spatial plane, and compress and project the remaining point cloud (X, Y, Z) in the bridge component space onto the representative spatial plane (X pj , Y pj , Z pj ); according to the distance coefficient D, construct an inspection plane parallel to the representative spatial plane, reduce the spatial dimension needed to consider when designing the flight path of the unmanned aerial vehicle; further, convert and project the component point cloud on the representative spatial plane onto the inspection plane, lock the camera azimuth angle, that is, the camera acquisition plane is parallel to the bridge plane to be detected, and the two key factors of inspection planning are unified to the inspection plane, and the inspection plane in three-dimensional space is converted into a two-dimensional plane model;

[0027] S134, based on the regularity of the bridge component, take the center line as the reference of the unmanned aerial vehicle inspection path on the two-dimensional inspection plane, quickly extract the bridge component center line through the boundary fitting method; the boundary fitting direction follows the aspect ratio criterion, that is, the numerical comparison of the bridge component width and the bridge component height is performed, when the bridge component width > bridge component height, the bridge component width direction is taken; when the bridge component height > bridge component width, the bridge component height direction is taken; the boundary is fitted using a polynomial;

[0028] S135, combine the field of view model to plan the camera acquisition points on the bridge component center line, consider the combination of different distance coefficients D and focal lengths f to ensure that the coverage range is greater than the boundary distance; in the rough inspection stage, increase the distance coefficient D and reduce the focal length f to increase the acquisition field of view range, and then reduce the overlap rate δ to cover the bridge component in the optimal number of acquisition points;

[0029] S136, restore the acquisition point coordinates on the two-dimensional plane to three-dimensional coordinates through reverse mapping to obtain the complete inspection path.

[0030] Step S21 specifically includes the following sub-steps:

[0031] S211, the Light-PVIT lightweight model is composed of six stages of learning tasks, the first five stages of learning tasks are composed of a convolution module CVB, a deep separation convolution module with a reverse residual structure (RDB2, RDB1), and a partial visual transformer module PVIT, and the deep separation convolution module with a reverse residual structure is used as an efficient replacement for the convolution layer;

[0032] The first step of each learning stage is to perform a down-sampling operation on the features, and a reverse deep separation convolution module without a residual structure is used to down-sample the features;

[0033] The partial visual transformer module PVIT includes three sub-modules, namely a local representation sub-module, a global representation sub-module, and a feature fusion sub-module, wherein the local representation sub-module is used to model the local information of the input A depth separation convolution with a convolution kernel size of 3x3 and a mapping convolution with a convolution kernel size of 1x1 are used to model the local information of the features, and the output features Wherein,

[0034] H, W, C s represent the height, width, and channel number of the output feature map, respectively;

[0035] The global representation sub-module introduces the self-attention mechanism of the transformer to capture the global context information of the feature map, to obtain the global information of the HxW feature map;

[0036] S213, a partial splitting operation is introduced in the channel dimension, specifically, by splitting coefficient n, the spatial compression feature X s is divided into global modeling features and local modeling features In the early stage of the model, the redundancy of high-resolution features is managed, the splitting coefficient n stage-3 of stage three is set to 4; the splitting coefficient n stage-4 of stage four and the splitting coefficient n stage-5 of stage five are set to 2 respectively, to avoid information loss caused by excessive compression;

[0037] S214, the global modeling features X st are fed into the transformer module to capture long-distance dependencies in the input features, and the local modeling features X sc are merged with the global modeling features X st in the channel direction at the output stage of the transformer module, while restoring the feature shape, and outputting the mixed modeling features

[0038] The feature fusion sub-module is for the hybrid modeling feature Integration of global information and local information is performed;

[0039] S215, the last stage of the model adopts a convolution sub-module, a pooling flattening sub-module, and a full connection sub-module to convert the features of stage six into a final prediction result.

[0040] Step S22 specifically includes the following sub-steps:

[0041] S221, automatically screening the data collected in the rough screening stage by using a disease detection network, and independently classifying the image data containing diseases to form a disease set that is different from the normal area image data;

[0042] Before the disease screening stage, the collection point sequence based on the rough screening route Assign a corresponding label x to each collected image i After the screening stage, the real spatial position of the disease collection point can be quickly indexed according to the disease image label;

[0043] An adaptive inspection method based on rough screening information feedback is proposed, which can make the unmanned aerial vehicle efficiently inspect while keeping attention on the disease area. First, combining the information fed back in the disease screening stage, the collection points of the unmanned aerial vehicle are divided into two categories: normal and abnormal,

[0044] At the normal collection point, that is, the area where the bridge component has no damage, the unmanned aerial vehicle takes the rough inspection path as the inspection reference;

[0045] At the abnormal collection point, that is, the area where the bridge component has damage, the unmanned aerial vehicle adaptively adjusts the detection path according to the disease feedback information, so as to obtain more accurate detailed information;

[0046] Reduce the size of the unmanned aerial vehicle collection field of view, record the local details of the disease; by reducing the distance coefficient D or increasing the focal length f, the collection field of view can be reduced. By keeping the camera focal length unchanged, a scheme of constructing a fine detection plane D s in the inspection space is adopted to realize flexible adjustment of the unmanned aerial vehicle fine collection field of view (W s ,H s ), where W s ,H s represent the height and width of the collection field of view;

[0047] S222, further, project the disease area as an optimization target onto the newly constructed fine detection plane, convert the detection plane in the three-dimensional space to a two-dimensional plane, and perform mesh cutting on the disease area corresponding to a single collection point, where the width height S w ,S h respectively represent the division coefficient in width and height, and the center coordinates of these subgrids represent the collection points on the inspection path;

[0048] S223, traverse all disease areas using the same operation as step S222 to develop a fine collection plan;

[0049] The collection points exceeding the limit threshold delta are filtered and deleted using the overlapping area measurement criterion;

[0050] A heuristic search algorithm is used to connect the discrete collection points, and a distance cost minimization between adjacent points is used to construct the inspection path;

[0051] S224, finally, the spatial conversion relationship between the collection points and the inspection target points is used, that is, the collection points are projected onto the bridge surface through the distance coefficient D to locate the disease area in the real space.

[0052] The beneficial effects of the present application are:

[0053] 1. The present application uses satellite maps and prior information of bridges to plan the UAV path. This method not only significantly improves the efficiency of UAV flight, ensures that all necessary detection areas are covered in the shortest time, but also greatly improves the safety of the entire flight process.

[0054] 2. The present application uses PointNeXt network as a tool to learn the representation of bridge component information, which achieves an excellent balance between model accuracy and size. Under a certain category attribute, the bridge may contain multiple components, and the differences between these components cannot be identified and detected by the UAV. In this process, a density-based spatial clustering post-processing method combined with a semantic segmentation network is used. Through this method, instance-level division can be achieved while maintaining simplicity.

[0055] 3. Convolutional Neural Networks (CNN) have long been the mainstream choice for image processing and computer vision tasks, and they are good at capturing local features of images, but the lack of global context understanding restricts the performance of CNN network structure. In recent years, the rising Transformer architecture has shown excellent performance in handling global information. However, they usually require more computing resources, which becomes impractical on resource-limited mobile devices. Therefore, the present application combines the efficiency of CNN with the excellent performance of Transformer, and proposes Light-PVIT as an efficient disease detection tool based on the Mobile-ViT structure as the network baseline.

[0056] 4.The unmanned aerial vehicle inspection strategy of the present application deeply integrates the patrol activities of unmanned aerial vehicles with data analysis, and builds an efficient interactive closed-loop system. The system uses a multi-layer representation learning method to comprehensively learn and analyze the spatial structure, component attributes and disease characteristics of the bridge, ensuring the optimization of the whole chain from data collection to processing. Through this strategy, the large amount of data collected during the patrol process is not only used for immediate analysis, but the conclusions drawn from the analysis can be quickly fed back to the patrol task, thereby realizing the continuous accumulation and self-improvement of the bridge detection experience. In addition, the present application further explores the lightweight problem of internal analysis methods, and through comprehensive lightweight processing from three-dimensional space to two-dimensional plane, significantly reduces the computational burden and hardware requirements of the algorithm, making the patrol system more efficient and practical.

[0057] In summary, the unmanned aerial vehicle inspection strategy proposed by the present application not only has innovation in theory, but also can effectively improve the efficiency and accuracy of bridge detection in practical application. Through the combination of multi-layer representation learning and lightweight analysis method, the present application method provides a new technical path for the field of bridge detection, which is helpful to promote the further development and application of related technologies. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The overall framework diagram proposed by the present application;

[0059] Figure 2 The bridge component semantic learning model proposed by the present application;

[0060] Figure 3 The bridge component instance segmentation diagram based on spatial distribution characteristics proposed by the present application;

[0061] Figure 4 The camera projection geometry proposed by the present application;

[0062] Figure 5 The path planning method based on spatial dimension reduction proposed by the present application;

[0063] Figure 6 The lightweight disease identification model structure diagram proposed by the present application;

[0064] Figure 7 The spatial channel double-dimension optimization module detail diagram proposed by the present application;

[0065] Figure 8 The adaptive patrol method based on disease information feedback proposed by the present application;

[0066] Figure 9 The bridge structure diagram proposed by the example;

[0067] Figure 10 The implementation process of the unmanned aerial vehicle rough inspection proposed by the example;

[0068] Figure 11 Example proposed coarse detection stage detection result map;

[0069] Figure 12 Example proposed adaptive coarse-fine detection two-stage result schematic diagram of bridge arch member.

[0070] Figure 13 Example proposed adaptive coarse-fine detection two-stage result schematic diagram of bridge arch member.

[0071] Figure 14 Example proposed adaptive coarse-fine detection two-stage result schematic diagram of bridge arch member. DETAILED DESCRIPTION

[0072] The present application will be further clarified by the following examples and specific embodiments, which should not be taken as limiting the scope of the present application. After reading the present application, those skilled in the art will be able to modify the general principles described above to suit particular situations, all of which fall within the scope of the application as defined by the appended claims.

[0073] An unmanned aerial vehicle adaptive inspection method based on multi-layer representation learning, the overall flowchart of the solution is shown in Figure 1 The solution includes the following components:

[0074] S1: Automatic rough inspection based on bridge spatial understanding. This stage uses the large field of view camera carried by the unmanned aerial vehicle to quickly obtain the geometric information of the bridge, and quickly constructs a three-dimensional model of the bridge. Further, the attribute information of the bridge member level is analyzed, and an efficient spatial inspection path is automatically planned for each independent member of the bridge in the plane geometry, ensuring that the unmanned aerial vehicle can conduct comprehensive coverage inspection of the bridge member.

[0075] S2: Adaptive detailed inspection based on disease information feedback. Combined with the identified damage area and the established spatial conversion criteria, the inspection path is adaptively adjusted to guide the unmanned aerial vehicle to detect the damage area in detail with a small field of view.

[0076] Further, the automatic rough inspection based on bridge spatial understanding in step S1 includes the following steps:

[0077] Step S11: Three-dimensional model reconstruction of the bridge: quickly obtain the geometric information of the whole bridge;

[0078] Step S12: Point cloud instance segmentation framework based on optimization clustering algorithm: in-depth analysis of the attribute information of the bridge member level, realizing end-to-end comprehensive extraction of the attribute information of each member of the bridge.

[0079] Step S13: Geometric space dimensionality reduction projection technology based on simulated field of view model: automatic planning of efficient spatial inspection path of each independent component of the bridge in planar geometry.

[0080] Further, S11: the specific method of three-dimensional model reconstruction of the bridge is as follows:

[0081] Satellite maps and prior information of the bridge are used to plan the UAV path. First, the satellite map provides the UAV with macro geographical information, so that the UAV can identify the key features of the bridge and its surrounding environment before flying, including the precise spatial positioning of the bridge, adjacent buildings and other possible flight interference factors. Based on this information, a preliminary flight path can be designed to avoid potential dangerous areas (such as high-voltage towers and other obstacles). This method not only significantly improves the efficiency of UAV flight, ensuring that all necessary detection areas are covered in the shortest time, but also greatly improves the safety of the entire flight process. According to the planned UAV route, the bridge is comprehensively reconstructed with multi-view three-dimensional data collection. Feature extraction is performed on each collected image, usually using SIFT, SURF, and other feature point algorithms. Then, by matching these feature points between each pair of images, the corresponding relationship between the images is established. Finally, using these corresponding relationships, the position of each feature point in the three-dimensional coordinate system is calculated, and the three-dimensional model of the entire scene is constructed.

[0082] Further, S12: the specific method of point cloud instance segmentation framework based on optimization clustering algorithm is as follows:

[0083] PointNeXt network is used as a tool to learn the representation of bridge component information, which achieves an excellent balance between model accuracy and size. Inspired by the U-Net architecture, the design concept of encoder and decoder is adopted to process point cloud data and achieve fine structure segmentation. In this process, the role of the encoder is to capture high-level semantic features in the point cloud, while the decoder is responsible for remapping these features to construct a detailed segmentation structure of the point cloud, such as Figure 2N denotes the number of point clouds output per layer, and x represents the feature dimension of the point cloud. The encoder is composed of multiple layers of perceptron (MLP), point set abstraction (SA) layer, and inverse residual multi-layer perception layer (IRM). The point set abstraction module is the core of the encoder, which includes three key components: the farthest point sampling layer, the spherical grouping layer, and the feature extraction layer. The sampling layer uses the farthest point algorithm to select 1 / 4 of the number of center points of the point cloud, and by optimizing the topological structure of the center points, it ensures that the sampling covers the entire point cloud. The spherical grouping layer creates a spherical neighborhood with a radius R around each center point, thereby sampling K feature points within each local region. The feature extraction layer extracts the features of the local region through the max-pooling operation, although this operation may cause partial loss of feature information, but by using MLP to expand the feature dimension before pooling, the features can be enriched and compensated, reducing the impact of information loss. The inverse residual multi-layer perception layer is a feature enhancement structure built on the point set abstraction layer. It effectively alleviates the problem of gradient disappearance by introducing a residual connection between the input and the output. Compared with the original structure, this design cancels the sampling layer and adds two MLP layers to enhance the feature extraction capability. At the same time, the grouping layer now uses a larger radius R spherical neighborhood, which expands the receptive field of the model, thereby establishing more robust local feature relationships. The stacked MLP layers can effectively enhance the feature extraction capability of the model, but at the same time, they will significantly increase the computational load. To optimize this point, the model uses a separable MLP structure to perform the feature extraction task step by step, with the MLP layer between the grouping layer and the pooling layer focusing on extracting features between each neighborhood, and the subsequent MLP layer focusing on extracting features of individual points. This hierarchical and fine-grained processing strategy not only enhances the feature extraction capability of the model, but also maintains the computational efficiency.

[0084] The decoder adopts a strategy based on distance interpolation and skip connection, using several feature propagation modules (FP) to gradually restore and map the compressed features. Each FP module mainly includes a distance interpolation layer and a multi-layer perception layer. In order to compensate for the loss of detail information that occurs during downsampling, the decoder fuses the downsampled features of each stage during feature propagation. These features are introduced into the corresponding FP module and integrated through the MLP layer, thereby producing a more refined feature representation.

[0085] Further intra-class division is made on the basis of semantic segmentation. A spatial clustering post-processing method based on density is combined with the semantic segmentation network, such as Figure 3The point cloud region condensation processing is first introduced. Specifically, a three-dimensional grid is constructed in the point cloud space according to a set three-dimensional pixel size S_p, and each pixel appears in the form of a cube. The point cloud data is mapped into the created grid space, and the geometric centroid of each sub-point cloud space in the cube is selected as a representative point. Then, a statistical filter is used as a supplementary step to filter out abnormal points by using the distance statistical characteristics of each point in the point cloud data and its neighboring point set. For each point in the point cloud, the N_k nearest points are selected as the neighboring point set. Then, for each point, the distance between it and its neighboring points is calculated and the global data, including the mean and standard deviation, is calculated. Based on these statistical data, the global average distance plus the standard deviation multiplied by a multiple is used as the screening threshold T_k. If the average distance of a point from its neighbors is greater than this threshold, it is considered an abnormal point and is removed from the point cloud. Finally, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm based on the local point density distribution characteristics is used. The spatial search range ε of each point cloud element and the minimum number of point clouds required for each dense region n are determined. Based on the spatial search range ε, the neighborhood of each point cloud is searched, and if it contains at least the number of points n required for a dense region, the point is defined as a core point P_ci. All points in the point cloud space that meet the definition of core points are the key to cluster formation, and all core points between adjacent core points whose distance is within the neighborhood range are connected to form a cluster through iteration. When the current clustering is completed, the next new unvisited point cloud is retrieved, and the above steps are repeated until all points in the point cloud model are visited.

[0086] Further, step S13: the specific method of the geometric space dimension reduction projection technology based on the simulated field of view model is as follows:

[0087] For different inspection tasks, it is crucial to select the optimal camera field of view. The camera field of view, i.e., the spatial range that can be covered by the camera at a certain distance and focal length, is determined by multiple factors, including the sensor size, focal length, and acquisition distance of the camera. The sensor size H s ,W s and focal length f of the camera determine the width and narrowness of the camera field of view. The acquisition distance D directly affects the coverage area of the field of view. Under the premise that these parameters are known, the actual acquisition field of view of the camera in the real space can be simulated, as shown in Figure 4 The formula for calculating the field of view is as follows:

[0088]

[0089] To ensure that the UAV inspection path can completely cover the target area, it is necessary to ensure that the images between adjacent collection points have sufficient overlap area W o ×H o . Appropriate overlap provides a buffer for environmental changes or minor errors in flight, thereby ensuring the quality and integrity of the UAV inspection. Therefore, reasonable setting of the overlap rate between adjacent collection points is the key to efficient and comprehensive inspection.

[0090] According to the simulation field of view model, a rough inspection route is planned for each component in the bridge point cloud space. The form of the bridge component basically belongs to regular spatial geometric bodies. Combined with these characteristics, the dimension reduction idea is used to compress and transform the point cloud space, so as to accurately depict the spatial relationship in the plane geometry, as shown in Figure 5 .

[0091] For different bridge components, a representative spatial plane with morphological characteristics needs to be found. Here, the idea of consistency based on random samples is adopted, aiming to accurately depict the main geometric morphological characteristics of the component. Preliminary involves randomly selecting three non-collinear points P1, P2, P3 from the point cloud data set to form an initial point set as the basis for subsequent model fitting. The normal vector of the initial point set is calculated using mathematical methods and an initial spatial plane reference model is constructed accordingly. Distance measurement operations are performed on each remaining point in the component point cloud data. Any point with a distance from the plane model below the tolerance threshold is identified as an inner point and included in the Consensus Set. The above iterative process is repeated, and the model with the largest consensus set is selected as the representative spatial plane. The remaining point cloud (X, Y, Z) in the component space is compressed and projected onto the representative spatial plane (X pj , Y pj , Z pj ), realizing the transformation of the point cloud space to the spatial plane. The projection formula is as follows:

[0092]

[0093] Here a, b, c, d are the model coefficients of the representative spatial plane. The plane is also considered as the inspection object. According to the distance coefficient D, an inspection plane parallel to the representative plane is constructed, reducing the spatial dimension that needs to be considered when designing the UAV flight path. Further, according to the above formula, the component point cloud on the representative spatial plane is transformed and projected onto the inspection plane, locking the camera azimuth angle, i.e. the camera collection plane is parallel to the inspection object plane. The two key factors of inspection planning are unified on the inspection plane, transforming the inspection plane in three-dimensional space into a two-dimensional plane model.

[0094] Based on the regularity of bridge components, the center line is used as the reference for the UAV inspection path on the two-dimensional inspection plane. The bridge component center line is quickly extracted by the boundary fitting method. The boundary fitting direction needs to follow the aspect ratio criterion, that is, the polynomial fitting boundary is used in the direction with a larger scale coefficient. Then, the camera collection point planning is performed on the bridge component center line combined with the field of view model, considering the combination of different distance coefficients D and focal lengths f, and the appropriate coarse inspection collection field of view (W p ,H p ) is calculated to ensure that the coverage range is greater than the boundary distance. At the same time, the appropriate field of view overlap rate is an essential condition to ensure the comprehensive inspection of the component outer surface. The coarse inspection stage mainly aims at efficiency, and through the combination of large distance coefficient D and small focal length f, the collection field of view range is increased. Then, the overlap rate δ is reduced to cover the component in the optimal number of collection points. Finally, the collection point coordinates on the two-dimensional plane are restored to three-dimensional coordinates through reverse mapping to obtain the complete inspection path.

[0095] Further, the adaptive detailed inspection based on disease information feedback in step S2 includes the following steps: step S21: Light-PVIT lightweight model: effectively capturing and integrating local features and global context information; step S22: fine inspection method based on disease information feedback: adaptively adjusting the inspection path to guide the UAV to detect the damage area in detail with a small field of view.

[0096] Further, the specific method of step S21: Light-PVIT lightweight model is as follows:

[0097] The efficiency of CNN and the excellent performance of Transformer are combined, and Mobile-ViT structure is used as the network baseline to propose Light-PVIT as an efficient disease detection tool. Light-PVIT consists of six stages of learning tasks to extract deep features of images in a multi-scale architecture, as shown in Figure 6 The first five stages of learning tasks are mainly composed of convolution modules (CVB), deep separation convolution modules with reverse residual structure (RDB2, RDB1), and partial visual transformer modules (PVIT). In order to optimize the subsequent learning tasks, the deep separation convolution module with reverse residual structure is used as an efficient replacement for the convolution layer. The addition of the residual structure can efficiently alleviate the gradient vanishing problem, thereby accelerating the convergence of the network.

[0098] In the depthwise separable convolution module of the residual structure, first, 1x1 mapping convolution is used for feature dimension reduction, then 3x3 depthwise separable convolution is used for efficient feature extraction, and finally 1x1 mapping convolution is used for feature dimension increase. The feature dimension adjustment operation of the reverse residual structure is opposite, first, 1x1 mapping convolution is used for feature dimension increase, and finally 1x1 mapping convolution is used for feature dimension reduction. The first step of each learning stage is to perform down-sampling operation on the features. The reverse depthwise separable convolution module without residual structure is used to down-sample the features.

[0099] A more efficient visual transformer module PVIT is invented, which can capture and fuse local and global information of feature maps, and at the same time will not produce too many parameters, as shown in Figure 7 The module mainly includes three sub-modules, local representation sub-module, global representation sub-module and feature fusion sub-module. The local representation sub-module models the local information of the input A depthwise separable convolution with a convolution kernel size of 3x3 and a mapping convolution with a convolution kernel size of 1x1 are used to model the local information of the features, and the output feature The depthwise separable convolution with a convolution kernel size of 3x3 can encode the local spatial information of the feature map in a lightweight manner, and then the mapping convolution with a convolution kernel size of 1x1 is used to learn the linear combination information between the input feature channels. In order to obtain the global information of the feature map with an effective receptive field of HxW, the global representation sub-module introduces the self-attention mechanism of the transformer to capture the global context information of the feature map. The calculation process of the self-attention mechanism can be represented by the following formula:

[0100]

[0101] where ⊕ represents the Concat operation, Q, K, and V represent the weight matrices of the Q, K, and V branches in the i-th linear layer, respectively. σ represents the softmax function, and <·,·> represents the dot product operation.

[0102] In order to solve the challenges of the traditional self-attention mechanism, a fast self-attention mechanism is reorganized from the two dimensions of space and channel. In the global representation submodule, the input feature map has established information dependencies between pixels through local operations, that is, each pixel has established connections with adjacent pixels. The idea of ​​hole convolution is introduced into the transformer module to hollow out the calculation mode between token vectors. Specifically, each token only performs self-attention calculations on the token at the corresponding hole distance position in the feature map. First, according to the pre-set window patch size and To X f Perform a flattening operation to form Here Indicates the number of blocks. This means that a block of size Instead of treating the patch as a single feature unit, it is divided into N independent tokens, each token corresponding to a pixel sub-region in the patch. Usually, the patch used in the network is square, that is, Furthermore, each token will only be spatially distant from it. The adjacent tokens are used to construct global information, which effectively improves the computational efficiency of self-attention.

[0103] A similar processing strategy is adopted in the spatial dimension, and a partial split operation is introduced in the channel dimension to optimize the computing resource usage of the model while maintaining the model performance. Specifically, the feature X after spatial compression is split by the split coefficient n. s Divided into global modeling features and local modeling features To effectively manage the redundancy of high-resolution features in the early stages of the model and improve the computational efficiency of the model, the splitting coefficient n in stage 3 is stage-3 Set it to 4. Appropriately reduce the value of the split coefficient and set n stage-4 ,n stage-5 Set to 2 to avoid excessive compression and loss of information. st Feed it into the transformer module and use the above Att formula to capture the long-range dependencies in the input features and enhance the global representation ability of the feature representation. Local modeling feature X sc Will be combined with X at the output stage of the transformer module st Merge in the channel direction and restore the characteristic shape at the same time, output

[0104] The fusion submodule targets the input A mapping convolution with a convolution kernel size of 1×1 and a depth separation convolution with a convolution kernel size of 3×3 are used to fuse the global and local information of the features. The output features The primary function of the 1×1 mapping convolution is to recombine and map feature channels while preserving the spatial dimension. Next, a 3×3 depthwise separable convolution is employed to efficiently fuse spatial information. By combining these two steps, the fusion submodule effectively integrates global and local information. The output feature representation not only contains rich global context but also captures key details of the local region.

[0105] The final stage of the model is crucial for converting the learned features into final predictions. This stage consists of three components: a convolutional submodule, a pooling and flattening submodule, and a fully connected submodule. These three submodules work together to ensure that the model can effectively extract useful information from the learned high-dimensional features.

[0106] Furthermore, step S22: the specific method of the fine inspection method based on disease information feedback is as follows:

[0107] The disease detection network is used to automatically perform high-precision disease screening on the data collected in the rough inspection stage, and the image data containing diseases are independently summarized to form a disease set that is different from the image data of normal areas, such as Figure 8 As shown. Analyzing the images at this stage can achieve the effect of disease identification, that is, understanding the existence of bridge diseases. In order to build a connection between the image and the real space, before the disease screening stage, the sequence of acquisition points based on the rough inspection route is Assign a label x to each collected image i After the screening stage, the real spatial location of the disease collection point can be quickly indexed based on the disease image label.

[0108] An adaptive inspection method based on rough inspection information feedback is proposed, which can maintain high-precision attention to the diseased area while the UAV performs efficient inspections. Figure 8 As shown. First, based on the information fed back during the disease screening phase, the drone's collection points are divided into two categories: normal and abnormal. At normal collection points, the drone uses the rough inspection path as the inspection benchmark. However, at abnormal collection points, the drone will adaptively adjust the inspection path based on the disease feedback information to obtain more accurate detail information. Reduce the size of the drone's acquisition field of view so that the image focuses on the recording task of local details of the disease. The size of the drone's field of view is mainly determined by two factors: the distance coefficient D and the focal length f. By reducing D or increasing f, the acquisition field of view can be reduced. Keep the camera focal length unchanged and construct a distance target plane D in the inspection space. sThe solution of fine detection plane realizes the fine acquisition of field of view (W s ,H s ) is flexibly adjusted. Further, the diseased area is projected onto the newly constructed fine detection plane as the optimization target. Here, the concept of dimensionality reduction analysis is adhered to and the detection plane in the three-dimensional space is transformed into a two-dimensional plane. The diseased area corresponding to a single collection point is cut into a grid, where the size of each grid is S w ,S h The center coordinates of these subgrids represent the sampling points on the detailed inspection path. The same operation as above is used to traverse all diseased areas, thereby developing a more refined sampling plan. The overlapping area metric is used to automatically filter and delete sampling points that exceed a limit threshold δ, thereby reducing resource waste and improving inspection efficiency. An efficient heuristic search algorithm is used to solve the connection problem between discrete sampling points, constructing a detailed inspection path by minimizing the distance cost between adjacent points.

[0109] The inspection method based on detailed inspection paths acquires a series of detailed local image data. This data is then screened for defects using a defect detection network, and the actual acquisition location is quickly located based on the label index. Finally, the spatial transformation relationship between the acquisition point and the inspection target is utilized. This distance coefficient, D, is used to reversely map the acquisition point onto a representative spatial plane of the target, enabling precise localization of the defect area in real space.

[0110] Example 1

[0111] Fangshan Bridge was used as a test object, such as Figure 9 As shown. Fangshan Bridge is a bottom-supported prestressed concrete tie-arch bridge with a main span of 96.76m and a deck width of 7m. The main arch ribs are made of reinforced concrete structure, and the arch axis is a quadratic parabola with a calculated span of 95m and a rise of 14m. When the drone performs adaptive inspection tasks, it gets rid of the dependence on direct human control and realizes autonomous flight. In order to take into account both safety and convenience, the compact DJI M3T industrial drone was specially selected as the equipment carrier. The sensor size of its camera is 1 / 2" (6.4×4.8mm), with an equivalent focal length of 24mm. The actual focal length obtained by conversion is 4.47mm. When the drone is located at a distance of 14m from the inspection surface of the bridge, 5 times the original focal length is used for data collection, and the actual field of view size is 4×3m (with an accuracy of 1mm / pixel). The drone in this state is used as the flight criterion for the rough inspection stage of the inspection method, that is, the flight distance D p =14m, camera focal length f p =22.35mm, collecting field width and height W p ×Hp = 4009 x 3007 mm ≈ 4 x 3 m. This stage focuses on the overall information of the bridge. With the UAV at a distance of 5 m from the target, using the same focal length, the real field of view obtained is 1.4 x 1 m (with a precision of 0.35 mm / pixel). The UAV in this state is used as the flight guideline for the detailed inspection stage of the inspection method, i.e., the distance from the target plane D s = 5 m, f s = 22.35 mm, W s x H s = 1432 x 1074 mm ≈ 1.4 x 1 m.

[0112] The UAV is used to collect multi-angle large-scene views of the experimental bridge at a long distance, and then a three-dimensional reconstruction task is performed through a motion recovery structure algorithm for multi-view geometry. A preliminary exploration of the bridge component attributes is performed through the PointNeXt network, and the instance segmentation results are shown in Figure 10 The inspection target includes cross braces, arch structures, suspension rods, and beams. According to the distance coefficient D p of the rough inspection stage, the inspection plane of each component is constructed. According to the fine component attribute information, customized inspection planning is performed for each component, as shown in Figure 10

[0113] The data collected in the rough inspection stage is input into the Light-PVIT model for rapid disease screening, as shown in Figure 11

[0114] In order to provide effective bridge operation suggestions, the real spatial position of the disease collection point is quickly inferred according to the association between the disease collection image and the actual space, as shown in Figure 12 , 13 , 14 The rough inspection plane of the bridge arch, beam, and cross brace components is shown in the figure. Dark collection points indicate that there is disease in the area covered at this position, while light collection points indicate that there is no damage. In the disease area, the UAV will adaptively adjust the detection path according to the disease information feedback in the rough inspection stage to obtain more accurate detailed information

[0115] The fine detection plane of the distance target plane D s is constructed in the inspection space, and the distance from the rough inspection plane to the fine inspection plane is D ps = D p - D s , and the collection field of view of the UAV is adjusted from W p x H p to W s x H s ​​All disease area ranges are meshed, with each mesh having a size of w=1 mm and h=0.75 mm. Meanwhile, the center points of the sub-meshes are calculated as the collection points for fine inspection. Collection points with an overlap rate threshold exceeding 30% are automatically filtered and deleted.

[0116] In summary, the specific embodiments verify the effectiveness and applicability of the scheme of the present application to complex projects.

[0117] The above disclosure is only a typical embodiment of the present application, but the embodiments of the present application are not limited thereto, and any modification of the present application made by those skilled in the art after reading the patent should fall within the protection scope of the present application.

Claims

1. A method for adaptive inspection of UAV based on multi-layer representation learning, characterized in that, Comprise the following steps: S1, automatic rough patrol based on bridge space understanding; S11, quickly obtain the geometric information of the bridge by using the large field of view camera carried by the unmanned aerial vehicle, and combine the motion recovery structure algorithm to quickly build a three-dimensional model of the bridge; S12, analyze the attribute information of the bridge component level, propose a point cloud instance segmentation framework based on an optimization clustering algorithm, and realize end-to-end comprehensive extraction of the attribute information of each component of the bridge; S13, compress and convert the point cloud space through the simulation field of view model and the geometric space dimension reduction projection method, and automatically plan an efficient space patrol path for each independent component of the bridge in the plane geometry, ensuring that the unmanned aerial vehicle can conduct comprehensive coverage patrol on the bridge components; S2: adaptive detailed patrol based on disease information feedback; S21, reorganize the self-attention mechanism from two key dimensions of space and channel, and build a lightweight model named Light-PVIT to identify the damage area of the bridge component; S22, combine the damage area of the bridge component identified in step S21 with the geometric space dimension reduction projection method to adaptively adjust the patrol path planned in step S13, and guide the unmanned aerial vehicle to detect the damage area in detail; Step S12 specifically comprises the following sub-steps: S121, use the PointNeXt network as a tool to learn the bridge component information representation, and use the encoder and decoder design concept to process point cloud data and finely segment the structure; In this process, the encoder captures high-level semantic features in the point cloud, and the decoder remaps these high-level semantic features to build a detailed segmentation structure of the point cloud; S122, further intra-class division is performed based on semantic segmentation, and a density-based spatial clustering post-processing method is combined with the semantic segmentation network until all points in the point cloud model are accessed; Step S21 specifically comprises the following sub-steps: S211, the Light-PVIT lightweight model is composed of six learning tasks, the first five learning tasks are composed of a convolution module CVB, a deep separation convolution module with a reverse residual structure (RDB2, RDB1), and a partial visual transformer module PVIT, and a deep separation convolution module with a reverse residual structure is used as an efficient replacement for the convolution layer; The first step of each learning stage is to perform down-sampling operation on the features, and a reverse deep separation convolution module without residual structure is used to down-sample the features; The partial visual transformer module PVIT includes three sub-modules, namely a local representation sub-module, a global representation sub-module and a feature fusion sub-module, wherein the local representation sub-module is used for input The local information of the feature is modeled by using a depth separation convolution with a convolution kernel size of 3*3 and a mapping convolution with a convolution kernel size of 1*1, and the output feature is wherein, H, W, C s respectively represent the height, width and channel number of the output feature map; The global feature sub-module introduces the self-attention mechanism of the transformer to capture the global context information of the feature map, so as to obtain the global information of the HxW feature map; S213, introducing a partial split operation in the channel dimension, specifically, by a split factor n to compress the spatial feature X s into global modeling features and local modeling features Managing the redundancy of high resolution features at the early stage of the model, setting the split factor n stage-3 of stage three to 4; setting the split factor n stage-4 of stage four and the split factor n stage-5 of stage five to 2 respectively, to avoid the loss of information caused by over-compression; S214, the global modeling feature X st The feed-in transformer module captures long-distance dependencies in the input feature, and the local modeling feature X sc The global modeling feature X st Merge in the channel direction while restoring the feature shape, output the mixed modeling feature The feature fusion sub-module is directed to the hybrid modeling feature Integration of global information and local information is performed; S215, the last stage of the model uses a convolution submodule, a pooling and flattening submodule, and a fully connected submodule to convert the features of stage six into the final prediction result. 2.The multi-layer representation learning based adaptive UAV inspection method of claim 1, wherein: Step S11 specifically comprises the following sub-steps: S111, plan the unmanned aerial vehicle path using satellite maps and prior information of the bridge; S112, collect multi-view three-dimensional reconstruction data of the bridge, and then build a three-dimensional model of the entire scene. 3.The multi-layer representation learning based adaptive UAV inspection method of claim 1, wherein: Step S13 specifically comprises the following sub-steps: S131, according to the simulated visual field model, rough route planning is performed for each component in the bridge point cloud space, the bridge component form belongs to a regular spatial geometric body, combined with these characteristics, the point cloud space is compressed and transformed using the dimension reduction idea, and the spatial relationship is described in the plane geometry; S132, randomly extract three non-collinear points P1, P2, P3 from the point cloud data set to form an initial point set as the basis for subsequent model fitting; calculate the normal vector of the initial point set And an initial space plane reference model is constructed accordingly; distance measurement operation is performed on each remaining point in the component point cloud data, and any point with a distance from the plane model lower than the tolerance threshold is identified as an inner point t; S133, repeat the above iteration process, select the model with the largest consensus set as the representative space plane, compress and project the remaining point cloud (X, Y, Z) in the bridge component space onto the representative space plane (X pj ,Y pj ,Z pj ); according to the distance coefficient D, a parallel inspection plane is constructed, which reduces the spatial dimension needed to be considered in the design of the flight path of the unmanned aerial vehicle; further, the component point cloud on the representative space plane is converted and projected onto the inspection plane, and the camera azimuth angle is locked, that is, the camera acquisition plane is parallel to the bridge plane to be detected, and two key factors of inspection planning are unified on the inspection plane, and the inspection plane in three-dimensional space is converted into a two-dimensional plane model; S134, based on the regularity of the bridge component, the center line thereof is taken as the unmanned aerial vehicle inspection path reference on the two-dimensional inspection plane, and the bridge component center line is quickly extracted through the boundary fitting method; the boundary fitting direction follows the aspect ratio criterion, that is, the numerical comparison of the bridge component width and the bridge component height is performed, when the bridge component width > bridge component height, the bridge component width direction is taken; when the bridge component height > bridge component width, the bridge component height direction is taken; the boundary is fitted using a polynomial; S135, combined with the visual field model, camera collection point planning is performed on the bridge component center line, considering the combination of different distance coefficients D and focal lengths f, so as to ensure that the coverage range is greater than the boundary distance; in the rough inspection stage, the distance coefficient D is increased and the focal length f is reduced, the collection field of view range is increased, and then the overlap rate δ is reduced, so as to cover the bridge component in the optimal number of collection points; S136, the collection point coordinates on the two-dimensional plane are restored to three-dimensional coordinates through reverse mapping, and the complete inspection path is obtained. 4.The multi-layer representation learning based adaptive UAV inspection method of claim 1, wherein: Step S22 specifically comprises the following sub-steps: S221, the data collected in the rough inspection stage is automatically screened for diseases using a disease detection network, and the image data containing diseases is independently summarized to form a disease set that is different from normal area image data; The order of the collection points based on the rough detection route before the disease screening stage Assigning a corresponding label x to each collection image i After the screening stage, the real spatial position of the disease collection point can be quickly indexed according to the disease image label; An adaptive inspection method based on rough inspection information feedback is proposed, which enables the unmanned aerial vehicle to efficiently inspect while paying attention to the disease area. First, combined with the information fed back in the disease screening stage, the collection points of the unmanned aerial vehicle are divided into two categories: normal and abnormal, At the normal collection point, that is, the area where the bridge component is not damaged, the unmanned aerial vehicle takes the rough inspection path as the inspection reference; At the abnormal collection point, that is, the area where the bridge component is damaged, the unmanned aerial vehicle adaptively adjusts the detection path according to the disease feedback information, so as to obtain more accurate detailed information; Reduce the size of the drone's field of view and record the local details of the disease; the field of view can be reduced by reducing the distance coefficient D or increasing the focal length f. Keep the camera focal length unchanged and construct a distance target plane D in the inspection space. s The solution of fine detection plane realizes the fine acquisition of field of view (W s ,H s ) flexible adjustment, W s ,H s Indicates the height and width of the acquisition field of view; S222、Further, the disease area is projected to the newly constructed fine detection plane as an optimization target, and the detection plane in three-dimensional space is converted to a two-dimensional plane to cut the disease area corresponding to a single collection point in a grid format, where the width of each grid is height S w ,S h respectively represent the division coefficients in width and height, and the center coordinates of these subgrids represent the collection points on the fine detection path; S223, the same operation as step S222 is used to traverse all disease areas and develop a fine collection plan; The collection points exceeding the limit threshold δ are filtered and deleted using the overlap area measurement criterion; The heuristic search algorithm is used to connect the discrete collection points, and the distance cost between adjacent points is minimized to construct the fine inspection path; S224, finally, the spatial conversion relationship between the collection points and the inspection target points is used, that is, the collection points are projected onto the bridge surface through the distance coefficient D, to locate the disease area in the real space.

Images