Bridge inspection method based on image recognition

Abstract

The present invention discloses a bridge inspection method based on image recognition, which relates to the technical field of bridge inspection, including: obtaining the geodetic coordinate data of the target bridge, and obtaining the spatial rectangular coordinate data of the target bridge through coordinate system conversion. Performing a first route planning to obtain a preliminary inspection route, commanding a drone to collect images of the target bridge according to the preliminary inspection route to obtain a first image set, and classifying the first image set through a first machine learning model to obtain a classification result of key parts of the bridge. Based on the classification result of key parts of the bridge, a second route planning and a third route planning are performed to obtain a formal inspection route; commanding the drone to perform a flight inspection according to the formal inspection route, and performing real-time adjustment through a safety potential field algorithm to obtain an updated inspection route and complete the inspection shooting to obtain a second image set. Performing defect detection on the second image set through a second machine learning model to obtain a bridge defect recognition result, and completing the bridge inspection.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge inspection, and in particular to a bridge inspection method based on image recognition. Background Art

[0002] In recent years, my country's transportation infrastructure construction has shown a trend of shifting from rapid development to steady development. As the service life of existing large-scale transportation buildings increases, the focus of attention on large-scale transportation buildings such as bridges has shifted to daily inspections. Thanks to innovative breakthroughs in the fields of artificial intelligence technology, image processing technology, and drone technology, drone inspection technology combined with image recognition has gradually been put into use in bridge inspection applications.

[0003] At present, the Chinese invention patent with application number CN202311600725.X discloses a bridge intelligent inspection system based on drones, including: a server, a ground station and a drone; the ground station includes: a modeling module, the modeling module establishes a three-dimensional model of the bridge according to the data collected when the drone flies around the bridge; a path planning module, the path planning module plans and generates the cruise path of the drone during the inspection according to the three-dimensional model of the bridge; the drone automatically flies around the bridge along the generated cruise route and collects image data, and the server analyzes the image data to find the location of the disease on the bridge; the invention can establish a bridge model before the inspection begins and automatically plan the inspection route, so as to avoid obstacles that may be encountered during the inspection, and at the same time minimize the impact of the staff's control ability on the drone during the inspection process, so as to ensure the inspection effect and efficiency. However, this solution requires three-dimensional modeling of the bridge, and the calculation complexity is high when the inspection target is a large bridge. There is a lack of targeted analysis of different bridge structures, and it relies on the satellite navigation system during the inspection process, which is easily interfered by external signals. Summary of the invention

[0004] The technical problem solved by the present invention is that the traditional scheme often requires three-dimensional modeling of the bridge when formulating the bridge inspection plan, which easily leads to the problem of high computational complexity. At the same time, there is a lack of targeted analysis of different structures in the bridge, and the inspection process relies on the satellite navigation system, which is easily interfered by external signals.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] Step S1, obtaining the geodetic coordinate data of the target bridge, and obtaining the spatial rectangular coordinate data of the target bridge through coordinate system conversion;

[0007] Step S2, performing a first route planning based on the spatial rectangular coordinate data of the target bridge to obtain a preliminary inspection route, commanding the drone to collect images of the target bridge according to the preliminary inspection route to obtain a first image set, and classifying the first image set through a first machine learning model to obtain a classification result of key parts of the bridge;

[0008] Step S3, based on the classification results of the key parts of the bridge, a second route planning is performed to obtain an upper area route and a lower area route, and a third route planning is performed to integrate and connect them to obtain a formal inspection route;

[0009] Step S4, commanding the drone to perform flight inspection along the formal inspection route, adjusting the formal inspection route in real time through the safety potential field algorithm to obtain an updated inspection route, and completing the inspection shooting along the updated inspection route to obtain a second image set;

[0010] Step S5, performing defect detection on the second image set through the second machine learning model to obtain a bridge defect recognition result, thereby completing the bridge inspection.

[0011] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, the geodetic coordinate data of the target bridge is obtained based on the geographic information system data management platform, a spatial rectangular coordinate system is established with the center point of the target bridge as the origin of the coordinate system and the longitudinal direction of the bridge as the longitudinal axis of the coordinate system, and the spatial rectangular coordinate data of the target bridge is obtained by performing coordinate system transformation based on the geodetic coordinate data of the target bridge.

[0012] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, wherein: a first route is planned based on the spatial rectangular coordinate data of the target bridge to obtain a preliminary inspection route, and the drone is commanded to collect images of the target bridge according to the preliminary inspection route to obtain a first image set;

[0013] The logic of performing the first route planning includes: taking the origin of the spatial rectangular coordinate system as the center of the circle, times the maximum height of the bridge as horizontal height, A preliminary inspection circle is obtained by drawing a circle with a radius of times the bridge length in the spatial rectangular coordinate system. Points are evenly selected on the preliminary inspection circle at preset angle intervals to obtain the coordinates of the nodes of the preliminary inspection route. The preliminary inspection route is obtained by connecting the coordinates of the nodes of the preliminary inspection route in the order of evenly selected points.

[0014] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, wherein: the first image set is input into a first machine learning model that has been trained in advance, and the first image set is classified by the first machine learning model to obtain a classification result of key parts of the bridge;

[0015] The classification results of key parts of the bridge include: piers, abutments, bearings, bridge deck, bridge bottom, arch rings, towers, steel structures, beams, cables and guardrails.

[0016] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, the first image set is subjected to mosaic data enhancement through the input end of the first machine learning model to obtain an enhanced image data set, the enhanced image data set is subjected to slice dimension reduction and convolution processing through the FOCUS module to obtain slice image data, GhostNet is introduced into the backbone network for lightweight improvement, the GhostModule module can replace the original convolution layer, the slice image data is subjected to channel number compression processing through a point convolution kernel with a step size of 1×1, feature extraction is performed through a depth-separable convolution layer, and a GM feature data set is obtained by performing nonlinear transformation through a BNReLU activation function, and a feature splicing processing is performed on the GM feature data set to obtain a first feature extraction data set;

[0017] A dilated spatial pyramid pool is introduced to replace the original SPP. The dilated spatial pyramid pool is a five-parallel computing branch structure, including a 3×3 dilated convolution processing with a dilation rate of 1, a 3×3 dilated convolution processing with a dilation rate of 3, a 3×3 dilated convolution processing with a dilation rate of 5, a 1×1 convolution processing, and a global pooling processing. Five sets of scale feature data sets are calculated through the five parallel computing branch structures, and the five sets of scale feature data sets are subjected to secondary feature splicing and 1×1 convolution processing to obtain the backbone network output data;

[0018] The output data of the backbone network is sampled and processed by the neck network, and the feature fusion is performed to obtain the output data of the neck network. The original detection head in the head network is replaced with the decoupled detection head. The output data of the neck network is subjected to global average pooling calculation to obtain the feature vector corresponding to each channel. The feature vector corresponding to each channel and the channel weight vector are weightedly summed up through the CA attention mechanism to obtain the final fused feature. The classifier and regressor in the decoupled detection head are used to perform category prediction and bounding box prediction on the fused feature to obtain the classification results of key parts of the bridge.

[0019] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, wherein: based on the classification results of key parts of the bridge, route planning is performed to obtain a formal inspection route, and the logic of obtaining the formal inspection route includes: establishing a three-dimensional grid map based on the spatial rectangular coordinates corresponding to the classification results of key parts of the bridge and the Astar algorithm with an improved cost function, dividing the target bridge into an upper area of ​​the bridge body and a lower area of ​​the bridge body based on the three-dimensional grid map and the bridge deck coordinate data in the classification results of key parts of the bridge, using the grids corresponding to the key parts of the bridge in the three-dimensional grid map as path nodes, and performing a second route planning on the path nodes in the upper area of ​​the bridge body and the lower area of ​​the bridge body through the Astar algorithm with an improved cost function to obtain the upper area route and the lower area route.

[0020] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, the final node coordinates of the upper area route are used as the starting point of the series route, and the initial node coordinates of the lower area route are used as the end point of the series route. The third route is planned by the Astar algorithm with an improved cost function to obtain the series route, and the upper area route and the lower area route are integrated and connected in series to obtain the formal inspection route.

[0021] Among them, the calculation expression of the improved cost function is:

[0022] ;

[0023] Where n represents the index of the current node, f(n) represents the estimated total cost, g(n) represents the actual cost corresponding to the current node, and h(n) represents the heuristic estimated cost corresponding to the current node. represents the weight corresponding to the classification result of the key parts of the bridge, and i represents the index corresponding to the classification result of the key parts of the bridge.

[0024] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, the drone is commanded to perform a flight inspection according to the formal inspection route, the real-time distance of obstacles is obtained by measuring the distance through the airborne millimeter-wave radar, and during the flight inspection, the formal inspection route is adjusted in real time through the safety potential field algorithm to obtain an updated inspection route, and the inspection is completed by updating the inspection route to obtain a second image set;

[0025] The logic of real-time adjustment of the formal inspection route through the safety potential field algorithm includes: calculating the real-time obstacle repulsion based on the real-time distance of the obstacle and the potential field repulsion function, and performing threshold judgment on the real-time obstacle repulsion through a preset repulsion threshold. When the real-time obstacle repulsion is less than the preset repulsion threshold, the priority of the safety potential field algorithm is set to the lowest, and the UAV continues to fly along the formal inspection route;

[0026] When the real-time obstacle repulsion is greater than or equal to the preset repulsion threshold, the priority of the safety potential field algorithm is set to the highest, and the path planning from the current position to the next path node is calculated by the safety potential field algorithm to obtain an updated inspection route;

[0027] Among them, the calculation expression of the potential field repulsion function is:

[0028] ;

[0029] Where U represents the potential field repulsion function, represents the repulsive field coefficient, Indicates the real-time distance between the drone and the obstacle. represents the acceleration of the drone, I represents the spatial rectangular coordinate of the obstacle, Represents the spatial rectangular coordinates of the drone, Represents the range of the repulsive field.

[0030] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, wherein: a second image set is subjected to defect detection by a second machine learning model to obtain a bridge defect recognition result, wherein the second machine learning model is a pre-trained Fast-RCNN model;

[0031] The bridge defect identification results include: cracks, spalling, rust, deformation, support loss, bridge deck potholes, tower tilt, deflection, cable breakage and guardrail damage.

[0032] As a preferred solution of the bridge inspection method based on image recognition described in the present invention, the logic of bridge defect recognition includes: introducing a residual module in the convolution layer, performing sliding convolution calculation on the second image set through the residual module, obtaining a residual feature map through residual connection and weighted calculation processing, and performing nonlinear processing on the residual feature map through the ReLU activation function to obtain a convolution layer feature map;

[0033] The convolutional layer feature map is gridded and processed by the maximum pooling through the RoI pooling layer to obtain a deep feature map, a parallel branch is introduced, the second image set is gray-scale mapped through the parallel branch to obtain gray-scale image data, the gray-scale image data is smoothed by Gaussian filtering to obtain smoothed image data, the smoothed image data is subjected to edge feature extraction by the Sobel operator to obtain a sharpened feature map, and the sharpened feature map is spliced ​​with the deep feature map to obtain a fused feature map;

[0034] In the fully connected layer, the fused feature map is batch normalized, the fused feature map data is segmented to obtain segment data, the mean and variance of each segment data are normalized to obtain a normalized vector, the normalized vector is linearly transformed by scaling weights and bias constants to obtain a generalized feature vector, and the generalized feature vector is calculated by the Softmax classifier and bounding box regression to obtain the bridge defect recognition result and the corresponding bounding box.

[0035] The beneficial effects of the present invention are as follows: the key structures of the bridge are accurately classified by the first machine learning model before route planning, which is conducive to the reasonable inspection of different types of bridges and improves the adaptability of the scheme. The grid map of the bridge is divided into the upper area of ​​the bridge body and the lower area of ​​the bridge body, and the path planning is performed separately, which reduces the vertical lifting path in the line, thereby reducing the inspection energy consumption and reducing the complexity of the drone camera control. The introduction of the safety potential field algorithm improves the obstacle avoidance ability of the drone during inspection. The priority setting of the safety potential field algorithm is adjusted by threshold judgment, which is conducive to flexible adjustment of the flight route and improvement of the safety of the inspection. The second image set obtained by close shooting is identified by the second machine model, and the extracted convolution features and edge features are fused, which is conducive to improving the accuracy of the classification of the second image set. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the basic flow of a bridge inspection method based on image recognition is provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0038] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a bridge inspection method based on image recognition, comprising:

[0039] Step S1, obtaining the geodetic coordinate data of the target bridge, and obtaining the spatial rectangular coordinate data of the target bridge through coordinate system conversion;

[0040] Step S2, performing a first route planning based on the spatial rectangular coordinate data of the target bridge to obtain a preliminary inspection route, commanding the drone to collect images of the target bridge according to the preliminary inspection route to obtain a first image set, and classifying the first image set through a first machine learning model to obtain a classification result of key parts of the bridge;

[0041] Step S3, based on the classification results of the key parts of the bridge, a second route planning is performed to obtain an upper area route and a lower area route, and a third route planning is performed to integrate and connect them to obtain a formal inspection route;

[0042] Step S4, commanding the drone to perform flight inspection along the formal inspection route, adjusting the formal inspection route in real time through the safety potential field algorithm to obtain an updated inspection route, and completing the inspection shooting along the updated inspection route to obtain a second image set;

[0043] Step S5, performing defect detection on the second image set through the second machine learning model to obtain a bridge defect recognition result, thereby completing the bridge inspection.

[0044] In this embodiment, the geodetic coordinate data of the target bridge is obtained based on the geographic information system data management platform, a spatial rectangular coordinate system is established with the center point of the target bridge as the origin of the coordinate system and the longitudinal direction of the bridge as the longitudinal axis of the coordinate system, and the spatial rectangular coordinate data of the target bridge is obtained by performing coordinate system transformation based on the geodetic coordinate data of the target bridge.

[0045] In this embodiment, a first route is planned based on the spatial rectangular coordinate data of the target bridge to obtain a preliminary inspection route, and the drone is commanded to collect images of the target bridge according to the preliminary inspection route to obtain a first image set;

[0046] The logic of performing the first route planning includes: taking the origin of the spatial rectangular coordinate system as the center of the circle, times the maximum height of the bridge as horizontal height, A preliminary inspection circle is obtained by drawing a circle with a radius of times the bridge length in the spatial rectangular coordinate system. Points are evenly selected on the preliminary inspection circle at preset angle intervals to obtain the coordinates of the nodes of the preliminary inspection route. The preliminary inspection route is obtained by connecting the coordinates of the nodes of the preliminary inspection route in the order of evenly selected points.

[0047] Among them, the preliminary inspection route is an approximate circle with the center point of the bridge as the center, which is conducive to comprehensively photographing the overall structure of the target bridge. At the same time, the camera angle on the drone gimbal is constantly pointed to the center of the space rectangular coordinate system, and the overall image of the bridge can be collected without adjusting the camera angle.

[0048] In this embodiment, the first image set is input into a first machine learning model that has been pre-trained, and the first image set is classified by the first machine learning model to obtain a classification result of key parts of the bridge;

[0049] The classification results of key parts of the bridge include: piers, abutments, bearings, bridge deck, bridge bottom, arch rings, towers, steel structures, beams, cables and guardrails.

[0050] In this embodiment, mosaic data enhancement is performed on the first image set through the input end of the first machine learning model to obtain an enhanced image data set, slice dimension reduction and convolution processing are performed on the enhanced image data set through the FOCUS module to obtain slice image data, GhostNet is introduced into the backbone network for lightweight improvement, the GhostModule module can replace the original convolution layer, the slice image data is subjected to channel number compression processing through a point convolution kernel with a step size of 1×1, feature extraction is performed through a depth-separable convolution layer, and a nonlinear transformation is performed through a BNReLU activation function to obtain a GM feature data set, and a feature splicing process is performed on the GM feature data set to obtain a first feature extraction data set;

[0051] A dilated spatial pyramid pool is introduced to replace the original SPP. The dilated spatial pyramid pool is a five-parallel computing branch structure, including a 3×3 dilated convolution processing with a dilation rate of 1, a 3×3 dilated convolution processing with a dilation rate of 3, a 3×3 dilated convolution processing with a dilation rate of 5, a 1×1 convolution processing, and a global pooling processing. Five sets of scale feature data sets are calculated through the five parallel computing branch structures, and the five sets of scale feature data sets are subjected to secondary feature splicing and 1×1 convolution processing to obtain the backbone network output data;

[0052] The output data of the backbone network is sampled and processed by the neck network, and the feature fusion is performed to obtain the output data of the neck network. The original detection head in the head network is replaced with the decoupled detection head. The output data of the neck network is subjected to global average pooling calculation to obtain the feature vector corresponding to each channel. The feature vector corresponding to each channel and the channel weight vector are weightedly summed up through the CA attention mechanism to obtain the final fused feature. The classifier and regressor in the decoupled detection head are used to perform category prediction and bounding box prediction on the fused feature to obtain the classification results of key parts of the bridge.

[0053] Among them, the first machine learning model is used to accurately classify the key structures of the bridge before route planning, which is conducive to the reasonable inspection of different types of bridges and improves the adaptability of the solution.

[0054] In this embodiment, route planning is performed based on the classification results of key parts of the bridge to obtain a formal inspection route, and the logic of obtaining the formal inspection route includes: establishing a three-dimensional grid map based on the spatial rectangular coordinates corresponding to the classification results of the key parts of the bridge and the Astar algorithm with an improved cost function, dividing the target bridge into an upper area and a lower area of ​​the bridge body based on the three-dimensional grid map and the bridge deck coordinate data in the classification results of the key parts of the bridge, using the grids corresponding to the key parts of the bridge in the three-dimensional grid map as path nodes, and performing a second route planning on the path nodes in the upper area of ​​the bridge body and the lower area of ​​the bridge body through the Astar algorithm with an improved cost function to obtain the upper area route and the lower area route.

[0055] In this embodiment, the final node coordinates of the upper area route are used as the starting point of the series route, and the initial node coordinates of the lower area route are used as the end point of the series route. The third route is planned by the Astar algorithm with an improved cost function to obtain the series route, and the upper area route and the lower area route are integrated and connected in series to obtain the formal inspection route;

[0056] Among them, the calculation expression of the improved cost function is:

[0057] ;

[0058] Where n represents the index of the current node, f(n) represents the estimated total cost, g(n) represents the actual cost corresponding to the current node, and h(n) represents the heuristic estimated cost corresponding to the current node. represents the weight corresponding to the classification result of the key parts of the bridge, and i represents the index corresponding to the classification result of the key parts of the bridge.

[0059] Among them, the grid map of the bridge is divided into the upper area and the lower area of ​​the bridge body, and the paths are planned separately, which reduces the vertical lifting paths in the line and thus reduces the energy consumption of inspection. At the same time, the drone uses a fixed depression angle to control the camera lens when inspecting and shooting along the route of the upper area, and uses a fixed elevation angle to control the camera lens when inspecting and shooting along the route of the lower area, reducing the complexity of drone camera control.

[0060] In this embodiment, the drone is commanded to perform a flight inspection along the formal inspection route, and the real-time distance of obstacles is obtained by measuring the distance through the airborne millimeter-wave radar. During the flight inspection, the formal inspection route is adjusted in real time through the safety potential field algorithm to obtain an updated inspection route, and the inspection is completed by updating the inspection route to obtain a second image set.

[0061] The logic of real-time adjustment of the formal inspection route through the safety potential field algorithm includes: calculating the real-time obstacle repulsion based on the real-time distance of the obstacle and the potential field repulsion function, and performing threshold judgment on the real-time obstacle repulsion through a preset repulsion threshold. When the real-time obstacle repulsion is less than the preset repulsion threshold, the priority of the safety potential field algorithm is set to the lowest, and the UAV continues to fly along the formal inspection route;

[0062] When the real-time obstacle repulsion is greater than or equal to the preset repulsion threshold, the priority of the safety potential field algorithm is set to the highest, and the path planning from the current position to the next path node is calculated by the safety potential field algorithm to obtain an updated inspection route;

[0063] Among them, the calculation expression of the potential field repulsion function is:

[0064] ;

[0065] Where U represents the potential field repulsion function, represents the repulsive field coefficient, Indicates the real-time distance between the drone and the obstacle. represents the acceleration of the drone, I represents the spatial rectangular coordinate of the obstacle, Represents the spatial rectangular coordinates of the drone, Represents the range of the repulsive field.

[0066] Among them, the introduction of the safety potential field algorithm improves the obstacle avoidance capability of the UAV during inspection. Adjusting the priority setting of the safety potential field algorithm through threshold judgment is conducive to flexibly adjusting the flight route and improving the safety of inspection.

[0067] In this embodiment, defect detection is performed on the second image set by a second machine learning model to obtain a bridge defect recognition result, where the second machine learning model is a pre-trained Fast-RCNN model;

[0068] The bridge defect identification results include: cracks, spalling, rust, deformation, support loss, bridge deck potholes, tower tilt, deflection, cable breakage and guardrail damage.

[0069] In this embodiment, the logic of bridge defect recognition includes: introducing a residual module in the convolution layer, performing sliding convolution calculation on the second image set through the residual module, obtaining a residual feature map through residual connection and weighted calculation processing, and performing nonlinear processing on the residual feature map through the ReLU activation function to obtain a convolution layer feature map;

[0070] The convolutional layer feature map is gridded and processed by the maximum pooling through the RoI pooling layer to obtain a deep feature map, a parallel branch is introduced, the second image set is gray-scale mapped through the parallel branch to obtain gray-scale image data, the gray-scale image data is smoothed by Gaussian filtering to obtain smoothed image data, the smoothed image data is subjected to edge feature extraction by the Sobel operator to obtain a sharpened feature map, and the sharpened feature map is spliced ​​with the deep feature map to obtain a fused feature map;

[0071] In the fully connected layer, the fused feature map is batch normalized, the fused feature map data is segmented to obtain segment data, the mean and variance of each segment data are normalized to obtain a normalized vector, the normalized vector is linearly transformed by scaling weights and bias constants to obtain a generalized feature vector, and the generalized feature vector is calculated by the Softmax classifier and bounding box regression to obtain the bridge defect recognition result and the corresponding bounding box.

[0072] Defects are identified on the second image set obtained by close-up shooting through the second machine model, and the extracted convolution features and edge features are fused, which is beneficial to improving the accuracy of classification of the second image set.

[0073] It should be understood by those skilled in the art that the embodiments of the present invention may provide methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. Among them, the storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A bridge inspection method based on image recognition, characterized in that: include: Step S1, obtaining the geodetic coordinate data of the target bridge, and obtaining the spatial rectangular coordinate data of the target bridge through coordinate system conversion; Step S2, performing a first route planning based on the spatial rectangular coordinate data of the target bridge to obtain a preliminary inspection route, commanding the UAV to collect images of the target bridge according to the preliminary inspection route to obtain a first image set, and classifying the first image set through a first machine learning model to obtain a classification result of key parts of the bridge, wherein the classification result of key parts of the bridge includes: piers, abutments, bearings, bridge decks, bridge bottoms, arch rings, towers, steel structures, beams, cables, and guardrails; Step S3, based on the classification results of the key parts of the bridge, a second route planning is performed to obtain an upper area route and a lower area route, and a third route planning is performed to integrate and connect them to obtain a formal inspection route; Step S4, commanding the drone to perform flight inspection along the formal inspection route, adjusting the formal inspection route in real time through the safety potential field algorithm to obtain an updated inspection route, and completing the inspection shooting along the updated inspection route to obtain a second image set; Step S5, performing defect detection on the second image set by a second machine learning model to obtain a bridge defect recognition result, and completing the bridge inspection; Wherein, a first route is planned based on the spatial rectangular coordinate data of the target bridge to obtain a preliminary inspection route, and the UAV is commanded to collect images of the target bridge according to the preliminary inspection route to obtain a first image set; The logic of performing the first route planning includes: taking the origin of the spatial rectangular coordinate system as the center of the circle, times the maximum height of the bridge as horizontal height, A circle is drawn in the spatial rectangular coordinate system with a radius of times the bridge length to obtain a preliminary inspection circle, and points are uniformly selected on the preliminary inspection circle at preset angle intervals to obtain the left side of the preliminary inspection route nodes. The preliminary inspection route is obtained by connecting the coordinates of the preliminary inspection route nodes in the uniformly selected point order; Based on the classification results of key parts of the bridge, route planning is performed to obtain a formal inspection route. The logic of obtaining the formal inspection route includes: establishing a three-dimensional grid map based on the spatial rectangular coordinates corresponding to the classification results of the key parts of the bridge and the Astar algorithm with an improved cost function, dividing the target bridge into an upper area of ​​the bridge body and an upper area of ​​the bridge body based on the three-dimensional grid map and the bridge deck coordinate data in the classification results of the key parts of the bridge, using the grids corresponding to the key parts of the bridge in the three-dimensional grid map as path nodes, and performing a second route planning on the path nodes in the upper area of ​​the bridge body and the lower area of ​​the bridge body respectively through the Astar algorithm with an improved cost function to obtain an upper area route and a lower area route; The final node coordinates of the upper area route are used as the starting point of the series route, and the initial node coordinates of the lower area route are used as the end point of the series route. The third route is planned by the Astar algorithm with an improved cost function to obtain the series route. The upper area route and the lower area route are integrated and connected in series to obtain the formal inspection route. Among them, the calculation expression of the improved cost function is: ； Where n represents the index of the current node, f(n) represents the estimated total cost, g(n) represents the actual cost corresponding to the current node, and h(n) represents the heuristic estimated cost corresponding to the current node. It represents the weight corresponding to the classification result of the key parts of the bridge, and i represents the index corresponding to the classification result of the key parts of the bridge.

2. The bridge inspection method based on image recognition as claimed in claim 1, characterized in that: The geodetic coordinate data of the target bridge are obtained based on the geographic information system data management platform, and a spatial rectangular coordinate system is established by taking the center point of the target bridge as the origin of the coordinate system and the longitudinal direction of the bridge as the longitudinal axis of the coordinate system. The spatial rectangular coordinate data of the target bridge are obtained by performing coordinate system transformation based on the geodetic coordinate data of the target bridge.

3. The bridge inspection method based on image recognition as claimed in claim 1, characterized in that: Inputting the first image set into a pre-trained first machine learning model, and classifying the first image set by the first machine learning model to obtain a classification result of key parts of the bridge; The classification results of key parts of the bridge include: piers, abutments, bearings, bridge deck, bridge bottom, arch rings, towers, steel structures, beams, cables and guardrails.

4. The bridge inspection method based on image recognition as claimed in claim 3, characterized in that: The first image set is subjected to mosaic data enhancement through the input end of the first machine learning model to obtain an enhanced image data set, the strong image data set is subjected to slice dimension reduction and convolution processing through the FOCUS module to obtain slice image data, GhostNet is introduced into the backbone network for lightweight improvement, the GhostModule module can replace the original convolution layer, the slice image data is subjected to channel number compression processing through a point convolution kernel with a step size of 1×1, feature extraction is performed through a depthwise separable convolution layer, and a nonlinear transformation is performed through a BNReLU activation function to obtain a GM feature data set, and a feature splicing processing is performed on the GM feature data set to obtain a first feature extraction data set; A dilated spatial pyramid pool is introduced to replace the original SPP. The dilated spatial pyramid pool is a five-parallel computing branch structure, including a 3×3 dilated convolution processing with a dilation rate of 1, a 3×3 dilated convolution processing with a dilation rate of 3, a 3×3 dilated convolution processing with a dilation rate of 5, a 1×1 convolution processing, and a global pooling processing. Five sets of scale feature data sets are calculated through the five parallel computing branch structures, and the five sets of scale feature data sets are subjected to secondary feature splicing and 1×1 convolution processing to obtain the backbone network output data; The output data of the backbone network is sampled and processed by the neck network, and the feature fusion is performed to obtain the output data of the neck network. The original detection head in the head network is replaced with the decoupled detection head. The output data of the neck network is subjected to global average pooling calculation to obtain the feature vector corresponding to each channel. The feature vector corresponding to each channel and the channel weight vector are weightedly summed up through the CA attention mechanism to obtain the final fused feature. The classifier and regressor in the decoupled detection head are used to perform category prediction and bounding box prediction on the fused feature to obtain the classification results of key parts of the bridge.

5. The bridge inspection method based on image recognition as claimed in claim 1, characterized in that: Command the drone to perform a flight inspection along the official inspection route, obtain the real-time distance of obstacles by measuring the distance through the airborne millimeter-wave radar, and adjust the official inspection route in real time through the safety potential field algorithm during the flight inspection to obtain an updated inspection route. The inspection is completed by updating the inspection route to obtain a second image set. The logic of real-time adjustment of the formal inspection route through the safety potential field algorithm includes: calculating the real-time obstacle repulsion based on the real-time distance of the obstacle and the potential field repulsion function, and performing threshold judgment on the real-time obstacle repulsion through a preset repulsion threshold. When the real-time obstacle repulsion is less than the preset repulsion threshold, the priority of the safety potential field algorithm is set to the lowest, and the UAV continues to fly along the formal inspection route; When the real-time obstacle repulsion is greater than or equal to the preset repulsion threshold, the priority of the safety potential field algorithm is set to the highest, and the path planning from the current position to the next path node is calculated by the safety potential field algorithm to obtain an updated inspection route; Among them, the calculation expression of the potential field repulsion function is: ； Where U represents the potential field repulsion function, represents the repulsive field coefficient, Indicates the real-time distance between the drone and the obstacle. represents the acceleration of the drone, I represents the spatial rectangular coordinate of the obstacle, Represents the spatial rectangular coordinates of the drone, Represents the range of the repulsive field.

6. The bridge inspection method based on image recognition according to claim 1, characterized in that: Performing defect detection on the second image set by using a second machine learning model to obtain a bridge defect recognition result, wherein the second machine learning model is a pre-trained Fast-RCNN model; The bridge defect identification results include: cracks, spalling, rust, deformation, support loss, bridge deck potholes, tower tilt, deflection, cable breakage and guardrail damage.

7. The bridge inspection method based on image recognition as claimed in claim 6, characterized in that: The logic of bridge defect recognition includes: introducing a residual module in a convolution layer, performing sliding convolution calculation on the second image set through the residual module, obtaining a residual feature map through residual connection and weighted calculation processing, and performing nonlinear processing on the residual feature map through a ReLU activation function to obtain a convolution layer feature map; The convolutional layer feature map is gridded and processed by the maximum pooling through the RoI pooling layer to obtain a deep feature map, a parallel branch is introduced, the second image set is gray-scale mapped through the parallel branch to obtain gray-scale image data, the gray-scale image data is smoothed by Gaussian filtering to obtain smoothed image data, the smoothed image data is subjected to edge feature extraction by the Sobel operator to obtain a sharpened feature map, and the sharpened feature map is spliced ​​with the deep feature map to obtain a fused feature map; In the fully connected layer, the fused feature map is batch normalized, the fused feature map data is segmented to obtain segment data, the mean and variance of each segment data are normalized to obtain a normalized vector, the normalized vector is linearly transformed by scaling weights and bias constants to obtain a generalized feature vector, and the generalized feature vector is calculated by the Softmax classifier and bounding box regression to obtain the bridge defect recognition result and the corresponding bounding box.

Images