Progressive multi-uav patrol bridge apparent disease rapid detection method

Abstract

The application provides a progressive multi-unmanned aerial vehicle (UAV) patrol bridge apparent disease rapid detection method, and particularly relates to the bridge apparent disease detection technical field, and the scheme comprises the following steps: a first UAV is used to carry a high-resolution camera to quickly shoot a large-range bridge global image, a deep learning network model is used to identify a disease local area, and after the area position information is solved, the absolute coordinate form is transmitted to a plurality of second UAVs, the second UAVs are controlled to patrol and detect a plurality of disease local areas, and an apparent disease detection result is obtained. The scheme adopts a progressive detection scheme, only one global detection is required to obtain a disease layout area, and then accurate detection is only carried out on the disease local area, so that the detection efficiency and accuracy are improved, and the view angle of the bridge detection is effectively widened.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bridge apparent disease detection, and particularly relates to a progressive multi-unmanned aerial vehicle (UAV) patrol bridge apparent disease rapid detection method. BACKGROUND

[0002] As a key transportation infrastructure, the structural health and safety of a bridge is crucial for ensuring traffic safety and economic development. Due to long-term exposure to natural environment and continuous load, a bridge is prone to apparent diseases such as cracks, corrosion and concrete spalling. If these diseases are not detected and treated in time, they may evolve into structural damage, affecting the carrying capacity of the bridge and threatening public safety. Therefore, it is necessary to rapidly detect the apparent diseases of the bridge, timely find the apparent disease abnormalities, and provide support for the maintenance and repair of the bridge facilities to ensure the safety of the bridge structure.

[0003] In recent years, with the continuous development of unmanned aerial vehicle (UAV) technology and computer network technology, the bridge apparent disease detection method based on UAV has been widely applied in practical engineering. The current UAV inspection mainly detects the apparent diseases of the whole bridge. However, for a bridge in normal operation, apparent diseases only occur in a few local areas of the bridge. Although the method of full-range coverage inspection can accurately detect diseases, it has problems such as resource redundancy and long detection time. SUMMARY

[0004] In view of the above problems of the prior art, the present application aims to provide a progressive multi-UAV patrol bridge apparent disease rapid detection method, which aims to solve the problems of resource waste and low detection efficiency in the prior art.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides a progressive multi-UAV patrol bridge apparent disease rapid detection method, comprising:

[0006] obtaining absolute position information of a target bridge;

[0007] based on the absolute position information, using at least one first UAV to detect the apparent diseases of the global area of the target bridge, and identifying a plurality of local disease areas;

[0008] calculating the position coordinates of all the local disease areas to obtain a set of detection waypoint coordinates;

[0009] based on the set of detection waypoint coordinates, path planning is performed on a plurality of second UAVs to determine the detection path of each second UAV;

[0010] using the second UAVs to patrol and detect all the local disease areas according to the respective detection paths to obtain an apparent disease detection result.

[0011] In an embodiment, based on the absolute position information, apparent disease detection is performed on the global area of the target bridge by at least one first unmanned aerial vehicle, and a plurality of local disease areas are identified, including:

[0012] Based on the absolute position information, an initial shooting distance of the target bridge and an area of a region to be detected are determined;

[0013] The field of view, pixel, and focal length of a shooting camera are obtained, and based on the initial shooting distance, the area of the region to be detected, the field of view, the pixel, and the focal length, an optimal shooting distance is determined;

[0014] Based on the absolute position information, the area of the region to be detected, the field of view, and the optimal shooting distance, a flight path of the first unmanned aerial vehicle is determined;

[0015] The global area of the target bridge is shot along the corresponding flight path by the first unmanned aerial vehicle, and a global bridge image is obtained;

[0016] The global bridge image is subjected to apparent disease detection by using a preset first deep learning network model, and a plurality of local disease areas are identified.

[0017] In an embodiment, based on the initial shooting distance, the area of the region to be detected, the field of view, the pixel, and the focal length, the optimal shooting distance is determined, including:

[0018] Based on the initial shooting distance, the pixel, and the focal length, a spatial resolution is determined;

[0019] Based on the initial shooting distance and the field of view, a camera field of view range is determined;

[0020] Based on the area of the region to be detected, the camera field of view range, and the spatial resolution, a target function is constructed;

[0021] The initial shooting distance is optimized to determine the optimal shooting distance, with the goal of minimizing the target function.

[0022] In an embodiment, based on the absolute position information, the area of the region to be detected, the field of view, and the optimal shooting distance, the flight path of the first unmanned aerial vehicle is determined, including:

[0023] Based on the area of the region to be detected, the field of view, and the optimal shooting distance, a shooting frequency is determined;

[0024] Based on the shooting frequency and the absolute position information, a number of positions to be shot is determined;

[0025] determine a flight path of the first unmanned aerial vehicle based on the number of positions to be photographed.

[0026] In an embodiment, the position coordinates of all the disease local areas are calculated to obtain a detection waypoint coordinate set, including:

[0027] The position coordinates of the disease local areas are calculated to obtain absolute coordinates of each disease local area.

[0028] The waypoint positions are determined based on the absolute coordinates of the disease local areas and a preset minimum safe shooting distance.

[0029] The detection waypoint coordinate set is obtained based on the waypoint positions.

[0030] In an embodiment, the second unmanned aerial vehicles are path planned based on the detection waypoint coordinate set to determine a detection path of each second unmanned aerial vehicle, including:

[0031] The detection paths in each disease local area are planned based on the waypoint positions corresponding to each disease local area according to the principle of shortest path.

[0032] The disease local areas required to be detected by each second unmanned aerial vehicle are determined based on the distances between the initial positions of all the second unmanned aerial vehicles and the disease local areas.

[0033] The detection paths of each second unmanned aerial vehicle are determined based on the disease local areas corresponding to each second unmanned aerial vehicle and the detection paths in the corresponding disease local areas.

[0034] In an embodiment, the second unmanned aerial vehicles are used to detect all the disease local areas according to the respective detection paths to obtain apparent disease detection results, including:

[0035] The detection images of each disease local area are obtained by using the second unmanned aerial vehicles to detect all the disease local areas according to the respective detection paths.

[0036] The detection images are detected and classified by using a preset second deep learning network model to obtain apparent disease detection results.

[0037] The second aspect of the present application provides a multi-unmanned aerial vehicle bridge apparent disease detection system, including:

[0038] An information acquisition module is configured to acquire position information of a target bridge.

[0039] The disease local area detection module is configured to detect apparent diseases of the target bridge by using at least one first unmanned aerial vehicle based on the position information, and identify a plurality of disease local areas.

[0040] The detection waypoint solving module is configured to solve position coordinates of all the disease local areas, and obtain a detection waypoint coordinate set.

[0041] The local detection path planning module is configured to plan paths for a plurality of second unmanned aerial vehicles based on the detection waypoint coordinate set, and determine a detection path of each of the second unmanned aerial vehicles.

[0042] The disease detection and result output module is configured to detect all the disease local areas by using the second unmanned aerial vehicles according to the respective detection paths, and obtain an apparent disease detection result.

[0043] The third aspect of the present application provides an intelligent terminal, which comprises a memory, a processor, and a multi-unmanned aerial vehicle (UAV) bridge apparent disease detection program stored in the memory and executable on the processor.

[0044] The fourth aspect of the present application provides a computer readable storage medium, which stores a multi-UAV bridge apparent disease detection program.

[0045] Compared with the prior art, the present application has the following advantages:

[0046] The present application uses at least one first unmanned aerial vehicle to carry a high-resolution camera to quickly capture a large-scale bridge global image, uses a deep learning network model to identify disease local areas, and transmits position information of the disease local areas to a plurality of second unmanned aerial vehicles to control the unmanned aerial vehicles to accurately identify apparent diseases in the detected disease local areas. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0048] Figure 1 Progressive multi-unmanned aerial vehicle patrol bridge surface disease rapid detection method flowchart of the present application;

[0049] Figure 2 Progressive multi-unmanned aerial vehicle patrol bridge surface disease rapid detection flowchart of the present application;

[0050] Figure 3 Progressive multi-unmanned aerial vehicle patrol bridge surface disease rapid detection module schematic diagram of the present application;

[0051] Figure 4 Intelligent terminal structure schematic diagram of the present application. DETAILED DESCRIPTION

[0052] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0053] It should be understood that the term "comprising" as used in the specification and the appended claims indicates the presence of the recited features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0054] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms, unless the context clearly indicates otherwise.

[0055] It should be further understood that the term "and / or" as used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0056] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] Current drone inspections mainly focus on detecting apparent defects across the entire bridge. However, for a bridge operating normally, apparent defects will only occur in a few localized areas of the bridge. While a full-coverage inspection method can accurately detect defects, it suffers from issues such as resource redundancy and time-consuming inspections. The present invention addresses the issues of resource waste and low detection efficiency in existing methods for detecting apparent defects in bridges, and proposes a method for rapid detection of apparent defects across bridges using a progressive multi-drone patrol. Specifically, at least one large drone equipped with a high-resolution camera rapidly captures a large-scale global image of the bridge, uses a deep learning network model to identify apparent defect areas, and transmits the absolute coordinates of the area locations to multiple small drones. The drones are then controlled to approach the area, and the type and location of the defect are accurately identified. The progressive multi-drone patrol bridge apparent defect rapid detection solution adopted by the present invention only requires a single global inspection to obtain the defect layout area, and then only performs precise inspections on the localized defect areas, thereby improving detection efficiency and accuracy and effectively broadening the perspective of bridge inspection. This technology enables efficient, high-precision, and comprehensive inspection of bridge defects, effectively broadening the scope of bridge inspection. Compared to traditional drone-based full-coverage bridge inspections, this invention significantly reduces the time and cost of bridge inspections, improves their efficiency, and effectively supports the safe operation and maintenance of bridges.

[0059] The embodiment of the present invention provides a method for rapid detection of apparent defects of bridges by using multiple drones for patrolling. The method is deployed on electronic devices such as computers and servers and is applied to the scene of detecting apparent defects of bridge structures. Any solution for defect detection based on the concept of the present invention is also within the scope of protection of the present invention. Specifically, Figure 1 and Figure 2 As shown, the steps of the method in this embodiment include:

[0060] Step S100: Obtaining the absolute position information of the target bridge;

[0061] Specifically, a target bridge to be detected is determined, and the global navigation satellite system (GNSS) and the structural design drawing of the target bridge are used to obtain the absolute position coordinate information of the piers and the main girder corner points of the bridge.

[0062] Step S200: Based on the absolute position information, at least one first unmanned aerial vehicle (UAV) is used to detect the apparent disease of the target bridge, and a plurality of disease local areas are identified.

[0063] Specifically, based on the collected absolute position information, a large unmanned aerial vehicle (UAV) equipped with a high-precision global navigation satellite system (GNSS), an inertial measurement unit (IMU), a high-resolution visual camera, and an embedded system is used as the first UAV to perform high-altitude full-coverage bridge image shooting, covering the entire bridge structure to ensure that high-quality images and data can be obtained. Considering factors such as the geographical location, structural characteristics, and bridge size of the target bridge, the flight route of the UAV is planned to ensure that the shooting can cover the entire surface of the bridge structure. During image acquisition, the UAV and the visual camera are considered as the same platform, the visual camera extrinsic matrix is obtained in real time through the GNSS and the IMU, the visual camera intrinsic matrix and the scale factor are calibrated in advance, and the flight height and speed of the UAV are set to balance the image resolution and coverage range, and to avoid image blur or data loss; the images and data are transmitted in real time to the ground control station or the cloud server to ensure that the first UAV is in a safe flight and normal working state. After the images are collected, the collected images are preprocessed, such as denoising and enhancement, to improve the image quality and data accuracy, and then the first deep learning network model is used to detect the apparent disease of the processed images to identify a plurality of disease local areas. The first deep learning network model refers to a deep learning network model used for image processing or detection classification, which is not limited in the present embodiment.

[0064] Step S300: The position coordinates of all the disease local areas are calculated to obtain a detection waypoint coordinate set.

[0065] Specifically, since the absolute position coordinate information of the target bridge is the coordinate position in the geodetic coordinate system, and combined with the height data of the UAV, the data of the inertial measurement unit (IMU), and the like, the position coordinates of the disease local areas collected in the images are the coordinates in the three-dimensional coordinate system where the visual camera is located, and there is a deviation in the positions in the two coordinate systems. Therefore, in order to accurately represent the position of the disease local area, the present embodiment uses photogrammetry to calculate the position of the disease local area represented in the three-dimensional coordinate system where the camera is located to the geodetic coordinate system for representation, and uses all the calculated absolute position coordinates of the disease local areas to construct a coordinate set to obtain a detection waypoint coordinate set. The detection waypoint coordinate set contains the unique identifier and the geographic position coordinates of each disease local area.

[0066] Step S400: based on the detected waypoint coordinate set, path planning is performed for a plurality of second unmanned aerial vehicles to determine a detection path for each of the second unmanned aerial vehicles;

[0067] Specifically, after obtaining the detected waypoint coordinate set, the apparent disease in each disease local area needs to be detected. In order to improve the detection accuracy and efficiency, the embodiment selects a plurality of small unmanned aerial vehicles carrying a GNSS high-precision global positioning system, a portable camera and an embedded system as the second unmanned aerial vehicles to detect the apparent disease in all disease local areas. Then, according to the spatial distribution of the detected waypoint coordinate set, the density and distribution range of the disease local area are determined, and according to the performance parameters of the selected second unmanned aerial vehicles, such as flight speed, endurance time and load capacity, the flight height, speed and possible flight time of the second unmanned aerial vehicles during task execution are evaluated, so as to formulate the detection path planning principle, such as the principle of minimizing the sum of the detection paths of all second unmanned aerial vehicles, or the principle of minimizing the time for all second unmanned aerial vehicles to fly to each disease local area. Finally, based on the detected waypoint coordinate set to be detected, the number and initial position of the second unmanned aerial vehicles, the path planning is performed for each of the second unmanned aerial vehicles, so as to determine the detection path of each unmanned aerial vehicle, to ensure that each unmanned aerial vehicle can safely and efficiently complete the detection task, and the detection area of all second unmanned aerial vehicles can cover all disease local areas on the target bridge.

[0068] Step S500: using the second unmanned aerial vehicles to perform a round of detection on all the disease local areas according to the respective detection paths to obtain the apparent disease detection results.

[0069] Specifically, the second unmanned aerial vehicles are controlled to navigate and fly close to the detected disease local areas according to the preset detection paths. In the flight process, images of the detected disease local areas are collected in real time to obtain a plurality of local area images, and the collected local area images are preprocessed, such as denoising and enhancement, to improve the image quality. Then, the processed images are subjected to disease identification using the trained second deep learning network model to identify the type, position and severity of the apparent disease in each disease local area and other disease information. The first deep learning network model refers to a deep learning network model used for image processing or detection classification, which is not specifically limited in the embodiment.

[0070] Further, the detected disease information can also be counted and analyzed according to the specific application requirements to understand the overall disease condition and distribution law of the bridge, and a detection plan is formulated according to the detection results to regularly inspect and evaluate the bridge to ensure the safe operation of the bridge.

[0071] In this embodiment, a large unmanned aerial vehicle is used to carry high-resolution cameras and other instruments to quickly take images of the entire bridge, and deep learning is used to identify local areas with apparent diseases on the bridge. Through coordinate mapping, the absolute position coordinates can be calculated, which can quickly locate the potential disease area and reduce the time and resource consumption of overall detection. At the same time, multiple unmanned aerial vehicles are used to patrol the local area, and deep learning is used to accurately identify the disease detection results. Through progressive patrol detection, the unmanned aerial vehicle can quickly capture the apparent disease area of the bridge, effectively make up for the lack of resource redundancy in traditional unmanned aerial vehicle bridge detection, and significantly improve the efficiency and accuracy of bridge apparent disease detection.

[0072] In one embodiment, the step S200 includes:

[0073] Step S210: determining the initial shooting distance and the area of the region to be detected of the target bridge based on the absolute position information;

[0074] Specifically, according to the absolute position coordinate information of the piers and the main beam corner points of the target bridge and various parameter configurations of the first unmanned aerial vehicle, the initial shooting distance and the area of the region to be detected of the target bridge are determined to ensure that the surface of the target bridge can be shot, thereby laying a good foundation for ensuring the completeness of the shooting data and the safety of the unmanned aerial vehicle.

[0075] Step S220: obtaining the field of view, pixel and focal length of the shooting camera, and determining the optimal shooting distance based on the initial shooting distance, the area of the region to be detected, the field of view, the pixel and the focal length;

[0076] Specifically, the function of the initial shooting distance d and the spatial resolution r is established according to the camera imaging principle. Assuming that the pixel size of the camera sensor is s and the focal length is f, the relationship between d and r is as follows, thereby determining the spatial resolution.

[0077]

[0078] Based on the initial shooting distance and the field of view, the camera field of view range is determined. Specifically, assuming that the horizontal field of view of the camera is FOV v , the vertical field of view of the camera is FOV h , and the physical distance range DOV of the image field of view under the initial shooting distance d is calculated:

[0079]

[0080] wherein, DOV v represents the horizontal distance of the visual camera field of view, and DOVv Indicates the vertical distance of the visual camera's field of view.

[0081] Calculate the coverage of the area to be detected and the field of view of the visual camera. Specifically, when the fields of view do not overlap, the visual camera is used to calculate the coverage of the area to be detected and the field of view of the visual camera. v and DOV h Taking the limited field of view as the standard, calculate the number of shooting positions required to cover the entire inspection area of ​​the target bridge, that is, Among them, S bridge It represents the size of the bridge area to be inspected, that is, the area of ​​the area to be inspected. As can be seen, the larger the camera's field of view, the fewer positions need to be captured.

[0082] Since the horizontal distance DOV of the visual camera's field of view v , vertical distance of visual camera field of view DOV h The spatial resolution r of the visual camera is expressed by the initial shooting distance d. Therefore, the objective function ∈ can be constructed based on the area of ​​the area to be detected, the camera field of view and the spatial resolution, that is:

[0083]

[0084] Among them, ω and ρ are weights.

[0085] Taking both shooting efficiency and resolution into account, a genetic algorithm is used to optimize the initial shooting distance d with the goal of minimizing the objective function ∈, minimizing the number of shots required and minimizing the spatial resolution. The optimal shooting distance is determined. The lower the resolution, the clearer the image. This allows for efficient completion of the shooting task and ensures that the captured images meet high precision requirements.

[0086] Step S230: determining an aerial photography path of the first UAV based on the absolute position information, the area of ​​the area to be detected, the field of view angle, and the optimal shooting distance;

[0087] Specifically, after determining the optimal shooting distance, the number of shots is determined based on the area of ​​the area to be detected, the field of view angle and the optimal shooting distance, that is, by The number of photographing times is calculated, and based on the number of photographing times and the absolute position information, the number of positions required to be photographed by the first unmanned aerial vehicle is determined, that is, how many times the first unmanned aerial vehicle needs to photograph to cover the entire area corresponding to the area to be detected, and based on the number of positions required to be photographed, the first unmanned aerial vehicle is controlled to photograph at appropriate waypoints without photographing at each waypoint, thereby improving the detection efficiency. It is worth stating that usually in the stage of identifying the disease area of the apparent disease of the bridge, one first unmanned aerial vehicle can be used to achieve the purpose, if it is desired to improve the detection efficiency, multiple first unmanned aerial vehicles can be selected to simultaneously identify the disease area according to actual needs, correspondingly, the aerial photography path of the first unmanned aerial vehicle can be planned according to the number of first unmanned aerial vehicles and the calculated number of photographing times, so as to ensure that the working time of each first unmanned aerial vehicle is approximately or the same, so as to shorten the total detection time and maximize the detection efficiency.

[0088] The number of positions required to be photographed by the unmanned aerial vehicle is calculated according to the range of the photographing field of view, and under the premise of ensuring flight safety, the path waypoints of the first unmanned aerial vehicle are interpolated through the absolute position coordinate information of the corner points, and the specific interpolation rule method is to connect adjacent corner points to form a straight line, and to solve the functional expression relationship of the straight line, and to select corresponding waypoints in the straight line to complete the interpolation according to the flight task of the unmanned aerial vehicle. Then, the interpolated waypoints are connected in turn to form an aerial photography path, and the aerial photography path is transmitted to the first unmanned aerial vehicle to complete the route planning.

[0089] Step S240: photographing the global area of the target bridge by the first unmanned aerial vehicle along the corresponding aerial photography path to obtain a global image of the bridge;

[0090] Specifically, the global image of the bridge is obtained by photographing the global area of the target bridge by the first unmanned aerial vehicle along the corresponding aerial photography path, and the global image of the bridge is transmitted to the embedded system carried by the first unmanned aerial vehicle through wired or wireless connection transmission, and the image is divided into a plurality of mutually independent rectangular local areas to improve the detection efficiency of deep learning.

[0091] Step S250: detecting the apparent disease of the global image of the bridge by using a preset first deep learning network model to identify a plurality of disease local areas.

[0092] Specifically, before the apparent disease detection of the local area image, the local area image is preprocessed by using filtering and noise reduction, sub-pixel algorithm and the like to enhance the quality of the local area image. Then, the preprocessed local area image is input into the pre-trained first deep learning network model (such as ResNet deep learning neural network model) to identify whether the local area exists disease. The model is pre-trained by performing two classification training on the public bridge apparent disease data set, taking the bridge image as the input and whether the apparent disease exists as the output, thereby identifying a plurality of disease local areas.

[0093] In this embodiment, the detection path of the first unmanned aerial vehicle is planned according to the position information of the target bridge and the internal and external parameters of the visual camera carried by the first unmanned aerial vehicle, and the first unmanned aerial vehicle is used to detect the apparent disease of the region corresponding to the area to be detected of the target bridge, so as to quickly and accurately identify the local disease area.

[0094] In an embodiment, the position coordinates of all the local disease areas in the step S300 are calculated to obtain a detection waypoint coordinate set, including:

[0095] Step S310: calculating the position coordinates of the local disease area to obtain the absolute coordinates of each local disease area;

[0096] Step S320: determining the waypoint position based on the absolute coordinates of the local disease area and the preset minimum safe shooting distance;

[0097] Step S330: obtaining a detection waypoint coordinate set based on the waypoint position.

[0098] Specifically, first, the top pixel coordinates p P = [(u lt , v lt ), (u rt , v rt ), (u lb , v lb ), (u rb , v rb )] of the local disease area are read, where u represents the horizontal pixel coordinate, v represents the vertical pixel coordinate, lt represents the upper left corner, rt represents the upper right corner, lb represents the lower left corner, and rb represents the lower right corner.

[0099] When shooting the global image of the bridge, the unmanned aerial vehicle and the camera are considered as the same platform, the camera external parameter matrix is obtained in real time through the GNSS global positioning system and the inertial measurement unit IMU, including the rotation matrix R and the translation vector T, the camera internal parameter matrix K is calibrated in advance, and the scale factor z c .

[0100] Then, the position coordinates of the local disease area are calculated by using the basic principle of photogrammetry. According to the conversion relationship between the pixel space coordinate system and the three-dimensional coordinate system in which the visual camera is located shown in formula (5), the top pixel coordinates p P of the local area are converted into the coordinates p c in the three-dimensional coordinate system in which the visual camera is located, and formula (5) is as follows:

[0101] p P ·z c = K·pc (5)

[0102] And according to the conversion relationship between the three-dimensional coordinate system where the visual camera is located and the geodetic coordinate system shown in formula (6), the coordinates of the local area vertex camera coordinate system p c are converted into absolute coordinates p w = [(x lt ,y lt ,z lt ), (x rt ,y rt ,z rt ), (x lb ,y lb ,z lb ), (x rb ,y rb ,z rb )], formula (6) is as follows:

[0103] p w =R·p c +T (6)

[0104] Wherein, x represents the absolute horizontal coordinate, y represents the absolute vertical coordinate, and z represents the absolute depth coordinate.

[0105] Finally, based on the absolute coordinates of the disease local area and the preset minimum safe shooting distance, the waypoint position is determined, and based on the waypoint position, the detection waypoint coordinate set is obtained.

[0106] Suppose there are M bridge disease local areas, and the vertex coordinates of each disease local area are arranged in turn to obtain a disease area set U a , The absolute position coordinates of the disease area set U a are used as the reference waypoint of the second unmanned aerial vehicle, in order to ensure the clarity and resolution of the image, the waypoint position is adjusted according to the minimum safe distance between the unmanned aerial vehicle and the pier and the main beam, and the detection waypoint coordinate set U b ,

[0107] In this embodiment, the position coordinates of the disease local area are calculated by using the basic principle of photogrammetry, the pixel coordinates of the disease local area are first converted into the three-dimensional coordinate system where the visual camera is located, and then converted into the geodetic coordinate system, which can ensure the effectiveness of the coordinate conversion and is beneficial to improve the accuracy of each position coordinate in the detection waypoint coordinate set.

[0108] In one embodiment, the detection path of each second unmanned aerial vehicle is determined by performing path planning on a plurality of second unmanned aerial vehicles based on the detection waypoint coordinate set in step S400, comprising:

[0109] Step S410: connecting the waypoint positions corresponding to each of the disease local areas according to the principle of shortest path, and planning the detection path in each of the disease local areas based on the detected waypoint coordinate set;

[0110] Step S420: determining the disease local area required to be detected by each of the second unmanned aerial vehicles based on the distance between the initial position of each of the second unmanned aerial vehicles and the disease local area;

[0111] Step S430: determining the detection path of each of the second unmanned aerial vehicles based on the corresponding disease local area of each of the second unmanned aerial vehicles and the detection path in the corresponding disease local area.

[0112] Specifically, since the disease local area is obtained from the equal-range bridge global image, the time efficiency of detecting a disease local area by each second unmanned aerial vehicle is consistent. Therefore, the embodiment mainly optimizes the detection paths of multiple second unmanned aerial vehicles according to the time of flying to the local area. Specifically, assuming that several second unmanned aerial vehicles are parked along the main beam axis in sequence, the flight distance between the unmanned aerial vehicle and the local area is calculated by using the Euclidean distance, and the detection path of each second unmanned aerial vehicle is planned according to the principle of shortest flight distance to the local area. b The corresponding waypoints p wb of each local area are connected in sequence as an air route according to the principle of shortest flight distance to the local area, the detection path in each of the disease local areas is planned, the local area path planning is formed, and the task allocation is realized. Detecting a disease local area by a single second unmanned aerial vehicle is regarded as a detection task, and the execution time of the task is estimated according to the performance of the unmanned aerial vehicle.

[0113] In the embodiment, the inspection path of each second unmanned aerial vehicle is planned according to the principle of shortest path based on the initial position of each second unmanned aerial vehicle and the detection waypoints in the disease local area, which can effectively improve the detection efficiency and the rationality of task division.

[0114] In an embodiment, the step S500 of using the second unmanned aerial vehicles to perform the patrol detection on all the disease local areas according to the respective detection paths to obtain the apparent disease detection result comprises:

[0115] Step S510: using the second unmanned aerial vehicles to perform the patrol detection on all the disease local areas according to the respective detection paths to obtain the detection image of each of the disease local areas;

[0116] Step S520: detecting and classifying all the detection images by using a preset second deep learning network model to obtain the apparent disease detection result.

[0117] Specifically, the bridge picture is transmitted to the embedded system carried by the unmanned aerial vehicle through wired or wireless data transmission connection, and the size of the image is adjusted to the required size of the model using an interpolation method. A filter is used to smooth the image to remove noise and enhance image quality. The processed image data is input into a preset second deep learning network model (such as a single-stage target detection algorithm (YOLOv5) model) one by one to complete the bridge disease detection task. The model is pre-trained through multi-classification on a public bridge surface disease data set. The input of the model is a bridge image, and the output of the model is a multi-dimensional data structure containing specific type and accurate position information of the disease. Specifically, the output provides a bounding box for each detected disease area, which accurately marks the position and type of the disease in the image. After detecting the specific position and type of the bridge surface disease, the pixel coordinates of the vertices of the bounding box are converted according to the above formula (5) and formula (6) using the basic principles of photogrammetry to accurately identify the absolute position of the disease, thereby achieving accurate and efficient detection of bridge surface diseases. It should be noted that the first deep learning network model and the second deep learning network model in the present application are not limited to ResNet model and YOLOv5 model. Other deep learning network models that can be used for image detection and classification, such as Faster R-CNN and UNet, can be selected flexibly according to the actual detection accuracy, cost, efficiency, and computational requirements.

[0118] In the present embodiment, the vertex position coordinates of the disease local area are used to plan efficient, high-precision, and all-around multi-unmanned aerial vehicle patrol detection of intelligent bridge surface diseases, effectively widening the perspective of bridge detection. Compared with traditional unmanned aerial vehicle full-coverage bridge detection, the present application significantly reduces the time cost of bridge detection and improves the efficiency of bridge detection, effectively supporting the safe operation and maintenance of bridges.

[0119] As shown in Figure 3 Corresponding to the above progressive multi-unmanned aerial vehicle patrol bridge surface disease rapid detection method, the present application embodiment also provides a multi-unmanned aerial vehicle patrol bridge surface disease detection system, which comprises:

[0120] An information acquisition module 310 is configured to acquire position information of a target bridge;

[0121] A disease local area detection module 320 is configured to detect surface diseases of the target bridge using at least one first unmanned aerial vehicle based on the position information, and identify a plurality of disease local areas;

[0122] A detection waypoint solving module 330 is configured to solve the position coordinates of all the disease local areas to obtain a set of detection waypoint coordinates;

[0123] The local detection path planning module 340 is configured to plan paths for a plurality of second unmanned aerial vehicles based on the set of detection waypoint coordinates, and determine a detection path for each of the second unmanned aerial vehicles.

[0124] The disease detection and result output module 350 is configured to detect all the local disease areas according to the respective detection paths by the second unmanned aerial vehicles, and obtain apparent disease detection results.

[0125] Specifically, the specific functions of the above multi-unmanned aerial vehicle bridge apparent disease detection system in the embodiment can also refer to the corresponding descriptions in the above progressive multi-unmanned aerial vehicle bridge apparent disease rapid detection method, and will not be described here.

[0126] Based on the above embodiment, the present application further provides an intelligent terminal, and a principle block diagram thereof can be shown as follows. Figure 4 The intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the intelligent terminal is configured to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a multi-unmanned aerial vehicle bridge apparent disease detection program. The internal memory provides an environment for the operating system and the multi-unmanned aerial vehicle bridge apparent disease detection program based on the multi-unmanned aerial vehicle bridge apparent disease detection program. The network interface of the intelligent terminal is configured to communicate with external terminals through network connection. The multi-unmanned aerial vehicle bridge apparent disease detection program, when executed by the processor, implements the steps of any one of the above progressive multi-unmanned aerial vehicle bridge apparent disease rapid detection methods. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0127] Those skilled in the art can understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the intelligent terminal to which the present application is applied. The specific intelligent terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0128] In one embodiment, an intelligent terminal is provided, and the above intelligent terminal includes a memory, a processor, and a multi-unmanned aerial vehicle bridge apparent disease detection program stored on the above memory and executable on the above processor. The multi-unmanned aerial vehicle bridge apparent disease detection program, when executed by the processor, implements the steps of any one of the above progressive multi-unmanned aerial vehicle bridge apparent disease rapid detection methods provided by the present application.

[0129] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a multi-robot bridge surface disease detection program. The multi-robot bridge surface disease detection program is executed by a processor to realize the steps of any one of the progressive multi-robot bridge surface disease rapid detection methods provided by the embodiment of the present application.

[0130] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the execution sequence, and the execution sequence of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the above described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the above method embodiments, which will not be described here.

[0132] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0133] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0134] In the embodiments of the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely schematic, and the division of the above modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0135] The above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A rapid detection method for bridge surface defects using progressive multi-UAV patrols, characterized in that: The following steps are involved: Obtain the absolute position information of the target bridge; Based on the absolute position information, using at least one first drone to perform apparent disease detection on a global area of ​​the target bridge, and identifying a plurality of local disease areas; Calculating the position coordinates of all the local diseased areas to obtain a detection waypoint coordinate set; performing path planning for a plurality of second drones based on the detection waypoint coordinate set to determine a detection path for each of the second drones; Utilizing the second drone to conduct patrol inspections on all the local diseased areas along respective inspection routes to obtain apparent disease detection results; The method of using the second drone to conduct patrol inspections on all the local diseased areas according to respective inspection paths to obtain apparent disease inspection results includes: Utilizing the second drone to perform patrol inspections on all the local diseased areas according to respective inspection paths, and obtaining inspection images of each local diseased area; All the detection images are detected and classified using a preset second deep learning network model to obtain apparent disease detection results.

2. The method for rapid detection of bridge surface defects by progressive multi-UAV patrol according to claim 1 is characterized in that: Based on the absolute position information, at least one first UAV is used to perform apparent disease detection on the global area of ​​the target bridge to identify a number of local disease areas, including: Based on the absolute position information, determining the initial shooting distance and the area of ​​the area to be detected of the target bridge; Obtaining the field of view angle, pixel, and focal length of the shooting camera, and determining the optimal shooting distance based on the initial shooting distance, the area of ​​the area to be detected, the field of view angle, the pixel, and the focal length; Determining an aerial photography path of the first UAV based on the absolute position information, the area of ​​the area to be detected, the field of view angle, and the optimal shooting distance; Using the first UAV to photograph the entire area of ​​the target bridge along the corresponding aerial photography path to obtain a global image of the bridge; A preset first deep learning network model is used to perform apparent disease detection on the global image of the bridge, and a number of local diseased areas are identified.

3. The method for rapid detection of bridge surface defects by progressive multi-UAV patrol according to claim 2 is characterized in that: The determining of the optimal shooting distance based on the initial shooting distance, the area of ​​the region to be detected, the field of view angle, the pixel, and the focal length includes: Determining a spatial resolution based on the initial shooting distance, the pixel, and the focal length; Determining a camera field of view based on the initial shooting distance and the field of view angle; Constructing an objective function based on the area of ​​the area to be detected, the field of view of the camera, and the spatial resolution; With the goal of minimizing the objective function, the initial shooting distance is optimized to determine the optimal shooting distance.

4. The method for rapid detection of bridge surface defects by progressive multi-UAV patrol according to claim 2 is characterized in that: The determining of the aerial photography path of the first UAV based on the absolute position information, the area of ​​the area to be detected, the field of view angle, and the optimal shooting distance includes: Determining the number of shots based on the area of ​​the area to be detected, the field of view angle, and the optimal shooting distance; Determining the number of positions to be photographed based on the number of photographing times and the absolute position information; Based on the number of locations to be photographed, an aerial photography path of the first UAV is determined.

5. The method for rapid detection of bridge surface defects by progressive multi-UAV patrol according to claim 1 is characterized in that: The step of calculating the position coordinates of all the local diseased areas to obtain a detection waypoint coordinate set includes: Calculating the position coordinates of the local diseased area to obtain the absolute coordinates of each local diseased area; Determining a waypoint location based on the absolute coordinates of the local area of ​​the disease and a preset minimum safe shooting distance; Based on the waypoint positions, a detection waypoint coordinate set is obtained.

6. The method for rapid detection of bridge surface defects by progressive multi-UAV patrol according to claim 5 is characterized in that: The performing path planning for the plurality of second UAVs based on the detection waypoint coordinate set to determine a detection path for each second UAV includes: Based on the detection waypoint coordinate set, the waypoint positions corresponding to each of the local disease areas are connected according to the principle of the shortest path, and the detection path within each of the local disease areas is planned; Determining the local disease area to be inspected by each second drone based on the distances between the initial positions of all second drones and the local disease area; Based on the local disease area corresponding to each second drone and the detection path within the corresponding local disease area, the detection path of each second drone is determined.

7. The multi-UAV patrol bridge apparent disease detection system is characterized by: The system is applied to implement the steps of the rapid detection method according to any one of claims 1 to 6, comprising: An information acquisition module is used to obtain the absolute position information of the target bridge; a local defect area detection module, configured to perform apparent defect detection on the global area of ​​the target bridge using at least one first UAV based on the absolute position information, and identify a plurality of local defect areas; A detection waypoint solving module is used to solve the position coordinates of all the local disease areas to obtain a detection waypoint coordinate set; a local detection path planning module, configured to perform path planning for a plurality of second UAVs based on the detection waypoint coordinate set, and determine a detection path for each of the second UAVs; The disease detection and result output module is used to use the second drone to perform patrol inspections on all the local disease areas according to their respective detection paths to obtain apparent disease detection results.

8. Intelligent terminal, characterized in that, The intelligent terminal includes a memory, a processor, and a multi-UAV patrol bridge apparent disease detection program stored in the memory and runnable on the processor. When the multi-UAV patrol bridge apparent disease detection program is executed by the processor, the steps of the progressive multi-UAV patrol bridge apparent disease rapid detection method as described in any one of claims 1-6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program for detecting apparent defects of bridges patrolled by multiple UAVs. When the program is executed by a processor, the steps of the progressive method for rapid detection of apparent defects of bridges patrolled by multiple UAVs are implemented as described in any one of claims 1 to 6.

Images