Bridge appearance inspection method and system based on drone

Abstract

The present invention relates to the field of image processing technology, and discloses a method and system for detecting the appearance of a bridge based on a drone, so as to improve the efficiency of detection and the reliability of before-and-after comparison evaluation and tracking of repair effects. The existing unmanned driving technology of binocular vision is introduced into the bridge monitoring operation based on drones, and the collected images are ensured to be roughly in line with expectations by taking pictures at substantially the same shooting points and the same navigation posture, and then the average distance between the pixel points in the rectangular window of the current image and the drone is compared with the average distance recorded before repair to confirm whether the current collected image is in line with expectations. In essence, automatic verification based on machine vision is realized. Furthermore, when the automatic verification does not meet the set requirements, it can be supplemented by manual intervention, thereby improving the intelligence and flexibility of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for detecting the appearance of a bridge based on an unmanned aerial vehicle. Background Art

[0002] Bridge inspection refers to the work of investigating and studying bridges, systematically understanding their use status, formulating application conditions, and proposing maintenance or reinforcement measures to ensure transportation safety. Bridge inspection also creates conditions for accumulating technical data, improving bridge calculation theory, strengthening scientific technical management, and improving bridge technology level.

[0003] Usually, the main contents of bridge inspection are:

[0004] 1. Routine periodic inspection: including bridge deck inspection, superstructure inspection, and substructure inspection.

[0005] 2. Regular structural inspection: including concrete strength inspection, concrete carbonization depth inspection, steel bar position and concrete cover thickness inspection.

[0006] 3. Underwater component inspection: Inspect and record the underwater pile foundation concrete for defects such as falling off, cracks, exposed reinforcement, voids, mechanical damage, etc.

[0007] 4. Bearing capacity assessment: The bearing capacity assessment is used to determine whether the bearing capacity of the bridge at this stage can meet the design requirements.

[0008] 5. Arrangement of long-term monitoring points and initial observation: In order to observe the displacement of bridge piers and main beams under the action of vehicles over a long period of time, and thus analyze the safety of the bridge, monitoring points are arranged at key positions of the bridge and the initial observation is carried out on the monitoring points.

[0009] 6. Submit the final bridge inspection report for each bridge, the content of which shall comply with the requirements of the industry standard of the People's Republic of China "Technical Specifications for Urban Bridge Maintenance" CJJ99-2003. In addition to the above content, the report should also include opinions on the setting of load limit, height limit and other signs for each bridge.

[0010] In the existing bridge appearance inspection process, a special vehicle can be used to control the mechanical arm to carry out manual work on the workbench. However, there are disadvantages such as low efficiency and high cost of supporting facilities. At present, drone aerial survey technology is gradually being used in the disease detection process of transportation infrastructure. It can quickly complete the image acquisition of the existing surface information of the object based on manual remote control; thus, it provides great convenience for the daily maintenance and inspection of bridges.

[0011] On the other hand, before and after the existing bridge repair, pairing and comparing the pictures before and after the bridge repair is conducive to objectively evaluating and tracking the repair effect. However, the existing inspection reports and data collection and storage are mostly based on manual methods, and some collected images also have problems such as high similarity between different areas and inconsistent field of view before and after repair of the same area. As a result, it is easy to make pairing errors in the process of manually pairing the manually collected image data, which affects the reliability of evaluating and tracking the repair effect. Summary of the invention

[0012] The present invention aims to disclose a method and system for detecting the appearance of a bridge based on an unmanned aerial vehicle, so as to improve the efficiency of detection and the reliability of before-and-after comparative evaluation and tracking of repair effects.

[0013] To achieve the above-mentioned purpose, the present invention discloses a bridge appearance detection method based on a drone, comprising:

[0014] A drone equipped with a binocular camera is used to take photos of each local area of ​​the bridge to be repaired; and in the process of taking photos of any local area, the pixels in the central rectangular window of the left view are matched with the right view, and the average distance between each pixel point in the rectangular window and the drone is calculated according to the matched disparity data, and then the corresponding relationship between the average distance and the corresponding left view, and the earth coordinate position information and navigation attitude information of the shooting point where the drone is located is saved; wherein the specification of the number of pixels in the rectangular frame is a uniformly set default value;

[0015] After the bridge is repaired, the navigation route of the UAV is planned according to the distribution information of the earth coordinates corresponding to each previously repaired area of ​​the bridge; and after the UAV arrives at any photo taking point according to the planned navigation route, the corresponding navigation attitude information is found from the corresponding relationship to take photos of the left and right views according to the corresponding navigation attitude information; then the pixels in the central rectangular window of the current left view are matched with the right view, and the average distance between each pixel in the rectangular window and the UAV is calculated according to the matched disparity data; and the error between the current average distance and the corresponding average distance before repair in the corresponding relationship is calculated to be within the set threshold range. If so, it is determined that The image taken before meets the requirements, and the left view of the local area taken this time is paired with the left view before restoration and stored in the same data block. If it exceeds the set threshold range, the human operator is notified to adjust the earth coordinate position and / or navigation attitude information of the photo point where the drone is located to obtain a photo image that meets the requirements of the operating user, and while the left view taken this time by manual intervention is paired with the left view before restoration and stored in the same data block, the earth coordinate position information and navigation attitude information of the photo point where the drone is located that were previously saved in the corresponding relationship are replaced with the earth coordinate position and / or navigation attitude information of the photo point where the drone is located after the manual adjustment for subsequent inspections.

[0016] As an equivalent replacement, the present invention also discloses a bridge appearance detection method based on a drone, comprising:

[0017] A drone equipped with a binocular camera is used to take photos of each local area of ​​the bridge to be repaired; and in the process of taking photos of any local area, the pixels in the central rectangular window of the right view are matched with the left view, and the average distance between each pixel point in the rectangular window and the drone is calculated according to the matched disparity data, and then the corresponding relationship between the average distance and the corresponding right view, and the earth coordinate position information and navigation attitude information of the shooting point where the drone is located is saved; wherein the specification of the number of pixels in the rectangular frame is a uniformly set default value;

[0018] After the bridge is repaired, the navigation route of the UAV is planned according to the distribution information of the earth coordinates corresponding to the various previously repaired areas of the bridge; and after the UAV arrives at any photo taking point according to the planned navigation route, the corresponding navigation attitude information is searched from the corresponding relationship to take photos of the left and right views according to the corresponding navigation attitude information; then the pixels in the central rectangular window of the current right view are matched with the left view, and the average distance between each pixel in the rectangular window and the UAV is calculated according to the matched disparity data; and the error between the currently obtained average distance and the corresponding average distance before repair in the corresponding relationship is calculated to be within the set threshold range. If so, it is determined that The image taken before meets the requirements, and the right view of the local area taken this time is paired with the right view before restoration and stored in the same data block. If it exceeds the set threshold range, the human operator is notified to adjust the earth coordinate position and / or navigation attitude information of the photo point where the UAV is located to obtain a photo image that meets the requirements of the operating user, and while the right view taken this time by manual intervention is paired with the right view before restoration and stored in the same data block, the earth coordinate position information and navigation attitude information of the photo point where the UAV is located that were previously saved in the corresponding relationship are replaced with the earth coordinate position and / or navigation attitude information of the photo point where the UAV is located after the manual adjustment for subsequent inspection.

[0019] Furthermore, the present invention also discloses a drone-based bridge appearance inspection system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps corresponding to the above method when executing the computer program.

[0020] The present invention has the following beneficial effects:

[0021] The existing binocular vision unmanned driving technology is introduced into the UAV-based bridge monitoring operation, and the collected images are roughly in line with expectations by taking the same photo point and the same navigation posture. Then, the average distance between the pixel points in the current image rectangular window and the UAV is compared with the average distance recorded before the repair to confirm whether the current collected image meets expectations. In essence, automatic verification based on machine vision is realized. For comparison: before and after the repair, some pixel values ​​in the images before and after the repair of the same area usually have mutations, so it is impossible to determine whether the current collected image meets expectations by the approximation of the images before and after the repair. At the same time, if the secondary verification is omitted, even with GPS positioning plus base station positioning, WIFI and other auxiliary positioning, there will be objectively small deviations between the position positioning of the front and rear photo points, and the small deviation of the UAV position point is easy to cause a huge deviation between the collected image and the expected collection area. Therefore, under automatic inspection, only the distance-based secondary verification can ensure that the collected image meets expectations. Furthermore, when the automatic verification does not meet the set requirements, it can be supplemented by manual intervention, which improves the intelligence and flexibility of the system.

[0022] Moreover, this embodiment uses a rectangular window for local image matching, which greatly improves the accuracy and real-time performance of data processing compared to existing technologies such as unmanned driving based on binocular vision, which requires matching the entire image. At the same time, during the data storage process, only the left view or the right view is selected for storage, saving storage space.

[0023] Thereby, this embodiment improves the efficiency of detection by taking multiple measures simultaneously and improves the reliability, intelligence and flexibility of before-and-after comparison evaluation and tracking of repair effects.

[0024] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 It is a flow chart of a bridge appearance detection method based on drone disclosed in an embodiment of the present invention.

[0027] Figure 2 This is a flow chart of another method for detecting bridge appearance based on a drone disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0029] Example 1

[0030] This embodiment discloses a bridge appearance detection method based on a drone, such as Figure 1 As shown, the following steps are included:

[0031] Step S1, taking photos of each local area of ​​the bridge to be repaired with a drone equipped with a binocular camera; and in the process of taking photos of any local area, matching the pixels in the central rectangular window of the left view with the right view, calculating the average distance between each pixel point in the rectangular window and the drone based on the matched disparity data, and then saving the corresponding relationship between the average distance and the corresponding left view, as well as the earth coordinate position information and navigation attitude information of the shooting point where the drone is located; wherein the specification of the number of pixels in the rectangular frame is a uniformly set default value.

[0032] In this step, the earth coordinate location information of the drone's photo-taking point can be used to assist in GPS navigation and positioning, combined with base station positioning, wifi positioning, etc. to improve positioning accuracy. The so-called "earth coordinate location information" refers to general information that can reflect the position relationship of each point in the geographic coordinate system and is convenient for other devices to directly call. The geographic coordinate system usually refers to a spherical coordinate system that uses longitude and latitude to represent ground points.

[0033] Optionally, in this step, the navigation attitude information includes LBS position information, navigation azimuth and pitch angle information. Among them, the specification of the number of pixels in the rectangular frame can be reasonably set according to the resolution and experience value, for example, it can be specifically set to 25*25. Usually, the pixel information in the rectangular frame records specific disease characteristics. Under the effect of calculating the average distance between the pixels in the rectangular frame and the drone, by reasonably setting the number of pixels in the rectangular frame, it can be ensured that in the repair process of cracks and exposed steel bars, the change in depth information within a certain range before and after the repair will not affect the accurate judgment of whether the subsequent collected images meet the requirements in step S2.

[0034] Step S2, after the bridge is repaired, the navigation route of the UAV is planned according to the distribution information of the earth coordinates corresponding to each previously repaired area of ​​the bridge; and after the UAV arrives at any photo taking point according to the planned navigation route, the corresponding navigation attitude information is searched from the corresponding relationship to take photos of the left and right views according to the corresponding navigation attitude information; then the pixels in the central rectangular window of the current left view are matched with the right view, and the average distance between each pixel in the rectangular window and the UAV is calculated according to the matched disparity data; and the error between the current average distance and the corresponding average distance before repair in the corresponding relationship is calculated to be within the set threshold range, if so, It is determined that the currently photographed image meets the requirements, and the left view of the local area photographed this time is paired with the left view before restoration and stored in the same data block. If it exceeds the set threshold range, the human operator is notified to adjust the earth coordinate position and / or navigation attitude information of the photographing point where the drone is located to obtain a photo image that meets the requirements of the operating user. While the left view photographed by this manual intervention is paired with the left view before restoration and stored in the same data block, the earth coordinate position information and navigation attitude information of the photographing point where the drone is located that were previously saved in the corresponding relationship are replaced with the earth coordinate position and / or navigation attitude information of the photographing point where the drone is located after the manual adjustment for subsequent inspection.

[0035] In this step, whether the current image meets the expected threshold range can be reasonably set according to statistical experience values ​​in daily applications to ensure the reliability of automatic verification.

[0036] Example 2

[0037] As an equivalent replacement for the above-mentioned embodiment 1, this embodiment discloses a bridge appearance detection method based on a drone, such as Figure 2 As shown, including:

[0038] Step S3, taking photos of each local area of ​​the bridge to be repaired with a drone equipped with a binocular camera; and in the process of taking photos of any local area, matching the pixels in the central rectangular window of the right view with the left view, calculating the average distance between each pixel point in the rectangular window and the drone based on the matched disparity data, and then saving the correspondence between the average distance and the corresponding right view, as well as the earth coordinate position information and navigation attitude information of the shooting point where the drone is located; wherein the specification of the number of pixels in the rectangular frame is a uniformly set default value.

[0039] Similarly, the navigation attitude information of this step includes LBS position information, navigation azimuth and pitch angle information.

[0040] Step S4, after the bridge is repaired, the navigation route of the UAV is planned according to the distribution information of the earth coordinates corresponding to each previously repaired area of ​​the bridge; and after the UAV arrives at any photo taking point according to the planned navigation route, the corresponding navigation attitude information is searched from the corresponding relationship to take photos of the left and right views according to the corresponding navigation attitude information; then the pixels in the central rectangular window of the current right view are matched with the left view, and the average distance between each pixel point in the rectangular window and the UAV is calculated according to the matched disparity data; and the error between the current average distance and the corresponding average distance before repair in the corresponding relationship is calculated to be within the set threshold range, if so, It is determined that the currently photographed image meets the requirements, and the right view of the local area photographed this time is paired with the right view before restoration and stored in the same data block. If it exceeds the set threshold range, the human operator is notified to adjust the earth coordinate position and / or navigation attitude information of the photographing point where the UAV is located to obtain a photo image that meets the requirements of the operating user. While the right view photographed by this manual intervention is paired with the right view before restoration and stored in the same data block, the earth coordinate position information and navigation attitude information of the photographing point where the UAV is located that were previously saved in the corresponding relationship are replaced with the earth coordinate position and / or navigation attitude information of the photographing point where the UAV is located after the manual adjustment for subsequent inspection.

[0041] Example 3

[0042] This embodiment discloses a bridge appearance inspection system based on a drone, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps corresponding to the method described in the above-mentioned embodiment 1 or embodiment 2 when executing the computer program.

[0043] In summary, the UAV-based bridge appearance detection method and system disclosed in the above embodiments of the present invention have at least the following beneficial effects:

[0044] The existing binocular vision unmanned driving technology is introduced into the UAV-based bridge monitoring operation, and the collected images are roughly in line with expectations by taking the same photo point and the same navigation posture. Then, the average distance between the pixel points in the current image rectangular window and the UAV is compared with the average distance recorded before the repair to confirm whether the current collected image meets expectations. In essence, automatic verification based on machine vision is realized. For comparison: before and after the repair, some pixel values ​​in the images before and after the repair of the same area usually have mutations, so it is impossible to determine whether the current collected image meets expectations by the approximation of the images before and after the repair. At the same time, if the secondary verification is omitted, even with GPS positioning plus base station positioning, WIFI and other auxiliary positioning, there will be objectively small deviations between the position positioning of the front and rear photo points, and the small deviation of the UAV position point is easy to cause a huge deviation between the collected image and the expected collection area. Therefore, under automatic inspection, only the distance-based secondary verification can ensure that the collected image meets expectations. Furthermore, when the automatic verification does not meet the set requirements, it can be supplemented by manual intervention, which improves the intelligence and flexibility of the system.

[0045] Moreover, this embodiment uses a rectangular window for local image matching, which greatly improves the accuracy and real-time performance of data processing compared to existing technologies such as unmanned driving based on binocular vision, which requires matching the entire image. At the same time, during the data storage process, only the left view or the right view is selected for storage, saving storage space.

[0046] In summary, the present invention improves the efficiency of detection by taking multiple measures and improves the reliability, intelligence and flexibility of before-and-after comparison evaluation and tracking of repair effects.

[0047] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A bridge appearance detection method based on drone, characterized in that: include: Use a drone equipped with a binocular camera to take photos of the various local areas of the bridge to be repaired; In the process of taking photos of any local area, the pixels in the central rectangular window of the left view are matched with the right view, and the average distance between each pixel point in the rectangular window and the drone is calculated according to the matched disparity data, and then the corresponding relationship between the average distance and the corresponding left view, and the earth coordinate position information and navigation attitude information of the photo taking point of the drone is saved; wherein the specification of the number of pixels in the rectangular frame is a uniformly set default value; After the bridge is repaired, the navigation route of the UAV is planned according to the distribution information of the earth coordinates corresponding to each previously repaired area of ​​the bridge; and after the UAV arrives at any photo taking point according to the planned navigation route, the corresponding navigation attitude information is found from the corresponding relationship to take photos of the left and right views according to the corresponding navigation attitude information; then the pixels in the central rectangular window of the current left view are matched with the right view, and the average distance between each pixel in the rectangular window and the UAV is calculated according to the matched disparity data; and the error between the current average distance and the corresponding average distance before repair in the corresponding relationship is calculated to be within the set threshold range. If so, it is determined that The image taken before meets the requirements, and the left view of the local area taken this time is paired with the left view before restoration and stored in the same data block. If it exceeds the set threshold range, the human operator is notified to adjust the earth coordinate position and / or navigation attitude information of the photo point where the drone is located to obtain a photo image that meets the requirements of the operating user, and while the left view taken this time by manual intervention is paired with the left view before restoration and stored in the same data block, the earth coordinate position information and navigation attitude information of the photo point where the drone is located that were previously saved in the corresponding relationship are replaced with the earth coordinate position and / or navigation attitude information of the photo point where the drone is located after the manual adjustment for subsequent inspections.

2. The method according to claim 1, characterized in that The navigation attitude information includes LBS position information, navigation azimuth and pitch angle information.

3. A bridge appearance inspection system based on an unmanned aerial vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to claim 1 or 2 are implemented.

4. A bridge appearance detection method based on drone, characterized in that: include: Use a drone equipped with a binocular camera to take photos of the various local areas of the bridge to be repaired; In the process of taking photos of any local area, the pixels in the central rectangular window of the right view are matched with the left view, and the average distance between each pixel point in the rectangular window and the drone is calculated according to the matched disparity data, and then the corresponding relationship between the average distance and the corresponding right view, and the earth coordinate position information and navigation attitude information of the photo taking point of the drone is saved; wherein the number of pixels in the rectangular frame is a uniformly set default value; After the bridge is repaired, the navigation route of the UAV is planned according to the distribution information of the earth coordinates corresponding to the various previously repaired areas of the bridge; and after the UAV arrives at any photo taking point according to the planned navigation route, the corresponding navigation attitude information is searched from the corresponding relationship to take photos of the left and right views according to the corresponding navigation attitude information; then the pixels in the central rectangular window of the current right view are matched with the left view, and the average distance between each pixel in the rectangular window and the UAV is calculated according to the matched disparity data; and the error between the currently obtained average distance and the corresponding average distance before repair in the corresponding relationship is calculated to be within the set threshold range. If so, it is determined that The image taken before meets the requirements, and the right view of the local area taken this time is paired with the right view before restoration and stored in the same data block. If it exceeds the set threshold range, the human operator is notified to adjust the earth coordinate position and / or navigation attitude information of the photo point where the UAV is located to obtain a photo image that meets the requirements of the operating user, and while the right view taken this time by manual intervention is paired with the right view before restoration and stored in the same data block, the earth coordinate position information and navigation attitude information of the photo point where the UAV is located that were previously saved in the corresponding relationship are replaced with the earth coordinate position and / or navigation attitude information of the photo point where the UAV is located after the manual adjustment for subsequent inspection.

5. The method according to claim 4, characterized in that The navigation attitude information includes LBS position information, navigation azimuth and pitch angle information.

6. A bridge appearance inspection system based on an unmanned aerial vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to claim 4 or 5 are implemented.

Images