Bridge crack detection method and system based on unmanned aerial vehicle

Abstract

The invention discloses a bridge crack detection method and system based on an unmanned aerial vehicle. The method comprises the following steps: acquiring an image of a to-be-detected bridge by using the unmanned aerial vehicle carrying a camera; constructing an AR view at the current moment by using an SLAM technology; utilizing a first CNN model with a plurality of branches to identify a bridge type in the AR view at the current moment, judging whether the AR view at the current moment has a potential crack or not according to the matched branches, and marking the position of the potential crack; using a second CNN model with a plurality of branches to judge whether the mark position has a crack or not and judge the crack position and the crack severity according to the branches of which the colors and the formats are adaptively matched in the marks in the AR view; and adjusting the flight route of the unmanned aerial vehicle according to the judgment result, so as to perform image acquisition and crack existence and crack severity confirmation on the mark positions which are judged to have cracks and the severity of the cracks greater than the preset severity, and generating a bridge crack detection report. According to the invention, the comprehensiveness and accuracy of bridge crack detection can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of drone bridge detection, and specifically to a drone-based bridge crack detection method and system. Background Art

[0002] With the accelerated development of urbanization, the safety monitoring of large-scale infrastructure has become increasingly important. Especially for key facilities such as bridges, their safety and reliability are directly related to the safety of life and property of the public. Traditional bridge inspection methods mainly rely on manual inspection, but due to high labor costs, low efficiency, and difficulty in covering all areas, traditional methods have become increasingly difficult to meet the management needs of modern large-scale infrastructure.

[0003] In order to solve the problems existing in traditional bridge inspection, the existing technology usually uses ground robots or fixed cameras for monitoring. Ground robots can patrol along preset paths and perform certain maintenance and monitoring tasks, but their movements are limited when facing complex structures, especially high areas of bridges, resulting in incomplete bridge crack detection results; the existing technology also uses drones to collect bridge images and combines image recognition technology for crack detection, but none of them takes into account the differences between different types of bridge cracks, resulting in poor reliability of detection results, and under more complex environmental factors, it is difficult to intuitively observe the located cracks, and it is difficult for maintenance personnel to make quick decisions on the detected cracks. Summary of the invention

[0004] In order to improve the comprehensiveness and accuracy of bridge crack detection, the present application provides a bridge crack detection method and system based on drone.

[0005] In a first aspect, the present application provides a bridge crack detection method based on a drone, comprising: Using drones equipped with cameras, the images of the bridges to be inspected are collected in real time according to the preset flight routes, and the collected images are transmitted to the cloud in real time; Using SLAM technology, the transmitted image is compared with the bridge model database stored in the cloud in real time to construct an AR view at the current moment; the bridge model database stores three-dimensional models of different types of bridges, and the three-dimensional model of each type of bridge is marked with key parts; Using a first CNN model with multiple branches built into the cloud, the bridge type in the AR view at the current moment is identified, and a matching branch is adaptively selected according to the identified bridge type to determine whether there is a potential crack in the AR view at the current moment. If there is, the location of the potential crack is marked in the AR view and marked with a preset color and format matching the bridge type; each branch in the first CNN model is trained and judged whether a bridge of a certain type of bridge has a potential crack; Using the second CNN model with multiple branches built into the cloud, the branch that adaptively matches the color and format of the marker in the AR view is used to determine whether there is a crack at the marker position, as well as the position and severity of the crack; each branch of the second CNN model introduces an attention mechanism to train and determine whether there is a crack at the marker position in a bridge image of a certain type of bridge, as well as the position and severity of the crack; The heading and flight route of the UAV are adjusted according to the presence of cracks at the marked position in the AR view at the current moment and the judgment results of the crack position and crack severity, so as to re-capture images and reconfirm the presence of cracks, crack position and crack severity at the marked position judged to have cracks and crack severity greater than the preset severity, and generate a bridge crack detection report based on the confirmation results.

[0006] By adopting the above solution, combining the constructed AR view with the first CNN model with different branches, the potential cracks in different types of bridges are specifically identified and intuitively marked and displayed. Then, the second CNN model with different branches is used to identify the cracks and severity of different types of bridges and perform secondary confirmation, which improves the comprehensiveness and accuracy of crack detection, realizes the precise identification and positioning of cracks, improves detection efficiency, and reduces labor costs.

[0007] Preferably, the process of marking the position where it is determined that a crack exists and the severity of the crack is greater than a preset severity includes: It is determined whether the marked position is a critical part. If it is a critical part, the preset crack severity is adaptively adjusted so that the preset crack severity after adjustment is lower than the preset crack severity before adjustment.

[0008] By adopting the above scheme, key parts are determined and the preset crack severity is adaptively adjusted to be lower than the preset crack severity before adjustment, thereby lowering the judgment threshold of the preset crack severity of key parts, thereby improving the accuracy and reliability of crack detection in key parts.

[0009] Preferably, it also includes: The camera-carrying drone is also equipped with a laser radar, and the drone is used to complete the real-time acquisition of the image of the bridge to be inspected, and the radar distance measurement value is obtained in real time; Synchronize real-time transmitted image data and radar distance measurements and transmit them to the cloud; The second CNN model is replaced by an improved optimized second CNN model with multiple branches built in the cloud, and the branches that adaptively match the color and format in the mark in the AR view determine whether there is a crack at the marked position and the position and severity of the crack; the improved optimized second CNN model with multiple branches is based on the second CNN model, and the input of the model adds a radar distance measurement value synchronized with the AR view at the current moment; during the training process of each branch of the improved second CNN model for whether there is a crack at the marked position in a bridge of a class of bridge types and the position and severity of the crack, bridge image data corresponding to the bridge type, marked with the existence of cracks, the position and severity of cracks, and potential crack position marks, and the synchronized radar distance measurement values ​​are used as training data.

[0010] By adopting the above scheme, taking into account the detection errors that may be caused by relying solely on image data in complex environments or bad weather, radar ranging measurement values ​​are collected simultaneously, and multimodal data are integrated to determine whether there are cracks on the bridge, thereby improving the accuracy and reliability of crack identification.

[0011] Preferably, it also includes: The camera-carrying drone is also equipped with a multi-band light source device, and the drone is used to complete the real-time acquisition of the image of the bridge to be inspected, and at the same time, the illumination data in the environmental data is obtained in real time; Synchronize the real-time transmitted image data and illumination data and transmit them to the cloud; Spectral adaptation technology is used to analyze the lighting data at the current moment, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi-band light source equipment on board to adjust the wavelength of light emitted by the multi-band light source, and the adjusted lighting data is the preset lighting data for the corresponding bridge type.

[0012] By adopting the above solution and using spectral adaptation technology, the current lighting is adjusted according to the environmental data obtained in real time, thereby ensuring high-quality detection images under different lighting conditions and improving the accuracy and reliability of crack detection.

[0013] Preferably, it also includes: For multiple bridge crack inspection reports of the same bridge to be inspected at different time periods, the number of occurrences of the same crack position in the report where cracks exist and the crack severity is greater than the preset crack severity is counted. If the number of occurrences is greater than the preset number, the marked position is identified as a critical part.

[0014] By adopting the above scheme, the locations where cracks are found in multiple tests are counted and identified as key parts, thereby achieving accurate positioning and focused monitoring of key parts of the bridge, and improving the reliability and pertinence of the detection.

[0015] Preferably, it also includes: The AR view generated at the current moment is transmitted to the ground control station in real time, and the potential crack locations manually drawn by the user in the AR view are received and transmitted back to the cloud; The manually drawn potential crack locations are displayed in the AR view in the cloud, marked with a preset color and format matching the bridge type, and additional marks are added to distinguish the marks of the potential crack locations identified by the first CNN model.

[0016] By adopting the above solution, users are allowed to supplement potential crack information based on actual experience, ensuring comprehensive coverage of crack detection and enhancing the flexibility and accuracy of the detection system.

[0017] Preferably, it also includes: According to the bridge crack detection report generated by the drone, after comparing the existence of cracks at the marked positions corresponding to the potential crack positions identified by the first CNN model and the proportion of the judgment results of the crack positions and the crack severity being consistent with the actual situation, the proportion of the existence of cracks at the marked positions corresponding to the potential crack positions drawn manually and the proportion of the judgment results of the crack positions and the crack severity being consistent with the actual situation is counted; If the preset ratio is not reached, the user ID of the manually drawn potential crack location is marked, and when the manually drawn potential crack location is subsequently received from the same user ID, a prompt message for reconfirming the manually drawn potential crack location is generated and returned to the ground control station, and the receiving user reconfirms the manually drawn potential crack location or adjusts the manually drawn potential crack location.

[0018] By adopting the above scheme, it is determined that the second CNN model has a relatively accurate recognition. The user IDs whose proportion of the manually drawn potential crack location judgment results consistent with the actual situation does not reach the preset proportion are counted, so as to send reconfirmation information to these error-prone users to prompt the accuracy and reliability of crack recognition.

[0019] In a second aspect, the present application provides a bridge crack detection system based on a drone, comprising: The bridge image acquisition module is used to use a drone equipped with a camera to complete the real-time image acquisition of the bridge to be inspected according to the preset flight route, and transmit the collected images to the cloud in real time; The bridge AR view construction module is used to use SLAM technology to compare the transmitted image with the bridge model database stored in the cloud in real time to construct the AR view at the current moment; the bridge model database stores three-dimensional models of different types of bridges and the three-dimensional model of each type of bridge is marked with key parts; A bridge potential crack identification module is used to use a first CNN model with multiple branches built into the cloud to identify the type of bridge in the AR view at the current moment, and adaptively select a matching branch according to the identified bridge type to determine whether there is a potential crack in the AR view at the current moment. If there is, the location of the potential crack is marked in the AR view and marked with a preset color and format matching the bridge type; each branch in the first CNN model is trained and judged whether a bridge of a certain type of bridge has a potential crack; A bridge crack recognition module is used to use a second CNN model with multiple branches built into the cloud to determine whether there is a crack at the marked position, as well as the position and severity of the crack according to the branch that adaptively matches the color and format in the mark in the AR view; each branch of the second CNN model introduces an attention mechanism to train and judge whether there is a crack at the marked position in the bridge image of a class of bridge types, as well as the position and severity of the crack; The bridge crack report generation module is used to adjust the heading and flight path of the drone according to the presence of cracks at the marked position in the AR view at the current moment and the judgment results of the crack position and crack severity, so as to re-capture images and re-confirm the presence of cracks, crack position and crack severity at the marked position judged to have cracks and crack severity greater than the preset severity, and generate a bridge crack detection report based on the confirmation results.

[0020] By adopting the above scheme, a multi-branch CNN model is used to identify and confirm bridge cracks in a targeted manner to improve detection accuracy. At the same time, the combination of CNN and AR technology is used to intuitively display potential cracks. The position facilitates more accurate crack identification and positioning, and improves the reliability and efficiency of detection.

[0021] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0022] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0023] In summary, this application has the following beneficial effects: 1. Design a branched first CNN model and a second CNN model to work together to perform preliminary identification of potential cracks, identification and reconfirmation of cracks for different types of bridges, and improve the accuracy and comprehensiveness of crack identification. Use SLAM technology and a pre-built bridge model database, combined with AR views to accurately mark the location of cracks, and combine with the attention mechanism to facilitate more accurate crack identification and facilitate subsequent ground control station staff to intuitively confirm the crack location and complete UAV heading adjustment and maintenance. 2. By introducing LiDAR, the fusion data of distance measurement data and spherical data can be combined in complex environments to achieve accurate crack identification and ensure stable detection performance; or spectral adaptation technology can be used to adaptively adjust the lighting to ensure high-quality detection images under different lighting conditions, thereby improving the accuracy and reliability of crack detection; 3. Use AR technology to provide real-time AR views for ground control station staff, allowing them to supplement potential crack information based on actual experience to ensure comprehensive coverage of crack detection. Combined with the results of crack identification using the CNN model, statistically determine the staff who are prone to errors in supplementing potential crack information, and generate confirmation prompts during the subsequent supplement process to ensure more accurate supplementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a bridge crack detection method based on a drone described in a specific embodiment; Figure 2 It is a UAV flight mission management business structure in a UAV-based bridge crack detection method described in a specific embodiment; Figure 3 It is a structural schematic diagram of a bridge crack detection system based on a drone described in a specific embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] like Figure 1 As shown, the embodiment of the present application discloses a bridge crack detection method based on a drone, comprising: S1. Use a drone equipped with a camera to complete the real-time image collection of the bridge to be inspected according to the preset flight route and transmit it to the cloud.

[0027] Specifically, the camera-equipped drone communicates with the ground control station and the cloud-based service platform. It is responsible for receiving control instructions from the ground control station, completing real-time image acquisition of the bridge to be inspected according to the preset flight route, and transmitting the acquired images to the cloud-based service platform in real time.

[0028] S2. Using SLAM technology, the transmitted image is compared with the bridge model database stored in the cloud in real time to construct an AR view of the current moment.

[0029] Specifically, the bridge model database stores three-dimensional models, structural information, and historical crack data of several different types of bridges, such as beam bridges, arch bridges, suspension bridges, and cable-stayed bridges. Each type has different key parts due to different bridge structures, such as the main beam, support, and pier of a beam bridge; the arch ring and arch foot of an arch bridge; the main cable, pylon, suspender, and anchor of a suspension bridge; the cable, pylon, and deck beam of a cable-stayed bridge. Therefore, key parts of different bridges are marked separately, such as in text form.

[0030] Utilizing SLAM technology, the transmitted image is compared in real time with the bridge model database stored in the cloud. Based on feature data such as key parts data, the three-dimensional bridge model of the same type as the current real-time transmitted image and with a similarity greater than the preset similarity is matched from the bridge model database stored in the cloud to determine the position on the bridge where the current drone image is located. After completing model matching and positioning, an AR view of the current moment is constructed, in which the three-dimensional model of the bridge and structural information (such as key parts information) are superimposed and displayed.

[0031] In order to provide the ground control station with an intuitive and comprehensive visual interface, the generated AR view of the current moment can be transmitted to the ground control station in real time.

[0032] S3. Using the first CNN model with multiple branches built into the cloud, potential cracks of the bridge in the AR view at the current moment are identified and the locations of the potential cracks are marked.

[0033] Specifically, several CNN models are stored in the cloud, including a first CNN model with multiple branches; the first CNN model includes a bridge type recognition sub-model, which pre-processes the input AR view, extracts key features, and quickly recognizes the bridge type; the matching branch is adaptively selected according to the recognized bridge type, and each branch of the first CNN model trains and judges whether a type of bridge has potential cracks. Specifically, the backbone network of each branch selects a basic CNN architecture with excellent performance, such as ResNet, VGG, etc., and fine-tunes the basic CNN architecture according to the crack characteristics of each bridge type. For example, the number of convolutional layers, the size of the filter, and the step size can be adjusted; an independent training plan is formulated for each branch, and the corresponding type of bridge crack image data marked with potential cracks is used for training to ensure that each branch can focus on extracting the crack characteristics of this type of bridge; all trained CNN branches are integrated into a unified model to generate the first CNN model.

[0034] In this embodiment, the first CNN model with multiple branches built into the cloud is used. The input of the model is the AR view at the current moment. The bridge type in the AR view at the current moment is identified. According to the identified bridge type, the matching branch is adaptively selected to determine whether there is a potential crack in the AR view at the current moment. If there is, the location of the potential crack is marked in the AR view and marked with a preset color and format matching the bridge type. Among them, the preset color and format marking matching each bridge type are different, such as: the preset color of the arch bridge is yellow and marked with a dot with a radius of X1, and the preset color of the beam bridge is red and marked with a triangle with a side length of X2.

[0035] S4. Using the second CNN model with multiple branches built into the cloud, determine whether there is a crack at the location of the potential crack marked in the AR view at the current moment and the severity of the crack.

[0036] Specifically, the several CNN models stored in the cloud also include a second CNN model with multiple branches; the second CNN model is provided with multiple branches, and each branch introduces an attention mechanism, so that it can automatically focus on key areas in the image, such as the marked position of potential cracks, and train and judge whether there are cracks at the marked position in the bridge image of a type of bridge, as well as the crack position and crack severity; specifically, each branch is trained using the corresponding type of bridge crack image data with potential crack marked positions annotated with crack positions and crack severity (such as: extremely slight, mild, general, relatively serious, extremely serious, with levels increasing gradually), to ensure that each branch can focus on extracting the crack features of this type of bridge, and all trained branches are integrated into a unified model to generate the second CNN model.

[0037] In this embodiment, a second CNN model with multiple branches built into the cloud is used. The input of the model is an AR view of the current environment that marks the location of potential cracks. The branches that adaptively match the color and format of the marks in the AR view are used to determine whether there are cracks at the marked location and the severity of the cracks, and the judgment result is output.

[0038] S5. According to the presence of cracks at the marked position in the AR view at the current moment and the determination results of the crack position and crack severity, the heading and flight path of the drone are adjusted, and the area suspected of having severe cracks is reconfirmed.

[0039] Specifically, since the location area with severe cracks requires emergency treatment, a second confirmation scan is performed on the location area suspected to be severe cracks to ensure that resources for subsequent emergency treatment are not wasted.

[0040] According to the judgment result of whether there is a crack at the marked position in the AR view at the current moment and the crack position and crack severity, once it is determined that there is a crack at the AR view at the current moment and the crack severity is greater than the marked position of the preset severity, such as Figure 2 As shown, a flight instruction for adjusting the original flight route is generated so that at the next moment, the drone will re-fly the marked positions where cracks exist in the AR view at the current moment and where the severity of the cracks is greater than the preset severity, and then continue to fly along the original flight route, and the flight instruction is transmitted to the ground control station. The ground control station controls the drone to adjust the heading according to the instruction, re-captures images of the marked positions where cracks exist in the AR view at the current moment and where the severity of the cracks is greater than the preset severity, and repeats S3-S4 for the re-captured marked positions to complete the confirmation of whether cracks exist at the marked positions and the severity of the cracks.

[0041] In addition, in order to strengthen the reconfirmation of suspected serious key parts, the process of marking the position where cracks are judged to exist and the severity of the cracks is greater than the preset severity includes: judging whether the marked position is a key part, and if it is a key part, adaptively adjusting the preset crack severity so that the adjusted preset crack severity is lower than the preset crack severity before adjustment. For example: the preset crack severity before adjustment is relatively serious, if it is judged to be a key part, the preset crack severity is general.

[0042] S6. Generate a bridge crack detection report based on the confirmation results.

[0043] Specifically, combined with the confirmation result, if the confirmation result is that the result of judging the presence of cracks in the AR view at the current moment and the marked position where the severity of the cracks is greater than the preset severity is correct, then the marked position where the cracks are confirmed to exist and the severity of the cracks is greater than the preset severity is retained; otherwise, the result of the reconfirmation is retained; and a bridge crack detection report containing all crack locations and crack severities is generated in combination with all judgment and confirmation results.

[0044] In a specific embodiment, in order to further avoid errors in determining and locating the presence or absence of cracks due to complex environmental conditions using single image data, and to improve the accuracy and stability of the detection results, the method further includes: The camera-equipped drone is also equipped with a laser radar. While the drone is used to complete real-time image acquisition of the bridge to be inspected, the radar distance measurement value is obtained in real time. Among them, the laser radar Liar can still obtain fine distance information on the surface of an object in a low-visibility environment to determine whether there are cracks.

[0045] Synchronize real-time transmitted image data and radar distance measurements and transmit them to the cloud; The second CNN model is replaced by an improved optimized second CNN model with multiple branches built in the cloud, and the branches that adaptively match the color and format in the mark in the AR view are used to determine whether there is a crack at the marked position and the position and severity of the crack; wherein the improved optimized second CNN model with multiple branches is based on the second CNN model, and in terms of the model input design, the model input adds the radar distance measurement value synchronized with the AR view at the current moment. Accordingly, in the process of training whether there is a crack at the marked position in a bridge of a type of bridge, as well as the position and severity of the crack, each branch of the improved second CNN model uses bridge image data of the corresponding bridge type, which is marked with the existence of cracks, the position and severity of cracks, and has potential crack position markers, and the synchronized radar distance measurement value as training data.

[0046] In addition, a drone equipped with an infrared camera can be used to complete real-time image acquisition of the bridge to be inspected, and at the same time obtain temperature distribution data in real time, and synchronously transmit the real-time transmitted image data, radar ranging data and temperature distribution data to the cloud. On the basis of the second CNN model, the model input is further optimized to be the real-time transmitted image data, radar ranging data and temperature distribution data, and corresponding optimization training is performed. The trained optimized second CNN model is used to determine whether there is a crack at the mark position, as well as the crack position and the severity of the crack according to the branch that adaptively matches the color and format of the mark in the AR view.

[0047] In a specific embodiment, due to the special geographical location, the same type of bridges at different locations are often affected by complex lighting conditions such as strong direct sunlight, shadows and fog. In order to ensure the accuracy of bridge cracks under different lighting conditions, the method further includes: The camera-equipped drone is also equipped with a multi-band light source device. While the drone is used to complete the real-time acquisition of the image of the bridge to be inspected, the lighting data in the environmental data is obtained in real time, including: light intensity, color temperature and wavelength distribution.

[0048] The image data and lighting data transmitted in real time are synchronized and transmitted to the cloud.

[0049] The spectral adaptation algorithm is used to analyze the lighting data at the current moment, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi-band light source equipment on board to adjust the wavelength of light emitted by the multi-band light source, and the adjusted lighting data is the lighting data preset for the bridge type; wherein the preset lighting data is determined based on the lighting conditions with the highest historical accuracy for identifying cracks in different types of bridges.

[0050] In addition, by further considering the materials and colors of different types of beams, the preset lighting data under corresponding conditions can be further determined based on the lighting conditions with the highest historical accuracy in identifying cracks in bridges of different types, materials and colors, and the material and color of the current bridge to be inspected can be obtained. The spectral adaptation algorithm can be used to analyze the lighting data at the current moment, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi-band light source equipment carried by the drone to adjust the wavelength of light emitted by the multi-band light source, and make the adjusted lighting data the preset lighting data for the bridge type, corresponding bridge material and color.

[0051] In a specific embodiment, considering that in the process of marking potential cracks in the AR view, manual interaction is introduced to improve the accuracy and flexibility of crack identification, the method further includes: The AR view generated at the current moment is transmitted to the ground control station in real time, and the potential crack locations manually drawn by the user in the AR view are received and transmitted back to the cloud; The manually drawn potential crack locations are displayed in the AR view in the cloud, marked with a preset color and format matching the bridge type, and additional marks are added to distinguish the marks of the potential crack locations identified by the first CNN model. Specifically, new marks can be added based on the preset color and format, such as adding an outer circle mark based on the original format.

[0052] In addition, considering that in the process of introducing manual interaction, it is also very important to ensure the accuracy of the user's manual marking to avoid the waste of computing resources caused by the user's incorrect marking, the method also includes: According to the bridge crack detection report generated by the drone, the proportion of the judgment results of the existence of cracks at the corresponding marked positions of the potential crack locations identified by the first CNN model and the consistency of the crack location and crack severity with the actual situation (i.e., the similarity is greater than the preset similarity of the corresponding type of bridge) (the proportion of the total number of judgment results) reaches the preset proportion, indicating that the accuracy of crack identification using the second CNN model currently meets the requirements. On this basis, the proportion of the judgment results of the existence of cracks at the corresponding marked positions of the potential crack locations drawn manually and the consistency of the crack location and crack severity with the actual situation is counted; If the preset ratio is not reached, it indicates that the error rate of the manually drawn potential crack location marked by the corresponding user is high. The user ID of the manually drawn potential crack location is marked, and when the manually drawn potential crack location is subsequently received from the same user ID, a prompt message for reconfirming the manually drawn potential crack location is generated and returned to the ground control station, and the receiving user reconfirms the manually drawn potential crack location or adjusts the manually drawn potential crack location.

[0053] Of course, in addition to allowing users to hand-draw potential cracks, the AR view can also intuitively display information, including crack trend information; specifically, when it is determined that there is a crack in the AR view at the current moment, the historical crack data of the location is queried according to the crack location, and the current crack data is input into the crack trend prediction model (such as a CNN model) of the corresponding bridge type trained with the queried historical crack data to obtain the crack trend, and the crack trend is displayed in a layer of the AR view and transmitted to the ground control station.

[0054] In a specific embodiment, in order to further improve the accuracy of detecting the locations where cracks are likely to occur, the method further includes: For multiple bridge crack inspection reports of the same bridge to be inspected at different time periods, the number of occurrences of the same crack position in the report where cracks exist and the crack severity is greater than the preset crack severity is counted. If the number of occurrences is greater than the preset number, it indicates that the position is prone to cracks, and the marked position is identified as a critical position.

[0055] like Figure 3 As shown, the embodiment of the present application discloses a bridge crack detection system based on a drone, comprising: The bridge image acquisition module 101 is used to use a drone equipped with a camera to complete the real-time image acquisition of the bridge to be inspected according to a preset flight route, and transmit the acquired image to the cloud in real time; The bridge AR view construction module 102 is used to use SLAM technology to compare the transmitted image with the bridge model database stored in the cloud in real time to construct an AR view at the current moment; the bridge model database stores three-dimensional models of different types of bridges and the three-dimensional model of each type of bridge is marked with key parts; The bridge potential crack identification module 103 is used to use the first CNN model with multiple branches built in the cloud to identify the bridge type in the AR view at the current moment, and adaptively select a matching branch according to the identified bridge type to determine whether there is a potential crack in the AR view at the current moment. If there is, the location of the potential crack is marked in the AR view and marked with a preset color and format matching the bridge type; each branch in the first CNN model is trained and judged whether a bridge of a certain bridge type has a potential crack; The bridge crack recognition module 104 is used to use a second CNN model with multiple branches built in the cloud to determine whether there is a crack at the marked position, the crack position and the severity of the crack according to the branch that adaptively matches the color and format in the mark in the AR view; each branch of the second CNN model introduces an attention mechanism to train and judge whether there is a crack at the marked position in the bridge image of a class of bridge types, the crack position and the severity of the crack; The bridge crack report generation module 105 is used to adjust the heading and flight path of the UAV according to the presence of cracks at the marked position in the AR view at the current moment and the judgment results of the crack position and crack severity, so as to re-capture images and re-confirm the presence of cracks, crack position and crack severity at the marked position judged to have cracks and crack severity greater than a preset severity, and generate a bridge crack detection report based on the confirmation results.

[0056] In a specific embodiment, the system further includes: The radar distance measurement value acquisition module 106 is used for the drone equipped with the camera and also equipped with a laser radar, and the drone is used to complete the real-time acquisition of the image of the bridge to be inspected, and at the same time, obtain the radar distance measurement value in real time; synchronize the real-time transmitted image data and the radar distance measurement value and transmit them to the cloud; The bridge crack recognition module 104 is also used to replace the second CNN model with an improved optimized second CNN model with multiple branches built into the cloud, and judge whether there is a crack at the marked position and the crack position and severity according to the branch that adaptively matches the color and format in the mark in the AR view; the improved optimized second CNN model with multiple branches is based on the second CNN model, and the input of the model adds a radar distance measurement value synchronized with the AR view at the current moment; during the training process of each branch of the improved second CNN model for whether there is a crack at the marked position in a bridge of a class of bridge types and the crack position and crack severity, bridge image data corresponding to the bridge type, which is annotated with the existence of cracks, crack position and crack severity, and has potential crack marked positions, and the synchronized radar distance measurement value is used as training data.

[0057] In a specific embodiment, the system further comprises: Lighting data acquisition module 107, used for the drone equipped with camera also equipped with multi-band light source equipment, using drone to complete real-time image acquisition of the bridge to be inspected, while obtaining lighting data in environmental data in real time; synchronize the real-time transmitted image data and lighting data and transmit them to the cloud; The bridge image acquisition module 101 is also used to use the spectral adaptation algorithm to analyze the lighting data at the current moment, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi-band light source device to adjust the wavelength of light emitted by the multi-band light source.

[0058] In a specific embodiment, the bridge AR view construction module 102 in the system is also used to count the number of occurrences of the same crack position in the reports where cracks exist and the crack severity is greater than a preset crack severity for multiple bridge crack detection reports of the same bridge to be inspected in different time periods. If the number of occurrences is greater than the preset number, the marked position is identified as a critical part.

[0059] The bridge potential crack interaction module 108 is used to transmit the constructed AR view at the current moment to the ground control station in real time, receive the potential crack locations manually drawn by the user in the AR view and transmit them back to the cloud; display the manually drawn potential crack locations in the AR view in the cloud, mark them with preset colors and formats matching the bridge type, and add additional marks to distinguish them from the marks of the potential crack locations identified by the first CNN model.

[0060] It is also used to generate a bridge crack detection report for an unmanned aerial vehicle, compare the existence of cracks at the marked positions corresponding to the potential crack locations identified by the first CNN model, and the proportion of the judgment results of the crack locations and the crack severity being consistent with the actual situation after reaching a preset proportion, and then count the proportion of the existence of cracks at the marked positions corresponding to the potential crack locations manually drawn, and the judgment results of the crack locations and the crack severity being consistent with the actual situation; if the preset proportion is not reached, mark the user ID of the manually drawn potential crack location, and when the manually drawn potential crack location is subsequently received from the same user ID, generate a prompt message for reconfirming the manually drawn potential crack location and return it to the ground control station, and receive the user's reconfirmation of the manually drawn potential crack location or the adjusted manually drawn potential crack location.

[0061] The embodiment of the present application also discloses a computer-readable storage medium.

[0062] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed such as the above-mentioned drone-based bridge crack detection method. The computer-readable storage medium includes, for example: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0063] The embodiment of the present application also discloses a computer device.

[0064] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned drone-based bridge crack detection method.

[0065] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A bridge crack detection method based on drone, characterized in that: include: Using drones equipped with cameras, the images of the bridges to be inspected are collected in real time according to the preset flight routes, and the collected images are transmitted to the cloud in real time; Using SLAM technology, the transmitted image is compared with the bridge model database stored in the cloud in real time to construct an AR view of the current moment; The bridge model database stores three-dimensional models of different types of bridges, and the three-dimensional model of each type of bridge is marked with key parts; Using the first CNN model with multiple branches built into the cloud, the bridge type in the AR view at the current moment is identified, and a matching branch is adaptively selected according to the identified bridge type to determine whether there is a potential crack in the AR view at the current moment. If so, the location of the potential crack is marked in the AR view with a preset color and format matching the bridge type; Each branch in the first CNN model is trained and judged whether a type of bridge has potential cracks; Using the second CNN model with multiple branches built into the cloud, the branch that adaptively matches the color and format of the marker in the AR view is used to determine whether there is a crack at the marker position, as well as the position and severity of the crack; each branch of the second CNN model introduces an attention mechanism to train and determine whether there is a crack at the marker position in a bridge image of a certain type of bridge, as well as the position and severity of the crack; The heading and flight route of the UAV are adjusted according to the presence of cracks at the marked position in the AR view at the current moment and the judgment results of the crack position and crack severity, so as to re-capture images and reconfirm the presence of cracks, crack position and crack severity at the marked position judged to have cracks and crack severity greater than the preset severity, and generate a bridge crack detection report based on the confirmation results.

2. The method for detecting bridge cracks based on an unmanned aerial vehicle according to claim 1, characterized in that: The process of marking the position of the cracks and determining that the cracks are present and the severity of the cracks is greater than a preset severity includes: It is determined whether the marked position is a critical part. If it is a critical part, the preset crack severity is adaptively adjusted so that the preset crack severity after adjustment is lower than the preset crack severity before adjustment.

3. The bridge crack detection method based on drone according to claim 1 is characterized in that: Also includes: The camera-carrying drone is also equipped with a laser radar, and the drone is used to complete the real-time acquisition of the image of the bridge to be inspected, and the radar distance measurement value is obtained in real time; Synchronize real-time transmitted image data and radar distance measurements and transmit them to the cloud; The second CNN model is replaced by an improved optimized second CNN model with multiple branches built in the cloud, and the presence of a crack at the marker position, the position of the crack, and the severity of the crack are determined according to a branch that adaptively matches the color and format of the marker in the AR view; the improved optimized second CNN model with multiple branches is based on the second CNN model, and a radar distance measurement value synchronized with the AR view at the current moment is added to the input of the model; In the process of training each branch of the improved second CNN model for the presence or absence of cracks at the marked position in a type of bridge, as well as the crack location and crack severity, bridge image data of the corresponding bridge type, marked with potential crack location marks, and synchronized radar distance measurement values ​​are used as training data.

4. The method for detecting bridge cracks based on an unmanned aerial vehicle according to claim 1, characterized in that: Also includes: The camera-carrying drone is also equipped with a multi-band light source device, and the drone is used to complete the real-time acquisition of the image of the bridge to be inspected, and at the same time, the illumination data in the environmental data is obtained in real time; Synchronize the real-time transmitted image data and illumination data and transmit them to the cloud; The spectral adaptation algorithm is used to analyze the lighting data at the current moment, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi-band light source equipment on board to adjust the wavelength of light emitted by the multi-band light source, and the adjusted lighting data is the preset lighting data for the corresponding bridge type.

5. The bridge crack detection method based on drone according to claim 1 is characterized in that: Also includes: For multiple bridge crack inspection reports of the same bridge to be inspected at different time periods, the number of occurrences of the same crack position in the report where cracks exist and the crack severity is greater than the preset crack severity is counted. If the number of occurrences is greater than the preset number, the marked position is identified as a critical part.

6. The method for detecting bridge cracks based on drone according to claim 1, characterized in that: Also includes: The AR view generated at the current moment is transmitted to the ground control station in real time, and the potential crack locations manually drawn by the user in the AR view are received and transmitted back to the cloud; The manually drawn potential crack locations are displayed in the AR view in the cloud, marked with a preset color and format matching the bridge type, and additional marks are added to distinguish the marks of the potential crack locations identified by the first CNN model.

7. The method for detecting bridge cracks based on an unmanned aerial vehicle according to claim 6, characterized in that: Also includes: According to the bridge crack detection report generated by the drone, after comparing the existence of cracks at the marked positions corresponding to the potential crack positions identified by the first CNN model and the proportion of the judgment results of the crack positions and the crack severity being consistent with the actual situation, the proportion of the existence of cracks at the marked positions corresponding to the potential crack positions drawn manually and the proportion of the judgment results of the crack positions and the crack severity being consistent with the actual situation is counted; If the preset ratio is not reached, the user ID of the manually drawn potential crack location is marked, and when the manually drawn potential crack location is subsequently received from the same user ID, a prompt message for reconfirming the manually drawn potential crack location is generated and returned to the ground control station, and the receiving user reconfirms the manually drawn potential crack location or adjusts the manually drawn potential crack location.

8. A bridge crack detection system based on drone, characterized in that: include: The bridge image acquisition module is used to use a drone equipped with a camera to complete the real-time image acquisition of the bridge to be inspected according to the preset flight route, and transmit the collected images to the cloud in real time; The bridge AR view construction module is used to use SLAM technology to compare the transmitted image with the bridge model database stored in the cloud in real time to construct the AR view at the current moment; the bridge model database stores three-dimensional models of different types of bridges and the three-dimensional model of each type of bridge is marked with key parts; A bridge potential crack identification module is used to use a first CNN model with multiple branches built into the cloud to identify the type of bridge in the AR view at the current moment, and adaptively select a matching branch to determine whether there is a potential crack in the AR view at the current moment according to the identified bridge type. If there is, the location of the potential crack is marked in the AR view and marked with a preset color and format matching the bridge type; Each branch in the first CNN model is trained and judged whether a type of bridge has potential cracks; A bridge crack recognition module is used to use a second CNN model with multiple branches built into the cloud to determine whether there is a crack at the marked position, as well as the position and severity of the crack according to the branch that adaptively matches the color and format in the mark in the AR view; each branch of the second CNN model introduces an attention mechanism to train and judge whether there is a crack at the marked position in the bridge image of a class of bridge types, as well as the position and severity of the crack; The bridge crack report generation module is used to adjust the heading and flight path of the drone according to the presence of cracks at the marked position in the AR view at the current moment and the judgment results of the crack position and crack severity, so as to re-capture images and re-confirm the presence of cracks, crack position and crack severity at the marked position judged to have cracks and crack severity greater than the preset severity, and generate a bridge crack detection report based on the confirmation results.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images