An Unmanned Aerial Vehicle-Based Bridge Crack Detection Method and System

Abstract

The present application discloses a method and system for detecting bridge cracks based on an unmanned aerial vehicle. The method includes: using an unmanned aerial vehicle equipped with a camera to collect images of the bridge to be detected; using SLAM technology to construct an AR view at the current moment; using a first CNN model with multiple branches to identify the bridge type in the AR view at the current moment and determine whether there are potential cracks in the AR view at the current moment in the matching branch, and marking the location where the potential cracks are located; using a second CNN model with multiple branches to determine whether there are cracks at the marked position and the crack position and severity according to the branches that adaptively match the colors and formats in the marks in the AR view; adjusting the flight route of the unmanned aerial vehicle according to the judgment result to re-collect images of the marked positions where cracks are determined to exist and the severity of the cracks is greater than the preset severity and confirm the existence of cracks and the severity of the cracks, and generating a bridge crack detection report. The present application can improve the comprehensiveness and accuracy of bridge crack detection.

Description

Technical Field

[0001] This application relates to the technical field of unmanned aerial vehicle (UAV) bridge inspection, and specifically to a method and system for bridge crack detection based on UAVs. Background Art

[0002] With the accelerating development of urbanization, the safety monitoring of large-scale infrastructure has become increasingly important. Especially for key facilities such as bridges, their safety and reliability are directly related to the safety of public life and property. Traditional bridge inspection methods mainly rely on manual inspection. However, due to high labor costs, low efficiency, and difficulty in covering all areas, traditional methods are increasingly unable to meet the management requirements of modern large-scale infrastructure.

[0003] To solve the problems existing in traditional bridge inspection, existing technologies usually use ground robots or fixed cameras for monitoring. Ground robots can patrol according to a preset path and perform certain maintenance and monitoring tasks. However, they are restricted in movement when facing complex structures, especially the high areas of bridges, resulting in incomplete bridge crack detection results. There are also existing technologies that use UAVs to collect bridge images and combine image recognition technology for crack detection. However, the differences in different types of bridge cracks are not considered, resulting in poor reliability of the detection results. Moreover, in a relatively complex environmental factor, it is difficult to visually observe the located cracks, and it is difficult for maintenance personnel to make quick decisions on the detected cracks. Summary of the Invention

[0004] To improve the comprehensiveness and accuracy of bridge crack detection, this application provides a method and system for bridge crack detection based on UAVs.

[0005] In a first aspect, this application provides a method for bridge crack detection based on UAVs, including:

[0006] Using a UAV equipped with a camera to complete real-time image acquisition of the bridge to be detected according to a preset flight route, and transmitting the acquired images to the cloud in real time;

[0007] Using SLAM technology to compare the transmitted images with the bridge model database stored in the cloud in real time, and constructing an AR view at the current moment; the bridge model database stores three-dimensional models of different types of bridges, and the three-dimensional models of each type of bridge are marked with key parts;

[0008] Using the first CNN model with multiple branches built into the cloud, identify the bridge type in the AR view at the current moment, and adaptively select a matching branch according to the identified bridge type to determine whether there is a potential crack in the AR view at the current moment. If there is, mark the location of the potential crack in the AR view and mark it with a preset color and format that matches the bridge type; each branch in the first CNN model is trained and judged on whether there is a potential crack in a bridge of a certain bridge type.

[0009] Using the second CNN model with multiple branches built into the cloud, judge whether there is a crack, the crack location, and the crack severity at the marked position according to the branch adaptively matched by the color and format in the mark in the AR view; an attention mechanism is introduced into each branch of the second CNN model, and training and judgment are carried out on whether there is a crack, the crack location, and the crack severity at the marked position in the bridge image of a certain bridge type.

[0010] Adjust the heading and flight route of the drone according to the judgment results of the presence of cracks, the crack location, and the crack severity at the marked position in the AR view at the current moment, so as to re-collect images of the marked positions judged to have cracks and the crack severity greater than the preset severity, and confirm the presence of cracks, the crack location, and the crack severity. Generate a bridge crack detection report based on the confirmation results.

[0011] By adopting the above solution, combined with the constructed AR view and the first CNN model with different branches, the potential cracks existing in different types of bridges are specifically identified and visually marked and displayed. Then, the second CNN model with different branches is used to correspondingly identify the cracks and severity existing in different types of bridges and perform secondary confirmation, improving the comprehensiveness and accuracy of crack detection, realizing the precise identification and positioning of cracks, improving the detection efficiency, and reducing the labor cost.

[0012] Preferably, the process of judging the marked positions where there are cracks and the crack severity is greater than the preset severity includes:

[0013] Judge whether the marked position is a key part. If it is a key part, adaptively adjust the preset crack severity so that the adjusted preset crack severity is lower than the preset crack severity before adjustment.

[0014] By adopting the above solution, determine the key part and adaptively adjust the preset crack severity to be lower than the preset crack severity before adjustment, reducing the judgment threshold of the preset crack severity of the key part, thereby improving the accuracy and reliability of crack detection for the key part.

[0015] Preferably, it further includes:

[0016] The drone equipped with a camera is also equipped with a lidar. While using the drone to complete real-time image acquisition of the bridge to be detected, radar distance measurement values are obtained in real time.

[0017] Synchronously transmit the image data and the radar distance measurement values in real time and transmit them to the cloud.

[0018] Use the improved optimized second CNN model with multiple branches built into the cloud to replace the second CNN model. According to the branch that adaptively matches the color and format in the AR view marker, judge whether there is a crack at the marker position, as well as the crack position and the severity of the crack; the improved optimized second CNN model is based on the second CNN model, and the input of the model adds the radar distance measurement value synchronized with the AR view at the current moment; during the training process of each branch of the improved second CNN model for whether there is a crack at the marker position and the crack position and the severity of the crack in the bridges of a certain type of bridge, the bridge image data corresponding to the bridge type with potential crack position markers marked with the presence of cracks and the crack position and the severity of the crack and the synchronized radar distance measurement values are used as training data.

[0019] By adopting the above scheme, considering the detection errors that may be caused by relying only on image data in complex environments or bad weather, synchronously collect radar ranging measurement values, and fuse multi-modal data to judge whether there is a crack in the bridge, thereby improving the accuracy and reliability of crack recognition.

[0020] Preferably, it further includes:

[0021] The drone equipped with a camera is also equipped with a multi-band light source device. While using the drone to complete real-time image acquisition of the bridge to be detected, light data in the environmental data is obtained in real time.

[0022] Synchronously transmit the image data and the light data in real time and transmit them to the cloud.

[0023] Using spectral adaptation technology, analyze the light data at the current moment, generate corresponding light adjustment instructions and send them back to the drone, so that the drone controls the multi-band light source device carried to adjust the light wavelength emitted by the multi-band light source, and the adjusted light data is the preset light data for the corresponding bridge type.

[0024] By adopting the above scheme, using spectral adaptation technology to adjust the current light according to the real-time acquired environmental data, thereby ensuring high-quality detection images under different light conditions and improving the accuracy and reliability of crack detection.

[0025] Preferably, it further includes:

[0026] For multiple bridge crack detection reports of the same bridge to be detected at different times, count the number of occurrences of the same crack position where there are cracks and the severity of the cracks is greater than the preset crack severity. If the number of occurrences is greater than the preset number, then determine that the marked position is a key part.

[0027] By adopting the above solution, count the positions where cracks exist in multiple detections and determine them as key parts, so as to achieve accurate positioning and key monitoring of the key parts of the bridge, and improve the reliability and pertinence of the detection.

[0028] Preferably, it further includes:

[0029] Transmit the generated AR view at the current moment to the ground control station in real time, receive the position of the potential crack manually drawn by the user in the AR view and transmit it back to the cloud;

[0030] Display the position of the potential crack manually drawn in the AR view in the cloud, mark it with a preset color and format matching the bridge type, and add additional marks to distinguish the marks of the position of the potential crack identified by the first CNN model.

[0031] By adopting the above solution, allow the user to supplement potential crack information according to actual experience, ensure comprehensive coverage of crack detection, and enhance the flexibility and accuracy of the detection system.

[0032] Preferably, it further includes:

[0033] According to the bridge crack detection report generated by the drone, after comparing the proportion of the judgment results of the existence of cracks and the crack position and severity at the corresponding marked positions of the potential cracks identified by the first CNN model with the actual situation reaches the preset proportion, count the proportion of the judgment results of the existence of cracks and the crack position and severity at the corresponding marked positions of the potential cracks manually drawn.

[0034] If the preset proportion is not reached, then mark the user ID of the position of the potential crack manually drawn. When receiving the position of the potential crack manually drawn sent by the same user ID subsequently, generate a prompt message to reconfirm the position of the potential crack manually drawn and return it to the ground control station, and receive the user to re-determine the position of the potential crack manually drawn or the adjusted position of the potential crack manually drawn.

[0035] By adopting the above solution, on the basis of determining that the second CNN model is relatively accurate, count the user IDs whose proportion of the judgment results of the position of the potential crack manually drawn and the actual situation does not reach the preset proportion, so as to send a reconfirmation message to this part of error-prone users to prompt the accuracy and reliability of crack identification.

[0036] In a second aspect, the present application provides a bridge crack detection system based on an unmanned aerial vehicle, including:

[0037] A bridge image acquisition module, configured to use an unmanned aerial vehicle equipped with a camera to complete real-time acquisition of images of a bridge to be detected according to a preset flight route, and transmit the acquired images to the cloud in real time;

[0038] A bridge AR view construction module, configured to use SLAM technology to compare the transmitted images with a bridge model database stored in the cloud in real time, and construct an AR view at the current moment; the bridge model database stores three-dimensional models of different types of bridges, and the three-dimensional models of each type of bridge are marked with key parts;

[0039] A bridge potential crack identification module, configured to use a first CNN model with multiple branches built in the cloud to identify the bridge type in the AR view at the current moment, and adaptively select a matching branch according to the identified bridge type to determine whether there are potential cracks in the AR view at the current moment. If so, mark the location of the potential crack in the AR view and mark it with a preset color and format matching the bridge type; each branch in the first CNN model is trained and judged on whether a bridge of a certain type of bridge has potential cracks;

[0040] A bridge crack identification module, configured to use a second CNN model with multiple branches built in the cloud to determine whether there is a crack at the marked position and the crack position and severity according to the branches adaptively matched with the color and format in the marks in the AR view; an attention mechanism is introduced into each branch of the second CNN model, and training and judgment are performed on whether there is a crack at the marked position in the bridge image of a certain type of bridge and the crack position and severity;

[0041] A bridge crack report generation module, configured to adjust the heading and flight route of the unmanned aerial vehicle according to the judgment results of the presence of cracks, crack positions, and crack severity at the marked positions in the AR view at the current moment, so as to re-acquire images of the marked positions judged to have cracks and crack severity greater than the preset severity and confirm the presence of cracks, crack positions, and crack severity, and generate a bridge crack detection report in combination with the confirmation results.

[0042] By adopting the above solution, a multi-branch CNN model is used to specifically identify and confirm bridge cracks, improving the detection accuracy. At the same time, by combining CNN and AR technologies, the potential cracks are intuitively displayed

[0043] The position is convenient for more accurate crack identification and positioning, and improves the reliability and efficiency of detection.

[0044] In a third aspect, the present application provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0045] In a fourth aspect, the present application provides a computer device, which includes a memory, a processor, and a program stored and executable on the memory, and the program implements the steps of the above method when executed by the processor.

[0046] In summary, this application has the following beneficial effects:

[0047] 1. Design a multi-model collaboration between a branched first CNN model and a second CNN model to perform preliminary identification of potential cracks, crack identification, and reconfirmation for different types of bridges, improving the accuracy and comprehensiveness of crack identification. Utilizing SLAM technology and a pre-built bridge model database, combined with AR views, the precise location of cracks can be annotated. Incorporating an attention mechanism facilitates more accurate crack identification and facilitates subsequent intuitive confirmation of crack locations by ground control station staff, enabling UAV heading adjustment and maintenance.

[0048] 2. By introducing LiDAR, the fusion of distance measurement data and image data can achieve accurate crack identification and ensure stable detection performance in complex environments. Alternatively, spectral adaptation technology is used to adaptively adjust the lighting to ensure high-quality detection images under different lighting conditions, thereby improving the accuracy and reliability of crack detection.

[0049] 3. Use AR technology to provide real-time AR views for ground control station staff, allowing them to supplement potential crack information based on actual experience to ensure comprehensive coverage of crack detection. Combined with the results of crack identification using the CNN model, the system statistically identifies staff who are prone to making errors when supplementing potential crack information, and generates confirmation prompts during the subsequent supplementation process to ensure more accurate supplementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a bridge crack detection method based on a drone in a specific embodiment;

[0051] Figure 2 It is a UAV flight mission management business structure in a UAV-based bridge crack detection method described in a specific embodiment;

[0052] Figure 3 Schematic diagram of the structure of a UAV-based bridge crack detection system described in a specific embodiment. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further details this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0054] As Figure 1 shown, an embodiment of this application discloses a method for detecting bridge cracks based on an unmanned aerial vehicle, including:

[0055] S1. Use an unmanned aerial vehicle equipped with a camera to complete real-time image acquisition of the bridge to be detected according to a preset flight route and transmit the images to the cloud.

[0056] Specifically, the unmanned aerial vehicle equipped with a camera is communicatively connected to the ground control station and the service platform in the cloud in pairs, responsible for receiving control instructions from the ground control station, completing real-time image acquisition of the bridge to be detected according to the preset flight route, and transmitting the acquired images to the service platform in the cloud in real time.

[0057] S2. Use SLAM technology to compare the transmitted images with the bridge model database stored in the cloud in real time and construct an AR view at the current moment.

[0058] Specifically, the bridge model database stores three-dimensional models, structural information, and historical crack data of several bridges of different types, such as beam bridges, arch bridges, suspension bridges, and cable-stayed bridges, etc.; and for each type, due to different bridge structures, the corresponding key parts are different, such as the main beam, bearings, and piers of a beam bridge; the arch ring and arch feet of an arch bridge; the main cable, bridge tower, suspenders, and anchorages of a suspension bridge; the stay cables, bridge tower, and deck girder of a cable-stayed bridge, etc. Therefore, key parts are marked separately for different bridges, such as in text form.

[0059] Use SLAM technology to compare the transmitted images with the bridge model database stored in the cloud in real time. According to feature data such as key part data, match a three-dimensional bridge model in the bridge model database stored in the cloud that is of the same type as the current real-time comparison transmitted image and has a similarity greater than the preset similarity, and determine the position on the bridge where the current unmanned aerial vehicle acquired the image; after completing model matching and positioning, construct an AR view at the current moment, and superimpose and display the three-dimensional model and structural information (such as key part information) of the bridge in this view.

[0060] To facilitate providing an intuitive and comprehensive visual interface for the ground control station, the generated AR view at the current moment can be transmitted to the ground control station in real time.

[0061] S3. Use the first CNN model with multiple branches built into the cloud to identify potential cracks on the bridge in the AR view at the current moment and mark the positions where the potential cracks are located.

[0062] Specifically, there are several CNN models stored in the cloud, including a first CNN model with multiple branches; the first CNN model includes a bridge type recognition sub-model, which preprocesses the input AR view, extracts key features, and quickly identifies the bridge type; according to the identified bridge type, a matching branch is adaptively selected, and each branch of the first CNN model trains and judges whether there are potential cracks in the bridges of a certain bridge type. Specifically, a performance-excellent basic CNN architecture, such as ResNet, VGG, etc., is selected for the backbone network of each branch, and the basic CNN architecture is fine-tuned according to the crack characteristics of each bridge type. For example, the number of convolutional layers, the size and stride of the filters can be adjusted; an independent training plan is formulated for each branch, and the corresponding type of bridge crack image data marked with the presence or absence of potential cracks is used for training to ensure that each branch can focus on extracting the crack characteristics of this type of bridge; all the trained CNN branches are integrated into a unified model to generate the first CNN model.

[0063] In this embodiment, the first CNN model with multiple branches built in the cloud is used. The input of the model is the AR view at the current moment. The bridge type in the AR view at the current moment is recognized, and a matching branch is adaptively selected according to the recognized bridge type to judge whether there are potential cracks in the AR view at the current moment. If so, the location of the potential crack is marked in the AR view and marked with a preset color and format matching the bridge type. Among them, the preset color and format markings for each bridge type are different. For example, the preset color for an arch bridge is yellow and it is marked with a dot with a radius of X1, and the preset color for a beam bridge is red and it is marked with a triangle with a side length of X2.

[0064] S4. Use the second CNN model with multiple branches built in the cloud to judge whether there is a crack at the location of the potential crack marked in the AR view at the current moment and the severity of the crack.

[0065] Specifically, among the several CNN models stored in the cloud, there is also a second CNN model with multiple branches; the second CNN model is provided with multiple branches, and an attention mechanism is introduced into each branch, enabling it to automatically focus on the key areas in the image, such as the marked location of the potential crack, and trains and judges whether there is a crack at the marked location in the bridge image of a certain bridge type and the crack location and the severity of the crack; specifically, each branch uses the corresponding type of bridge crack image data marked with the crack location and the severity of the crack (such as: extremely minor, minor, general, relatively serious, extremely serious, with the level increasing gradually) and having the marked location of the potential crack for training to ensure that each branch can focus on extracting the crack characteristics of this type of bridge, and all the trained branches are integrated into a unified model to generate the second CNN model.

[0066] In this embodiment, a second CNN model with multiple branches built into the cloud is utilized. The input of the model is the current environmental AR view that marks the location of the potential crack. According to the branch that adaptively matches the color and format in the mark in the AR view, the matching branch is used to determine whether there is a crack at the marked position and the severity of the crack, and the judgment result is output.

[0067] S5. Adjust the heading and flight route of the drone according to the judgment result of the existence of cracks at the marked positions in the AR view at the current moment, the crack positions, and the severity of the cracks, and conduct a secondary confirmation of the suspected severely cracked area.

[0068] Specifically, since the position area with severe cracks needs to be urgently processed, a secondary confirmation scan is performed on the suspected severely cracked position area to ensure that the subsequent emergency treatment resources are not wasted.

[0069] According to the judgment result of the existence of cracks at the marked positions in the AR view at the current moment, the crack positions, and the severity of the cracks, once it is determined that there are cracks in the AR view at the current moment and the marked positions with the severity of the cracks greater than the preset severity, as Figure 2 shown, generate a flight instruction to adjust the original flight route so that at the next moment, the drone will fly over the marked positions in the AR view at the current moment where there are cracks and the severity of the cracks is greater than the preset severity again and then continue to fly according to the original flight route, and transmit the instruction to the ground control station. The ground control station controls the drone to adjust its heading according to the instruction, and re - collect images of the marked positions in the AR view at the current moment where there are cracks and the severity of the cracks is greater than the preset severity. For the re - collected marked positions, repeat S3 - S4 to complete the confirmation of the existence of cracks and the severity of the cracks at the marked positions.

[0070] In addition, in order to strengthen the re - confirmation of the suspected key parts with severe cracks, the process of judging the marked positions where there are cracks and the severity of the cracks is greater than the preset severity includes: judging whether the marked position is a key part. If it is a key part, adaptively adjust the preset severity of the cracks so that the adjusted preset severity of the cracks is lower than the preset severity before adjustment. For example: the preset severity of the cracks before adjustment is relatively severe. If it is judged to be a key part, the preset severity of the cracks is general.

[0071] S6. Generate a bridge crack detection report in combination with the confirmation result.

[0072] Specifically, in combination with the confirmation result, if the confirmation result is that the marker position result of determining the existence of cracks and the severity of cracks at the current moment in the AR view is correct and the severity of cracks is greater than the preset severity, then retain the marker positions where cracks are confirmed to exist and the severity of cracks is greater than the preset severity; otherwise, retain the result of reconfirmation; generate a bridge crack detection report including all crack positions and crack severities based on all judgment and confirmation results.

[0073] In a specific embodiment, to further avoid errors in judging and locating the existence of cracks using a single image data under complex environmental conditions and improve the accuracy and stability of the detection result, the method further includes:

[0074] The drone equipped with a camera is also equipped with a lidar. While using the drone to complete real-time image acquisition of the bridge to be detected, the radar distance measurement value is obtained in real time; among them, the lidar Liar can still obtain fine distance information on the surface of the object in a low visibility environment, so as to judge whether there are cracks.

[0075] Synchronously transmit the image data and the radar distance measurement value in real time and transmit them to the cloud;

[0076] Use the improved optimized second CNN model with multiple branches built into the cloud to replace the second CNN model. According to the branch that adaptively matches the color and format in the marker in the AR view, judge whether there are cracks at the marker position and the crack position and severity; among them, the improved optimized second CNN model is based on the second CNN model. In terms of the design of the model input, the radar distance measurement value synchronized with the AR view at the current moment is added to the model input. Correspondingly, during the training process of each branch of the improved second CNN model for judging whether there are cracks at the marker position and the crack position and severity in a bridge of a certain type of bridge, the bridge image data with potential crack position markers marked with the existence of cracks and the crack position and severity and the synchronized radar distance measurement value corresponding to the bridge type are used as training data.

[0077] In addition, a drone equipped with an infrared camera can also be used. While using the drone to complete real-time image acquisition of the bridge to be detected, the temperature distribution data is obtained in real time. Synchronously transmit the image data, radar ranging data and temperature distribution data in real time to the cloud. Based on the second CNN model, further optimize the model input to the real-time transmitted image data, radar ranging data and temperature distribution data, and correspondingly perform optimized training. Use the trained optimized second CNN model to judge whether there are cracks at the marker position and the crack position and severity according to the branch that adaptively matches the color and format in the marker in the AR view.

[0078] A specific embodiment. Due to special geographical locations, bridges of the same type in different positions are often affected by complex lighting conditions such as strong direct sunlight, shadow occlusion, and fog. To ensure the accuracy of bridge cracks under different lighting conditions, the method further includes:

[0079] The drone equipped with a camera is also equipped with a multi - band light source device. While using the drone to complete real - time image acquisition of the bridge to be detected, real - time acquisition of lighting data in the environmental data is performed, including: the intensity, color temperature, and wavelength distribution of light, etc.

[0080] Synchronously and real - time transmit the image data and lighting data and transmit them to the cloud.

[0081] Using a spectral adaptation algorithm, analyze the lighting data at the current moment, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi - band light source device carried to adjust the wavelength of the light emitted by the multi - band light source, and the adjusted lighting data is the lighting data preset for the bridge type; among them, the preset lighting data is determined according to the lighting conditions with the highest crack recognition accuracy for different types of bridges in history.

[0082] In addition, further considering the materials and colors of different types of beams, the preset lighting data under corresponding conditions can be further determined according to the lighting conditions with the highest crack recognition accuracy for bridges of different types, different materials, and colors in history. Obtain the material and color of the bridge to be detected currently, use the spectral adaptation algorithm, analyze the lighting data at the current moment, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi - band light source device carried to adjust the wavelength of the light emitted by the multi - band light source, and make the adjusted lighting data be the lighting data preset for the bridge type, corresponding bridge material, and color.

[0083] A specific embodiment. Considering that in the process of marking potential cracks in the AR view, introducing manual interaction can improve the accuracy and flexibility of crack recognition, the method further includes:

[0084] Transmit the currently constructed AR view in real - time to the ground control station, receive the position of the potential crack manually drawn by the user in the AR view and transmit it back to the cloud;

[0085] Display the position of the potential crack manually drawn in the AR view in the cloud, mark it with a preset color and format matching the bridge type, and add additional marks to distinguish the marks of the position of the potential crack recognized by the first CNN model. Specifically, on the basis of the preset color and format, new marks can be added, such as adding an outer - ring mark on the original format.

[0086] In addition, considering that it is also very important to ensure the accuracy of the user's manual marking during the introduction of manual interaction and to avoid wasting computing resources due to incorrect markings by the user, the method further includes:

[0087] According to the generated bridge crack detection report of the drone, compare the proportion (the proportion of the total number of judgment results) of the judgment results that the cracks exist at the corresponding marked positions of the potential cracks identified by the first CNN model and the judgment results of the crack positions and crack severity being consistent with the actual situation (i.e., the similarity is greater than the preset similarity of the corresponding type of bridge). After reaching the preset proportion, it indicates that the accuracy of using the second CNN model to identify cracks meets the requirements. On this basis, count the proportion of the judgment results that the cracks exist at the corresponding marked positions of the potential cracks drawn manually and the judgment results of the crack positions and crack severity being consistent with the actual situation;

[0088] If the preset proportion is not reached, it indicates that the error rate of the manually drawn potential crack positions marked by the corresponding user is relatively high. Mark the user ID of the manually drawn potential crack positions. When receiving the manually drawn potential crack positions sent by the same user ID subsequently, generate a prompt message to reconfirm the manually drawn potential crack positions and return it to the ground control station, and receive the user to re-determine or adjust the manually drawn potential crack positions.

[0089] Of course, in addition to allowing the user to hand-draw potential cracks, the AR view can also visually display information, including crack trend information; specifically, when it is determined that there is a crack in the AR view at the current moment, query the historical crack data of this position according to the crack position, input the crack data at the current moment into the crack trend prediction model (such as a CNN model) of the corresponding bridge type trained with the queried historical crack data, obtain the crack trend, and display the crack trend in a layer of the AR view and transmit it to the ground control station.

[0090] In a specific embodiment, in order to further improve the correctness of detecting high-incidence crack positions, the method further includes:

[0091] For multiple bridge crack detection reports of the same bridge to be detected at different time periods, count the number of times the same crack position with cracks and crack severity greater than the preset crack severity appears in the reports. If the number of appearances is greater than the preset number, it indicates that this position belongs to a position prone to cracks, and this marked position is identified as a key part.

[0092] As Figure 3 shown, the embodiment of the present application discloses a bridge crack detection system based on a drone, including:

[0093] The bridge image acquisition module 101 is used to use a drone equipped with a camera to complete real-time image acquisition of the bridge to be detected according to a preset flight route, and transmit the acquired images to the cloud in real time;

[0094] The bridge AR view construction module 102 is used to use SLAM technology to compare the transmitted images with the bridge model database stored in the cloud in real time and construct an AR view at the current moment; the bridge model database stores three-dimensional models of different types of bridges, and the three-dimensional models of each type of bridge are marked with key parts;

[0095] The bridge potential crack identification module 103 is used to use the first CNN model with multiple branches built in the cloud to identify the bridge type in the AR view at the current moment, and adaptively select a matching branch according to the identified bridge type to judge whether there are potential cracks in the AR view at the current moment. If there are, mark the location of the potential crack in the AR view and mark it in a preset color and format matching the bridge type; each branch in the first CNN model is trained and judged on whether a certain type of bridge has potential cracks.

[0096] The bridge crack identification module 104 is used to use the second CNN model with multiple branches built in the cloud to judge whether there is a crack at the marked position and the crack position and severity according to the branch adaptively matched with the color and format in the mark in the AR view; each branch of the second CNN model introduces an attention mechanism, and is trained and judged on whether there is a crack at the marked position in the bridge image of a certain type of bridge and the crack position and severity.

[0097] The bridge crack report generation module 105 is used to adjust the heading and flight route of the drone according to the judgment results of the presence of cracks, crack positions and severity at the marked positions in the AR view at the current moment, so as to re-acquire images of the marked positions judged to have cracks and crack severity greater than the preset severity and confirm the presence of cracks, crack positions and severity, and generate a bridge crack detection report in combination with the confirmation results.

[0098] In a specific embodiment, the system further includes:

[0099] The radar distance measurement value acquisition module 106 is used for the drone equipped with a camera to be further equipped with a lidar. While using the drone to complete real-time image acquisition of the bridge to be detected, it obtains radar distance measurement values in real time; synchronously transmits the image data and radar distance measurement values in real time and transmits them to the cloud;

[0100] The bridge crack identification module 104 is further configured to replace the second CNN model with an improved optimized second CNN model with multiple branches built in the cloud, and determine whether there is a crack at the marker position, as well as the crack position and the severity of the crack, according to the branch that adaptively matches the color and format in the AR view; the improved optimized second CNN model is based on the second CNN model, and the input of the model adds the radar distance measurement value synchronized with the AR view at the current moment; during the training process of each branch of the improved second CNN model for whether there is a crack at the marker position in a bridge of a certain type of bridge, as well as the crack position and the severity of the crack, the bridge image data corresponding to the bridge type with potential crack marker positions marked with the presence of cracks, the crack position and the severity of the crack, and the synchronized radar distance measurement values are used as training data.

[0101] In a specific embodiment, the system further includes:

[0102] The illumination data acquisition module 107 is configured to the drone carrying the camera is also equipped with a multi-band light source device. While using the drone to complete the real-time acquisition of the images of the bridge to be detected, the illumination data in the environmental data is acquired in real time; the image data and the illumination data transmitted in real time are synchronously transmitted and sent to the cloud.

[0103] The bridge image acquisition module 101 is further configured to use the spectral adaptation algorithm to analyze the illumination data at the current moment, generate corresponding illumination adjustment instructions and send them back to the drone, so that the drone controls the multi-band light source device carried to adjust the light wavelength emitted by the multi-band light source.

[0104] In a specific embodiment, the bridge AR view construction module 102 in the system is further configured to, for multiple bridge crack detection reports of the same bridge to be detected at different times, count the number of times the same crack position with cracks and the severity of the crack greater than the preset crack severity appears in the reports. If the number of appearances is greater than the preset number, it is determined that the marker position is a key part.

[0105] The bridge potential crack interaction module 108 is configured to transmit the currently generated AR view in real time to the ground control station, receive the position of the potential crack manually drawn by the user in the AR view and send it back to the cloud; display the position of the potential crack manually drawn in the AR view in the cloud, marked with a preset color and format matching the bridge type, and add additional marks to distinguish the marks of the position of the potential crack identified by the first CNN model.

[0106] It is also used to generate a bridge crack detection report for the drone. After comparing the proportion of the judgment results of the presence of cracks at the marked positions corresponding to the potential cracks identified by the first CNN model and the crack positions and severity with the actual situation reaches a preset proportion, the proportion of the judgment results of the presence of cracks at the marked positions corresponding to the potential cracks manually drawn and the crack positions and severity with the actual situation is statistically calculated; if the preset proportion is not reached, the user ID of the manually drawn potential crack position is marked. When receiving the manually drawn potential crack position sent by the same user ID subsequently, a prompt message for reconfirming the manually drawn potential crack position is generated and returned to the ground control station, and the user is received to re-determine the manually drawn potential crack position or adjust the manually drawn potential crack position.

[0107] The embodiment of the present application also discloses a computer-readable storage medium.

[0108] Specifically, the computer-readable storage medium stores a computer program that can be loaded and executed by a processor, such as the above-mentioned drone-based bridge crack detection method. The computer-readable storage medium includes, for example: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0109] The embodiment of the present application also discloses a computer device.

[0110] Specifically, the computer device includes a memory and a processor. The memory stores a computer program that can be loaded and executed by the processor, such as the above-mentioned drone-based bridge crack detection method.

[0111] The above are all the preferred embodiments of the present application. Without limiting the protection scope of the present application accordingly, any feature disclosed in this specification (including the abstract and drawings), unless specifically described, can be replaced by other equivalent or similar-purpose alternative features. That is, unless specifically described, each feature is only an example in a series of equivalent or similar features.

Claims

1. A method for detecting bridge cracks based on drones, characterized in that, Including: Using a drone equipped with a camera to complete real-time image acquisition of the bridge to be detected according to a preset flight route, and transmitting the acquired images to the cloud in real time; Using SLAM technology to compare the transmitted images with the bridge model database stored in the cloud in real time and construct an AR view at the current moment; The bridge model database stores 3D models of different types of bridges, and the 3D models of each type of bridge are marked with key parts; Using the first CNN model with multiple branches built into the cloud to identify the bridge type in the AR view at the current moment, and adaptively selecting a matching branch according to the identified bridge type to determine whether there are potential cracks in the AR view at the current moment. If so, mark the location of the potential crack in the AR view and mark it in a preset color and format matching the bridge type; Each branch in the first CNN model is trained and judged on whether there are potential cracks in a certain type of bridge; Using the second CNN model with multiple branches built into the cloud to judge whether there is a crack, the crack location and the severity of the crack at the marked position according to the branch adaptively matched with the color and format in the mark in the AR view; Each branch of the second CNN model introduces an attention mechanism to train and judge whether there is a crack, the crack location and the severity of the crack at the marked position in the bridge image of a certain type of bridge; Adjust the heading and flight route of the drone according to the judgment results of the existence of cracks, the crack location and the severity of the crack at the marked position in the AR view at the current moment, so as to re-acquire images and confirm the existence of cracks, the crack location and the severity of the crack at the marked positions where the existence of cracks and the severity of the cracks are greater than the preset severity, and generate a bridge crack detection report in combination with the confirmation results; Also including: The drone equipped with a camera is also equipped with a lidar. While using the drone to complete real-time image acquisition of the bridge to be detected, the radar distance measurement value is obtained in real time; Synchronously transmit the image data and the radar distance measurement value in real time and transmit them to the cloud; Using the improved optimized second CNN model with multiple branches built into the cloud to replace the second CNN model, and judging whether there is a crack, the crack location and the severity of the crack at the marked position according to the branch adaptively matched with the color and format in the mark in the AR view; The improved optimized second CNN model with multiple branches is based on the second CNN model, and the input of the model adds the radar distance measurement value synchronized with the AR view at the current moment; During the training process of each branch of the improved second CNN model for judging whether there is a crack, the crack location and the severity of the crack at the marked position in a certain type of bridge, the bridge image data corresponding to the bridge type with potential crack locations marked with the existence of cracks, the crack location and the severity of the crack and the synchronized radar distance measurement value are used as training data; The drone equipped with a camera is also equipped with a multi-band light source device. While using the drone to complete real-time image acquisition of the bridge to be detected, the light data in the environmental data is obtained in real time; Synchronize the image data and light data for real-time transmission and transmit them to the cloud; Obtain the material and color of the bridge to be detected currently. Use the spectral adaptation algorithm to analyze the light data at the current moment, generate corresponding light adjustment instructions and send them back to the drone, so that the drone controls the multi-band light source device carried to adjust the light wavelength emitted by the multi-band light source, and the adjusted light data is the preset light data corresponding to the bridge type, the bridge material and color; Determine the preset light data under the corresponding conditions according to the light conditions with the highest crack recognition accuracy for bridges of different types, materials and colors in history; Transmit the generated AR view at the current moment to the ground control station in real time, receive the position of the potential crack manually drawn by the user in the AR view and transmit it back to the cloud; Display the position of the potential crack manually drawn in the AR view in the cloud, mark it with a preset color and format matching the bridge type, and add additional marks to distinguish the marks of the position of the potential crack identified by the first CNN model; According to the generated bridge crack detection report of the drone, after the ratio of the judgment results of the existence of cracks and the crack position and severity at the corresponding marked positions identified by the first CNN model to the actual situation reaches the preset ratio, count the ratio of the judgment results of the existence of cracks and the crack position and severity at the corresponding marked positions based on the manually drawn potential crack positions to the actual situation; If the preset ratio is not reached, mark the user ID of the position of the potential crack manually drawn. When receiving the position of the potential crack manually drawn sent by the same user ID subsequently, generate a prompt message to confirm the position of the potential crack manually drawn again and return it to the ground control station, and receive the user to confirm the position of the potential crack manually drawn again or the adjusted position of the potential crack manually drawn.

2. The method for detecting bridge cracks based on an unmanned aerial vehicle according to claim 1, wherein The process of determining the marked positions with cracks and the crack severity greater than the preset severity includes: Judge whether the marked position is a key part. If it is a key part, adaptively adjust the preset crack severity so that the adjusted preset crack severity is lower than the preset crack severity before adjustment.

3. The method for detecting bridge cracks based on an unmanned aerial vehicle according to claim 1, wherein, It also includes: For multiple bridge crack detection reports of the same bridge to be detected at different times, count the number of times the same crack position with cracks and the crack severity greater than the preset crack severity appears in the reports. If the number of appearances is greater than the preset number, determine that the marked position is a key part.

4. A drone-based bridge crack detection system, characterized in that, It includes: Bridge image acquisition module, which is used to use the drone equipped with a camera to complete the real-time acquisition of the images of the bridge to be detected according to the preset flight route, and transmit the acquired images to the cloud in real time; Bridge AR view construction module, which is used to use the SLAM technology to compare the transmitted images with the bridge model database stored in the cloud in real time and construct the AR view at the current moment; The bridge model database stores three-dimensional models of different types of bridges, and the three-dimensional models of each type of bridge are marked with key parts; The bridge potential crack identification module is used to identify the bridge type in the AR view at the current moment by using the first CNN model with multiple branches built in the cloud, adaptively select a matching branch according to the identified bridge type to judge whether there are potential cracks in the AR view at the current moment. If there are, mark the location of the potential crack in the AR view and mark it with a preset color and format matching the bridge type; Each branch in the first CNN model is trained and judged on whether there are potential cracks in a certain type of bridge; The bridge crack identification module is used to judge whether there is a crack at the marked position and the crack position and severity according to the branch adaptively matched according to the color and format in the mark in the AR view by using the second CNN model with multiple branches built in the cloud; Each branch of the second CNN model introduces an attention mechanism to train and judge whether there is a crack at the marked position in the bridge image of a certain type of bridge and the crack position and severity; The bridge crack report generation module is used to adjust the heading and flight route of the drone according to the judgment results of the existence of cracks and the crack position and severity in the AR view at the current moment, so as to re-collect images of the marked positions judged to have cracks and the crack severity greater than the preset severity and confirm the existence of cracks and the crack position and severity. Combine the confirmation results to generate a bridge crack detection report; The radar distance measurement value acquisition module is used for the drone equipped with a camera to also be equipped with a lidar. While using the drone to complete real-time image acquisition of the bridge to be detected, real-time radar distance measurement values are obtained; The image data and radar distance measurement values transmitted in real time are synchronously transmitted to the cloud; The bridge crack identification module is also used to replace the second CNN model with an improved optimized second CNN model with multiple branches built in the cloud, and judge whether there is a crack at the marked position and the crack position and severity according to the branch adaptively matched according to the color and format in the mark in the AR view; The improved optimized second CNN model with multiple branches is based on the second CNN model, and the input of the model adds the radar distance measurement value synchronized with the AR view at the current moment; During the training process of each branch of the improved second CNN model on whether there is a crack at the marked position in a certain type of bridge and the crack position and severity, the bridge image data corresponding to the bridge type with potential crack marked positions marked with the existence of cracks and the crack position and severity and the synchronized radar distance measurement values are used as training data; The illumination data acquisition module is used for the drone equipped with a camera to also be equipped with a multi-band light source device. While using the drone to complete real-time image acquisition of the bridge to be detected, real-time illumination data in the environmental data is obtained; The image data and illumination data transmitted in real time are synchronously transmitted to the cloud; The bridge image acquisition module is also used to obtain the material and color of the bridge to be detected currently, analyze the current lighting data by using the spectral adaptation algorithm, generate corresponding lighting adjustment instructions and transmit them back to the drone, so that the drone controls the multi-band light source device carried to adjust the light wavelength emitted by the multi-band light source, and the adjusted lighting data is the lighting data preset for the corresponding bridge type, corresponding bridge material and color; determine the preset lighting data under the corresponding conditions according to the lighting conditions with the highest crack recognition accuracy for bridges of different types, different materials and colors in history; The bridge potential crack interaction module is used to transmit the generated AR view at the current moment to the ground control station in real time, receive the position of the potential crack manually drawn by the user in the AR view and transmit it back to the cloud; display the position of the potential crack manually drawn in the AR view in the cloud, mark it with a preset color and format matching the bridge type, and add additional marks to distinguish the marks of the position of the potential crack identified by the first CNN model; It is also used to generate a bridge crack detection report for the drone. After comparing the proportion of the judgment results of the existence of cracks and the crack positions and severity at the corresponding marked positions of the potential cracks identified by the first CNN model with the actual situation reaches a preset proportion, count the proportion of the judgment results of the existence of cracks and the crack positions and severity at the corresponding marked positions of the potential cracks manually drawn that match the actual situation; if the preset proportion is not reached, mark the user ID of the position of the potential crack manually drawn. When receiving the position of the potential crack manually drawn sent by the same user ID subsequently, generate a prompt message to confirm the position of the potential crack manually drawn again and return it to the ground control station, and receive the user to confirm the position of the potential crack manually drawn again or adjust the position of the potential crack manually drawn.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of claims 1 to 3.

6. A computer device, characterized in that, The computer device includes a memory, a processor, and a program stored and executable on the memory. When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 3.

Images