Airport pavement disease detection method and system

Abstract

The present invention provides a method for detecting airport pavement defects, including: obtaining a trained YOLOv5 target detection model and a DDRNet semantic segmentation model; optimizing the trained YOLOv5 and DDRNet based on an inference optimizer; deploying the optimized YOLOv5 and DDRNet and an image acceleration decoder on a movable inspection body; controlling the movable inspection body to travel according to a preset driving path, and controlling an image acquisition device to acquire images; the image acceleration decoder acquires and decodes the acquired images in real time; the optimized YOLOv5 detects the decoded images to obtain a first detection result; the optimized DDRNet extracts contour information of an image region with defects based on the first detection result to obtain a second detection result. The present invention also provides an airport pavement defect detection system. The present invention can obtain detection results in real time during the inspection process.

Description

Technical Field

[0001] The present application relates to the technical field of airport pavement defect detection, and specifically to an airport pavement defect detection method and system. Background Art

[0002] The airport pavement is an important facility of the airport. Under the influence of climate, the airport pavement will gradually develop cracks, incomplete marking lines and other defects. The airport needs to pay attention to the dynamic changes of the above information at all times to eliminate safety hazards and ensure the safe operation of the airport.

[0003] With the rapid growth of air traffic, airport pavement inspection has changed from the initial manual inspection method to the automated inspection method. Currently, radar detection technology and deep learning technology are the mainstream automatic detection technologies, but the price of laser radar is relatively expensive, and the detection accuracy is difficult to meet the detection requirements of small-sized cracks on the pavement; in terms of deep learning, target detection algorithms and semantic segmentation algorithms are used to process airport pavement images, but the target detection algorithm cannot extract the contour information of the cracks, so it cannot determine the severity of the cracks on the airport pavement; the semantic segmentation algorithm can extract contour information, but the real-time detection is poor, such as the airport pavement disease foreign body detection method and system based on deep learning disclosed in patent document CN113111704A. Summary of the invention

[0004] In view of the above technical problems, the technical solution adopted by this application is:

[0005] The embodiment of the present invention provides a method for detecting airport pavement defects, comprising the following steps:

[0006] S100, respectively optimizing and building the YOLOv5 convolutional neural network model and the DDRNet convolutional neural network model and training them to obtain the trained YOLOv5 target detection model and DDRNet semantic segmentation model;

[0007] S200, optimizes the trained YOLOv5 object detection model and DDRNet semantic segmentation model based on the preset inference optimizer to improve the inference speed of the model;

[0008] S300, deploying the optimized YOLOv5 target detection model, DDRNet semantic segmentation model and image acceleration decoder on an image detection device of a movable inspection body; the movable inspection body is also provided with an image acquisition device;

[0009] S400, controlling the movable inspection body to travel along a preset travel path, and controlling the image acquisition device to acquire an image of the airport pavement;

[0010] S500, the image acceleration decoder acquires and decodes the collected airport pavement image in real time, and sends the decoded image to the optimized YOLOv5 target detection model;

[0011] S600, using the optimized YOLOv5 target detection model to detect the decoded image, and obtain a first detection result, wherein the first detection result includes a disease type and a disease location;

[0012] S700, using the optimized DDRNet semantic segmentation model to extract contour information of the image area where the disease exists based on the first detection result, to obtain a second detection result, where the second detection result includes the disease contour information.

[0013] Another embodiment of the present invention provides an airport pavement disease detection system, comprising: a movable inspection body and a remote control terminal connected in communication, wherein the movable inspection body is provided with an image acquisition device and an image detection device;

[0014] The image detection device includes an image acceleration decoder, a YOLOv5 target detection model optimized by a preset inference optimizer, and a DDRNet semantic segmentation model;

[0015] Wherein, the movable inspection body is used to perform inspection according to a preset driving path based on the control instruction sent by the remote control terminal;

[0016] The image acquisition device is used to collect the airport pavement image during the inspection process of the movable inspection body and send it to the image detection device;

[0017] The image detection device is used for:

[0018] The collected airport pavement images are acquired and decoded in real time through the image acceleration decoder, and the decoded images are sent to the optimized YOLOv5 target detection model;

[0019] Detecting the decoded image using an optimized YOLOv5 target detection model to obtain a first detection result, where the first detection result includes a type of disease and a location of the disease in the image;

[0020] The optimized DDRNet semantic segmentation model is used to extract the contour information of the image area where the disease exists based on the first detection result to obtain a second detection result, wherein the second detection result includes the contour information of the disease; and the first detection result and the second detection result are sent to the remote control end.

[0021] The airport pavement defect detection method and system provided in the embodiments of the present invention use a thrust optimizer to optimize the YOLOv5 target detection model and the DDRNet semantic segmentation model, and use an image acceleration decoder to quickly decode the collected airport pavement images. Therefore, during the inspection process, detection results containing defect type, defect location and defect contour information can be obtained in real time, and defects can be discovered in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A schematic diagram of a flow chart of a method for detecting airport pavement defects provided in an embodiment of the present application;

[0024] Figure 2 A structural block diagram of an airport pavement defect detection system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0026] Figure 1 The following is a flow chart of a method for detecting airport pavement defects provided in an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a method for detecting airport pavement defects, comprising the following steps:

[0027] S100, respectively optimize and build the YOLOv5 convolutional neural network model and the DDRNet convolutional neural network model and train them to obtain the trained YOLOv5 target detection model and DDRNet semantic segmentation model.

[0028] In the embodiment of the present invention, the training sample set can be obtained in the following way: obtain and annotate images of pavement defects of airports and urban roads from multiple airports and third-party institutions, and use Mosaic and MixUp data enhancement technologies to enhance the data, so as to obtain 80,000 training data and 20,000 test data. In the embodiment of the present invention, tensorflow2.5 is selected as the deep learning framework to construct YOLOv5 and DDRNet convolutional neural network models. In the embodiment of the present invention, pavement defects may include broken joints, peeling of plate corners, cracks, patches, potholes, etc.

[0029] In the embodiment of the present invention, the YOLOv5 target detection model uses the k-means clustering algorithm to perform cluster analysis on the target detection frame in the defect data set image, obtains the new number and size of anchors suitable for airport pavement defect detection, and modifies the number and size of anchors in the YOLOv5 network configuration file according to the parameters obtained by the k-means clustering algorithm; in order to improve the model's detection ability for small-sized defects such as cracks and potholes, the DDRNet semantic segmentation model uses the OhemCELoss loss function for training. The construction, training and testing of the YOLOv5 and DDRNet convolutional neural networks are all carried out on a training server equipped with 4 NVIDIA Tesla V100s (32GB video memory).

[0030] S200 optimizes the trained YOLOv5 object detection model and DDRNet semantic segmentation model based on the preset inference optimizer to improve the inference speed of the convolutional neural network model.

[0031] In an embodiment of the present invention, the inference optimizer may use TensorRT launched by Nvidia. The optimization of the convolutional neural network model by TensorRT mainly includes reducing the amount of model inference calculation and reconstructing the model, specifically including:

[0032] Reduce the FP32 data type parameters in the convolutional neural network model to FP16 or INT8 data types through mapping operations to reduce the amount of model inference calculations; and

[0033] The network layers in the convolutional neural network model are simplified through inter-layer fusion and tensor fusion to simplify the model structure. Specifically, the common block structures in the convolutional neural network model can be merged into one network layer through inter-layer fusion, multiple network layers with the same structure can be merged into one network layer through tensor fusion, and the model structure can be simplified by automatically deleting the connection layer and other model simplification methods. By reconstructing the model, the CUDA core occupancy of the model on the GPU and the calling frequency are reduced, making the entire model structure smaller, faster, and more efficient.

[0034] S300, deploying the optimized YOLOv5 target detection model, DDRNet semantic segmentation model and image acceleration decoder on the image detection device of the movable inspection body; the movable inspection body is also provided with an image acquisition device.

[0035] In an illustrative embodiment of the present invention, the image acceleration decoder may be DeepStream. DeepStream is an SDK launched by Nvidia to accelerate streaming video analysis. Through this SDK, it is possible to quickly complete the process from video stream capture to output of deep learning algorithm recognition results on the GPU. A stream processing pipeline is built through DeepStream, and the camera video stream is input into the pipeline; the fast decoding module quickly decodes the video stream through the GPU and obtains sequence frame images; the image preprocessing module performs color channel conversion, size scaling and normalization on the image through the GPU, so that the image meets the input requirements of the YOLOv5 algorithm.

[0036] In an embodiment of the present invention, the movable inspection body may be an inspection vehicle that can move automatically. The image acquisition device may be composed of three industrial cameras, wherein one camera observes the front lower part of the inspection vehicle, and the other two cameras observe the side lower part of the inspection vehicle.

[0037] In a specific embodiment, the optimized YOLOv5 target detection model and DDRNet semantic segmentation model and image acceleration decoder can be deployed on the AI ​​edge computing platform of the image detection device. In an illustrative embodiment, the Jetson AGX Xavier development kit can be used as the AI ​​edge computing platform of the present invention; on the AI ​​edge computing platform, TensorRT and DeepStream are installed; TensorRT is used to optimize the trained YOLOv5 and DDRNet model files (.PB) to generate .engine files; the optimized YOLOv5 and DDRNet models are loaded through DeepStream to complete the deployment of the model.

[0038] S400, controlling the movable inspection body to travel along a preset travel path, and controlling the image acquisition device to acquire an image of the airport pavement.

[0039] The preset driving path can be determined based on the inspection task and can be generated by the remote control terminal.

[0040] S500, the image acceleration decoder acquires and decodes the collected airport pavement image in real time, and sends the decoded image to the optimized YOLOv5 target detection model.

[0041] S600, using the optimized YOLOv5 target detection model to detect the decoded image to obtain a first detection result, where the first detection result includes a disease type and a disease location.

[0042] Using the optimized YOLOv5 target detection model to detect the decoded image and obtain the first detection result may be a prior art. When the optimized YOLOv5 target detection model detects a disease, it will mark the disease on the image, for example, marking an outer rectangular frame of the disease on the image.

[0043] S700, using the optimized DDRNet semantic segmentation model to extract contour information of the image area where the disease exists based on the first detection result, to obtain a second detection result, where the second detection result includes the disease contour information.

[0044] It is prior art to use the optimized DDRNet semantic segmentation model to extract contour information of the image area with the disease based on the first detection result to obtain the second detection result.

[0045] Furthermore, it also includes:

[0046] S800: Send the first detection result and the second detection result to a remote control terminal.

[0047] In the embodiment of the present invention, after receiving the first detection result and the second detection result, the remote control terminal will perform visual presentation on the display terminal. When performing the visual presentation, the disease type and disease location may be presented in the form of boxes and labels.

[0048] Figure 2 This is a structural block diagram of an airport pavement disease detection system provided in an embodiment of the present application. Figure 2 As shown, another embodiment of the present invention provides an airport pavement defect detection system, including: a remote control terminal 1 and a movable inspection body 2 and a display terminal 3 that are communicatively connected to the remote control terminal 1, and the movable inspection body 2 is provided with an image acquisition device 4 and an image detection device 5.

[0049] In the embodiment of the present invention, the movable inspection body 2 is used to perform inspection according to a preset driving path based on the control instruction sent by the remote control terminal. The movable inspection body 2 can be an inspection vehicle capable of automatic movement, which has an automatic driving system capable of automatic movement.

[0050] The patrol body 2 is also provided with a positioning device 6, a vehicle control device 7 and a communication device 8. The positioning device 6 may be composed of a Beidou antenna and an inertial navigation device, and is used to locate the patrol vehicle and send the positioning information to the remote control terminal 1. The vehicle control device 7 is used to execute the patrol control instructions of the remote control terminal 1 to the patrol vehicle, and control the patrol body to travel according to the set driving path. The communication device 8 is used for real-time communication between the patrol vehicle and the remote control terminal 1, for example, 5G can be used for real-time communication.

[0051] The remote control terminal 1 is equipped with a patrol control system, including a vehicle path planning module and a remote control module, which is used to send patrol control instructions to the on-board control device 7 to automatically control or remotely temporarily control the movement route, i.e., the driving path, of the patrol vehicle. In the specific implementation, the remote control terminal 2 is equipped with a web-based program of the airport pavement intelligent patrol system, which has two main functions: one is to receive and save pavement patrol images and pavement disease detection results, and transmit pavement patrol images and disease information to the display terminal 3; the other is to send the vehicle control system instructions to the on-board control module on the patrol vehicle through the 5G network to control the movement of the patrol vehicle.

[0052] Furthermore, in the embodiment of the present invention, the image acquisition device 4 is used to collect the airport pavement image during the inspection process of the inspection vehicle, and transmit the image information to the remote control terminal 1 through the 5G network, and at the same time transmit it to the image detection device 5 through the data line. The image acquisition device can be composed of three industrial cameras, wherein one camera observes the front and lower part of the inspection vehicle, and the other two cameras observe the side and lower part of the inspection vehicle.

[0053] Further, in an embodiment of the present invention, the image detection device 5 may include an AI edge computing platform and a YOLOv5 target detection model and a DDRNet semantic segmentation model, and an image acceleration decoder optimized by a preset inference optimizer deployed on the AI ​​edge computing platform, for detecting the type, location, and contour information of pavement defects, and sending the detection results to the remote control terminal 1 through a communication system. Among them, the inference optimizer may be TensorRT, and the image acceleration decoder may be DeepStream. The specific optimization process of the YOLOv5 target detection model and the DDRNet semantic segmentation model, as well as the specific deployment of the optimized YOLOv5 target detection model and the DDRNet semantic segmentation model, and the image acceleration decoder can be referred to the aforementioned embodiments.

[0054] Specifically, the image detection device 5 is used for:

[0055] The collected airport pavement images are acquired and decoded in real time through the image acceleration decoder, and the decoded images are sent to the optimized YOLOv5 target detection model;

[0056] Detecting the decoded image using an optimized YOLOv5 target detection model to obtain a first detection result, where the first detection result includes a disease type and a disease location;

[0057] Extracting contour information of the image region where the disease exists based on the first detection result by using the optimized DDRNet semantic segmentation model to obtain a second detection result, where the second detection result includes the disease contour information; and

[0058] The first detection result and the second detection result are sent to the remote control terminal 1.

[0059] The acquisition of the first detection result and the second detection result can be referred to the aforementioned embodiment. To avoid redundancy, the detailed introduction thereof is omitted in this embodiment.

[0060] In the embodiment of the present invention, the display terminal 3 is used to display the first detection result and the second detection result. In the specific implementation, the display terminal 3 is used to display the web page program of the airport pavement intelligent inspection system built on the remote control terminal 1, and the main page content is the real-time pavement picture, the disease record column on the right, and the vehicle control module column below.

[0061] In specific applications, when the airport pavement disease detection system provided by the embodiment of the present invention is used to automatically inspect pavement diseases, the following steps may be included:

[0062] Step 1: In the vehicle control module column of the airport pavement intelligent inspection system web page, select an inspection task and determine the inspection path of the inspection vehicle. The navigation device on the inspection vehicle sends the positioning information to the remote control terminal through the 5G network. The remote control terminal obtains the corresponding inspection control instructions based on the inspection vehicle's location information and the planned inspection path, and sends the inspection control instructions to the on-board control device on the inspection vehicle through the 5G network. The on-board control device controls the movement of the inspection vehicle to perform automatic inspections.

[0063] Step 2: During the vehicle inspection, the image acquisition device collects road surface images and transmits the images to the remote control terminal through the 5G network, and transmits the images to the image detection device through the data line. The image detection device detects defects on the road surface images and transmits the detected defect type, location and contour information to the remote control terminal through the 5G network.

[0064] Step three: The remote control terminal receives and stores pavement images and defect detection information. The pavement inspection images are stored as videos, and the pavement defect detection results and the location information of the inspection vehicles are saved in the database. At the same time, the remote control terminal displays the pavement inspection images in real time on the web program of the airport pavement intelligent inspection system, and marks the detected defect types and locations in real time in the form of boxes and labels, and displays the defect pictures with outline marks in the defect record column in a timely manner.

[0065] Step 4: When the operator finds that the system has detected a defect on the display terminal, he or she can manually remotely control the movement of the inspection vehicle through the control buttons on the remote control module in the vehicle control module column of the airport pavement intelligent inspection system web page to further confirm the pavement defect.

[0066] Step 5: After the defect is confirmed, the operator selects to continue the inspection according to the inspection route in the vehicle control module column of the airport pavement intelligent inspection system web page. When the inspection vehicle reaches the end of the inspection route, the entire pavement inspection operation is completed.

[0067] Although some specific embodiments of the present application have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are only for illustration and not for limiting the scope of the present application. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A method for detecting airport pavement defects, characterized in that: The following steps are involved: S100, respectively building a YOLOv5 target detection model and a DDRNet semantic segmentation model and setting hyperparameter training to obtain a trained YOLOv5 target detection model and a DDRNet semantic segmentation model; S200, optimizes the trained YOLOv5 object detection model and DDRNet semantic segmentation model based on the preset inference optimizer to improve the inference speed of the model; S300, deploying the optimized YOLOv5 target detection model, DDRNet semantic segmentation model and image acceleration decoder on an image detection device of a movable inspection body; the movable inspection body is also provided with an image acquisition device; S400, controlling the movable inspection body to travel along a preset travel path, and controlling the image acquisition device to acquire an image of the airport pavement; S500, the image acceleration decoder acquires and decodes the collected airport pavement image in real time, and sends the decoded image to the optimized YOLOv5 target detection model; S600, using the optimized YOLOv5 target detection model to detect the decoded image, and obtain a first detection result, wherein the first detection result includes a disease type and a disease location; S700, using the optimized DDRNet semantic segmentation model to extract contour information of the image area where the disease exists based on the first detection result, to obtain a second detection result, where the second detection result includes the disease contour information.

2. The airport pavement disease detection method according to claim 1, characterized in that: The inference optimizer is TensorRT.

3. The airport pavement disease detection method according to claim 2, characterized in that: The TensorRT is used to: Reduce the FP32 parameters in the YOLOv5 object detection model and DDRNet semantic segmentation model to FP16 or INT8 data types through mapping operations to reduce the amount of model inference calculations; and The network layers in the YOLOv5 target detection model and DDRNet semantic segmentation model are simplified through inter-layer fusion and tensor fusion to simplify the model structure.

4. The airport pavement disease detection method according to claim 1, characterized in that: The image acceleration decoder is DeepStream.

5. The airport pavement disease detection method according to claim 1, characterized in that: Also includes: S800: Send the first detection result and the second detection result to a remote control terminal.

6. An airport pavement disease detection system, characterized in that: include: A movable inspection body and a remote control terminal connected in communication, wherein the movable inspection body is provided with an image acquisition device and an image detection device; The image detection device includes an image acceleration decoder, a YOLOv5 target detection model optimized by a preset inference optimizer, and a DDRNet semantic segmentation model; Wherein, the movable inspection body is used to perform inspection according to a preset driving path based on the control instruction sent by the remote control terminal; The image acquisition device is used to collect the airport pavement image during the inspection process of the movable inspection body and send it to the image detection device; The image detection device is used for: The collected airport pavement images are acquired and decoded in real time through the image acceleration decoder, and the decoded images are sent to the optimized YOLOv5 target detection model; Detecting the decoded image using an optimized YOLOv5 target detection model to obtain a first detection result, where the first detection result includes a disease type and a disease location; Extracting contour information of the image region where the disease exists based on the first detection result by using the optimized DDRNet semantic segmentation model to obtain a second detection result, where the second detection result includes the disease contour information; and The first detection result and the second detection result are sent to the remote control end.

7. The airport pavement defect detection system according to claim 6, characterized in that: The movable inspection body is also provided with a positioning device.

8. The airport pavement defect detection system according to claim 6, characterized in that: It also includes a display terminal for displaying the first detection result and the second detection result.

9. The airport pavement defect detection system according to claim 6, characterized in that: The inference optimizer is TensorRT.

10. The airport pavement defect detection system according to claim 6, characterized in that: The image acceleration decoder is DeepStream.

Images