Pavement multi-feature disease detection method and device based on multi-neural network combination

Abstract

The present invention discloses a method for detecting multi-feature road surface diseases based on the combination of multiple neural networks, including: generating road surface disease images by training a generative adversarial network, expanding the data set to train a road surface disease detection model; using the trained optimal road surface disease detection model to load into a system board for road surface disease detection; sending the road surface detection information to a monitoring information platform via MQTT; and finally, the user accesses the relevant road surface information through the platform. The present invention does not need to waste a lot of time and manpower to collect and annotate data sets, can quickly detect multi-feature road surface diseases, and establishes a monitoring platform so that users can view road surface information anytime and anywhere.

Description

Technical Field

[0001] The present invention relates to the field of pavement disease detection, and in particular to a pavement multi-feature disease detection method and device based on the combination of multiple neural networks. Background Art

[0002] In recent years, my country's road construction has been continuously improved, and transportation capacity has been continuously enhanced. However, with the passage of time, road facilities have gradually been damaged, such as road structure damage, zebra crossing white line wear, road cracks and potholes, road rutting, etc., which have brought relatively bad social impacts and serious economic losses. Therefore, strengthening the planning and construction of road maintenance can effectively improve the quality of roads and avoid economic losses caused by road damage. At present, due to technical limitations, road disease inspection work is mainly based on manual inspection, and only a small part of intelligent inspection equipment is used, but the equipment has a single function and low intelligence. The cost of road technical condition inspection vehicles is high, the inspection cycle is long, and the average inspection cycle is about once a year, which cannot meet the requirements of normalized disease inspections. How to quickly and efficiently automate the inspection of the road surface has become an important research topic in the transportation industry.

[0003] CN112215203A A road surface disease detection method and device based on deep learning, discloses a road surface disease detection method based on deep learning, including: obtaining a road surface image or a road surface video; obtaining a road surface image to be inspected according to the road surface image or the road surface video; using a road surface disease detection model based on a deep learning network to identify the road surface disease in the road surface image to be inspected, and simultaneously obtaining corresponding positioning data to form comprehensive road surface disease information. This patent can only detect road surface diseases, but the outside world cannot obtain road information in real time and the workload of the early training model is large. Summary of the invention

[0004] Purpose of the invention: In order to solve the problem that the general road surface disease detection method has a large workload of preliminary training models and cannot obtain road information in real time, the present invention provides a road surface multi-feature disease detection method and device based on the combination of multiple neural networks, which uses an improved adversarial generative network to reduce the workload of preliminary training for road surface disease detection and can obtain road information in real time.

[0005] Technical solution: A multi-feature pavement disease detection method based on a combination of multiple neural networks includes the following steps:

[0006] (1) Use the vehicle-mounted camera to obtain images of various road surface diseases, select a small number of images of various types of diseases from the acquired images, label them and classify them;

[0007] (2) Build a generative adversarial network model and a pavement disease detection model;

[0008] (3) Training the generative adversarial network model, sending the pavement disease images into the generative adversarial network model by type and batch, and adjusting the hyperparameter γ to generate multiple data sets with different quality and diversity;

[0009] (4) Train the pavement disease detection model. The real pavement disease images and the generated images constitute a data set, which is input into the pavement disease detection model for training. The feature map is detected and the default box is obtained. After calculation, non-maximum suppression screening is performed to determine the pavement disease. The attention mechanism is introduced between Conv7, Conv8, Conv9, Conv10, and Conv11 of the pavement disease detection model to improve the efficiency of extracting disease features.

[0010] (5) Return the pavement damage data to the pavement information monitoring platform.

[0011] The generative adversarial network model in step (2) includes an encoder, a generator and a discriminator. After the real image is sent to the encoder, the latent variable value of the real image is obtained, which replaces the original random noise to train the adversarial network. For the type of disease with poor detection effect, the generative adversarial network model is used to expand the data set of this type to enhance the detection effect of the training of this disease type. The generative adversarial network model serves as the data set of the pavement disease detection model; the pavement disease detection model is an improved SSD model, which is responsible for multi-feature disease detection of the pavement.

[0012] The generative adversarial network model is obtained based on BEGAN network training, and the pavement disease detection model is obtained based on SSD network training.

[0013] The expected training effect judgment criteria of the pavement disease detection model include recall rate and accuracy rate.

[0014] The pavement information monitoring platform adopts the MQTT protocol for data transmission and implements the management of pavement disease detection through docker and FRP tools.

[0015] A road surface multi-feature disease detection device based on a combination of multiple neural networks, including a power supply module, a data acquisition module, a detection module and an external module;

[0016] The power supply module is used to supply power to the acquisition module, the detection module and the external module;

[0017] The data acquisition module uses the vehicle-mounted camera to obtain pictures of various road surface diseases;

[0018] The detection module is configured as follows:

[0019] Build a generative adversarial network model and a road surface disease detection model.

[0020] Train the generative adversarial network model, send the road surface disease images into the generative adversarial network model by type and batch, and adjust the hyperparameter γ to generate multiple data sets with different quality diversity.

[0021] Train the pavement disease detection model, form a data set with real pavement disease images and generated images, input them into the pavement disease detection model for training, detect the feature map and obtain the default box, perform non-maximum suppression screening after calculation, and determine the pavement disease.

[0022] The detection module is connected with the data acquisition module;

[0023] The external module includes a built-in display, a wireless network card, a speaker and a GPS. The GPS is used to locate the location of the road surface image, the speaker is used to set an alarm, the display screen is used to view data, and the wireless network card is used to transmit data.

[0024] The data acquisition module is a vehicle-mounted camera, which processes the road surface pictures taken by the camera and sends them to the detection module.

[0025] The generative adversarial network model in the detection module includes an encoder, a generator and a discriminator. The generative adversarial network model serves the data set of the pavement disease detection model; the pavement disease detection model is an improved SSD model, which is responsible for multi-feature disease detection of the pavement.

[0026] The road surface multi-feature disease detection device based on the combination of multiple neural networks also includes an external detection platform for users to read road section information, GPS coordinates and disease types, and transmit data to the outside world through the wireless network card of the external module.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0028] The multi-feature pavement disease detection method based on the combination of multiple neural networks provided by the present invention does not require a lot of time to collect and annotate data sets, and has a strong effect on the detection of multi-feature pavement diseases. External users can query road section information, GPS coordinates and disease types through the pavement information monitoring platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the multi-feature pavement disease detection method based on the combination of multiple neural networks;

[0030] Figure 2 To generate adversarial network model graph;

[0031] Figure 3 This is a road surface detection network model diagram;

[0032] Figure 4This is the design diagram of a multi-feature road surface disease detection device based on the combination of multiple neural networks;

[0033] Figure 5 This is the working flow chart of the road information monitoring platform. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0035] Embodiment 1:

[0036] The present invention provides a road surface multi-feature disease detection method based on the combination of multiple neural networks, such as Figure 1 As shown, the following steps are included:

[0037] Before performing multi-feature road disease detection, the neural network needs to be trained and loaded into the Jetson nano system board. First, a small number of road disease images are obtained through the camera mounted on the vehicle, the image size is adjusted to 300*300, and various road diseases are labeled for classification.

[0038] Table 1 Pavement damage categories:

[0039]

[0040] like Figure 2 As shown in the figure, a generative adversarial network model is built, where the encoder is the encoder module of the VAE variational autoencoder. After the real road surface disease picture is input into the encoder, the mean and variance obeyed by the real picture are obtained, and then random sampling is performed according to the normal distribution to obtain the hidden variable value. Since the use of generative adversarial network training is prone to non-convergence and collapse problems, the random noise input generator of the original generative adversarial network input is replaced by a hidden variable, and then sent to the discriminator together with the real road surface disease picture. The generator and discriminator are the generator and discriminator of the BEGAN network. The discriminator, generator, and the loss function formula of the tth step during training are as follows:

[0041] The loss function of the generated network is:

[0042] L G =L(G(z G )),

[0043] The loss function of the discriminator is:

[0044] L D =L(X)-k t (G(z D )),

[0045] During training, the loss function at step t is:

[0046] kt+1 =k t +λ k (γL(x)-L(G(z G ))).

[0047] Among them, different γ can change the image generation result. When the γ value is low, the image obtained is single, but a more detailed image can be generated; as γ increases, the diversity of generated data increases, and more blurred image data is generated. The hyperparameter γ is used to make an optimal choice between the quality and diversity of the generated image. The optimal selection method is determined according to the training effect of the pavement disease model.

[0048] Use the trained generative adversarial network to generate road surface disease images with a size of 300*300, label and classify them, and finally form a data set for the road surface detection model together with the real road surface disease images.

[0049] like Figure 3 As shown in the figure, a road multi-feature detection model is built, the input image size is 300*300, the data set is divided into a training set and a test set at a ratio of 8:2, and the attention mechanism is introduced between Conv7, Conv8, Conv9, Conv10, and Conv11. The selected attention module is SE-NET. The attention mechanism can increase the weight coefficient of the disease location, improve the detection efficiency and loss accuracy, set the batch processing number to 20, the learning rate to 0.0004, the momentum optimization value to 0.8, the number of iterations to 10000, and the regularization standard to select L2 regularization.

[0050] The criteria for judging the expected effect of the road multi-feature detection model training are recall rate and precision rate. During the training and testing process, multiple experiments are conducted to optimize the quality and diversity of the generative adversarial networks with different γ, and the model with the best training effect is selected. For the types of diseases with weak detection effects, the generative adversarial network is used to increase the corresponding road disease data set for enhanced training, so as to further improve the model training effect.

[0051] The optimal pavement disease multi-feature detection model is loaded into the Jetson nano system board to complete the algorithm part of the pavement multi-feature disease detection device based on the combination of multiple neural networks.

[0052] Embodiment 2:

[0053] The present invention discloses a road surface multi-feature disease detection device based on a combination of multiple neural networks, such as Figure 4 Shown is the hardware design diagram, including power supply module, acquisition module, detection module and external module.

[0054] The power supply module provides power supply support for the acquisition module, detection module and external modules.

[0055] The acquisition module is composed of a USB camera, which is connected to the Jetson nano system board via USB. The camera is installed on the front or rear of the vehicle to capture and collect road information, and the image information is transmitted to the road disease detection model for detection.

[0056] The detection module is equipped with a trained pavement disease feature detection model and an image preprocessing module. The image preprocessing module mainly preprocesses the image, including scaling the image to 300*300, performing Gaussian denoising, mean denoising, and improving image contrast. The image is then sent to the detection model for pavement disease detection and connected to an external module to send the processed data to the outside world.

[0057] The external module includes GPS, audio, display and wireless network card. GPS is used to obtain location information and timely understand the quality of the road surface. The audio is used to alarm or issue a prompt sound. For example, when the system fails or needs to remind the user, it can be fed back to the customer through the audio equipment. The display is convenient for users to operate the system and developers to debug the system. The wireless network card uses a USB network card to ensure the network communication of the system and can transmit the detection data to the road surface detection information platform in time.

[0058] like Figure 5 The following is the workflow of the pavement information monitoring platform. First, it is initialized and determines whether MQTT is working properly; then it determines whether the camera is working properly; if the camera is working abnormally, the abnormal information is sent to MQTT for users to subscribe to. If it is working normally, it starts to collect pavement defects and transmits them in real time through MQTT and waits for subscription. At the same time, the validity of the defect data is determined based on whether the vehicle is stationary. The monitoring information platform subscribes to the pavement defect topic information from MQTT and performs graphical processing and display.

[0059] The wireless communication system needs to complete the mutual communication function between various parts of the road surface detection system and the road surface monitoring information platform, and realize the management of the detection system through tools such as docker and FRP.

[0060] Through the FRP tool, local web projects can be provided to the external network for access. FRP supports domain name binding and has high-performance intranet penetration performance.

[0061] In the road surface detection system, configuring FRP between various servers can realize the remote management and daily maintenance of the detection system.

[0062] Use the dc (docker-compose) tool to manage and conveniently use Docker.

[0063] In the method and device for detecting multi-feature road surface defects based on the combination of multiple neural networks of the present invention, road surface data is first collected by a camera on a running vehicle, then sent to a development board for road surface detection and identification, and then uploaded to the backend and platform via MQTT, and users can view relevant information about the road surface on the platform.

Claims

1. A multi-feature pavement disease detection method based on the combination of multiple neural networks, characterized in that: The following steps are involved: (1) Use the vehicle-mounted camera to obtain images of various road surface defects; (2) Build a generative adversarial network model and a pavement disease detection model; (3) Training the generative adversarial network model, sending the pavement disease images into the generative adversarial network model by type and batch, and adjusting the hyperparameter γ to generate multiple data sets with different quality diversity; (4) Training a pavement disease detection model. The real pavement disease images and the generated images form a data set, which is input into the pavement disease detection model for training. The feature map is detected and the default box is obtained. After calculation, non-maximum suppression screening is performed to determine the pavement disease. (5) Return the pavement damage data to the pavement information monitoring platform; The generative adversarial network model comprises an encoder, a generator and a discriminator; wherein the encoder is an encoder module of a VAE variational autoencoder, after inputting a real road surface disease picture into the encoder, the mean and variance obeyed by the real picture are obtained, and then random sampling is performed according to a normal distribution to obtain a hidden variable value, and the hidden variable is used to replace the random noise input of the original generative adversarial network input into the generator, and then sent to the discriminator together with the real road surface disease picture, wherein the generator and the discriminator are the generator and the discriminator of the BEGAN network, and the discriminator, the generator, and the loss function formula of the t-th step during training are as follows: The loss function of the generator is: L G =L(G(z G )), The loss function of the discriminator is: L D =L(X)-k t (G(z D )), During training, the loss function at step t is: k t+1 =k t +λ k (γL(x)-L(G(z G ))), The hyperparameter γ is used to optimize the quality and diversity of the generated images. The optimization method is determined according to the training effect of the pavement disease model. The pavement disease detection model is an improved SSD model; an attention mechanism is introduced among Conv7, Conv8, Conv9, Conv10, and Conv11, and the selected attention module is SE-NET.

2. The method for detecting multi-feature road surface defects based on the combination of multiple neural networks according to claim 1 is characterized in that: The generative adversarial network model is obtained based on BEGAN network training, and the pavement disease detection model is obtained based on SSD network training.

3. The method for detecting multi-feature road surface defects based on the combination of multiple neural networks according to claim 1 is characterized in that: The expected training effect judgment criteria of the pavement disease detection model include recall rate and accuracy rate.

4. The method for detecting multi-feature road surface defects based on the combination of multiple neural networks according to claim 1 is characterized in that: The pavement information monitoring platform adopts the MQTT protocol for data transmission and implements the management of pavement disease detection through docker and FRP tools.

5. A multi-feature road surface disease detection device based on a combination of multiple neural networks, characterized in that: It includes power supply module, data acquisition module, detection module and external module; The power supply module is used to supply power to the acquisition module, the detection module and the external module; The data acquisition module uses the vehicle-mounted camera to obtain pictures of various road surface diseases; The detection module is configured as follows: Build a generative adversarial network model and a road surface disease detection model. Train the generative adversarial network model, send the road surface disease images into the generative adversarial network model by type and batch, and adjust the hyperparameter γ to generate multiple data sets with different quality diversity. Train the pavement disease detection model, form a data set with real pavement disease images and generated images, input them into the pavement disease detection model for training, detect the feature map and obtain the default box, perform non-maximum suppression screening after calculation, and determine the pavement disease. The detection module is connected with the data acquisition module; The external module includes a built-in display, a wireless network card, an audio system and a positioning system; The generative adversarial network model comprises an encoder, a generator and a discriminator; wherein the encoder is an encoder module of a VAE variational autoencoder, after inputting a real road surface disease picture into the encoder, the mean and variance obeyed by the real picture are obtained, and then random sampling is performed according to a normal distribution to obtain a hidden variable value, and the hidden variable is used to replace the random noise input of the original generative adversarial network input into the generator, and then sent to the discriminator together with the real road surface disease picture, wherein the generator and the discriminator are the generator and the discriminator of the BEGAN network, and the discriminator, the generator, and the loss function formula of the t-th step during training are as follows: The loss function of the generator is: L G =L(G(z G )), The loss function of the discriminator is: L D =L(X)-k t (G(z D )), During training, the loss function at step t is: k t+1 =k t +λ k (γL(x)-L(G(z G ))), The hyperparameter γ is used to optimize the quality and diversity of the generated images. The optimization method is determined according to the training effect of the pavement disease model. The pavement disease detection model is an improved SSD model; an attention mechanism is introduced among Conv7, Conv8, Conv9, Conv10, and Conv11, and the selected attention module is SE-NET.

6. The multi-feature road surface disease detection device based on multi-neural network combination according to claim 5 is characterized in that: The data acquisition module is a vehicle-mounted camera, which processes the road surface pictures taken by the camera and sends them to the detection module.

7. The multi-feature road surface disease detection device based on multi-neural network combination according to claim 5 is characterized in that: The generative adversarial network model in the detection module includes an encoder, a generator and a discriminator. The generative adversarial network model serves the data set of the pavement disease detection model; the pavement disease detection model is an improved SSD model, which is responsible for multi-feature disease detection of the pavement.

Images