Road surface disease lightweight detection method and device, medium and equipment

Abstract

The invention discloses a road surface disease lightweight detection method and device, a medium and equipment, and relates to the field of road detection.The method comprises the steps that unmanned aerial vehicle video data and GPS path data of a to-be-detected road surface are obtained; performing key frame extraction on the unmanned aerial vehicle video data through a key frame extraction algorithm, and determining a road table image sequence data set; an improved convolutional long-short term memory Conv-LSTM network is constructed; training the improved convolutional long-short-term memory Conv-LSTM network to obtain a road surface disease detection model for detecting road surface diseases; and inputting the to-be-detected road surface image sequence data set into the road surface disease detection model to obtain a road surface disease detection result, and obtaining a road surface disease detection report according to the road surface disease detection result and in combination with the GPS path data.

Description

Technical Field

[0001] The present invention relates to the field of road detection, and in particular to a lightweight detection method, device, medium and equipment for road surface defects. Background Art

[0002] Road surface diseases are a series of surface damages that appear in the early stage of road use, including cracks, repairs and potholes. Measuring the number and geometric parameters of road surface diseases is an important indicator for evaluating road performance, and road surface diseases have an important impact on road driving safety. For the detection of road surface diseases, because they have a certain degree of randomness in time and space, regular inspections are generally carried out to record the locations where the diseases appear. For areas where the diseases appear in large numbers, specific tests are carried out. This method of detection is easily affected by the subjective influence of the detection staff, resulting in the inability to guarantee the stability of the detection.

[0003] At present, the main methods for road surface disease detection can be divided into two types: one is the two-dimensional image detection method using industrial high-definition cameras, and the other is the three-dimensional image detection method using three-dimensional lasers. The industrial high-definition camera is installed on the road inspection vehicle. As the vehicle travels, data is collected to obtain the two-dimensional image data of a lane road surface and record the corresponding position information. The two-dimensional image data is then processed through a visual processing algorithm or a deep neural network to finally obtain the specific location of the disease and the degree of damage of the disease. However, the above method is easily affected by the road material and the external lighting environment, and the stability of the road surface disease detection results is limited. Summary of the invention

[0004] The present invention provides a road surface disease lightweight detection method, device, medium and equipment to solve the above-mentioned problem existing in the prior art, that is, how to improve the stability of road surface disease detection in the prior art. The present invention provides a road surface disease lightweight detection method, the method comprising: The road surface to be inspected is segmented and the drone video collection nodes are determined. When the inspection starts, the drone flies in an uplink and downlink manner and starts GPS positioning to obtain the drone video data and GPS path data of the road surface to be inspected. The key frames of the drone video data are extracted by the key frame extraction algorithm to determine the road surface image sequence data set; Construct an improved convolutional long short-term memory Conv-LSTM network; replace the encoder of the original Conv-LSTM network with a local feature extraction module, and replace the upsampling in the decoder of the original Conv-LSTM network with a bilinear interpolation method; the local feature extraction module is composed of a convolution layer, a global self-attention mechanism layer Global-SA, a multi-layer perceptron MLP, a local self-attention mechanism layer Local-SA and an MLP connected in sequence; The improved convolutional long short-term memory (Conv-LSTM) network is trained to obtain a road surface disease detection model for detecting road surface diseases; The road surface image sequence data set to be detected is input into the road surface disease detection model to obtain the result of the road surface disease detection. According to the result of the road surface disease detection and combined with the GPS path data, a road surface disease detection report is obtained.

[0005] Optionally, extracting key frames from the drone video data using a key frame extraction algorithm to determine a road surface image sequence data set specifically includes: The key frames of the drone video data are extracted by a key frame extraction algorithm to obtain image data. An image data set is constructed based on the image data. The image data set is divided into several subsets, and the several subsets constitute the road surface image sequence data set.

[0006] Optionally, the key frame extraction is performed on the drone video data by a key frame extraction algorithm to obtain image data, and an image data set is constructed according to the image data, and the image data set is divided into a plurality of subsets, and the plurality of subsets constitute the road table image sequence data set, specifically including: By using the key frame extraction algorithm at a frame rate of 10FPS, key frames are extracted from the video to obtain image data, and the image data set is obtained using the following formula: , in, is an image dataset, is the image data, is the timestamp corresponding to the image data, is the pixel position, is the pixel position The corresponding RGB value; Image dataset It is further divided into several subsets, each of which is an image data sequence, containing 10 orderly arranged image data. The specific calculation formula is as follows: , , in, For a subset, is the total number of subsets, is the mth image in the 10 image data i, and n is the sequence number of the subset; A plurality of image data sequences constitute the road surface image sequence data set.

[0007] Optionally, the road surface disease detection report is obtained based on the road surface disease detection result and combined with GPS path data, specifically including: According to the results of road surface disease detection, the timestamp of the surface disease location is obtained using the following formula: , in, is the timestamp of the surface disease location, n is the total number of subsets in the road surface image sequence dataset, and m is the mth image of the subset in the road surface image sequence dataset; Based on the timestamp of the surface defect location and combined with GPS path data, the defect location is determined and a road surface defect detection report is obtained.

[0008] Optionally, before extracting key frames from the drone video data using the key frame extraction algorithm, the road section range of the detection position in the drone video data is marked according to the GPS path data, and the drone video data is numbered, and each numbered road section is used as an independent data set.

[0009] Optionally, the image data sequence data set is processed to remove abnormal data.

[0010] The present invention provides a lightweight detection device for road surface defects, comprising: The acquisition module is used to segment the road surface to be inspected and determine the drone video acquisition nodes. When the inspection starts, the drone flies in an uplink and downlink manner and starts GPS positioning to obtain the drone video data and GPS path data of the road surface to be inspected; A key frame extraction module is used to extract key frames from UAV video data through a key frame extraction algorithm to determine a road surface image sequence data set; A construction module is used to construct an improved convolutional long short-term memory Conv-LSTM network; the encoder of the original Conv-LSTM network is replaced by a local feature extraction module, and the upsampling in the decoder of the original Conv-LSTM network is replaced by a bilinear interpolation method; the local feature extraction module is composed of a convolution layer, a global self-attention mechanism layer Global-SA, a multi-layer perceptron MLP, a local self-attention mechanism layer Local-SA and an MLP connected in sequence; A training module, used for training an improved convolutional long short-term memory (Conv-LSTM) network to obtain a road surface disease detection model for detecting road surface diseases; The detection module is used to input the road surface image sequence data set to be detected into the road surface disease detection model, obtain the result of the road surface disease detection, and obtain the road surface disease detection report based on the result of the road surface disease detection combined with the GPS path data.

[0011] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned road surface disease lightweight detection method is implemented.

[0012] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned lightweight detection method for road surface defects when executing the program.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a lightweight detection method for road surface defects, which obtains a continuous image sequence by extracting key frames from the acquired drone video data. When the lighting environment changes, mainly because there are shadows on the road, serious information loss problems may occur for some viewing angles. However, the continuous image sequence enables the acquisition of image data from different angles, and the associated features of adjacent images are accurately extracted by constructing an improved Conv-LSTM network, which enables accurate detection and lightweight detection of roads at all levels, thereby improving the stability of road surface defect detection. Compared with vehicle-mounted industrial cameras and three-dimensional lasers, the method proposed by the present invention is applicable to more engineering situations and has a high utilization rate. And the cost is low; then, the method has a high degree of automation. In the process of data processing, computer automation processing of the entire process can be realized, which effectively saves time cost and improves the accuracy of detection; this method avoids the problem that the detection vehicle needs to repeatedly drive a section of road multiple times to cover multiple lanes during the detection process of the intelligent detection vehicle, especially in the multi-lane detection process. While not affecting the existing traffic operation, it effectively reduces the amount of energy consumption in the detection process and reduces the carbon emissions of road detection projects; in addition, in the collection stage, through reasonable road segmentation, node selection and the drone collecting video data by uplink and downlink round-trip flights, comprehensive coverage of multiple lanes is achieved, which provides a solid foundation for subsequent road surface detection and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0015] Figure 1 A flow chart of a lightweight detection method for road surface defects provided by an embodiment of the present invention; Figure 2 The flowchart for obtaining road surface diseases provided by the embodiments of the present invention; Figure 3 The flowchart of the improved Conv-LSTM network provided by the embodiments of the present invention; Figure 4 The flowchart of the encoding module of the improved Conv-LSTM network provided by the embodiments of the present invention; Figure 5 The schematic diagram of a computer device for the lightweight detection method of road surface diseases provided by the embodiments of the present invention. Detailed implementation manners

[0016] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0017] The technical solutions of the present invention and how the technical solutions of the present invention solve the above technical problems will be described in detail below with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0018] As Figure 1 and Figure 2 shown, a lightweight detection method of road surface diseases shown in this embodiment includes: S1: Segment the road surface to be detected, and determine the UAV video acquisition nodes. At the start of the detection, the UAV flies in an up-and-down round-trip manner, starts GPS positioning, and acquires the UAV video data and GPS path data of the road surface to be detected.

[0019] Exemplarily, before acquiring the UAV video data of the road surface to be detected, segment the road surface to be detected; at the start of the detection, start GPS positioning to track and position the detection route and calibrate the detection position.

[0020] Exemplarily, the preparation work of the drone equipment can be carried out in an open area near the pre-inspection section, including assembling drone accessories and replacing the power supply and data storage card; then the drone is started, connected to the tablet, and the wireless connection between the drone and the tablet is debugged. After flying to a predetermined height, the focal length of the camera is calibrated and fixed; after adjusting the picture to be clear, enter the pre-inspection section for pre-flight. The pre-flight distance is approximately 1 kilometer, and the flight speed is adjusted between 1m / s and 5m / s. During the flight, the flight path setting and flight altitude are adjusted to ensure that the pre-inspection area appears completely within the lens, and the flight is carried out as much as possible along the center line of the road; return after completing all calibration and calibration work.

[0021] For example, before the formal collection, the road section to be inspected can be segmented. Generally, the single collection distance does not exceed 10KM. The node position of each segmentation should be selected in an open area suitable for take-off and landing. The ground operator can monitor the collection process of the drone. The collection is collected by uplink and downlink round-trip flight. The collection can cover multiple lanes during the flight in each direction. The coverage range is 0 to 10m, and the number of covered lanes should not exceed three. After completing the inspection of a section of road, it can move to the next node position and repeat the operation to continue the inspection of the next section of road. At the beginning of each test, the drone is started at the pre-selected node position and enters the test section. The video can be recorded in 4K mode at the beginning, and the resolution can be selected as 4096×2160. The flight speed and altitude are kept unchanged during the process. GPS positioning is started to track and locate the test route and calibrate the test position. The drone can transmit the test screen to the tablet computer for monitoring. The collected video data of the test road surface is stored in the data card. After the uplink is completed, the video collection is stopped, and the direction is turned to enter the downlink, and then the video collection is restarted. After the drone completes the collection and returns, take out the data card, upload the collected video MOV file to the computer, and note the road section range; number the collected video data in sequence, for example, the uplink number can be S1, S2, S3,..., and the downlink number can be X1, X2, X3,... Each numbered section is an independent data set, and then export the GPS path data file, name it accordingly and annotate it, and store it in the computer.

[0022] In this step, the use of drones to collect road surface detection video data not only improves the collection efficiency, but also ensures the accuracy and completeness of the data; at the same time, through meticulous debugging and pre-flight verification, the stability and accuracy of the drone are ensured; in addition, during the collection stage, through reasonable road segmentation, node selection and collection methods, comprehensive coverage of multiple lanes is achieved, providing a solid foundation for subsequent road surface detection and analysis work.

[0023] S2: Extract key frames from drone video data using a key frame extraction algorithm to determine a road surface image sequence dataset.

[0024] Optionally, extracting key frames from drone video data by a key frame extraction algorithm to obtain image data, and constructing an image data set based on the image data; dividing the image data set into a plurality of subsets, wherein the plurality of subsets constitute the road surface image sequence data set, comprises the following steps: By using the key frame extraction algorithm at a frame rate of 10FPS, key frames are extracted from the video to obtain image data, and the image data set is obtained using the following formula: ; in, is an image dataset, is the image data, is the timestamp corresponding to the image data, is the pixel position, is the pixel position The corresponding RGB value; Image dataset It is further divided into several subsets, each of which is an image data sequence, containing 10 orderly arranged image data. The specific calculation formula is as follows: ; ; in, For a subset, is the total number of subsets, is the mth image in the 10 image data i, and n is the sequence number of the subset; A plurality of image data sequences constitute the road surface image sequence data set.

[0025] Exemplarily, the image data sequence data set is preprocessed through image processing algorithms and statistical algorithms to eliminate abnormal data, including converting the RGB values ​​of the image into grayscale values, and calculating the average and standard deviation of the grayscale values ​​of each pixel position of each image data; based on the average and standard deviation of each image data in each image data sequence, a consistency check is performed to eliminate image data that does not meet the consistency; for image sequences with problematic data, the timestamp starting point is adjusted, and the operation key frame selection is restarted, and the process is re-performed to supplement the sequence.

[0026] For example, all images are processed by LabelMe software, and various road surface diseases appearing in the images are marked with square labels, and then saved as JSON data files. Each processed set of data samples includes image data and corresponding JSON data; based on the previous image data sequence, the processed images and corresponding JSON files are merged in a sequence of 10, and output as two NPY files, image sequence and mask sequence; in order to reduce the computational cost and time cost of network training, the image can be segmented into , a total of 32 regions, each region size is ; Then, the following formula is used to normalize the data in all data sets:

[0027] ; in, , It is the mean and standard deviation of the pixel values ​​at all locations in each channel of the RGB image.

[0028] S3: Construct an improved convolutional long short-term memory Conv-LSTM network; replace the encoder of the original Conv-LSTM network with a local feature extraction module, and replace the upsampling in the decoder of the original Conv-LSTM network with a bilinear interpolation method; the local feature extraction module is composed of a convolutional layer, a global self-attention mechanism layer Global-SA, a multi-layer perceptron MLP, a local self-attention mechanism layer Local-SA and an MLP connected in sequence.

[0029] like Figure 3 As shown in the figure, the improved convolutional long short-term memory Conv-LSTM network includes an encoding module, a decoding module, and a fully connected layer.

[0030] like Figure 4 As shown in the figure, the encoding module first inputs a convolutional layer Conv-Layer, then inputs the global self-attention mechanism layer Global-SA to connect a multi-layer perceptron MLP, and the local self-attention mechanism layer Local-SA to connect a multi-layer perceptron, and finally gets the output through the Conv-LSTM layer.

[0031] S4: Train the improved convolutional long short-term memory Conv-LSTM network to obtain a road surface disease detection model for detecting road surface diseases.

[0032] For example, the training can be stopped after the loss function drops to a stable state through the WANDB toolkit. The improved convolutional long short-term memory Conv-LSTM network is trained according to the pre-constructed data set. The specific collection and processing methods of the data set can refer to the above steps S1-S2.

[0033] Exemplarily, the detected pavement defects may include, for example, transverse cracks, longitudinal cracks, cracks, potholes, transverse repairs, longitudinal repairs, block repairs, and looseness.

[0034] S5: Inputting the road surface image sequence data set to be detected into the road surface disease detection model, obtaining the result of the road surface disease detection, and obtaining a road surface disease detection report based on the result of the road surface disease detection and combined with the GPS path data.

[0035] For example, all road surface image sequence data sets to be detected are input into the road surface disease detection model, batch processed, and the detection results are obtained and saved in RGB image format. For images with diseases, the corresponding image number is output. , save the summary as a CSV file; calculate the timestamp of the location where the surface disease is detected , and then the corresponding GPS path data is collected to obtain the specific disease location; by obtaining the type and location of all diseases, and counting the area and number of areas involved in the diseases, a road surface disease detection report is output.

[0036] Optionally, the timestamp of the surface damage location is obtained using the following formula: ; Among them, among them, is the timestamp of the surface disease location, n is the total number of subsets in the road surface image sequence dataset, and m is the mth image of the subset in the road surface image sequence dataset.

[0037] The above is a lightweight detection method for road surface defects provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding lightweight detection device for road surface defects, including: The acquisition module is used to segment the road surface to be inspected and determine the drone video acquisition nodes. When the inspection starts, the drone flies in an uplink and downlink manner and starts GPS positioning to obtain the drone video data and GPS path data of the road surface to be inspected; A key frame extraction module is used to extract key frames from UAV video data through a key frame extraction algorithm to determine a road surface image sequence data set; A construction module is used to construct an improved convolutional long short-term memory Conv-LSTM network; the encoder of the original Conv-LSTM network is replaced by a local feature extraction module, and the upsampling in the decoder of the original Conv-LSTM network is replaced by a bilinear interpolation method; the local feature extraction module is composed of a convolution layer, a global self-attention mechanism layer Global-SA, a multi-layer perceptron MLP, a local self-attention mechanism layer Local-SA and an MLP connected in sequence; A training module, used for training an improved convolutional long short-term memory (Conv-LSTM) network to obtain a road surface disease detection model for detecting road surface diseases; The detection module is used to input the road surface image sequence data set to be detected into the road surface disease detection model, obtain the result of the road surface disease detection, and obtain the road surface disease detection report based on the result of the road surface disease detection combined with the GPS path data.

[0038] For the specific definition of the lightweight detection device for road surface defects, please refer to the definition of the lightweight detection method for road surface defects above, which will not be repeated here. Each module in the above-mentioned lightweight detection device for road surface defects can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0039] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the above-mentioned lightweight detection method for road surface defects.

[0040] The present invention also provides Figure 5 The structural diagram of the computer device shown in FIG. Figure 5 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the lightweight detection method for road surface defects provided in the above embodiment.

[0041] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A lightweight detection method for road surface defects, characterized in that: include: The road surface to be inspected is segmented and the drone video collection nodes are determined. When the inspection starts, the drone flies in an uplink and downlink manner and starts GPS positioning to obtain the drone video data and GPS path data of the road surface to be inspected. The key frames of the drone video data are extracted by the key frame extraction algorithm to determine the road surface image sequence data set; Construct an improved convolutional long short-term memory Conv-LSTM network; The encoder of the original Conv-LSTM network is replaced by a local feature extraction module, and the upsampling in the decoder of the original Conv-LSTM network is replaced by a bilinear interpolation method; the local feature extraction module is composed of a convolutional layer, a global self-attention mechanism layer Global-SA, a multi-layer perceptron MLP, a local self-attention mechanism layer Local-SA and an MLP connected in sequence; The improved convolutional long short-term memory (Conv-LSTM) network is trained to obtain a road surface disease detection model for detecting road surface diseases; The road surface image sequence data set to be detected is input into the road surface disease detection model to obtain the result of the road surface disease detection. According to the result of the road surface disease detection and combined with the GPS path data, a road surface disease detection report is obtained.

2. The lightweight detection method for road surface defects according to claim 1, characterized in that: The key frame extraction algorithm is used to extract key frames from the drone video data to determine the road surface image sequence data set, specifically including: The key frames of the drone video data are extracted by a key frame extraction algorithm to obtain image data. An image data set is constructed based on the image data. The image data set is divided into several subsets, and the several subsets constitute the road surface image sequence data set.

3. The lightweight detection method for road surface defects as claimed in claim 2, characterized in that: The key frame extraction algorithm is used to extract key frames from the drone video data to obtain image data, and an image data set is constructed based on the image data. The image data set is divided into several subsets, and the several subsets constitute the road table image sequence data set, which specifically includes: By using the key frame extraction algorithm at a frame rate of 10FPS, key frames are extracted from the video to obtain image data, and the image data set is obtained using the following formula: ， in, is an image dataset, is the image data, is the timestamp corresponding to the image data, is the pixel position, is the pixel position The corresponding RGB value; Image dataset It is further divided into several subsets, each of which is an image data sequence, containing 10 orderly arranged image data. The specific calculation formula is as follows: ， ， in, For a subset, is the total number of subsets, is the mth image in the 10 image data i, and n is the sequence number of the subset; A plurality of image data sequences constitute the road surface image sequence data set.

4. The lightweight detection method for road surface defects according to claim 1, characterized in that: The road surface disease detection report is obtained based on the road surface disease detection result and combined with GPS path data, specifically including: According to the results of road surface disease detection, the timestamp of the surface disease location is obtained using the following formula: ， in, is the timestamp of the surface disease location, n is the total number of subsets in the road surface image sequence dataset, and m is the mth image of the subset in the road surface image sequence dataset; Based on the timestamp of the surface defect location and combined with GPS path data, the defect location is determined and a road surface defect detection report is obtained.

5. The lightweight detection method for road surface defects according to claim 1, characterized in that: Before extracting key frames from the drone video data using the key frame extraction algorithm, the road section range of the detection position in the drone video data is marked according to the GPS path data, and the drone video data is numbered, and each numbered road section is used as an independent data set.

6. The lightweight detection method for road surface defects according to claim 1, characterized in that: The image data sequence data set is processed to remove abnormal data.

7. A lightweight road surface disease detection device, characterized in that: include: The acquisition module is used to segment the road surface to be inspected and determine the drone video acquisition nodes. When the inspection starts, the drone flies in an uplink and downlink manner and starts GPS positioning to obtain the drone video data and GPS path data of the road surface to be inspected; A key frame extraction module is used to extract key frames from UAV video data through a key frame extraction algorithm to determine a road surface image sequence data set; Building module for building improved convolutional long short-term memory Conv-LSTM network; The encoder of the original Conv-LSTM network is replaced by a local feature extraction module, and the upsampling in the decoder of the original Conv-LSTM network is replaced by a bilinear interpolation method; the local feature extraction module is composed of a convolutional layer, a global self-attention mechanism layer Global-SA, a multi-layer perceptron MLP, a local self-attention mechanism layer Local-SA and an MLP connected in sequence; A training module, used for training an improved convolutional long short-term memory (Conv-LSTM) network to obtain a road surface disease detection model for detecting road surface diseases; The detection module is used to input the road surface image sequence data set to be detected into the road surface disease detection model, obtain the result of the road surface disease detection, and obtain the road surface disease detection report based on the result of the road surface disease detection combined with the GPS path data.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the lightweight detection method for road surface defects described in any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for lightweight detection of road surface defects as claimed in any one of claims 1 to 6 is implemented.

Images