Unmanned aerial vehicle-based road disease detection method, device and terminal equipment

Abstract

The application is suitable for the technical field of road disease, and provides a road disease detection method, device and terminal equipment based on a unmanned aerial vehicle, which comprises the following steps: obtaining original road information of a target road section by using a unmanned aerial vehicle, wherein the original road information comprises a path interval number and an original road image; sampling and splicing the original road images with the same path interval number to obtain a plurality of target road images, each target road image corresponding to a path interval number; inputting the plurality of target road images into a pre-trained road disease analysis model to obtain a plurality of road disease detection results of the target road images; and outputting a road disease report of the target road section according to the plurality of road disease detection results and the corresponding path interval numbers. The application discards manual detection, uses a unmanned aerial vehicle to obtain road information, analyzes road disease detection results through a road disease analysis model, improves road disease detection efficiency, reduces detection cost and ensures the reliability of detection accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of road surface diseases, and particularly relates to a road surface disease detection method, device and terminal equipment based on a UAV. BACKGROUND

[0002] Whether it is a cement or asphalt road surface, after being used for a period of time, various damages, deformations and other defects will appear one after another, which are collectively referred to as road surface diseases. In order to ensure the flatness of the road surface and ensure the smooth and safe driving of vehicles, it is necessary to periodically detect the road surface to find the diseases existing in the road surface and timely maintain and process them.

[0003] The traditional manual detection method mainly relies on experienced staff to drive at a low speed on the road surface, visually observe the road surface condition, and record the road surface disease condition. The existing manual detection method has the following problems: the detection process needs to close the road, the detection time is long, and the efficiency is low; it depends on a large amount of labor cost and traditional detection equipment, and the cost is high; at the same time, the reliability of the detection result depends on the comprehensive quality of the staff, and the fluctuation is large.

[0004] Therefore, there is an urgent need for a road surface disease detection method that can improve the detection efficiency, reduce the detection cost and ensure the reliability of the detection accuracy. SUMMARY

[0005] In order to overcome the problems in the related art, the embodiments of the application provide a road surface disease detection method, device and terminal equipment based on a UAV, which discards manual detection, uses a UAV to obtain road surface information, analyzes road surface disease detection results through a road surface disease analysis model, improves the road surface disease detection efficiency, reduces the detection cost and ensures the reliability of the detection accuracy.

[0006] The application is implemented through the following technical solutions:

[0007] In a first aspect, the embodiments of the application provide a road surface disease detection method based on a UAV, which comprises: obtaining original road surface information of a target road section by using a UAV, the original road surface information comprising a path interval number and an original road surface image, the path interval number being a number obtained by dividing a flight path of the UAV into a plurality of path intervals and sorting the plurality of path intervals according to a time point of aerial photography; sampling and splicing the original road surface images of the same path interval number to obtain a plurality of target road surface images, each target road surface image corresponding to a path interval number; inputting the plurality of target road surface images into a pre-trained road surface disease analysis model to obtain a plurality of road surface disease detection results of the target road surface image; and outputting a road surface disease report of the target road section according to the plurality of road surface disease detection results and the corresponding path interval numbers.

[0008] In one possible implementation of the first aspect, a drone is used to acquire the original road surface information of the target road segment, including: determining the maximum flight distance of the drone based on the drone's preset flight speed and sustainable flight time, and taking a length less than or equal to half of the maximum flight distance as the drone's flight distance; acquiring road segment information of the target road segment, including the road segment start point, road segment end point, road segment length, and road segment curve information; determining the drone's flight path on the target road segment based on the flight distance and road segment information; setting the drone's flight parameters through a ground station, including preset flight speed and flight path; sending the flight parameters to the drone, and receiving the original road surface information collected by the drone in response to the flight parameters.

[0009] In one possible implementation of the first aspect, sampling and stitching original road surface images with the same path interval number to obtain a target road surface image includes: acquiring original road surface images with the same path interval number; extracting images from the original road surface images according to a preset sampling rule to obtain multiple first road surface images; the preset extraction rule is based on the camera frame rate of the camera on the UAV and the preset flight speed of the UAV, and extracting multiple first road surface images from the original road surface images at preset intervals; preprocessing the multiple first road surface images to generate multiple second road surface images, the multiple second road surface images being grayscale images; performing similarity comparison on the multiple second road surface images based on a regional correlation method to obtain the overlapping area of ​​two adjacent second road surface images; and fusing the overlapping area of ​​two adjacent second road surface images to generate the target road surface image.

[0010] In one possible implementation of the first aspect, preprocessing multiple first road surface images to generate multiple second road surface images includes: converting the multiple road surface images to grayscale to obtain grayscale images of the multiple first road surface images; and performing a piecewise linear transformation on the grayscale images to obtain multiple second road surface images, wherein the expression for the piecewise linear transformation is:

[0011]

[0012] In the formula, Represents the pixels in the second road surface image grayscale value, Represents pixels in a grayscale image grayscale value, Represents the grayscale value of a grayscale image. This represents the grayscale value of the second road surface image.

[0013] In a possible implementation manner of the first aspect, the method for similarity comparison of the plurality of second road surface images based on the region correlation comprises: performing similarity calculation on adjacent two second road surface images in the plurality of second road surface images respectively with one pixel as a step, to obtain two two-dimensional similarity matrices, and an expression of the similarity calculation is:

[0014]

[0015] wherein, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value,

[0016] In a possible implementation manner of the first aspect, the method for fusing the overlapping region of the adjacent two second road surface images to generate the target road surface image comprises: fusing the overlapping region of the adjacent two second road surface images based on a fusion formula, and an expression of the fusion formula is:

[0017]

[0018] wherein, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value, I (i, j) represents a similarity value.

[0019] In a possible implementation manner of the first aspect, the constructing the road disease analysis model comprises: obtaining a data set of road disease picture samples, dividing the data set into a training set and a test set according to a preset ratio; building a neural network model, the neural network model comprising a forget gate, an input gate and an output gate, the forget gate selectively forgetting an input of a previous node, the input gate judging importance of information of the previous node and selectively memorizing the information, and the output gate determining to output road disease information; training the neural network model by using the training set to generate an optimized neural network model; testing the optimized neural network model by using the test set to obtain an identification accuracy of the optimized neural network model, and if the identification accuracy is greater than or equal to a preset identification accuracy, the road disease analysis model based on the neural network model is trained.

[0020] In the second aspect, the embodiments of the present application provide a road disease detection device based on a UAV, comprising: an information acquisition module, configured to acquire original road information of a target road section by using a UAV, wherein the original road information comprises a path interval number and an original road image, the path interval number is a number obtained by dividing a flight path of the UAV into a plurality of path intervals and sorting the plurality of path intervals according to a time point of aerial photography; an image processing module, configured to sample and splice the original road images of the same path interval number to obtain a plurality of target road images, each target road image corresponding to a path interval number; an image analysis module, configured to input the plurality of target road images into a pre-trained road disease analysis model to obtain a plurality of road disease detection results of the target road images; and a report output module, configured to output a road disease report of the target road section according to the plurality of road disease detection results and the corresponding path interval numbers.

[0021] In the third aspect, the embodiments of the present application provide a terminal device, comprising a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the road disease detection method based on the UAV as described in any one of the first aspect when executing the computer program.

[0022] In the fourth aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the road disease detection method based on the UAV as described in any one of the first aspect.

[0023] In the fifth aspect, the embodiments of the present application provide a computer program product, when the computer program product runs on a terminal device, the terminal device executes the road disease detection method based on the UAV as described in any one of the first aspect.

[0024] It can be understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description in the first aspect, and will not be repeated here.

[0025] The beneficial effects of the embodiments in this application compared with the prior art are:

[0026] This application's technical solution utilizes drones, which offer higher detection efficiency and lower costs, to inspect road surfaces, eliminating the need for traditional manual inspections. The target road segment is divided into multiple sections, each numbered according to the time of aerial photography. This allows for the determination of the original image's location within the target road segment based on time-sorted images and the drone's flight speed. Image processing of the original road surface images for the same path section numbering allows for the determination of the global image for any section within the target road segment. A pre-trained road defect analysis model analyzes the road sections to identify road defects, avoiding reliance on the comprehensive skills of personnel and improving the accuracy and reliability of road defect detection.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic flowchart of a road surface defect detection method based on unmanned aerial vehicles (UAVs) provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of the original roadside image with the same path interval number provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of a preset extraction rule provided in an embodiment of this application;

[0032] Figure 4 This is a grayscale line graph of the second road surface image after piecewise linear transformation provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of the overlapping area of ​​two adjacent road surface images provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of generating a target road surface image according to an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the structure of the road surface defect detection device based on UAV provided in the embodiments of this application;

[0036] Figure 8 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular sequences of acts, techniques, etc., in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the embodiments of the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the embodiments of the present application with unnecessary detail.

[0038] It should be understood that the term "comprises" when used in this specification and the appended claims indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0039] It should also be understood that the term "and / or" when used in this specification and the appended claims indicates that the associated listed items can be present one or more of the associated listed items, and that the combinations of the associated listed items are included.

[0040] As used in this specification and the appended claims, the term "if" can be interpreted as meaning "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]," depending on the context.

[0041] In addition, in the description of the specification and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0042] Reference within the specification of this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places within specified

[0043] For the purpose of making the purpose, technical scheme and advantages of the present application more clear and explicit, the present application is described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described below are only used to explain the present application and do not limit the present application.

[0044] Figure 1 is a flowchart of a method for detecting road surface diseases based on a UAV provided by an embodiment of the present application, referring to Figure 1 The method can be implemented by performing steps 101 to 104, which are described in detail as follows:

[0045] In step 101, the original road surface information of the target road section is obtained by using a UAV, and the original road surface information includes path interval numbers and original road surface images.

[0046] In some embodiments, a UAV equipped with a camera is used to take aerial photographs of the target road section to obtain the original road surface information of the target road section. The obtained original road surface information can include path interval numbers and original road surface images.

[0047] Specifically, the camera on the UAV can be a terminal such as a camera or an industrial camera that can take aerial photographs. Taking a camera as an example, a camera with a frame rate of 330 fps can be used, i.e., the camera can take 330 images per second.

[0048] Illustratively, the path interval number is the number obtained by dividing the flight path of the UAV into multiple path intervals and sorting the multiple path intervals according to the aerial photograph time point.

[0049] For ease of understanding, an example is given: the UAV flies at a constant speed of 1 m / s for 10 minutes, the flight path is 600 meters, and 198000 original road surface images are obtained. The flight path of 600 meters is divided into 300 intervals, each interval corresponds to a flight path of 2 meters, and the 300 intervals are sorted according to the time point of the aerial photography. The path interval number of 0-2 seconds is 1, and the aerial photography path is 0-2 meters of the target road section; the path interval number of 2-4 seconds is 2, and the aerial photography path is 2-4 meters of the target road section; …, the path interval number of 598-600 seconds is 300, and the aerial photography path is 598-600 meters of the target road section. It can be seen that the path interval number, the aerial photography path, and the time point of the aerial photography are one-to-one corresponding. Knowing any one of the path interval number, the aerial photography path, or the time point of the aerial photography, the information of the other two can be inferred.

[0050] In some embodiments, the UAV is used to obtain the original road surface information of the target road section, which can be achieved by performing steps 1011 to 1014, which are described in detail as follows:

[0051] In step 1011, based on the preset flight speed and the sustainable flight time of the UAV, the maximum flight distance of the UAV is determined, and a length less than or equal to half of the maximum flight distance is taken as the flight distance of the UAV.

[0052] Optionally, the maximum flight distance of the UAV is affected by various factors such as external wind speed, aerial photography height, and flight attitude of the UAV, in addition to the preset flight speed and the sustainable flight time based on the preset flight speed, which will not be limited one by one in this application.

[0053] Explanatorily, this application takes a length less than or equal to half of the maximum flight distance as the flight distance, considering that the UAV can fly back and forth to the target road section to obtain more abundant original road surface images.

[0054] In step 1012, the road section information of the target road section is obtained. The road section information includes the road section start point, the road section end point, the road section length, and the road section curve information.

[0055] Optionally, the road section information of the target path can be obtained according to the navigation map, the target road section construction drawing, or the field measurement, which will not be limited one by one in this application.

[0056] Explanatorily, the road section curve information includes various information such as the turning node direction, the turning node angle, the turning radius, the straight line length, and the turning length, which serves as the basis for subsequent setting of the flight path of the UAV.

[0057] In step 1013, based on the flight distance and the road section information, the flight path of the UAV on the target road section is determined.

[0058] Optionally, the flight path determined based on the flight distance and the road segment information can ensure that the UAV returns to the starting point of the path to obtain the original road surface images of the target road segment.

[0059] In step 1014, the flight parameters of the UAV are set by the ground station. The flight parameters include the preset flight speed and flight path.

[0060] Illustratively, the ground station can include a computer, a radio, and a remote control. The computer is provided with software for controlling the aircraft. The flight path of the UAV can be planned by using a flight planning tool. The flight height, flight speed, flight location, etc. of the UAV can be set. The flight parameters are compiled and transmitted to the flight control system built in the UAV through the data port after being connected to the radio.

[0061] Optionally, the flight parameters can further include the ground speed when the UAV takes off, the take-off angle of attack, the climb height, the end height, the radius or diameter of the hovering, the empty airspeed indicator, and the navigation deviation warning, etc. The present application does not limit them one by one.

[0062] In step 1015, the flight parameters are sent to the UAV, and the original road surface information collected by the UAV in response to the flight parameters is received.

[0063] Optionally, the ground station transmits the flight parameters to the flight control system built in the UAV through the radio. The UAV flies according to the received flight parameters. The camera on the UAV performs aerial photography to collect the original road surface information.

[0064] In step 102, the original road surface images of the same path interval number are sampled and spliced to obtain a plurality of target road surface images. Each target road surface image corresponds to a path interval number.

[0065] In some embodiments, since the frame rate of the camera is 330 fps, 330 original road surface images can be collected per second. If the original road surface images are not spliced, all images need to be analyzed and identified for road surface diseases. Not only is the data processing amount huge, but the repetition rate of the identified road surface diseases is also too high, which is not practically meaningful. At the same time, if all images are directly spliced, the processing amount is also too large.

[0066] Illustratively, the technical solution of step 102 can be achieved by performing steps 1021 to 1024 to reduce the image data processing amount, which is described in detail as follows:

[0067] In step 1021, the original road surface images of the same path interval number are obtained. The images in the original road surface images are extracted according to a preset sampling rule to obtain a plurality of first road surface images.

[0068] Illustratively, Figure 2is a schematic diagram of the original road surface images of the same path interval number provided by an embodiment of the present application. Referring to Figure 2 , the original image information in the figure is a certain section of the target road section, there are 5 road diseases in the section, the types of the 5 road diseases are different, and the specific positions of each road disease in the section are not clear. The present application groups the original images according to the path interval number, and the original images of the same path interval number are grouped into a group. The 5 road diseases are divided into 3 groups of original images of the same path interval number.

[0069] For example, part of the original road surface images is extracted according to the preset extraction rule. The extracted part of the image is the first road surface image.

[0070] For example, the preset extraction rule is based on the camera frame rate of the unmanned aerial vehicle and the preset flight speed of the unmanned aerial vehicle. According to the preset interval, a plurality of first road surface images are extracted from the original road surface images.

[0071] For ease of understanding, an example is given: Figure 3 is a schematic diagram of the preset extraction rule provided by an embodiment of the present application. As described above, the camera frame rate is 330 fps, and 330 original road surface images can be collected per second. The high frame rate of the camera will result in a very high repetition rate of adjacent two images. In order to reduce the processing amount of image stitching, the original road surface images of the same path interval number are extracted according to the preset extraction rule. The preset extraction rule is based on the camera frame rate and the preset flight speed, and is a rule set according to the preset interval. For example, if the camera frame rate is 330 fps and the preset flight speed is 1 m / s, the preset interval is set to extract the first image from the first 10 consecutive original road surface images. If the camera frame rate is 330 fps and the preset flight speed is 2 m / s, the preset interval is set to extract the first image from the first 5 consecutive original road surface images. As shown in Figure 3 , from the original road surface images of the same path interval number, the first one is extracted from the first n consecutive original road surface images as the first road surface image, and the n-1 images in the first n consecutive original road surface images are discarded, Figure 3 , 4 first road surface images are selected.

[0072] It should be noted that the plurality of extracted first road surface images belong to the same path interval number and are sorted according to the time points of collection.

[0073] In step 1022, the plurality of first road surface images are preprocessed to generate a plurality of second road surface images, and the second road surface images are grayscale images. Step 1022 can be implemented by performing steps A1 and A2:

[0074] In step A1, the plurality of first road surface images are subjected to grayscale processing to obtain grayscale images of the plurality of first road surface images.

[0075] In step A2, a piecewise linear transformation is performed on the gray image of each first road surface image to obtain a corresponding second road surface image.

[0076] The expression of the piecewise linear transformation can be as follows:

[0077]

[0078] In the formula, G represents the gray value of the pixel point in the second road surface image, G represents the gray value of the pixel point in the gray image, and G represents the gray value of the gray image.

[0079] Based on the above expression, FIG. 3 shows a gray value broken line graph of the second road surface image after the piecewise linear transformation according to an embodiment of the present application. As shown in FIG. 3, the gray values in the range of [0, 255] are compressed to the gray range of [0, 255], and the gray values in the range of [255, 511] are amplified to the gray range of [255, 511]. Figure 4 Figure 4 It can be seen that the piecewise linear transformation compresses or amplifies the gray values in different ranges, so that the gray values of the second road surface image are more obvious.

[0080] In step 1023, the similarity of two adjacent second road surface images in the plurality of second road surface images is compared based on the region correlation method, and an overlapping region of the two adjacent road surface images is obtained.

[0081] In some embodiments, the similarity of two adjacent second road surface images in the plurality of second road surface images is calculated with a step of one pixel, and two two-dimensional similarity matrices are obtained. The expression of the similarity calculation is as follows:

[0082]

[0083] In the formula, S represents the similarity value, G represents the gray value of the pixel point in the second road surface image, and G represents the gray value of the pixel point in the second road surface image.

[0084] In some embodiments, the two two-dimensional similarity matrices are compared, and the pixel region with the same similarity value in the two two-dimensional similarity matrices is the overlapping region of the two adjacent second road surface images. ​​​​​​​​​​​​

[0085] Figure 5 is a schematic diagram of an overlapping region of two adjacent road surface images provided by an embodiment of the present application, referring to Figure 5 , four first road surface images extracted from the same path interval number are preprocessed to generate four second road surface images, and the four second road surface images are compared with each other in similarity to determine the overlapping region, and the region in the dashed box is the overlapping region of the two adjacent second road surface images. Figure 3

[0086] In step 1024, the overlapping regions are fused multiple times to generate a target road surface image.

[0087] Based on the overlapping region, multiple adjacent second road surface images are fused to obtain multiple intermediate road surface images, and then the multiple adjacent intermediate road surface images are compared with each other in similarity according to step 1023 to obtain the overlapping region of the two adjacent intermediate road surface images, and then the overlapping region of the two adjacent intermediate road surface images is fused, and steps 1023 and 1024 are repeated to finally generate a target road surface image of the same path interval number. It should be noted that the fused image has all the information of the two adjacent images.

[0088] Figure 6 is a schematic diagram of generating a target road surface image provided by an embodiment of the present application. Referring to Figure 6 , the overlapping region of the two adjacent second road surface images in the four pictures is fused for the first time to generate two intermediate road surface images, and then the two intermediate road surface images are fused for the second time to generate a target road surface image.

[0089] In some embodiments, the overlapping region of the two adjacent second road surface images is fused based on a fusion formula, and the expression of the fusion formula is:

[0090]

[0091] In the formula, and respectively represent the two adjacent second road surface images, represents the pixel value of the target road surface image at the pixel point , and respectively represent the weight values assigned by the two adjacent second road surface images at the pixel point , , , and .

[0092] It should be specially noted that each target road surface image generated in the technical scheme of the present application corresponds to the same path interval number, which is used to determine the specific position of the target road surface image according to different path interval numbers. ​

[0093] In step 103, a plurality of target road surface images are input into the pre-trained road disease analysis model to obtain a plurality of road disease detection results of the target road surface images.

[0094] In some embodiments, the construction of the road disease analysis model can be achieved by performing steps 1031 to 1034, which are described in detail as follows:

[0095] In step 1031, a data set of road disease picture samples is obtained, and the data set is divided into a training set and a test set according to a preset ratio.

[0096] In some embodiments, the data set of road disease picture samples can be obtained in various ways, such as downloading from the network or taking pictures of road diseases, etc., which will not be repeated here.

[0097] In some embodiments, the obtained data set is divided into a training set and a test set according to a preset ratio. The picture samples in the training set are used to train and optimize the subsequent recognition model, and the picture samples in the test set are used to test the recognition performance of the recognition model. The preset ratio can be adaptively set according to the number of picture samples in the data set and actual needs.

[0098] In step 1032, a neural network model is built, which includes a forget gate, an input gate and an output gate.

[0099] For explanation, the forget gate selectively forgets the input of the previous node, the input gate judges the importance of the information of the previous node and selectively remembers it, and the output gate determines the output of the road disease information.

[0100] Optionally, the neural network model can be an LSTM model. Since the LSTM model is prior art and relatively mature, this application will not be repeated.

[0101] In step 1033, the training set is used to train the neural network model to generate an optimized neural network model.

[0102] In some embodiments, the neural network model can be trained by adjusting the hyperparameters to generate an optimized neural network model. The hyperparameters can include learning rate, total iteration number, number of memory blocks, batch size, prediction step sequence, etc.

[0103] In step 1034, the test set is used to test the optimized neural network model to obtain the recognition accuracy of the optimized neural network model. If the recognition accuracy is greater than or equal to a preset recognition accuracy, the road disease analysis model based on the neural network model is trained.

[0104] In some embodiments, if the recognition accuracy is less than the preset recognition accuracy, the neural network model is retrained using the training set until the recognition accuracy is greater than or equal to the preset recognition accuracy.

[0105] In step 104, a road surface disease report of the target road section is output according to the multiple road surface disease detection results and the corresponding path interval numbers.

[0106] In some embodiments, each target road surface image corresponds to a path interval number, and each target road surface image can generate one or more road surface disease detection results through the road surface disease analysis model. Therefore, each multiple road surface disease detection result corresponds to a path interval number. The multiple road surface disease detection results and the corresponding path interval numbers can be used to generate and output a road surface disease report of the target road section.

[0107] The above-mentioned unmanned aerial vehicle-based road surface disease detection method uses an unmanned aerial vehicle to inspect the road surface, which has higher detection efficiency and lower detection cost, and discards the traditional manual inspection. The target road section is divided into multiple intervals, and each interval is numbered according to the aerial photography time, so that images in time series can be obtained. In combination with the flight speed of the unmanned aerial vehicle, the position of the original road surface image on the target road section can be determined. The original road surface images with the same path interval number are subjected to image processing, so that the global image of any interval road section in the target road section can be determined. The interval road section is analyzed by using the pre-trained road surface disease analysis model to identify the road surface disease, which avoids the dependence of the detection result on the comprehensive quality of the staff and improves the detection accuracy and reliability of the road surface disease.

[0108] It should be understood that the size of the serial number of each step in the above-mentioned embodiments does not mean the order of execution. The execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0109] According to the unmanned aerial vehicle-based road surface disease detection method described in the above embodiments, Figure 7 A structure block diagram of the unmanned aerial vehicle-based road surface disease detection device 200 provided by the embodiments of the present application is shown. For ease of illustration, only the parts related to the embodiments of the present application are shown.

[0110] Referring to Figure 7 The unmanned aerial vehicle-based road surface disease detection device in the embodiments of the present application can include an information acquisition module 201, an image processing module 202, an image analysis module 203, and a report output module 204.

[0111] The information acquisition module 201 is used to acquire the original road surface information of the target road segment using a drone. The original road surface information includes the path interval number and the original road surface image. The path interval number is the number obtained by dividing the drone's aerial flight path into multiple path intervals and sorting the multiple path intervals according to the aerial photography time point.

[0112] The process of acquiring raw road surface information of a target road segment using drones includes: determining the maximum flight distance of the drone based on its preset flight speed and sustainable flight time, and using a length less than or equal to half of the maximum flight distance as the drone's flight distance; acquiring road segment information of the target road segment, including the road segment start point, road segment end point, road segment length, and road segment curve information; determining the drone's flight path on the target road segment based on the flight distance and road segment information; setting the drone's flight parameters through a ground station, including preset flight speed and flight path; sending the flight parameters to the drone; and receiving the raw road surface information collected by the drone in response to the flight parameters.

[0113] The image processing module 202 is used to sample and stitch together the original road surface images with the same path interval number to obtain multiple target road surface images, each target road surface image corresponding to a path interval number.

[0114] The process of sampling and stitching original road surface images with the same path interval number to obtain the target road surface image includes: acquiring original road surface images with the same path interval number; extracting images from the original road surface images according to a preset sampling rule to obtain multiple first road surface images; the preset extraction rule is based on the camera frame rate of the camera on the UAV and the preset flight speed of the UAV, and extracting multiple first road surface images from the original road surface images at preset intervals; preprocessing the multiple first road surface images to generate multiple second road surface images, which are grayscale images; comparing the similarity of two adjacent second road surface images in the multiple second road surface images based on a region correlation method to obtain the overlapping area of ​​two adjacent second road surface images; and fusing the overlapping area multiple times to generate the target road surface image.

[0115] Optionally, preprocessing multiple first road surface images to generate multiple second road surface images includes: converting the multiple first road surface images to grayscale to obtain grayscale images of the multiple first road surface images; performing a piecewise linear transformation on each grayscale image to obtain the corresponding second road surface image, wherein the expression for the piecewise linear transformation is:

[0116]

[0117] In the formula, Represents the pixels in the second road surface image grayscale value, Represents the pixels in a grayscale image grayscale value, Represents the grayscale value of a grayscale image. This represents the grayscale value of the second road surface image.

[0118] Optionally, a region-related method is used to compare the similarity of two adjacent second road surface images in multiple second road surface images to obtain the overlapping region of the two adjacent second road surface images, including:

[0119] For each adjacent second road surface image in a plurality of images, the similarity is calculated with a step size of one pixel, resulting in two two-dimensional similarity matrices. The expression for the similarity calculation is as follows:

[0120]

[0121] In the formula, Represents the similarity value. Represents the pixels in the second road surface image grayscale value, Represents the pixels in the second road surface image grayscale value;

[0122] By comparing two two-dimensional similarity matrices, the pixel regions with the same similarity value in the two two-dimensional similarity matrices are the overlapping regions of two adjacent second road surface images.

[0123] Optionally, multiple fusions of overlapping regions to generate the target road surface image include: fusing the overlapping regions of two adjacent second road surface images based on a fusion formula, the expression of which is:

[0124]

[0125] In the formula, and These represent two adjacent images of the second road surface. Indicates the target road surface image at pixels pixel values, and These represent the pixel points of two adjacent second road surface images. The assigned weights, , ,and .

[0126] The image analysis module 203 is used to input multiple target pavement images into a pre-trained pavement distress analysis model to obtain multiple pavement distress detection results of the target pavement images.

[0127] The construction of a road surface distress analysis model includes: obtaining a dataset of road surface distress image samples and dividing the dataset into a training set and a test set according to a preset ratio;

[0128] The neural network model includes a forgetting gate, an input gate and an output gate. The forgetting gate selectively forgets the input of the previous node. The input gate judges the importance of the information of the previous node and selectively remembers. The output gate determines the output of the pavement disease information.

[0129] The neural network model is trained by using the training set to generate an optimized neural network model.

[0130] The optimized neural network model is tested by using the test set to obtain the recognition accuracy of the optimized neural network model. If the recognition accuracy is greater than or equal to the preset recognition accuracy, the pavement disease analysis model based on the neural network model is trained.

[0131] The report output module 204 is configured to output a pavement disease report of the target road section according to the plurality of pavement disease detection results and the corresponding path interval numbers.

[0132] It should be noted that the information interaction, execution process and the like between the above devices / units are based on the same concept as the method embodiments of the present application. For specific functions and technical effects, please refer to the method embodiments part, which will not be repeated here.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or software. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0134] The present application also provides a terminal device, which is described with reference to Figure 8 The terminal device 300 can include at least one processor 310, a memory 320, a computer program 321 stored in the memory 320 and executable on the at least one processor 310, and the processor 310 executes the computer program 321 to implement the steps in any of the above method embodiments, for example Figure 1Steps 101 to 104 in the illustrated embodiment. Alternatively, the processor 310 implements the functions of the modules / units in each of the above-described apparatus embodiments when executing the computer program 321, for example Figure 7 the functions of the illustrated modules 201 to 204.

[0135] By way of example, the computer program 321 can be segmented into one or more modules / units, one or more of which are stored in the storage 320 and executed by the processor 310 to accomplish the present application. One or more of the modules / units can be a series of computer program segments capable of accomplishing a specific function, which are used to describe the execution process of the computer program in the terminal device 300.

[0136] Those skilled in the art can understand that Figure 8 The terminal device is merely an example and does not constitute a limitation on the terminal device, and can include more or fewer components than those shown, or combine certain components, or include different components, such as input / output devices, network access devices, buses, etc.

[0137] The processor 310 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0138] The storage 320 can be an internal storage unit of the terminal device 300, or an external storage device of the terminal device 300, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD), a flash card, etc. The storage 320 is used to store the computer program 321 and other programs and data required by the terminal device 300. The storage 320 can also be used to temporarily store data that has been output or will be output.

[0139] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0140] The method for detecting road diseases based on a UAV provided in the embodiments of the present application can be applied to a terminal device such as a computer, a wearable device, a vehicle-mounted device, a tablet computer, a notebook computer, a netbook, a personal digital assistant (PDA), an augmented reality (AR) / virtual reality (VR) device, a mobile phone, and the like. The embodiments of the present application do not limit the specific type of the terminal device.

[0141] The embodiments of the present application further provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in any of the embodiments of the method for detecting road diseases based on a UAV described above can be implemented.

[0142] The embodiments of the present application provide a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal executes the steps in any of the embodiments of the method for detecting road diseases based on a UAV described above.

[0143] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the computer program for instructing the related hardware to complete all or part of the processes in the above-mentioned embodiment methods can be stored in the computer readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk and the like. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0144] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0145] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0146] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0148] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for detecting road surface defects based on unmanned aerial vehicles (UAVs), characterized in that, include: The original road surface information of the target road segment is obtained by using a drone. The original road surface information includes path interval numbers and original road surface images. The path interval numbers are obtained by dividing the drone's aerial flight path into multiple path intervals and sorting the multiple path intervals according to the aerial photography time points. The original road surface images with the same path interval number are sampled and stitched together to obtain multiple target road surface images, each of which corresponds to one path interval number. The step of sampling and stitching the original road surface images with the same path interval number to obtain multiple target road surface images includes: acquiring the original road surface images with the same path interval number; extracting images from the original road surface images according to a preset sampling rule to obtain multiple first road surface images; the preset extraction rule is based on the camera frame rate of the camera on the UAV and the preset flight speed of the UAV, and extracting the multiple first road surface images from the original road surface images at preset intervals; preprocessing the multiple first road surface images to generate multiple second road surface images, the multiple second road surface images being grayscale images; performing similarity comparison on two adjacent second road surface images in the multiple second road surface images based on a region correlation method to obtain the overlapping region of the two adjacent second road surface images; fusing the overlapping region multiple times to generate a target road surface image; the step of performing similarity comparison on two adjacent second road surface images in the multiple second road surface images based on a region correlation method to obtain the overlapping region of the two adjacent second road surface images includes: calculating the similarity of two adjacent second road surface images in the multiple second road surface images with a step size of one pixel to obtain two two-dimensional similarity matrices, the expression for the similarity calculation being: In the formula, Represents the similarity value. Represents the pixels in the second road surface image grayscale value, Represents the pixels in the second road surface image The grayscale value is compared with the two two-dimensional similarity matrices. The pixel regions with the same similarity value in the two two-dimensional similarity matrices are the overlapping regions of the two adjacent second road surface images. The process of fusing the overlapping regions multiple times to generate the target road surface image includes: fusing the overlapping regions of two adjacent second road surface images based on a fusion formula, wherein the expression of the fusion formula is: In the formula, and These represent two adjacent images of the second road surface. Indicates the target road surface image at pixels pixel values, and These represent the pixel points of two adjacent second road surface images. The assigned weights, , ,and ; The multiple target pavement images are input into a pre-trained pavement distress analysis model to obtain multiple pavement distress detection results for the target pavement images; Based on the multiple pavement defect detection results and the corresponding path interval numbers, a pavement defect report for the target road segment is output.

2. The road surface defect detection method based on unmanned aerial vehicles as described in claim 1, characterized in that, The method of using drones to acquire the original road surface information of the target road section includes: Based on the drone's preset flight speed and sustainable flight time, the maximum flight distance of the drone is determined, and the length less than or equal to half of the maximum flight distance is taken as the drone's flight distance. Obtain the road segment information of the target road segment, including the road segment start point, road segment end point, road segment length, and road segment curve information; Based on the flight distance and the road segment information, the flight path of the UAV on the target road segment is determined; The flight parameters of the UAV are set through a ground station, including the preset flight speed and the flight path; Send the flight parameters to the drone and receive the raw road surface information collected by the drone in response to the flight parameters.

3. The road surface defect detection method based on unmanned aerial vehicles as described in claim 1, characterized in that, The preprocessing of the plurality of first road surface images to generate a plurality of second road surface images includes: The plurality of first road surface images are converted to grayscale to obtain grayscale images of the plurality of first road surface images; A piecewise linear transformation is performed on each of the grayscale images to obtain the corresponding second road surface image. The expression for the piecewise linear transformation is: In the formula, Represents the pixels in the second road surface image grayscale value, Represents the pixels in a grayscale image grayscale value, Represents the grayscale value of a grayscale image. This represents the grayscale value of the second road surface image.

4. The road surface defect detection method based on unmanned aerial vehicles as described in claim 1, characterized in that, The pavement distress analysis model is constructed as follows: Obtain a dataset of road surface defect image samples, and divide the dataset into a training set and a test set according to a preset ratio; A neural network model is constructed, which includes a forget gate, an input gate, and an output gate. The forget gate selectively forgets the input of the previous node, the input gate is used to determine the importance of the information of the previous node and selectively memorize it, and the output gate determines the output of road surface defect information. The neural network model is trained using the training set to generate an optimized neural network model; The optimized neural network model is tested using the test set to obtain the recognition accuracy of the optimized neural network model. If the recognition accuracy is greater than or equal to the preset recognition accuracy, the road surface defect analysis model based on the neural network model is successfully trained.

5. A road surface defect detection device based on unmanned aerial vehicles (UAVs), characterized in that, include: The information acquisition module is used to acquire the original road surface information of the target road segment using a drone. The original road surface information includes path interval numbers and original road surface images. The path interval numbers are obtained by dividing the drone's aerial flight path into multiple path intervals and sorting the multiple path intervals according to the aerial photography time points. The image processing module is used to sample and stitch together the original road surface images with the same path interval number to obtain multiple target road surface images, each of the target road surface images corresponding to one path interval number; The step of sampling and stitching the original road surface images with the same path interval number to obtain multiple target road surface images includes: acquiring the original road surface images with the same path interval number; extracting images from the original road surface images according to a preset sampling rule to obtain multiple first road surface images; the preset extraction rule is based on the camera frame rate of the camera on the UAV and the preset flight speed of the UAV, and extracting the multiple first road surface images from the original road surface images at preset intervals; preprocessing the multiple first road surface images to generate multiple second road surface images, the multiple second road surface images being grayscale images; performing similarity comparison on two adjacent second road surface images in the multiple second road surface images based on a region correlation method to obtain the overlapping region of the two adjacent second road surface images; fusing the overlapping region multiple times to generate a target road surface image; the step of performing similarity comparison on two adjacent second road surface images in the multiple second road surface images based on a region correlation method to obtain the overlapping region of the two adjacent second road surface images includes: calculating the similarity of two adjacent second road surface images in the multiple second road surface images with a step size of one pixel to obtain two two-dimensional similarity matrices, the expression for the similarity calculation being: In the formula, Represents the similarity value. Represents the pixels in the second road surface image grayscale value, Represents the pixels in the second road surface image The grayscale value is compared with the two two-dimensional similarity matrices. The pixel regions with the same similarity value in the two two-dimensional similarity matrices are the overlapping regions of the two adjacent second road surface images. The process of fusing the overlapping regions multiple times to generate the target road surface image includes: fusing the overlapping regions of two adjacent second road surface images based on a fusion formula, wherein the expression of the fusion formula is: In the formula, and These represent two adjacent images of the second road surface. Indicates the target road surface image at pixels pixel values, and These represent the pixel points of two adjacent second road surface images. The assigned weights, , ,and ; The image analysis module is used to input the multiple target pavement images into a pre-trained pavement distress analysis model to obtain multiple pavement distress detection results of the target pavement images; The report output module is used to output a pavement defect report for the target road segment based on the multiple pavement defect detection results and the corresponding path interval numbers.

6. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images