A road surface defect image acquisition method

Abstract

The present invention relates to a road surface defect image acquisition method; comprising the following steps: step 1: using a camera to continuously shoot the road surface at a certain tilt angle to obtain an oblique view of the road surface defect; step 2: performing image preprocessing on the oblique view of the road surface defect acquired by the camera to obtain a road surface defect image; step 3: establishing a defect data set based on the obtained road surface defect image. The present invention uses an oblique viewing angle to shoot the road surface, which overcomes the shortcomings of traditional vertical downward shooting that causes image blur and frame loss, improves data acquisition efficiency, and reduces acquisition costs.

Description

Technical Field

[0001] The present invention relates to the fields of road surface defect detection and artificial intelligence technology, and in particular to a road surface defect image acquisition method. Background Art

[0002] With the enhancement of my country's comprehensive national strength, the construction of highway transportation has achieved unprecedented development and made great contributions to the national economy. With the surge in traffic volume, a large number of surface defects have been generated during use. Existing road defects mainly include various cracks, bulges, potholes and ruts, etc. These road surface defects seriously affect driving safety. Therefore, being able to detect and repair various defects in a timely manner is of great significance to road health and driving safety.

[0003] Nowadays, the detection of road defects has developed from early camera measurement, ground penetrating radar, laser measurement and infrared measurement to today's deep neural network defect detection. In particular, neural networks such as FCN, U-Net and DeepLabV3+ are the main ones. However, no matter which detection method is used, it needs to be based on rich road defect image data. Only with a certain amount of data can the detection method be faster and more efficient. The acquisition of image data is closely related to the image acquisition method. When early researchers collected road defect data, they often needed to use a handheld camera to shoot the road on the highway. This acquisition method requires a lot of manpower, which not only increases labor costs, but also has extremely low efficiency. Working on the highway also poses a hidden danger to the personal safety of workers. In subsequent developments, researchers designed a road image acquisition vehicle to address these problems. The device uses a downward camera at the rear of the vehicle perpendicular to the road surface to shoot the road surface. Most public datasets such as the pavement crack dataset CRACK500 and the German asphalt pavement disease dataset GAPs384 are obtained using this method. Especially GAPs384, the images in this dataset contain a variety of defect types such as cracks, pits and patches, with a resolution of up to 1920*1080 and a number of 1969 images. Moreover, these datasets have made great contributions to road defect detection. However, this method of shooting perpendicular to the road surface has a very limited effective defect area, and blurring and frame drops often occur when driving at high speeds, resulting in a large amount of defect information loss. In order to obtain clear images at high speeds, the cost of equipment must be increased. Therefore, this method often cannot take into account both low cost and high efficiency at the same time. Therefore, a low-cost and high-efficiency image acquisition method is needed to overcome the above problems.

[0004] Rich pavement defect data can produce better neural network models, so increasing training data for actual scenarios is a necessary task. In order to obtain richer crack data and more diverse crack types, exploring an efficient and fast data collection method to establish a defect data set is still a research focus. Summary of the invention

[0005] In order to solve the above technical defects in the prior art, the present invention provides a road surface defect image acquisition method, which can effectively solve the problems in the background technology.

[0006] In order to solve the above technical problems, the technical solutions provided by the present invention are as follows:

[0007] The embodiment of the present invention discloses a method for collecting road surface defect images, comprising the following steps:

[0008] Step 1: Use a camera to continuously shoot the road surface at a certain tilt angle to obtain an oblique view of road surface defects;

[0009] Step 2: performing image preprocessing on the oblique view of the road surface defect captured by the camera to obtain a road surface defect image;

[0010] Step 3: Establish a defect dataset based on the obtained road surface defect images.

[0011] In any of the above schemes, it is preferred that, when in use, road surface defects are photographed at a certain tilt angle, and the video is frame-processed according to the vehicle speed and the effective shooting area where road surface defects appear and disappear. In this way, the shooting is not prone to blurring and frame drops, so the equipment requirements are relatively low and this method increases the effective shooting area, so a relatively clear road surface crack image can still be obtained when driving at high speed, thereby improving data collection efficiency.

[0012] In any of the above solutions, preferably, when photographing the road surface with a camera, the following steps are included:

[0013] Step 1: First, attach the camera to the hood or trunk of the vehicle, and make the camera lens mirror form an angle of α with the horizontal road surface;

[0014] Step 2: Extract frames from the video according to the vehicle speed and the effective shooting area where road defects appear and disappear.

[0015] In any of the above schemes, it is preferred that when the video is subjected to frame extraction processing based on the vehicle speed and the effective shooting area where road defects appear and disappear, the number of frames extracted per second is n>v / d; wherein n is the number of frames extracted per second, v is the vehicle speed, and d is the length of the effective shooting area of ​​the camera.

[0016] In any of the above solutions, preferably, when performing image preprocessing on the oblique view of the road surface defect collected by the camera, the following steps are included:

[0017] Step 1: Build a YOLOv5 target detection model and train it;

[0018] Step 2: Use the trained YOLOv5 target detection model to perform target detection on each frame after frame extraction to obtain the category and location information of various defects;

[0019] Step 3: Use perspective transformation to transform the oblique view of road surface defects into a vertical top view of road surface defects, and perform image cropping according to the position coordinate information of various defects to obtain crack image data.

[0020] In any of the above solutions, preferably, the matrix transformation formula of the perspective transformation is: Where [x,y,z] T is the source point matrix, [X,Y,Z] T is the target point matrix, and A matrix is ​​the perspective transformation matrix.

[0021] In any of the above schemes, preferably, when transforming the oblique view of the road surface defect into a vertical top view of the road surface defect by using perspective transformation, the following steps are included:

[0022] Step 1: Use Opencv to estimate the source view plane coordinates and the new view plane coordinates according to the camera height and the effective area of ​​the road defect, and obtain the perspective transformation matrix A;

[0023] Step 2: Use the perspective matrix to transform the original image to the new viewing plane;

[0024] Step 3: Use the formula Find the coordinate information corresponding to the original crack target in the new view plane, where (X′, Y′) is the coordinate of any point in the source view plane corresponding to the new view plane point.

[0025] In any of the above solutions, preferably, image cropping is to crop the transformed image according to the crack target and coordinate information to obtain clear image data containing only defect information.

[0026] In any of the above solutions, preferably, when establishing a defect data set according to the obtained road surface defect image, the following steps are included:

[0027] Step 1: Manually screen the obtained road surface defect image data to obtain clear road surface defect image data of different types and resolutions;

[0028] Step 2: Manually label the crack images of different types and resolutions to obtain a PNG mask image corresponding to the road surface defect image;

[0029] Step 3: After all images are labeled, the defect image dataset CRACK2000 is established.

[0030] In any of the above schemes, it is preferred that when manually annotating the obtained crack images of different types and resolutions, an image data annotation tool written in OpenCV is used to annotate the defect image after image preprocessing. When annotating, pixel by pixel annotation is performed along the inner contour of the defect to obtain a PNG mask image corresponding to the road surface defect image.

[0031] In any of the above schemes, it is preferred that, in the PNG mask image corresponding to the road defect image, each PNG mask image is a black and white binary image, and is divided into a background area and a defect area, wherein the white in the PNG mask image represents the defect area and the black represents the background area.

[0032] In any of the above schemes, it is preferred that the oblique view of the road surface defects is collected from a part of the road section in Baotou City, Inner Mongolia, China, with a total length of 33 kilometers, and a DJI Osmo sports camera is used to continuously shoot the asphalt road surface with the camera lens mirror forming an oblique angle α with the horizontal road surface, wherein the camera frame rate is, the resolution is 1920*1080, the vehicle speed is about, the road surface is continuously shot, and the camera height is recorded, the length of the effective area of ​​the camera shooting is about d=4m, then the number of frames extracted per second should be n>v / d≈5.56, so as to ensure that the defect information will not be lost during driving. Therefore, n=6 is taken, that is, 1 frame is extracted every 10 frames as the road surface defect image data.

[0033] In any of the above schemes, it is preferred that the acquired oblique view data of road surface defects are screened, and finally 8165 images containing only clear road surface defects are obtained, with a resolution of 1920*1080, and the defect types in the original data include horizontal and vertical cracks, mesh cracks, potholes, and various inlays, etc. There are relatively rich background interferences, including water stains, snow stains, shadows, and uneven lighting, which can well reflect the road surface conditions and are more representative.

[0034] In any of the above schemes, it is preferred that the original image contains multiple defects, and the distances between various defects are relatively far. In order to reduce the amount of subsequent image segmentation calculations, invalid areas are eliminated, and images containing only defects are obtained. Target detection algorithms are used for detection. The original image is detected using a pre-trained YOLOv5 target detection model, and image blocks containing only road surface defects are preliminarily selected, and the defect category and the original position coordinate information of the defects are obtained.

[0035] In any of the above schemes, it is preferred that after raw data acquisition and image preprocessing, clear defect image data of different defect types and different resolutions are obtained, and then the preprocessed defect image is annotated pixel by pixel to generate a PNG mask image corresponding to the defect image. After all images are annotated, a defect image dataset CRACK2000 is established. This dataset has more complex background information and more diverse defect types, providing a large amount of data support for the development of the field of road defect detection.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) This application uses an oblique perspective to shoot the road surface, overcoming the shortcomings of traditional vertical downward shooting that causes image blur and frame loss, thereby improving data collection efficiency and reducing collection costs.

[0038] (2) This application combines target detection with perspective transformation to design a road surface image acquisition method, through which a large number of road surface defect images can be quickly obtained, which has the characteristics of high efficiency and low cost.

[0039] (3) The defect dataset CRACK2000 was constructed using this acquisition method. This dataset has more complex background information and provides data support for pavement defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used for further understanding of the present invention and are used to explain the present invention together with the embodiments of the present invention, but do not constitute a limitation of the present invention.

[0041] Figure 1 It is an overall structural block diagram of a road surface defect image acquisition method provided by an embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of a road surface defect image acquisition method provided by an embodiment of the present invention.

[0043] Description of the numbers in the figure:

[0044] 1. Camera; 2. Road surface; 3. Camera shooting effective area; 4. Target detection; 5. Perspective transformation; 6. Image cropping; 7. Angle between the camera lens and the horizontal road surface; 8. Defect image. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0047] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0048] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0049] In order to better understand the above technical solution, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0050] A road surface defect image acquisition method comprises the following steps:

[0051] Step 1: Use a camera to continuously shoot the road surface at a certain tilt angle to obtain an oblique view of road surface defects;

[0052] Step 2: performing image preprocessing on the oblique view of the road surface defect captured by the camera to obtain a road surface defect image;

[0053] Step 3: Establish a defect dataset based on the obtained road surface defect images.

[0054] Specifically, when using a camera to shoot the road surface, the following steps are included:

[0055] Step 1: First, attach the camera to the hood or trunk of the vehicle, and make the camera lens mirror form an angle of α with the horizontal road surface;

[0056] Step 2: Extract frames from the video according to the vehicle speed and the effective shooting area where road defects appear and disappear.

[0057] Furthermore, when the video is subjected to frame extraction processing according to the vehicle speed and the effective shooting area where road defects appear and disappear, the number of frames extracted per second is n>v / d; wherein n is the number of frames extracted per second, v is the vehicle speed, and d is the length of the effective shooting area of ​​the camera.

[0058] When in use, road defects are photographed at a certain tilt angle, and the video is framed according to the vehicle speed and the effective shooting area where road defects appear and disappear. This way, the shooting is less prone to blurring and frame drops, so the equipment requirements are lower and this method increases the effective shooting area, so clearer road crack images can still be obtained when driving at high speeds, thereby improving data collection efficiency.

[0059] Specifically, when performing image preprocessing on the oblique view of road surface defects collected by the camera, the following steps are included:

[0060] Step 1: Build a YOLOv5 target detection model and train it;

[0061] Step 2: Use the trained YOLOv5 target detection model to perform target detection on each frame after frame extraction to obtain the category and location information of various defects;

[0062] Step 3: Use perspective transformation to transform the oblique view of road surface defects into a vertical top view of road surface defects, and perform image cropping according to the position coordinate information of various defects to obtain crack image data.

[0063] Furthermore, the matrix transformation formula for perspective transformation is: Where [x,y,z] T is the source point matrix, [X,Y,Z] T is the target point matrix, and A matrix is ​​the perspective transformation matrix.

[0064] Furthermore, when the oblique view of the road surface defect is transformed into a vertical top view of the road surface defect by using perspective transformation, the following steps are included:

[0065] Step 1: Use Opencv to estimate the source view plane coordinates and the new view plane coordinates according to the camera height and the effective area of ​​the road defect, and obtain the perspective transformation matrix A;

[0066] Step 2: Use the perspective matrix to transform the original image to the new viewing plane;

[0067] Step 3: Use the formula Find the coordinate information corresponding to the original crack target in the new view plane, where (X′, Y′) is the coordinate of any point in the source view plane corresponding to the new view plane point.

[0068] Furthermore, image cropping is to crop the transformed image according to the crack target and coordinate information to obtain clear image data containing only defect information.

[0069] When in use, since the ultimate goal of defect detection is to quantify the defects and calculate the geometric information of various defects, directly calculating the defects in the oblique view of the road surface will increase the amount of calculation and complexity. Therefore, the perspective transformation is used to transform the oblique view of the road surface into a vertical top view, and then the image is cropped according to the position coordinate information of various defects. While reducing the amount of calculation and the complexity of the calculation, the final crack image data with the same effect as the traditional acquisition method is obtained.

[0070] Specifically, when establishing a defect data set according to the obtained road surface defect image, the following steps are included:

[0071] Step 1: Manually screen the obtained road surface defect image data to obtain clear road surface defect image data of different types and resolutions;

[0072] Step 2: Manually label the crack images of different types and resolutions to obtain a PNG mask image corresponding to the road surface defect image;

[0073] Step 3: After all images are labeled, the defect image dataset CRACK2000 is established.

[0074] Furthermore, when manually annotating the crack images of different types and resolutions, the image data annotation tool written in OpenCV is used to annotate the defect images after image preprocessing. During the annotation, pixel-by-pixel annotation is performed along the inner contour of the defect to obtain a PNG mask image corresponding to the road surface defect image.

[0075] Furthermore, in the PNG mask images corresponding to the road defect images, each PNG mask image is a black and white binary image, and is divided into a background area and a defect area, wherein the white in the PNG mask image represents the defect area, and the black represents the background area.

[0076] This dataset has more complex background information and more diverse defect types, providing a large amount of data support for the development of the road defect detection field.

[0077] Example 1

[0078] A road surface defect image acquisition method comprises the following steps:

[0079] Step 1: Use a camera to continuously shoot the road surface at a certain tilt angle to obtain an oblique view of road surface defects; the oblique view of road surface defects is collected from a part of a road section in Baotou City, Inner Mongolia, China, with a total length of 33 kilometers. A DJI Osmo motion camera is used to continuously shoot the asphalt road surface with the camera lens mirror forming an oblique angle α with the horizontal road surface. The camera frame rate is, the resolution is 1920*1080, the vehicle speed is about, the road surface is continuously shot, and the camera height is recorded. The length of the effective area of ​​the camera shooting is about d=4m, then the number of frames extracted per second should be n>v / d≈5.56, in order to ensure that the defect information will not be lost during driving. Therefore, n=6 is taken, that is, 1 frame is extracted every 10 frames as the road surface defect image data.

[0080] The acquired oblique view data of road defects were screened, and finally 8165 images containing only clear road defects were obtained, with a resolution of 1920*1080. The defect types in the original data included horizontal and vertical cracks, mesh cracks, potholes, and various inlays, etc. There are relatively rich background interferences, including water stains, snow stains, shadows, and uneven lighting, which can well reflect the road conditions and are more representative.

[0081] Step 2: performing image preprocessing on the oblique view of the road surface defect captured by the camera to obtain a road surface defect image;

[0082] The original image contains multiple defects, and the distances between various defects are relatively far. In order to reduce the amount of subsequent image segmentation calculations, invalid areas are eliminated, and images containing only defects are obtained. The target detection algorithm is used for detection. The pre-trained YOLOv5 target detection model is used to detect the original image, and the image blocks containing only road surface defects are preliminarily selected, and the defect category and the original position coordinate information of the defects are obtained.

[0083] Step 3: Establish a defect dataset based on the obtained road surface defect images.

[0084] After raw data collection and image preprocessing, clear defect image data of different defect types and different resolutions were obtained. Then, the preprocessed defect images were annotated pixel by pixel to generate PNG mask images corresponding to the defect images. After all images were annotated, the defect image dataset CRACK2000 was established. This dataset has more complex background information and more diverse defect types, providing a large amount of data support for the development of the field of pavement defect detection.

[0085] Compared with the prior art, the present invention provides the following beneficial effects:

[0086] (1) This application uses an oblique perspective to shoot the road surface, overcoming the shortcomings of traditional vertical downward shooting that causes image blur and frame loss, thereby improving data collection efficiency and reducing collection costs.

[0087] (2) This application combines target detection with perspective transformation to design a road surface image acquisition method, through which a large number of road surface defect images can be quickly obtained, which has the characteristics of high efficiency and low cost.

[0088] (3) The defect dataset CRACK2000 was constructed using this acquisition method. This dataset has more complex background information and provides data support for pavement defect detection.

[0089] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A road defect image acquisition method, characterized in that: The following steps are involved: Step 1: Use a camera to continuously shoot the road surface at a certain tilt angle to obtain an oblique view of road surface defects; Step 2: performing image preprocessing on the oblique view of the road surface defect captured by the camera to obtain a road surface defect image; Step 3: Create a defect dataset based on the obtained road surface defect images ； When using a camera to shoot the road surface, the following steps are included: Step 1 (I): First, attach the camera to the hood or trunk of the vehicle, and make the camera lens mirror surface form an angle of α with the horizontal road surface; Step 1 (2): Extract frames from the video based on the vehicle speed and the effective shooting area where road defects appear and disappear ； When the video is framed according to the vehicle speed and the effective shooting area where road defects appear and disappear, the number of frames extracted per second is n>v / d; where n is the number of frames extracted per second, v is the vehicle speed, and d is the length of the effective shooting area of ​​the camera. ； When performing image preprocessing on the oblique view of road surface defects collected by the camera, the following steps are included: Step 2 (a): Build a YOLOv5 target detection model and train it; Step 2 (ii): Use the trained YOLOv5 target detection model to perform target detection on each frame after frame extraction to obtain the category and location information of various defects; Step 2 (3): Use perspective transformation to transform the oblique view of the road surface defects into a vertical top view of the road surface defects, and perform image cropping according to the position coordinate information of various defects to obtain crack image data. ； The matrix transformation formula for perspective transformation is: Among them, [x, y, z]T is the source point matrix, [X, Y, Z]T is the target point matrix, and A matrix is ​​the perspective transformation matrix ； When the oblique view of the road surface defect is transformed into a vertical top view of the road surface defect by using perspective transformation, the following steps are included: Step 2 (3) 1: Use Opencv to estimate the source view plane coordinates and the new view plane coordinates according to the camera height and the effective area of ​​the road defect, and obtain the perspective transformation matrix A; Step 2 (3) 2: Use the perspective matrix to transform the original image to the new viewing plane; Step 2(3)3: Use the formula Find the coordinate information corresponding to the original crack target in the new view plane, where (X′, Y′) is the coordinate of any point in the source view plane corresponding to the point in the new view plane; After raw data collection and image preprocessing, clear defect image data of different defect types and different resolutions are obtained. Then, the preprocessed defect images are annotated pixel by pixel to generate PNG mask images corresponding to the defect images. After all images are annotated, the defect image dataset CRACK2000 is established.

2. The road surface defect image acquisition method according to claim 1, characterized in that: Image cropping is to crop the transformed image according to the crack target and coordinate information to obtain clear image data that only contains defect information.

3. The road surface defect image acquisition method according to claim 2, characterized in that: When establishing a defect data set based on the obtained road surface defect image, the following steps are included: Step 1: Manually screen the obtained road surface defect image data to obtain clear road surface defect image data of different types and resolutions; Step 2: Manually label the crack images of different types and resolutions to obtain a PNG mask image corresponding to the road surface defect image; Step 3: After all images are labeled, the defect image dataset CRACK2000 is established.

4. The road surface defect image acquisition method according to claim 3, characterized in that: When manually annotating the crack images of different types and resolutions, the image data annotation tool written in OpenCV is used to annotate the defect images after image preprocessing. During the annotation, pixel-by-pixel annotation is performed along the inner contour of the defect to obtain a PNG mask image corresponding to the road surface defect image.

5. The road surface defect image acquisition method according to claim 4 is characterized in that: In the PNG mask images corresponding to the road defect images, each PNG mask image is a black and white binary image and is divided into a background area and a defect area, wherein the white in the PNG mask image represents the defect area and the black represents the background area.

Images