Real-time identification and segmentation method and system for road surface cracks during movement
By using a three-level deep learning model, combined with generative adversarial networks and transformer networks, the problems of insufficient detection accuracy and speed in pavement crack detection are solved, enabling real-time and accurate localization and risk assessment of pavement cracks.
Patent Information
- Application Number
- CN202311873275.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2043-12-29
AI Technical Summary
Existing technologies for detecting road surface cracks suffer from high costs, long processing times, significant safety risks to inspectors, and insufficient accuracy, especially when vehicles are traveling at high speeds, resulting in poor detection performance due to blurred images.
A three-level deep learning model, including a lightweight classification model, a generative deblurring model, and a high-precision segmentation model, is adopted, combined with generative adversarial networks and transformer networks, to achieve real-time identification and accurate localization of road surface cracks.
It enables real-time and accurate identification and location of road surface cracks during high-speed vehicle movement, improving detection speed and accuracy, and allowing for assessment of road risks and repair.
Smart Images

Figure CN117830845B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of pavement crack recognition, and in particular relates to a method and system for real-time recognition and segmentation of pavement cracks during travel. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Traditional pavement crack detection usually uses manual detection methods. This method is simple, but it also has disadvantages such as high cost, time-consuming, obstruction of traffic, and high safety risks for inspectors.
[0004] With the development of deep learning, image segmentation, as a computer vision technology, has broad application prospects in the field of road crack detection. Automatic road crack identification is usually achieved by using a vehicle equipped with imaging equipment to capture the passing road while driving at high speed.
[0005] Although crack recognition technology based on object detection or image segmentation has been applied to automatic detection technology, the crack size detection and positioning accuracy need to be improved, and the requirements for data acquisition are relatively high.
[0006] In practical applications, due to external factors such as high-speed vehicle driving, moving detection equipment, uneven road surface and operating environment, the image data actually collected is likely to have image degradation represented by image blur. Currently, the generative adversarial network model has been applied to image generation, but it has not been applied to the specific task of removing blur from road cracks. Summary of the invention
[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a real-time identification method of pavement cracks while moving based on deep learning, which is used to evaluate and repair the danger level of pavement cracks.
[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0009] In a first aspect, a method for real-time identification and segmentation of road cracks during driving is disclosed, comprising:
[0010] Conduct on-the-go data collection of pavement cracks;
[0011] A three-level deep learning model for real-time identification of road cracks during driving is established, including:
[0012] The first-level lightweight classification model: extracts fuzzy frames from the collected data;
[0013] The second-level generative deblurring model imports the blurred frames separated by the first-level lightweight classification model into the trained generative adversarial network model to repair the blurred frames;
[0014] The third-level high-precision segmentation model: segments the road crack image repaired by the second-level generative deblurring model into a binary image containing only crack information, identifies the extension direction, bifurcation direction and edge shape of the road cracks, performs feature extraction, and also uses a lightweight network to achieve pixel-level size detection of road cracks.
[0015] As a further technical solution, pixel-level size detection of road cracks specifically includes:
[0016] At a fixed focal length, the real geometric size corresponding to each pixel in the crack binary image is recorded, realizing the mapping of the crack pixel-level geometric size to the real millimeter-level geometric size;
[0017] The crack size measurement is performed on the final output crack binary image as above to evaluate and repair the degree of danger.
[0018] As a further technical solution, the first-level lightweight classification model is a lightweight classification module built by a convolutional neural network and a transformer network, which automatically classifies the input crack image as blurry or not blurry, and uses a lightweight network to speed up.
[0019] As a further technical solution, the second-level generative deblurring model is constructed by a generative adversarial network, based on conditional and cyclic generative adversarial networks, and trains the generator and discriminator at the same time. The function of the generator is to generate deblurred images, and the function of the discriminator is to distinguish between generated images and real images. This level of model is used to repair blurred frame images into non-blurred images.
[0020] As a further technical solution, the third-level high-precision segmentation model uses an image segmentation network trained with transformer and lightweight backbone to segment the road surface crack images, and divides the input road surface images into binary images with only cracks.
[0021] As a further technical solution, the extension direction, bifurcation direction and edge shape of the pavement cracks are identified, and after feature extraction, the size of the pavement cracks is identified by proportional calculation.
[0022] As a further technical solution, it also includes: splicing the identified road crack information map by frames to achieve the simultaneous display of the collected real-time road conditions and the segmented binary map in the same view.
[0023] In the second aspect, a real-time identification and segmentation system for road surface cracks during travel is disclosed, comprising:
[0024] The data collection module is configured to: collect on-the-go data of road cracks;
[0025] The three-level deep learning model building module is configured to build a three-level deep learning model for real-time recognition of road cracks during driving, including:
[0026] The first-level lightweight classification model: extracts fuzzy frames from the collected data;
[0027] The second-level generative deblurring model imports the blurred frames separated by the first-level lightweight classification model into the trained generative adversarial network model to repair the blurred frames;
[0028] The third-level high-precision segmentation model: segments the road crack image repaired by the second-level generative deblurring model into a binary image containing only crack information, identifies the extension direction, bifurcation direction and edge shape of the road cracks, performs feature extraction, and also uses a lightweight network to achieve pixel-level size detection of road cracks.
[0029] One or more of the above technical solutions have the following beneficial effects:
[0030] The present invention introduces a neural network commonly used in deep learning to achieve the whole process of crack information collection, crack image blur correction and accurate positioning of road cracks in the process of detecting road cracks on the move. The present invention uses a generative adversarial network and a transformer feature extraction algorithm to remove blur.
[0031] The present invention also introduces a lightweight neural network to improve the detection speed and realize real-time detection during movement.
[0032] The present invention can realize the real-time and accurate positioning of road surface cracks by the crack identification carrier during high-speed movement, thereby further evaluating the road surface risk.
[0033] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0035] Figure 1 This is a flow chart of a system for detecting road cracks while in motion according to an embodiment of the present invention;
[0036] Figure 2 A network framework diagram for classifying fuzzy frames as fuzzy or non-fuzzy in an embodiment of the present invention;
[0037] Figure 3 This is a framework diagram of a fuzzy repair network and a crack identification network according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0039] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.
[0040] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0041] Embodiment 1
[0042] This embodiment example discloses a method for real-time identification and segmentation of road cracks while moving. On a mobile device equipped with an imaging system, a three-level deep learning model of a generative adversarial network and an image segmentation network is used to build a road crack collection and positioning system while moving. The three-level deep learning model includes a first-level lightweight classification model, a second-level generative defuzzification model, and a third-level high-precision segmentation model.
[0043] Among them, the first-level lightweight classification model is a lightweight classification module built by a convolutional neural network and a transformer network. Figure 2 As shown in the figure, feature extraction is performed through convolutional neural networks and transformer networks. At the end of the model, a fully connected layer classification block for crack image classification is connected, which can automatically classify the input crack image as blurry or non-blurry. A lightweight network is used to speed up the process. The purpose of this level of model is to extract blurry frames of road crack information collected while driving. The advantage is that the second-level repair model is more complex and time-consuming than the first-level model. Through the first-level pre-classification, only the blurry cracks need to be repaired in the second-level model. The repair process of non-blurred images is omitted, which improves the overall processing time and facilitates real-time detection while driving.
[0044] The second-level generative deblurring model is a generative deblurring module built by a generative adversarial network. This level of model is based on conditional and cyclic generative adversarial networks, and trains both the generator and the discriminator. The generator is used to generate deblurred images, and the discriminator is used to distinguish between generated images and real images. This level of model can restore blurred frame images to non-blurred images, achieving more accurate blur removal and noise reduction for road cracks collected during driving.
[0045] For details, please refer to the attached Figure 3 As shown, both the artificially generated fuzzy crack image and the real clear crack image are input into the generator;
[0046] The test set contains artificially generated blurred crack images or real clear crack images. The images in the test set are deblurred and then input into the generator;
[0047] Output the deblurred crack image in the generator;
[0048] The artificially generated blurred crack image and the deblurred crack image outputted from the generator are used as inputs to the discriminator. The artificially generated blurred crack image and the real clear crack image are also used as inputs to the discriminator.
[0049] The discriminator judges yes or no based on the loss function, specifically:
[0050]
[0051]
[0052] z is a sample of the input blurred image, p z (z) is the spatial distribution of the blurred image, x is the clear crack image, p data (x) is the spatial distribution corresponding to x.
[0053] The third-level high-precision segmentation model uses an image segmentation network trained with transformers and lightweight backbones to segment road crack images, dividing the input road surface images into binary images with only cracks, thus achieving accurate road surface crack detection based entirely on artificial intelligence.
[0054] See again the attached Figure 3 As shown in Fig. 2, the deblurred crack image is input into the image segmentation network, and the image segmentation loss function adopts a linear combination of the BCE loss function and the Dice loss function, which is specifically:
[0055] l BCE =-[y true ·logy pred +(1-y true )·log(1-y pred )]
[0056]
[0057] l combine =0.5 l BCE +l Dice
[0058] Among them, y true and predThey represent the manually annotated crack labels and the prediction results of the model respectively.
[0059] Through the loss function of this linear combination, the prediction results and crack labels can be fitted under supervised learning, thereby improving the model prediction ability through iteration.
[0060] The real-time recognition and segmentation method of road cracks during driving is shown in the following flowchart: Figure 1 shown.
[0061] This embodiment discloses a method for real-time recognition of road cracks while moving based on a generative adversarial network and a transformer segmentation network, comprising:
[0062] Equip the vehicle-mounted platform with imaging equipment to collect non-artificial road crack information during movement;
[0063] The first-level lightweight classification model is an image classification model built by a convolutional neural network and a transformer network, which can automatically classify the input crack images as blurry or not blurry.
[0064] The first-level lightweight classification model built with convolutional neural networks and transformer networks can separate the collected real-time videos of road cracks into normal frames and blurred frames. The normal frames are retained and the blurred frames are input into the second-level model for blur removal, while the normal frames are directly input into the third-level image segmentation model. At the same time, the lightweight network can significantly improve the classification speed of the model.
[0065] Among them, the second-level generative deblurring model is a blur removal model constructed by a generative adversarial network. This level of model is based on conditional and cyclic generative adversarial networks, and trains both the generator and the discriminator. The generator is used to generate deblurred images, and the discriminator is used to distinguish between generated images and real images. This level of model can repair blurred frame images into non-blurred images, achieving more accurate blur removal and noise reduction for road cracks collected during driving.
[0066] A generative adversarial network for blur repair is trained in advance based on the experimental dataset. The blurry frames collected while moving are input into the network for blur repair, and clear images that can be used for detection are output. A lightweight neural network is used to speed up the process.
[0067] The image segmentation network of the third-level high-precision segmentation model for crack pixel-level recognition was also trained in advance. The repaired non-blurred road surface image was used as input, and the binary image after pixel area and background area segmentation was output. The lightweight neural network was used for speeding up. The segmented binary image can filter out noise interference and intuitively reflect the pixel-level characteristics of the road damage condition, so that the crack size can be measured through mapping in the next step.
[0068] The mapping between the image pixels and the real geometric size is achieved by fixing the focal length of the imaging device, and the corresponding real size of a single pixel in the binary image is clarified. The real size of the identified road cracks is calculated in this proportion, and the average width and maximum width of the road cracks in the frame are displayed in a prominent position.
[0069] To calculate the true size of the pavement cracks, the true geometric size corresponding to each pixel in the crack image is recorded at a fixed focal length, realizing the mapping of the crack pixel-level geometric size to the true millimeter-level geometric size. The crack size measurement is performed on the final output crack binary image to assess and repair the degree of danger.
[0070] When calculating, for example, through mapping, it is measured that the actual area corresponding to the size of each pixel in the image is 1mm×1mm. The binary image after segmentation of a longitudinal crack reaches a width of 10 pixels at the widest point, that is, the actual width of the crack at the widest point is 1cm.
[0071] Finally, the visualization interface is post-processed. In this process, the collected real-time road conditions and the segmented binary images are displayed in the same view. The recognized real-time road conditions and the binary images of road cracks are spliced by frames, and the road size, collection time, current road conditions and other information are displayed in a prominent position.
[0072] Embodiment 2
[0073] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.
[0074] Embodiment 3
[0075] The purpose of this embodiment is to provide a computer-readable storage medium.
[0076] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.
[0077] Embodiment 4
[0078] The purpose of this embodiment is to provide a real-time road crack recognition system based on deep learning, including:
[0079] Conduct on-the-go data collection of pavement cracks;
[0080] Extracting blurred frames from the collected data;
[0081] Import the blurred frame into the trained generative adversarial network model to repair the blurred frame;
[0082] The repaired pavement crack images are detected at the pixel level through the image segmentation model. The extension direction, bifurcation direction and edge shape of the pavement cracks are identified to evaluate and repair the danger level of the pavement cracks.
[0083] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.
[0084] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0085] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A real-time identification and segmentation method for road cracks during movement, characterized in that: include: Conduct on-the-go data collection of pavement cracks; The on-the-go data collection specifically includes: equipping the vehicle-mounted platform with an imaging device to collect non-artificial road crack information during movement; A three-level deep learning model for real-time identification of road cracks during driving is established, including: First-level lightweight classification model: extracts blur frames from the collected data; the first-level lightweight classification model is a lightweight classification module built by convolutional neural network and transformer network, which automatically classifies the input crack image as blurry or not blurry, and uses lightweight network to speed up; The second-level generative deblurring model: The blurred frames separated by the first-level lightweight classification model are imported into the trained generative adversarial network model to repair the blurred frames; the second-level generative deblurring model is constructed by a generative adversarial network, based on conditional and cyclic generative adversarial networks, and the generator and discriminator are trained at the same time. The function of the generator is to generate deblurred images, and the function of the discriminator is to distinguish between generated images and real images. This level of model is used to repair blurred frame images into non-blurred images; The third-level high-precision segmentation model: Segment the road crack image repaired by the second-level generative deblurring model into a binary image containing only crack information, identify the extension direction, bifurcation direction and edge shape of the road crack, perform feature extraction, and also use a lightweight network to achieve pixel-level size detection of road cracks; the third-level high-precision segmentation model uses an image segmentation network trained with transformer and lightweight backbone to segment the road crack image, and segment the input road image into a binary image containing only cracks; In the generative adversarial network model, the discriminator judges yes or no based on the loss function, specifically: Where z is a sample of the input blurred image, p z (z) is the spatial distribution of the blurred image, x is the clear crack image, p data (x) is the spatial distribution corresponding to x; Pixel-level size detection of road cracks, including: At a fixed focal length, the real geometric size corresponding to each pixel in the crack binary image is recorded, realizing the mapping of the crack pixel-level geometric size to the real millimeter-level geometric size; The crack size measurement is performed on the final output crack binary image as above to evaluate and repair the degree of danger.
2. The method for real-time identification and segmentation of road cracks during movement according to claim 1, characterized in that: After accurately locating the cracks and extracting their features, the size of the pavement cracks is identified by proportional calculation.
3. The method for real-time identification and segmentation of road cracks during movement according to claim 1, characterized in that: include: The identified pavement crack information map is spliced by frames to achieve the simultaneous display of the collected real-time pavement conditions and the segmented binary image in the same view.
4. A real-time road crack identification and segmentation system, characterized by: include: The data collection module is configured to: collect on-the-go data of road cracks; The on-the-go data collection specifically includes: equipping the vehicle-mounted platform with an imaging device to collect non-artificial road crack information during movement; The three-level deep learning model building module is configured to build a three-level deep learning model for real-time recognition of road cracks during driving, including: First-level lightweight classification model: extracts blur frames from the collected data; the first-level lightweight classification model is a lightweight classification module built by convolutional neural network and transformer network, which automatically classifies the input crack image as blurry or not blurry, and uses lightweight network to speed up; The second-level generative deblurring model: The blurred frames separated by the first-level lightweight classification model are imported into the trained generative adversarial network model to repair the blurred frames; the second-level generative deblurring model is constructed by a generative adversarial network, based on conditional and cyclic generative adversarial networks, and the generator and discriminator are trained at the same time. The function of the generator is to generate deblurred images, and the function of the discriminator is to distinguish between generated images and real images. This level of model is used to repair blurred frame images into non-blurred images; The third-level high-precision segmentation model: Segment the road crack image repaired by the second-level generative deblurring model into a binary image containing only crack information, identify the extension direction, bifurcation direction and edge shape of the road crack, perform feature extraction, and also use a lightweight network to achieve pixel-level size detection of road cracks; the third-level high-precision segmentation model uses an image segmentation network trained with transformer and lightweight backbone to segment the road crack image, and segment the input road image into a binary image containing only cracks; In the generative adversarial network model, the discriminator judges yes or no based on the loss function, specifically: Where z is a sample of the input blurred image, p z (z) is the spatial distribution of the blurred image, x is the clear crack image, p data (x) is the spatial distribution corresponding to x; Pixel-level size detection of road cracks, including: At a fixed focal length, the real geometric size corresponding to each pixel in the crack binary image is recorded, realizing the mapping of the crack pixel-level geometric size to the real millimeter-level geometric size; The crack size measurement is performed on the final output crack binary image as above to evaluate and repair the degree of danger.
5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method described in any one of claims 1 to 3 are implemented.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 3 are performed.
Citation Information
Patent Citations
Pavement crack rapid extraction method based on two-step convolutional neural network
CN110415233A
Road surface fault detection method based on convolutional neural network
CN111583229A
Method and device for removing image motion blur based on GAN and medium
CN116523780A
Defect detection in image space
US20230014823A1
Robot fleet management and additive manufacturing for value chain networks
WO2022133330A1
