Cement pavement disease identification method, medium and system

Abstract

The present application discloses a cement pavement disease identification method, a medium and a system. The method comprises: separately inputting collected cement pavement images into a disease identification module, and outputting a first image marked with a crack area and / or a second image marked with a joint area; inputting the first image into a refinement processing module, such that the refinement processing module extracts single cracks from the crack area in the first image and then outputs a third image marked with the extracted cracks; inputting the second image into the refinement processing module, such that the refinement processing module divides a constructed mask image on the basis of the joint area in the second image and then outputs a fourth image having non-overlapping cement plate areas; and inputting the third image and the fourth image into the disease identification module, so as to output the type of diseases of each cement plate area on the basis of the number of the cracks in the third image in each cement plate area in the fourth image and the positional relation between each cement plate area and the cracks. The present application improves the disease identification accuracy.

Description

Cement pavement disease identification method, medium and system

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on September 8, 2023, with application number 202311159574.9, entitled “A Method, Medium and System for Identifying Cement Pavement Defects,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the technical field of cement pavement disease identification, and in particular to a cement pavement disease identification method, medium and system. Background Art

[0004] According to the "Highway Technical Condition Assessment Standard" (JTG5210-2018), as shown in Figure 1, common cement pavement disease types in pavement images include cracks, edge peeling, plate corner fractures, and broken plates: (1) Cracks are when there is only one crack on the cement plate; (2) Plate corner fractures are when the crack intersects with the longitudinal and transverse joints, and the intersection is less than or equal to half the length of the plate edge; (3) Broken plates are when the cement plate is divided into three or more pieces by cracks. The damage rate of cracks and edge peeling is calculated based on the actual detection length, while the damage rate of plate corner fractures and broken plates is calculated based on the actual detection area.

[0005] Existing deep learning models have low accuracy when identifying these defects in images. The main reason is that all defects are actually caused by crack development and are morphologically similar to cracks. This makes it difficult to clearly define the boundaries between defect types during the manual annotation stage. For example, it is difficult to accurately determine in an image that the intersection of a crack and a plate edge is less than half the length of the plate edge. Furthermore, during the model training phase, if identification is performed directly based on semantics (i.e., defect type), cracks and non-crack defects can be easily confused. For example, three or more cracks that constitute a broken plate are morphologically identical to cracks, causing the trained model to overfit to the crack class and fail to distinguish broken plates.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a cement pavement disease identification method, medium and system to solve the problem that the existing technology cannot accurately identify cement pavement diseases.

[0008] In a first aspect, a method for identifying cement pavement defects is provided, comprising:

[0009] Inputting the collected cement pavement images into the disease recognition module respectively, so that the disease recognition module recognizes crack areas and / or joint areas in the cement pavement images and outputs a first image marked with crack areas and / or a second image marked with joint areas;

[0010] Inputting the first image into a refinement processing module, causing the refinement processing module to extract a single crack from the crack region in the first image and output a third image labeled with the extracted crack;

[0011] Inputting the second image into a refinement processing module, causing the refinement processing module to divide the constructed mask image based on the seam areas in the second image, and outputting a fourth image having non-overlapping cement slab areas;

[0012] The third image and the fourth image are input into a defect recognition module, so that the defect recognition module outputs the type of defect in each cement slab area based on the number of cracks in the third image in each cement slab area in the fourth image and the positional relationship between the cement slab area and the cracks.

[0013] In a second aspect, a computer-readable storage medium is provided, on which computer program instructions are stored; when the computer program instructions are executed by a processor, a cement pavement disease identification method as in the embodiment of the first aspect is implemented.

[0014] In a third aspect, a cement pavement disease identification system is provided, comprising: a computer-readable storage medium as in the embodiment of the second aspect.

[0015] In this way, the embodiment of the present application fully utilizes the deep learning model's ability to identify cracks. In the first stage, all crack areas in the image are extracted only through the relevant model, eliminating the semantic ambiguity between cracks and non-crack diseases; independent cement slabs are divided by extracting the joint areas in the image; the specific disease type is determined by analyzing the number of cracks in each cement slab and the relative spatial relationship between the crack morphology and the slab edge; and the accuracy of the judgment results is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 is a schematic diagram of common cement pavement diseases, including (a) cracks, (b) broken plate corners, represented by white solid lines, and (c) broken plates, represented by white solid lines;

[0018] FIG2 is a flow chart of a cement pavement disease identification method according to an embodiment of the present application;

[0019] FIG3 is a schematic diagram of identifying whether a plate corner fracture occurs according to an embodiment of the present application, wherein (a) an image edge is obtained, (b) a joint is obtained and an independent cement slab region is constructed with the image edge, and (c) the first intersection point is extracted to identify whether a plate corner fracture occurs.

[0020] FIG4 is a schematic diagram of identifying whether a plate is broken or not according to an embodiment of the present application, wherein (a) image edges are obtained, (b) seams are obtained and used to construct independent cement slab regions with the image edges, and (c) connected domains are extracted to identify whether a plate is broken or not;

[0021] FIG5 is a schematic structural diagram of a disease identification unit according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The present invention discloses a method for identifying cement pavement defects. As shown in FIG1 , the method includes the following steps:

[0024] Step S101: input the collected cement pavement images into the defect recognition module respectively, so that the defect recognition module recognizes the crack areas and / or joint areas in the cement pavement images and outputs a first image marked with the crack areas and / or a second image marked with the joint areas.

[0025] The defect recognition module in this embodiment of the application uses a classic encoder-decoder image segmentation model to implement two tasks: crack identification and joint identification. Specifically, the defect recognition module includes two defect recognition units. One defect recognition unit uses a convolutional neural network-based semantic segmentation model 1 to identify all crack areas in the cement pavement and outputs a first image. The other defect recognition unit uses another convolutional neural network-based semantic segmentation model 2 to identify joint areas in the cement pavement and outputs a second image.

[0026] Specifically, as shown in Figure 5, the disease recognition unit adopts the improved DeepLabv3+ semantic segmentation model, which is composed of a ResNet101 encoder, a pyramid feature pooling (ASPP) module, and a multi-scale feature fusion decoder connected in sequence.

[0027] For the ResNet101 encoder, it includes an input convolution layer and multiple sequentially connected convolution groups (usually four groups). The input convolution layer is generally implemented by a larger convolution kernel, such as a 7x7 convolution kernel. Each convolution group consists of a different number of residual modules. The number of repetitions of the residual modules of each convolution group of ResNet101 is 3-4-23-3 respectively. A convolution kernel with a stride of 2 is used between each convolution group to achieve downsampling of the feature map. Each residual module uses three cascaded convolution operations of 1x1, 3x3, and 1x1, and introduces the input to the output through a residual connection. If the input and output have different dimensions and feature map sizes, the residual connection is implemented using a 1x1 convolution kernel.

[0028] The ASPP module uses multiple parallel convolution operations with different dilation rates to extract features of different scales on the feature map of the last layer of the encoder. The purpose is to fully aggregate the rich contextual semantic information in the image and improve the segmentation accuracy of the target object at the current position.

[0029] The multi-scale feature fusion decoder is implemented using staged feature concatenation and fusion.

[0030] The classic DeepLabv3+ model decoder fuses features from the final layer and intermediate layers through feature concatenation, balancing intermediate and high-level image features and improving the ability to model complex objects. To further enhance the ability to characterize the subtle morphology of cracks and better capture the subtle and complex morphological features of pavement defects such as cracks, this application simultaneously fuses the low-level features of the input layer with the intermediate and high-level semantic features in the DeepLabv3+ decoder.

[0031] Specifically, in the first step, the ASPP module is used to extract rich contextual semantic features from the feature map output by the last convolution group of the ResNet101 encoder, and its feature channels are adjusted to a specific dimension, such as 48 dimensions, using a 1x1 convolution kernel, and then it is upsampled eight times to obtain the first feature map. In the second step, the channel of the feature map output by the first convolution group of the encoder is adjusted to 48 dimensions using a 1x1 convolution kernel to obtain the second feature map. In the third step, the first and second feature maps are spliced ​​along the channel direction, and then the number of channels is adjusted from 96 dimensions (48+48) to 48 dimensions using a 3x3 convolution kernel, and it is upsampled twice to restore it to half the size of the original cement pavement image to obtain the third feature map. In the fourth step, a 1x1 convolution kernel is used to adjust the channels of the feature map output by the encoder input layer to 48 dimensions, which is then concatenated with the third feature map. Then, a 3x3 convolution kernel is used to adjust the number of channels to 256 dimensions, and the image is upsampled twice to restore it to the original cement pavement image size. The fourth feature map (i.e., the first and second images) is obtained as the final feature map used to detect the target object.

[0032] In addition, to drive model training, the present embodiment collected two cement pavement image datasets. Dataset 1 contains approximately 2,000 pavement images and pixel-level annotations of all crack areas, and Dataset 2 contains approximately 2,000 pavement images and pixel-level annotations of all joint areas. These training images can be presented in the format of image masks, that is, the background is labeled as 0 and the crack or joint area is labeled as 1. The disease recognition unit is trained with these training images so that it can more accurately identify the corresponding targets.

[0033] Step S102: inputting the first image into the refinement processing module, causing the refinement processing module to extract a single crack from the crack region in the first image, and outputting a third image annotated with the extracted crack.

[0034] Specifically, this step includes the following process:

[0035] The first image is input into the refinement processing module, and a single crack with a width of one pixel is extracted from the crack area in the first image using the skeletonize function in the morphology module of the skimage library. Then, different single cracks are labeled using the label function in the measure module of the skimage library, and the third image is output.

[0036] The first image should be in the form of a binary image mask. The image mask can be refined using the skeletonize function, which uses a skeletonization algorithm. The label function uses a connected component algorithm with a connectivity parameter set to 2, extracting cracks or seams through morphological operations.

[0037] Step S103: inputting the second image into the refinement processing module, causing the refinement processing module to divide the constructed mask image based on the seam areas in the second image, and output a fourth image having non-overlapping cement slab areas.

[0038] Specifically, this step includes the following process:

[0039] The second image is input into the refinement processing module. A single seam with a width of one pixel is extracted from the seam area in the second image using the skeletonize function in the morphology module of the skimage library. The seam is then fitted using the hough_line function in the transform module of the skimage library to obtain seam information. Finally, a mask image with all zeros is constructed. The line function of the OpenCV library is used to fill the seams at the corresponding positions in the mask image according to the seam information. The mask image is then divided into non-overlapping cement slab areas based on the edges of the seams and the mask image, and the fourth image is output.

[0040] The second image should be in the form of a binarized image mask. The image mask can be refined using the skeletonize function, which uses a skeletonization algorithm. The hough_line function belongs to the Hough line detection model. Its purpose is to fit the straight line equation of the seam. During the fitting, the shortest line length is set to half the image height to avoid detecting seams that are too small, and the maximum line spacing is set to 100 to avoid duplicate lines. When processed by the hough_line function, the seam area is filled with 1. The fourth image is the mask image, with the value corresponding to the cement slab area being 1 and the value of 0 elsewhere.

[0041] The above steps S102 and S103 can be performed simultaneously.

[0042] Step S104: Input the third image and the fourth image into the defect recognition module, so that the defect recognition module outputs the type of defect of each cement slab area based on the number of cracks in each cement slab area in the fourth image and the positional relationship between the cement slab area and the cracks in the third image.

[0043] This step sets rules according to standard specifications, determines the specific type of disease based on the number of cracks and their relative spatial relationship with the joints, and finally outputs the corresponding test results.

[0044] Specifically, this step includes the following process:

[0045] 1. Copy the cracks in the third image to the corresponding positions in the fourth image.

[0046] Specifically, the crack position can be determined based on the crack information in the third image and copied. Through this step, the corresponding crack is filled into the corresponding cement slab area.

[0047] Second, determine the number of cracks in the cement slab area in each fourth image.

[0048] Specifically, this can be achieved through existing image recognition technology. For example, the number of cracks with identification information not equal to 0 can be calculated by obtaining the intersection of the mask area of ​​the fourth image and the cracks with identification information in the third image (the result after using the label function).

[0049] It should be understood that the fourth image is a mask image, meaning that the value corresponding to the cement slab area is 1, and the value elsewhere is 0. Taking the last image in Figure 3 as an example, the mask image for slab 1 is filled entirely with 1s in the area corresponding to slab 1, and all other areas are filled with 0s. This mask image makes it easy to identify all cracks in the corresponding area while ignoring cracks in other areas.

[0050] 3. Identify the type of damage in the cement slab area based on the number of cracks in the cement slab area and the positional relationship between the cement slab area and the cracks in the fourth image.

[0051] Specifically, this step includes the following situations:

[0052] 1. The first case:

[0053] If the number of cracks in the cement slab area is 0, it is determined that the cement slab area has no disease.

[0054] 2. The second situation (as shown in Figure 3):

[0055] (1) If the number of cracks in the cement plate region is 1, the first intersection point between the crack in the cement plate region and the cement plate region is extracted.

[0056] Extracting the first intersection point can be achieved through existing image recognition technology. For example, the intersection of the second image and the third image can be calculated to obtain the coordinate position of the intersection point of the crack and the seam.

[0057] (2) If the crack in the cement plate area and the first intersection point of the cement plate area meet the preset conditions, the crack in the cement plate area is identified as a plate corner fracture.

[0058] (3) If the crack in the cement plate area and the first intersection point of the cement plate area do not meet the preset conditions, the crack in the cement plate area is identified as a common crack.

[0059] The preset conditions include: the number of first intersections between the cracks in the cement slab area and the cement slab area is 2; the two first intersections are respectively located on two intersecting sides of the cement slab area (generally, the two intersecting sides are perpendicular to each other); and the length from the first intersection to the second intersection of the two sides is less than half the length of the side of the cement slab area where the first intersection is located.

[0060] 3. The third situation (as shown in Figure 4):

[0061] (1) If the number of cracks in the cement plate area is not less than 2, the number of connected domains in the cement plate area is extracted using the label function in the measure module of the skimage library.

[0062] (2) If the number of connected domains in the cement plate area is not less than 3, the type of disease in the cement plate area is identified as broken plate.

[0063] (3) If the number of connected domains in the cement plate region is less than 3, the first intersection point of each crack in the cement plate region and the cement plate region is extracted.

[0064] (4) If the crack in the cement plate area and the first intersection point of the cement plate area meet the preset conditions, the crack in the cement plate area is identified as a plate corner fracture.

[0065] (5) If the crack in the cement plate area and the first intersection point of the cement plate area do not meet the preset conditions, the crack in the cement plate area is identified as a common crack.

[0066] The preset conditions include: the number of first intersections between the cracks in the cement slab area and the cement slab area is 2, the two first intersections are respectively located on two intersecting sides of the cement slab area, and the length from the first intersection to the second intersection of the two sides is less than half the length of the side of the cement slab area where the first intersection is located.

[0067] In addition, an embodiment of the present application further provides a computer-readable storage medium on which computer program instructions are stored; when the computer program instructions are executed by a processor, the cement pavement disease identification method as described in the above embodiment is implemented.

[0068] In addition, an embodiment of the present application also provides a cement pavement disease identification system, including: a computer-readable storage medium as described in the above embodiment.

[0069] In summary, the embodiments of the present application fully utilize the deep learning model's ability to identify cracks. In the first stage, all crack areas in the image are extracted only through the relevant model, eliminating the semantic ambiguity between cracks and non-crack diseases; independent cement slabs are divided by extracting the joint areas in the image; the specific disease type is determined by analyzing the number of cracks in each cement slab and the relative spatial relationship between the crack morphology and the slab edge; and the accuracy of the discrimination results is improved.

[0070] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A cement pavement disease identification method, characterized in that: include: Inputting the collected cement pavement images into the disease recognition module respectively, so that the disease recognition module recognizes the crack area and / or the joint area in the cement pavement image, and then outputs a first image marked with the crack area and / or a second image marked with the joint area; Inputting the first image into a refinement processing module, causing the refinement processing module to extract a single crack from a crack region in the first image and then output a third image labeled with the extracted crack; Inputting the second image into a refinement processing module, so that the refinement processing module divides the constructed mask image based on the joint area in the second image, and outputs a fourth image having non-overlapping cement slab areas; The third image and the fourth image are input into a disease recognition module, so that the disease recognition module outputs the type of disease of each cement plate area according to the number of cracks in the third image in each cement plate area in the fourth image and the positional relationship between the cement plate area and the cracks.

2. The cement pavement disease identification method according to claim 1, characterized in that: The disease recognition module includes: two disease recognition units, one of which is used to output the first image, and the other of which is used to output the second image; Among them, the disease recognition unit is an improved DeepLabv3+ semantic segmentation model, and the DeepLabv3+ semantic segmentation model is composed of a ResNet101 encoder, a pyramid feature pooling ASPP module and a multi-scale feature fusion decoder connected in sequence; wherein, the multi-scale feature fusion decoder is implemented using staged feature splicing and fusion.

3. The cement pavement disease identification method according to claim 2, characterized in that: The step of outputting a third image annotated with the extracted cracks comprises: The first image is input into the refinement processing module, a single crack with a width of one pixel is extracted from the crack area in the first image through the skeletonize function in the morphology module of the skimage library, and then different single cracks are labeled through the label function in the measure module of the skimage library, and the third image is output.

4. The cement pavement disease identification method according to claim 2, characterized in that: The step of outputting a fourth image having non-overlapping cement slabs comprises: The second image is input into the refinement processing module, and a single seam with a width of one pixel is extracted from the seam area in the second image by using the skeletonize function in the morphology module of the skimage library. The seam is then fitted by the hough_line function in the transform module of the skimage library to obtain the seam information. Finally, a mask image with all zeros is constructed, and the seam is filled in the corresponding position of the mask image according to the seam information by using the line function of the opencv library, and the mask image is divided into non-overlapping regions according to the seam and the edge of the mask image. The cement slab area outputs the fourth image.

5. The cement pavement disease identification method according to claim 1, characterized in that: The step of outputting the type of disease of each cement block area includes: Copying the crack in the third image to a corresponding position in the fourth image; determining a number of cracks in a region of the cement slab in each of the fourth images; According to the number of cracks in the cement slab area in the fourth image and the positional relationship between the cement slab area and the cracks, the type of the disease in the cement slab area is identified.

6. The cement pavement disease identification method according to claim 5, characterized in that: The step of identifying the type of disease in the cement slab area includes: If the number of cracks in the cement slab area is 0, it is identified that the cement slab area has no disease.

7. The cement pavement disease identification method according to claim 5, characterized in that: The step of identifying the type of disease in the cement slab area includes: If the number of cracks in the cement block region is 1, extracting the first intersection point between the cracks in the cement block region and the cement block region; If the crack in the cement block area and the first intersection point of the cement block area meet a preset condition, then the crack in the cement block area is identified as a plate corner fracture; If the crack in the cement block area and the first intersection point of the cement block area do not meet a preset condition, identifying the crack in the cement block area as a common crack; Wherein, the preset conditions include: the number of first intersections between the cracks in the cement plate area and the cement plate area is 2, and the two first intersections are respectively located on two intersecting edges of the cement plate area, and the length from the first intersection to the second intersection of the two edges is less than half the length of the edge of the cement plate area where the first intersection is located.

8. The cement pavement disease identification method according to claim 5, characterized in that: The step of identifying the type of disease in the cement slab area includes: If the number of cracks in the cement block area is not less than 2, the number of connected domains in the cement block area is extracted by the label function in the measure module of the skimage library; If the number of connected domains in the cement plate area is not less than 3, then the disease type of the cement plate area is identified as a broken plate; If the number of connected domains in the cement block region is less than 3, extracting the first intersection point between each crack in the cement block region and the cement block region; If the crack in the cement block area and the first intersection point of the cement block area meet the preset conditions, then the identification The cracks in the cement slab area are fractures at the slab corners; If the crack in the cement block area and the first intersection point of the cement block area do not meet a preset condition, identifying the crack in the cement block area as a common crack; Wherein, the preset conditions include: the number of first intersections between the cracks in the cement plate area and the cement plate area is 2, and the two first intersections are respectively located on two intersecting edges of the cement plate area, and the length from the first intersection to the second intersection of the two edges is less than half the length of the edge of the cement plate area where the first intersection is located.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the cement pavement disease identification method according to any one of claims 1 to 8 is implemented.

10. A cement pavement disease identification system, characterized in that: include: The computer readable storage medium of claim 9.

Images