Pavement joint recognition method, system and equipment based on key point estimation and medium

Abstract

The invention belongs to the technical field of image processing, and provides a pavement joint recognition method, system, equipment and medium based on key point estimation in order to solve the problem that straight line categories cannot be distinguished in existing pavement joint recognition. A center point diagram and a displacement diagram are obtained by detecting an output head through a double-branch structure, and the positions of the middle point and the end point of the pavement joint are determined accordingly; obtaining pavement joint category information through a classification output head; according to the scheme, while the center point and the end point of the pavement joint are accurately detected, the detected target is endowed with accurate category information, and the recognition accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to road surface recognition, and in particular relates to a road surface joint recognition method, system, equipment and medium based on key point estimation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Automated pavement disease detection is a key technical means to improve the efficiency of pavement technical condition assessment and maintenance decision-making. Although cement pavements account for more than 60% of the current highway system, there are few dedicated algorithms for pavement disease identification for cement pavements. Developing a dedicated algorithm for cement pavement disease identification can effectively alleviate the heavy burden of pavement disease detection and technical condition assessment currently faced by cement pavement highways.

[0004] Cement pavement defects are generally divided into 11 categories, among which the identification of 7 types of defects, such as broken plates, cracks, plate corner fractures, misalignment, arching, edge and corner peeling, and joint material damage, are closely related to road joints. In addition, cement pavement joints are an effective tool for dividing cement pavement slabs. The design structure of cement pavement determines that a single cement slab has relatively independent mechanical properties. By combining the damage of a single slab with its mechanical structure, the performance status of the single slab can be accurately evaluated and corresponding maintenance measures can be formulated. Therefore, the development of a fast and accurate cement pavement joint recognition algorithm is an important part of realizing intelligent road disease detection equipment.

[0005] At present, there are relatively few algorithms for identifying cement pavement joints, and most of them adopt semantic segmentation pixel-by-pixel technology. He et al. introduced a pixel-level detection method based on deep learning, and used networks such as U-Net to achieve high-precision detection of asphalt pavement joints. Nguyen, Avila, and Begot proposed a comprehensive method combining anisotropic measurement and neural network to detect and classify various pavement defects including cracks, joints, and bridge expansion joints. Hsieh et al. used 3D imaging technology and convolutional neural network to enhance the accuracy and robustness of cement pavement joint detection. AAZhang et al. proposed an automated system method using full-width 3D pavement data and fully convolutional network (FCN) for efficient measurement of pavement displacement of transverse joints, which significantly improved the accuracy and efficiency compared with existing methods. These methods can better segment pavement joints in a single image, but due to the segmentation technology, they have a long image processing time, and in some special cases, a single joint will be identified as multiple discontinuous small segments and it is difficult to clearly identify the end points of the joint. Line detection algorithms based on deep learning have been developed for many years and have achieved high detection accuracy in wireframe parsing tasks. Among them, the bottom-up line detection algorithm generates or restores the line by finding feature points on the line, and has the significant advantage of clear endpoints without post-processing, which is an effective technical means to solve this problem. L-CNN first proposed a bottom-up generative method. On this basis, Huang et al. proposed a tri-points based line segment detector (TP-LSD), which uses the root point as the unique identifier of the line and predicts the position of the endpoint through the displacement map on the root point, thereby achieving high detection speed and accuracy. Gu et al. further simplified the parameter scale of the backbone network of TP-LSD, and proposed a training mode of line segment enhancement, matching and geometric loss optimization of TP-LSD, and finally proposed a mobile line segment detector (M-LSD), which significantly improved the detection speed of the model while maintaining the detection accuracy of the model. Although the above algorithms have achieved high accuracy in line detection tasks, they still lack practical applications in the field of cement pavement joint detection, and the above algorithms cannot distinguish the categories of target lines to be detected.

[0006] In summary, the existing cement pavement joint recognition algorithm has at least the following shortcomings: The segmentation technique has a long image processing time, and in some special cases, a single seam will be identified as multiple discontinuous segments and it is difficult to clearly identify the endpoints of the seam; the line detection algorithm based on deep learning cannot distinguish the category of the target line to be detected. In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a pavement joint recognition method, system, device and medium based on key point estimation, which can accurately detect the center points and endpoints of pavement joints and give accurate category information to the detected targets, thereby improving the accuracy and efficiency of recognition.

[0007] To achieve the above object, a first aspect of the present invention provides a road surface joint recognition method based on key point estimation, comprising: Acquire a road surface image to be detected; The acquired road surface image to be detected is used to obtain the detection result using the trained road surface gap detection model; In the road surface gap detection model, specifically: Extracting the global features and local features of the acquired road surface image to be detected and fusing them to obtain fused features; Based on the fusion features, a center point map and a displacement map are obtained by using a detection output head; and the positions of the midpoint and the end point of the pavement joint are determined according to the center point map and the displacement map; Based on the fusion features, using a classification output head to obtain pavement joint category information; Based on the determined positions of the midpoint and end points of the pavement joint and the pavement joint category information, a detection result is obtained.

[0008] A second aspect of the present invention provides a road surface joint recognition system based on key point estimation, comprising: An acquisition unit, used for acquiring an image of a road surface to be detected; A detection unit, used to obtain a detection result by using a trained road surface gap detection model for the acquired road surface image to be detected; In the road surface gap detection model, specifically: Extracting the global features and local features of the acquired road surface image to be detected and fusing them to obtain fused features; Based on the fusion features, a center point map and a displacement map are obtained by using a detection output head; and the positions of the midpoint and the end point of the pavement joint are determined according to the center point map and the displacement map; Based on the fusion features, using a classification output head to obtain pavement joint category information; Based on the determined positions of the midpoint and end points of the pavement joint and the pavement joint category information, a detection result is obtained.

[0009] The third aspect of the present invention provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, a pavement joint identification method based on key point estimation is executed.

[0010] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, executes a road surface joint recognition method based on key point estimation.

[0011] One or more of the above technical solutions have the following beneficial effects: In the present invention, the global features and local features of the road surface image to be detected are extracted and fused to obtain fused features. Using a dual-branch structure, a center point map and a displacement map are obtained through a detection output head, and the positions of the midpoints and endpoints of the road surface joints are determined; the road surface joint category information is obtained through a classification output head; the scheme of the present invention not only accurately detects the center points and endpoints of the road surface joints, but also gives accurate category information to the detected targets, thereby improving the accuracy and efficiency of recognition.

[0012] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0014] Figure 1 This is a schematic diagram of a double-branch structure in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a feature fusion strategy in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of redundant line segments in Embodiment 1 of the present invention; Figure 4 These are various types of cement pavement joints in Example 1 of the present invention. DETAILED DESCRIPTION

[0015] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0016] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0017] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0018] Embodiment 1 This embodiment discloses a road surface joint recognition method based on key point estimation, including: Acquire a road surface image to be detected; The acquired road surface image to be detected is used to obtain the detection result using the trained road surface gap detection model; In the road surface gap detection model, specifically: Extracting the global features and local features of the acquired road surface image to be detected and fusing them to obtain fused features; Based on the fusion features, a center point map and a displacement map are obtained by using a detection output head; and the positions of the midpoint and the end point of the pavement joint are determined according to the center point map and the displacement map; Based on the fusion features, using a classification output head to obtain pavement joint category information; Based on the determined positions of the midpoint and end points of the pavement joint and the pavement joint category information, a detection result is obtained.

[0019] This embodiment refers to the design concept of CenterNet, uses key point estimation to find the center point of the line segment, and determines the type of the joint based on the characteristics of the center point and regresses the relative position relationship between the end point of the line segment and the center point. This method not only improves the accuracy and efficiency of recognition, but also provides a technical solution for cement pavement joint classification.

[0020] Combine the following Figure 1-Figure 2 A road surface joint recognition method based on key point estimation proposed in this embodiment is described in detail, specifically including: Step 1: Transformer-based global information feature fusion strategy.

[0021] Before step 1, the method also includes using a backbone network to extract a feature map from the road surface image to be detected. The backbone network can select any feature extraction network as needed.

[0022] like Figure 2 As shown in the figure, the seam recognition scheme based on key point estimation requires a single pixel to have a relatively wide receptive field, but in the fully convolutional network architecture, the receptive field of a single pixel is ultimately limited. Using transformers to capture the global information in the feature map and fuse it into a single pixel improves the detection and classification performance of the model.

[0023] The specific design plan is as follows: S1-1: Transformer-based global information capture.

[0024] A single-layer transformer is used to capture the global information in the feature map. The position embedding and normalization strategies are retained in the design, and a multi-layer perceptron structure with an activation function is used at the end of the transformer.

[0025] S1-2: Local information capture based on Conv3*3.

[0026] A convolutional neural network is used to capture local information in the feature map. The specific modules include 3*3 convolution, ReLU activation function and batch normalization strategy connected in sequence. The purpose of this part is to retain enough local information to describe the target morphological characteristics during feature fusion.

[0027] S1-3: Feature information fusion strategy considering residual connection.

[0028] Referring to the design scheme of residual connection, the feature map, the global features captured by transformer, and the local features captured by Conv3*3 are spliced, and the three sets of features are fused using a separate Conv3*3 module to obtain the fused features.

[0029] Step 2: Consider the output head structure design for pavement joint detection and classification.

[0030] like Figure 1 As shown in the figure, a dual-branch detection scheme with parallel detection and classification is designed. The detection head detects the center point of the seam and regresses the relative distance between the endpoint and the center point; the classification head is responsible for assigning category information to the detected target.

[0031] The specific design plan is as follows: S2-1: Output head structure design for pavement joint detection. The fusion feature is input to the detection output head, which is processed by transformer, first convolution, pooling, batch normalization, and outputs the detection results of pavement joints, including center point map, displacement map, node map, and straight line map. The center point map and displacement map are used to determine the positions of the midpoints and end points of pavement joints, and the node map and straight line map are used to assist in improving the detection accuracy of the model.

[0032] According to the characteristic information of the center point, the relative position relationship is directly regressed using L1 loss to determine the positions of the midpoint and end points of the pavement joint.

[0033] During the training of the detection output head, the positive sample or predicted target refers to the center point in the fusion feature map. The binary cross entropy loss function is also calculated directly based on the center point. The center point is 1 and the background point is 0. After the last layer output is filtered by sigmoid and local maximum, and the appropriate confidence threshold is selected, only the target center point will remain. There may be a little redundancy in several targets and several center points.

[0034] The node graph and line graph use a multi-task learning strategy to assist in improving the detection accuracy of the model. Specifically, node and line labels are provided in the training phase, but the corresponding content is removed in the inference phase, and no inference is performed. They are only used in the training phase.

[0035] S2-2: Output head structure design for pavement joint classification. The fusion feature is input to the classification output head, and is processed by transformer, second convolution, pooling, and batch normalization in sequence to obtain the category information of the pavement joint, including the classification center point map and classification segmentation map. The classification center point map provides the category information of the pavement joint, and the classification segmentation map is used to assist in improving the classification accuracy of the model.

[0036] The implementation logic of the classification segmentation graph is similar to that of the node graph. It is implemented through multi-task learning. Specifically, there is a corresponding loss function in the training process, but it is not used in the inference stage.

[0037] The classification output head obtains a multi-channel center point map with several categories and channels. Each category only appears in the corresponding channel to achieve the classification goal. Compared with the single-channel center point map, each channel scores each pixel. According to the position of the center point, there will be n scores on the multi-channel center point map, where n is the number of categories. The channel with the highest score corresponds to the category of the line segment. There will be many redundant calculation results, but this step-by-step classification and detection is more stable.

[0038] The relative distance is estimated using the regression method, and the loss function is represented by L1 loss. The label is the relative distance between the endpoint and the center point. The coordinates of the endpoint and the center point are: (x, y), (x2, y2). When the position of the center point is known, the position of the endpoint can be calculated. Similarly, if the calculation is proper, the position of the endpoint can be calculated very clearly.

[0039] Among them, the convolutional layers of the first convolution in the detection output head and the second convolution in the classification output head are different.

[0040] S2-3: Combined scheme of pavement joint detection and classification information in the inference stage First, the pavement joint detection result is generated according to the output result of the detection output head, and on this basis, the pavement joint category is distinguished according to the category information provided by the classification output head.

[0041] Step 3: Non-maximum suppression strategy based on endpoint distance and line segment overlap.

[0042] The redundant center points generated in the center point map are used to filter out redundant detection results. Each center point generates a straight line. Direct prediction of the center points will inevitably lead to redundant center points in the detection stage. After the line segments are generated, some post-processing methods are used to merge the line segments that are close in spatial position.

[0043] like Figure 3 As shown in the figure, after the detection is completed, there will be multiple detection results for the same seam. The main reason is that the model will regard the 1 / 4 position of the seam or somewhere near the midpoint as the midpoint of a separate seam, and use this as the center point to construct an independent detection result. Therefore, a certain non-maximum suppression strategy is needed to filter out redundant detection results.

[0044] The specific design plan is as follows: S3-1: Line segment filtering strategy based on endpoint distance.

[0045] The segment filtering strategy based on endpoint distance calculates the square of the difference between the coordinates of the endpoints of two segments, adds these square differences to get the total sum of square differences, and selects the minimum value of these square differences as the distance metric between the segments. When the distance metric is less than the first set threshold, only the segments with high confidence are retained. The specific calculation formula is as follows:

[0046] Where: represents the distance measure between two line segments, Indicates the first line segment endpoint of and coordinate, Represents the second line segment endpoint of and coordinate.

[0047] S3-2: Line segment filtering strategy based on line segment overlap.

[0048] The line segment filtering strategy first represents the predicted line segments and the real line segments as heat maps, where the line segment area is set to 1 and the other areas are set to 0. Then, by calculating the intersection of the two heat maps and the sum of each heat map, the intersection over union (IoU) is obtained as the similarity measure of the line segments. The specific steps are as follows: If the intersection over union is greater than the second set threshold, the line segment with lower confidence is discarded.

[0049] S3-3: Non-maximum suppression strategy. For all detection results in an image, arrange them from large to small according to the confidence level, and judge whether the two line segments meet the endpoint threshold and line segment overlap judgment rules. If one of them is met, the corresponding line segment filtering strategy is activated to retain the line segments with higher confidence levels.

[0050] This application scheme is applied to an actual data set, the data set structure is as follows Figure 4 As shown in Figure 2. This dataset contains 1,000 cement pavement joint foreground images, collected by the CiCS IC model multifunctional road condition detection vehicle. The image resolution is 3,000 × 2,000 pixels, covering four types of cement pavement joints: road boundary lines, bridge expansion joints, longitudinal joints, and transverse joints. LabelMe is used to annotate the images, and each joint is annotated as a straight line with two endpoints.

[0051] The experiments were conducted on a system equipped with four NVIDIA GeForce RTX 3090 graphics cards and an i7-10700 CPU, running the Ubuntu 18.04 operating system with CUDA version 11.4. The dataset was split into a training set (800 images) and a test set (200 images) in an 8:2 ratio. All input images in the training and testing phases were resized to 512×512 pixels. Standard data augmentation techniques were used during training, including random horizontal and vertical flips, rotations, and shears. No pre-trained weights were used. The training process lasted 155 epochs, using the Adam optimizer with an initial learning rate of 0.003 and a weight decay of 0.000001. A step-down learning rate decay strategy was used, with linear warm-up for the first 100 steps. The step milestones were set to the 50th, 100th, and 150th epochs, with a step decay factor of 0.2.

[0052] Figure 4 The actual effect of the model on the cement pavement joint detection task is demonstrated. The results show that the model can effectively realize the accurate detection of cement pavement joints, achieve the expected design effect, and meet the actual needs of engineering applications.

[0053] Embodiment 2 The purpose of this embodiment is to provide a road surface joint recognition system based on key point estimation, including: An acquisition unit, used for acquiring an image of a road surface to be detected; A detection unit, used to obtain a detection result by using a trained road surface gap detection model for the acquired road surface image to be detected; In the road surface gap detection model, specifically: Extracting the global features and local features of the acquired road surface image to be detected and fusing them to obtain fused features; Based on the fusion features, a center point map and a displacement map are obtained by using a detection output head; and the positions of the midpoint and the end point of the pavement joint are determined according to the center point map and the displacement map; Based on the fusion features, using a classification output head to obtain pavement joint category information; Based on the determined positions of the midpoint and end points of the pavement joint and the pavement joint category information, a detection result is obtained.

[0054] Embodiment 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0055] Embodiment 4 The purpose of this embodiment is to provide a computer-readable storage medium.

[0056] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0057] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0058] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0059] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A road joint recognition method based on key point estimation, characterized in that: include: Acquire a road surface image to be detected; The acquired road surface image to be detected is used to obtain the detection result using the trained road surface gap detection model; In the road surface gap detection model, specifically: Extracting the global features and local features of the acquired road surface image to be detected and fusing them to obtain fused features; Based on the fusion features, a center point map and a displacement map are obtained by using a detection output head; and the positions of the midpoint and the end point of the pavement joint are determined according to the center point map and the displacement map; Based on the fusion features, using a classification output head to obtain pavement joint category information; Based on the determined positions of the midpoint and end points of the pavement joint and the pavement joint category information, a detection result is obtained.

2. A road surface joint recognition method based on key point estimation as claimed in claim 1, characterized in that: The global features and local features of the acquired road surface image to be detected are extracted and fused to obtain fused features, specifically: Extracting a feature map of the acquired road surface image to be detected; Extracting global features of the feature map using transformer; Extracting local features of the feature map using a first convolution module; The feature map, the global feature and the local feature are fused and processed by a second convolution module to obtain a fused feature.

3. A road surface joint identification method based on key point estimation as claimed in claim 1, characterized in that: The fused features are sequentially processed by transformer, first convolution, pooling, and batch normalization to obtain a center point map and a displacement map; the fused features are sequentially processed by transformer, second convolution, pooling, and batch normalization to obtain category information of road surface joints; wherein the convolution layers of the first convolution and the second convolution are different.

4. A road surface joint identification method based on key point estimation as claimed in claim 1 or 3, characterized in that: It also includes: using a non-maximum suppression strategy to filter out redundant center points generated by the center point map, specifically: Determine whether two line segments meet the endpoint distance threshold and line segment overlap determination; If one of the conditions is met, the confidence is calculated based on the endpoint distance threshold or the line segment coincidence determination, the calculated confidences are arranged from large to small, and the line segments are filtered out based on the arrangement results.

5. A road joint recognition method based on key point estimation as claimed in claim 4, characterized in that: The confidence is calculated based on the endpoint distance threshold, specifically: Calculate the square of the difference between the endpoint coordinates corresponding to the two line segments, add the calculated square differences to obtain the total sum of square differences, select the minimum value of the calculated sum of square differences as the distance metric between the line segments, and when the distance metric is less than the set threshold, it meets the endpoint distance threshold judgment.

6. A road joint identification method based on key point estimation as claimed in claim 4, characterized in that: The confidence is calculated based on the line segment coincidence judgment, specifically: the detected line segment and the real line segment are represented as heat maps, the intersection of the two heat maps is calculated to obtain the intersection-union ratio as the similarity measure of the line segments, and if the similarity measure is greater than the second set threshold, the line segment coincidence judgment is met.

7. A road joint identification method based on key point estimation as claimed in claim 1, characterized in that: The backbone network is used to extract feature maps from the acquired road surface image to be detected, and global features and local features are extracted from the extracted feature maps and fused to obtain fused features.

8. A road joint recognition system based on key point estimation, characterized in that: include: An acquisition unit, used for acquiring an image of a road surface to be detected; A detection unit, used to obtain a detection result by using a trained road surface gap detection model for the acquired road surface image to be detected; In the road surface gap detection model, specifically: Extracting the global features and local features of the acquired road surface image to be detected and fusing them to obtain fused features; Based on the fusion features, a center point map and a displacement map are obtained by using a detection output head; and the positions of the midpoint and the end point of the pavement joint are determined according to the center point map and the displacement map; Based on the fusion features, using a classification output head to obtain pavement joint category information; Based on the determined positions of the midpoint and end points of the pavement joint and the pavement joint category information, a detection result is obtained.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, a pavement joint recognition method based on key point estimation as described in any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying pavement joints based on key point estimation according to any one of claims 1 to 7 is executed.

Images