Pavement Joint Identification Method, System, Device and Medium Based on Key Point Estimation

Abstract

The present invention belongs to the technical field of image processing. In order to solve the problems existing in the existing road surface joint recognition, such as the inability to distinguish straight line categories, etc., a road surface joint recognition method, system, device and medium based on key point estimation are proposed. The global features and local features of the road surface image to be detected are extracted and fused to obtain fused features. Using a two-branch structure, a center point map and a displacement map are obtained through a detection output head, and the positions of the midpoints and endpoints of the road surface joints are determined based on this; the road surface joint category information is obtained through a classification output head. The solution of the present invention accurately detects the midpoints and endpoints of the road surface joints, and assigns accurate category information to the detected targets, improving the accuracy and efficiency of recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field related to road surface recognition, and particularly relates to a road surface joint recognition method, system, device and medium based on key point estimation. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] Automated road surface disease detection is a key technical means to improve the efficiency of road surface technical condition assessment and maintenance decision-making. Although the proportion of cement road surfaces in the current highway system reaches more than 60%, there are few dedicated algorithms for road surface disease recognition for cement road surfaces. Developing a dedicated algorithm for cement road surface disease recognition can effectively alleviate the heavy burden of road surface disease detection and technical condition assessment currently faced by cement road surface highways.

[0004] Cement road surface diseases are generally divided into 11 categories. The identification of 7 types of diseases, such as broken slabs, cracks, slab corner fractures, faulting, heaving, edge spalling and joint sealant damage, is closely related to road joints. In addition, cement road surface joints are effective tools for dividing cement road surface slabs. The design structure of cement road surfaces determines that each single cement slab has relatively independent mechanical properties. By combining the damage conditions of single slabs with their mechanical structures, the performance state of single slabs can be accurately evaluated and corresponding maintenance measures can be formulated. Therefore, developing a fast and accurate cement road surface joint recognition algorithm is an important link in realizing intelligent road disease detection equipment.

[0005] Currently, there are relatively few algorithms for identifying cement pavement joints. Basically, semantic segmentation pixel-by-pixel technical solutions are adopted. He et al. introduced a pixel-level detection method based on deep learning and achieved high-precision detection of asphalt pavement joints using networks such as U-Net. Nguyen, Avila, and Begot proposed a comprehensive method combining anisotropic measurement and neural networks for detecting and classifying various pavement defects including cracks, joints, and bridge expansion joints. Hsieh et al. used 3D imaging technology and convolutional neural networks to enhance the accuracy and robustness of cement pavement joint detection. A. A. Zhang et al. proposed an automated system method using full-frame 3D pavement data and a fully convolutional network (FCN) for efficiently measuring the pavement displacement of transverse joints, which significantly improved the accuracy and efficiency compared with existing methods. These methods can segment pavement joints in a single image well. However, due to the adopted segmentation technical solution, they have a long image processing time. And in the face of some special situations, a single joint will be recognized as multiple discontinuous small segments and it is difficult to clarify the endpoints of the joint. The straight line detection algorithm based on deep learning has been developed for many years and has achieved high detection accuracy in the wireframe parsing task. Among them, the bottom-up straight line detection algorithm generates or restores a straight line by finding feature points on the straight line. It has the significant advantage that the endpoints are clear and no post-processing is required, which is an effective technical means to solve this problem. L-CNN first proposed a bottom-up generative method. Huang et al. proposed a tri-points based line segment detector (TP-LSD) on this basis, using the root point as the unique identifier of the straight line and predicting the position of the endpoint through the displacement map on the root point, thus obtaining a high detection speed and accuracy. Gu et al. further simplified the backbone network parameter scale of TP-LSD and proposed a training mode that optimizes TP-LSD with line segment enhancement, matching, and geometric loss, and finally proposed a mobile line segment detector (M-LSD), which significantly improved the detection speed of the model while maintaining the detection accuracy of the model. Although the above algorithms have achieved high accuracy in the straight line detection task, there is still a lack of practical applications in the field of cement pavement joint detection, and the above algorithms cannot distinguish the straight line categories of the targets to be detected.

[0006] In summary, the existing cement pavement joint recognition algorithms have at least the following deficiencies:

[0007] Adopting a segmentation technical solution, it has a long image processing time, and in the face of some special situations, a single seam will be recognized as multiple discontinuous small segments and it is difficult to clarify the endpoints of the seam; based on the deep learning line detection algorithm, it cannot distinguish the line categories of the targets to be detected. Summary of the Invention

[0008] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a road surface joint recognition method, system, device and medium based on key point estimation, which can accurately detect the center points and endpoints of road surface joints, and at the same time assign accurate category information to the detected targets, improving the recognition accuracy and efficiency.

[0009] To achieve the above object, the first aspect of the present invention provides a road surface joint recognition method based on key point estimation, including:

[0010] Obtain the road surface image to be detected;

[0011] Use the obtained road surface image to be detected and the trained road surface gap detection model to obtain the detection result;

[0012] In the road surface gap detection model, specifically:

[0013] Extract the global features and local features of the obtained road surface image to be detected and fuse them to obtain the fused features;

[0014] Based on the fused features, use the detection output head to obtain the center point map and displacement map; determine the positions of the midpoints and endpoints of the road surface joints according to the center point map and displacement map;

[0015] Based on the fused features, use the classification output head to obtain the road surface joint category information;

[0016] Based on the determined positions of the midpoints and endpoints of the road surface joints and the road surface joint category information, obtain the detection result.

[0017] The second aspect of the present invention provides a road surface joint recognition system based on key point estimation, including:

[0018] An acquisition unit for acquiring the road surface image to be detected;

[0019] A detection unit for using the obtained road surface image to be detected and the trained road surface gap detection model to obtain the detection result;

[0020] In the road surface gap detection model, specifically:

[0021] Extract the global features and local features of the obtained road surface image to be detected and fuse them to obtain the fused features;

[0022] Based on the fused feature, a center point map and a displacement map are obtained by using a detection output head; the positions of the midpoints and endpoints of the road surface joints are determined according to the center point map and the displacement map;

[0023] Based on the fused feature, road surface joint category information is obtained by using a classification output head;

[0024] Based on the determined positions of the midpoints and endpoints of the road surface joints and the road surface joint category information, a detection result is obtained.

[0025] The third aspect of the present invention provides a computer device, including: a processor, a memory and a bus, where the memory stores machine-readable instructions executable by the processor. When the computer device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, a road surface joint recognition method based on key point estimation is executed.

[0026] The fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, a road surface joint recognition method based on key point estimation is executed.

[0027] The above one or more technical solutions have the following beneficial effects:

[0028] In the present invention, the global features and local features of the road surface image to be detected are extracted and fused to obtain a fused feature. Using a double-branch structure, a center point map and a displacement map are obtained through a detection output head, and the positions of the midpoints and endpoints of the road surface joints are determined accordingly; the road surface joint category information is obtained through a classification output head; the solution of the present invention accurately detects the midpoints and endpoints of the road surface joints and assigns accurate category information to the detected targets while improving the recognition accuracy and efficiency.

[0029] The advantages of the additional aspects of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. Description of the Drawings

[0030] The specification drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0031] Figure 1 It is a schematic diagram of the double-branch structure in the first embodiment of the present invention;

[0032] Figure 2 It is a schematic diagram of the feature fusion strategy in the first embodiment of the present invention;

[0033] Figure 3Schematic diagram of redundant line segments in Embodiment 1 of the present invention;

[0034] Figure 4 Joints of various types of cement pavement in Embodiment 1 of the present invention. Detailed implementation manners

[0035] It should be noted that the following detailed description is exemplary and is intended to provide further illustration of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0036] It should be noted that the terms used herein are only for describing the specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention.

[0037] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0038] Embodiment 1

[0039] This embodiment discloses a method for identifying pavement joints based on key point estimation, including:

[0040] Obtain the pavement image to be detected;

[0041] Use the obtained pavement image to be detected and the trained pavement gap detection model to obtain the detection result;

[0042] In the pavement gap detection model, specifically:

[0043] Extract the global features and local features of the obtained pavement image to be detected and fuse them to obtain the fused features;

[0044] Based on the fused features, use the detection output head to obtain the center point map and the displacement map; determine the positions of the midpoint and endpoints of the pavement joint according to the center point map and the displacement map;

[0045] Based on the fused features, use the classification output head to obtain the pavement joint category information;

[0046] Based on the determined positions of the midpoint and endpoints of the pavement joint and the pavement joint category information, obtain the detection result.

[0047] The solution of this embodiment refers to the design concept of centernet, uses key point estimation to find the center point of the line segment, and judges the category of the joint according to the characteristics of the center point and regresses the relative position relationship between the endpoint of the line segment and the center point. This method not only improves the accuracy and efficiency of recognition, but also provides a technical solution for classifying the joints of the cement pavement.

[0048] The following combines with Figure 1 - Figure 2A pavement joint recognition method based on key point estimation proposed in this embodiment is described in detail, specifically including:

[0049] Step 1: A global information feature fusion strategy based on Transformer.

[0050] Before step 1, it also includes using a backbone network to extract a feature map from the pavement image to be detected, and any feature extraction network can be selected according to needs for the backbone network.

[0051] As Figure 2 shown, the joint recognition scheme based on key point estimation requires that a single pixel has a relatively wide receptive field. In the fully convolutional network architecture, the receptive field of a single pixel is ultimately limited. Using Transformer to capture the global information in the feature map and fuse it into a single pixel can improve the detection and classification performance of the model.

[0052] The specific design scheme is as follows:

[0053] S1-1: Global information capture based on Transformer.

[0054] Use a single-layer Transformer to capture the global information in the feature map. In the design, the position embedding and normalization strategy are retained, and a multi-layer perceptron structure with an activation function is used at the end of the Transformer.

[0055] S1-2: Local information capture based on Conv3*3.

[0056] Use a convolutional neural network to capture the local information in the feature map. The specific modules include a 3*3 convolution, a ReLU activation function, and a batch normalization strategy connected in sequence. The purpose of this part is to retain sufficient local information that can depict the morphological features of the target during feature fusion.

[0057] S1-3: A feature information fusion strategy considering residual connection.

[0058] Referring to the design scheme of residual connection, the feature map, the global feature captured by Transformer, and the local feature captured by Conv3*3 are concatenated, and a separate Conv3*3 module is used to fuse the three groups of features to obtain a fused feature.

[0059] Step 2: A design scheme for the output head structure considering pavement joint detection and classification.

[0060] As Figure 1As shown in the figure, a dual-branch detection scheme that combines detection and classification in parallel is designed. The detection head detects the center point of the seam and regresses the relative distance between the end point and the center point; the classification head is responsible for assigning class information to the detected targets.

[0061] The specific design scheme is as follows:

[0062] S2-1: Design scheme of the output head structure for road surface joint detection. The fused features are input into the detection output head and sequentially pass through a transformer, a first convolution, pooling, and batch normalization processes to output the detection results of the road surface joints, including the center point map, displacement map, node map, and straight line map. The center point map and displacement map are used to determine the positions of the midpoint and end points of the road surface joints, and the node map and straight line map are used to assist in improving the detection accuracy of the model.

[0063] Based on the feature information of the center point, the relative position relationship is directly regressed using the L1 loss to determine the positions of the midpoint and end points of the road surface joints.

[0064] During the training process of the detection output head, the positive samples, or rather the prediction targets, are the center points in the fused feature map. The loss function of binary cross-entropy is also directly calculated based on the center points. The center points are 1 and the background points are 0. After the output of the last layer passes through sigmoid and local maximum filtering and a suitable confidence threshold is selected, only the target center points will remain. There will be as many center points as there are targets, and there may be a little redundancy.

[0065] The node map and straight line map use the strategy of multi-task learning to assist in improving the detection accuracy of the model. Specifically, node and straight line labels are provided during the training stage, but the corresponding content is removed during the inference stage and not inferred, only used during the training stage.

[0066] S2-2: Design scheme of the output head structure for road surface joint classification. The fused features are input into the classification output head and sequentially pass through a transformer, a second convolution, pooling, and batch normalization processes to obtain the class information of the road surface joints, including the classification center point map and classification segmentation map. The classification center point map provides the class information of the road surface joints, and the classification segmentation map is used to assist in improving the classification accuracy of the model.

[0067] The implementation logic of the classification segmentation map is similar to that of the node map and is achieved through the method of multi-task learning. Specifically, there is a corresponding loss function during the training process, but it is not used during the inference stage.

[0068] The classification output head obtains a multi-channel center point map. There are several channels corresponding to several categories, and each category only appears in the corresponding channel to achieve the classification goal. Compared with the single-channel center point map, each channel scores each pixel point. According to the center point position, there will be n scores on the multi-channel center point map, where n is the number of categories. The category corresponding to the channel with the highest score is the category of the line segment. There will be many redundant calculation results, but classifying and detecting step by step in this way is more stable.

[0069] The relative distance is estimated using a regression method. The loss function is characterized as L1 loss, and the label is the relative distance between the endpoint and the center point. The coordinates of the endpoint and the center point are (x, y) and (x2, y2) respectively. Given the position of the center point, the position of the endpoint can be calculated. Similarly, if calculated properly, the position of the endpoint can be clearly calculated.

[0070] Among them, the convolutional layers of the first convolution in the detection output head and the second convolution in the classification output head are different.

[0071] S2-3: The combination scheme of road surface joint detection and classification information in the inference stage. First, generate the detection results of road surface joints according to the output results of the detection output head, and on this basis, distinguish the categories of road surface joints according to the category information provided by the classification output head.

[0072] Step 3: The non-maximum suppression strategy based on endpoint distance and line segment overlap.

[0073] Filter redundant detection results for the redundant center points generated in the center point map. Each center point generates a straight line. It is inevitable to have redundant center points when directly predicting the center point in the detection stage. After generating the line segments, some post-processing means are used to merge the line segments that are close in spatial position.

[0074] As Figure 3 shown, after the detection is completed, there will sometimes be multiple detection results for the same joint. The main reason is that the model will regard the 1 / 4 position of the joint or somewhere near the midpoint as the midpoint of a separate joint and construct an independent detection result based on this as the center point. Therefore, a certain non-maximum suppression strategy is needed to filter out the redundant detection results.

[0075] The specific design scheme is as follows:

[0076] S3-1: The line segment filtering strategy based on endpoint distance.

[0077] The line filtering strategy based on endpoint distance calculates the square of the difference between the coordinates of the endpoints of two lines respectively, and adds these squared differences to obtain the total sum of squared differences. The minimum value of these squared differences is selected as the distance metric between the lines. When the distance metric is less than the first set threshold, only the lines with high confidence are retained. The specific calculation formula is as follows:

[0078]

[0079] In the formula: represents the distance metric between two lines, represents the endpoints of the first line of and coordinates, represents the endpoints of the second line of and coordinates.

[0080] S3-2: The line filtering strategy based on line overlap degree.

[0081] This line filtering strategy first represents the predicted line and the ground truth line as heatmaps, where the line region is set to 1 and other regions are set to 0. Then, by calculating the intersection of the two heatmaps and the sum of each heatmap, the intersection over union (IoU) is obtained as the similarity metric of the lines. The specific steps are as follows: If the IoU is greater than the second set threshold, the line with lower confidence is discarded.

[0082] S3-3: Non-maximum suppression strategy. For all detection results in an image, they are sorted in descending order of confidence. At the same time, it is judged whether the two lines meet the endpoint threshold and the line overlap degree determination rules. If either one is met, the corresponding line filtering strategy is activated, and the line with higher confidence is retained.

[0083] The solution of this application is applied to an actual dataset, and the dataset structure is as Figure 4 shown. This dataset contains 1000 foreground images of cement pavement joints, which are collected by a CiCS I-C type multi-functional road condition detection vehicle. The image resolution is 3000×2000 pixels, covering four types of cement pavement joints: road boundary lines, bridge expansion joints, longitudinal joints, and transverse joints. LabelMe is used to annotate the images, and each joint is annotated as a straight line with two endpoints.

[0084] The experiments were conducted on a system equipped with four NVIDIA GeForce RTX 3090 GPUs and an i7-10700 CPU, running the Ubuntu 18.04 operating system with CUDA version 11.4. The dataset was divided into a training set (800 images) and a test set (200 images) in an 8:2 ratio. All input images in the training and test phases were resized to 512×512 pixels. Standard data augmentation techniques were employed during training, including random horizontal and vertical flips, rotations, and shears. Pre-trained weights were not used. The training process lasted for 155 epochs, using the Adam optimizer with an initial learning rate of 0.003 and a weight decay of 0.000001. A stepwise learning rate decay strategy was adopted, with linear warm-up for the first 100 steps. The step milestones were set at the 50th, 100th, and 150th epochs, and the step decay factor was 0.2.

[0085] Figure 4 It shows the actual effect of the model in the cement pavement joint detection task. The results indicate that the model can effectively achieve accurate detection of cement pavement joints, reaching the expected design effect and meeting the actual requirements of engineering applications.

[0086] Embodiment 2

[0087] The purpose of this embodiment is to provide a pavement joint recognition system based on key point estimation, including:

[0088] An acquisition unit for acquiring the pavement image to be detected;

[0089] A detection unit for using the trained pavement gap detection model to obtain the detection result from the acquired pavement image to be detected;

[0090] In the pavement gap detection model, specifically:

[0091] Extract the global and local features of the acquired pavement image to be detected and fuse them to obtain the fused features;

[0092] Based on the fused features, use the detection output head to obtain the center point map and displacement map; determine the positions of the midpoints and endpoints of the pavement joints according to the center point map and displacement map;

[0093] Based on the fused features, use the classification output head to obtain the pavement joint category information;

[0094] Based on the determined positions of the midpoints and endpoints of the pavement joints and the pavement joint category information, obtain the detection result.

[0095] Embodiment 3

[0096] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0097] Embodiment 4

[0098] The purpose of this embodiment is to provide a computer-readable storage medium.

[0099] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above method are executed.

[0100] For the devices in the above Embodiments 2, 3, and 4, the steps involved correspond to those in Method Embodiment 1. For the specific implementation manners, reference may be made to the relevant description part of Embodiment 1. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0101] Those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0102] Although the specific implementation manners of the present invention are described above in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A road surface joint recognition method based on key point estimation, characterized in that Including: Obtain the road surface image to be detected; Use the trained road surface gap detection model to obtain the detection result for the obtained road surface image to be detected; In the road surface gap detection model, specifically: Extract the global features and local features of the obtained road surface image to be detected and fuse them to obtain fused features; Based on the fused features, use the detection output head to obtain the center point map and the displacement map; determine the positions of the midpoints and endpoints of the road surface joints according to the center point map and the displacement map; among them, according to the feature information of the center points, use the L1 loss to directly regress the relative position relationship to determine the positions of the midpoints and endpoints of the road surface joints; Based on the fused features, use the classification output head to obtain the road surface joint category information; Based on the determined positions of the midpoints and endpoints of the road surface joints and the road surface joint category information, obtain the detection result; It also includes: using the non-maximum suppression strategy to filter out the redundant center points generated by the center point map, specifically: Judge whether two line segments meet the endpoint distance threshold judgment and the line segment overlap degree judgment; If one of them is met, calculate the confidence according to the endpoint distance threshold judgment or the line segment overlap degree judgment, arrange the calculated confidences from large to small, and filter the line segments according to the arrangement result.

2. The pavement joint recognition method based on key point estimation according to claim 1, characterized in that Extract the global features and local features of the obtained road surface image to be detected and fuse them to obtain fused features, specifically: Extract the feature map of the obtained road surface image to be detected; Use the transformer to extract the global features of the feature map; Use the first convolution module to extract the local features of the feature map; Fuse the feature map, the global features and the local features, and process them through the second convolution module to obtain fused features.

3. The pavement joint recognition method based on key point estimation according to claim 1, characterized in that Pass the fused features through the transformer, the first convolution, pooling, and batch normalization processes in sequence to obtain the center point map and the displacement map; pass the fused features through the transformer, the second convolution, pooling, and batch normalization processes in sequence to obtain the category information of the road surface joints; among them, the convolutional layers of the first convolution and the second convolution are different.

4. The pavement joint recognition method based on key point estimation according to claim 1, characterized in that Calculate the confidence according to the endpoint distance threshold judgment, specifically: Calculate the square of the difference between the endpoint coordinates corresponding to the two line segments respectively, and add the calculated squared differences to obtain the total squared difference sum, select the minimum value among the calculated squared difference sums as the distance metric between the line segments, and when the distance metric is less than the set threshold, it meets the endpoint distance threshold judgment.

5. The pavement joint recognition method based on key point estimation according to claim 1, characterized in that Calculate the confidence according to the line segment overlap degree judgment, specifically: represent the detected line segment and the real line segment as heat maps, calculate the intersection of the two heat maps to obtain the intersection over union ratio as the similarity metric of the line segments, and if the similarity metric is greater than the second set threshold, it meets the line segment overlap degree judgment.

6. The pavement joint recognition method based on key point estimation according to claim 1, characterized in that Extract the feature map of the obtained road surface image to be detected through the backbone network, extract the global features and local features of the extracted feature map and fuse them to obtain fused features.

7. A pavement joint recognition system based on key point estimation, characterized in that Including: An acquisition unit for acquiring the road surface image to be detected; A detection unit for using the trained road surface gap detection model to obtain the detection result for the acquired road surface image to be detected; In the road surface gap detection model, specifically: Extract the global features and local features of the to-be-detected road surface image obtained, and fuse them to obtain fused features; Based on the fused features, use the detection output head to obtain a center point map and a displacement map; determine the positions of the midpoints and endpoints of the road surface joints according to the center point map and the displacement map; Based on the fused features, use the classification output head to obtain road surface joint category information; Based on the determined positions of the midpoints and endpoints of the road surface joints and the road surface joint category information, obtain the detection result; It further includes a filtering unit for filtering redundant center points generated by the center point map by using a non-maximum suppression strategy, specifically: Judge whether two line segments meet the endpoint distance threshold determination and the line segment coincidence degree determination; If one of them is met, calculate the confidence according to the endpoint distance threshold determination or the line segment coincidence degree determination, arrange the calculated confidences from large to small, and filter the line segments according to the arrangement result.

8. A computer device, characterized in that, Comprising: A processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor. When the computer device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, it executes a method for identifying road surface joints based on key point estimation according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is run by the processor, it executes a method for identifying road surface joints based on key point estimation according to any one of claims 1 to 6.

Images