Road surface detection method and device, electronic equipment and storage medium

Abstract

A road surface detection method and device, electronic equipment and storage medium are provided. The method comprises: acquiring a road surface image; performing feature extraction on the road surface image to obtain feature data; performing center point detection, offset detection and semantic segmentation on the feature data respectively; wherein the center point region of a crack and / or a joint in the road surface image is determined through the center point detection, the edge offset of the crack and / or the joint in the road surface image is determined through the offset detection, and the crack and / or the joint in the road surface image is subjected to semantic recognition through the semantic segmentation to obtain an identification result; and a detection result of the road surface image is generated in combination with the center point region, the edge offset and the identification result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a road surface detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] For highway pavements, cracks are a typical type of pavement distress. However, simply detecting pavement cracks cannot fully achieve a scientific assessment of the pavement's technical condition. In many cases, it is necessary to consider the location of pavement joints and the methods of treatment.

[0003] When inspecting road surfaces, it is necessary to distinguish between joints and cracks and to locate them precisely. Currently, most applications use a "one defect, one model" approach, which has low production efficiency and lacks a solution for simultaneously and accurately inspecting both. Summary of the Invention

[0004] In view of this, this application proposes a road surface detection method, apparatus, electronic device, and storage medium to solve or partially solve the above-mentioned problems.

[0005] To achieve the above objectives, this application provides a road surface detection method, comprising:

[0006] Acquire road surface images;

[0007] Feature extraction is performed on the road surface image to obtain feature data;

[0008] The feature data are subjected to center point detection, offset detection, and semantic segmentation respectively; wherein, the center point detection determines the center point region of cracks and / or joints in the road surface image, the offset detection determines the edge offset of cracks and / or joints in the road surface image, and the semantic segmentation performs semantic recognition on cracks and / or joints in the road surface image to obtain the recognition result.

[0009] The detection result of the road surface image is generated by combining the center point region, the edge offset, and the recognition result.

[0010] In some exemplary embodiments, the feature extraction of the road surface image includes:

[0011] The road surface image is used to extract features using an adjusted EfficientNet convolutional neural network; specifically, the adjusted EfficientNet convolutional neural network is adjusted using dilated spatial pyramid pooling.

[0012] In some exemplary embodiments, the adjustment of the EfficientNet convolutional neural network using dilated spatial pyramid pooling includes:

[0013] A dilated spatial pyramid pooling layer is added after the EfficientNet convolutional neural network.

[0014] Determine the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and use the first feature map of the last convolutional layer in each image size as the feature map to be fused.

[0015] The second feature map output from the hollow space pyramid pooling layer is fused with the feature map to be fused to perform feature extraction; wherein the feature map to be fused is fused with the second feature map in reverse order of the image size.

[0016] In some exemplary embodiments, the step of performing center point detection on the feature data includes:

[0017] The first convolutional branch, after training, is used to predict the center point of the feature data to obtain the predicted center point of any line segment in the road image, as well as the confidence level of whether the line segment is a straight line.

[0018] Based on the preset rules corresponding to cracks and / or joints, cracks or joints are distinguished according to the confidence level. The region is expanded with the predicted center point as the center to obtain the central region, thereby completing the center point detection.

[0019] In some exemplary embodiments, the offset detection of the feature data includes:

[0020] Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the completed training second convolution branch to perform the offset detection on the feature data; wherein, the offset detection specifically refers to: based on the predicted center point and according to the offset calculation method corresponding to the distinction result, determining the offset from the two endpoints of any line segment to the predicted center point, or determining the offset from the outer frame where any line segment is located to the predicted center point.

[0021] In some exemplary embodiments, the semantic segmentation of the feature data includes:

[0022] Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the trained third convolutional branch to perform semantic segmentation on the feature data; wherein, the semantic segmentation specifically refers to: based on the predicted center point and the distinction result, using the third convolutional branch to determine whether each point in the road surface image belongs to a crack or a joint, thereby completing the semantic segmentation.

[0023] In some exemplary embodiments, the detection result for generating the road surface image includes:

[0024] Non-maximum suppression detection is performed on any two line segments in the detection results to determine the perpendicular distance from one line segment to the other line segment.

[0025] In response to the minimum value of the vertical distance satisfying a preset condition, the confidence level of any two line segments determined in the center point detection is obtained, and the line segment with the lower confidence level among the two line segments is deleted.

[0026] Based on the same concept, this application also provides a road surface detection device, comprising:

[0027] The acquisition module is used to acquire road surface images;

[0028] The extraction module is used to extract features from the road surface image to obtain feature data;

[0029] The detection module is used to perform center point detection, offset detection, and semantic segmentation on the feature data respectively; wherein, the center point detection determines the center point region of cracks and / or joints in the road surface image, the offset detection determines the edge offset of cracks and / or joints in the road surface image, and the semantic segmentation performs semantic recognition on cracks and / or joints in the road surface image to obtain the recognition result.

[0030] The output module is used to combine the center point region, the edge offset, and the recognition result to generate the detection result of the road surface image.

[0031] Based on the same concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any of the preceding claims.

[0032] Based on the same concept, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the method described in any of the preceding claims.

[0033] As can be seen from the above description, this application provides a road surface detection method, apparatus, electronic device, and storage medium. The method includes: acquiring a road surface image; extracting features from the road surface image to obtain feature data; performing center point detection, offset detection, and semantic segmentation on the feature data; wherein, the center point region of cracks and / or joints in the road surface image is determined by the center point detection, the edge offset of cracks and / or joints in the road surface image is determined by the offset detection, and the semantic segmentation is used to perform semantic recognition on cracks and / or joints in the road surface image to obtain a recognition result; combining the center point region, the edge offset, and the recognition result, a detection result of the road surface image is generated. This application unifies the recognition modes of joints and cracks, modeling their recognition as a special instance segmentation problem. Based on the feature extraction backbone network, a multi-branch prediction structure is constructed. The center point detection branch predicts the center point of the joint and the center point of the crack region respectively; the offset detection branch predicts the offsets at both ends of the joint and the offset of the rectangular border covering the crack region; and the semantic segmentation branch achieves pixel-level initial recognition of joints and cracks. During the inference phase, for seams, the predicted center point and the offsets of the two endpoints are fused to directly obtain the seam position and length; for cracks, the center point, the regressed border position, and the pixel-level recognition results are fused to directly obtain the segmentation result of each crack region. In this way, accurate identification of both seams and cracks is achieved simultaneously in a single recognition process, improving overall recognition efficiency and user experience. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating an exemplary method provided in an embodiment of this application.

[0036] Figure 2 This is a schematic diagram of the structure of the feature extraction backbone network provided in an embodiment of this application.

[0037] Figure 3 This is a schematic diagram of a model output process according to an exemplary method provided in an embodiment of this application.

[0038] Figure 4 A schematic diagram of the structure of an exemplary device provided in an embodiment of this application.

[0039] Figure 5This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this specification clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0041] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element, object, or method step preceding the term covers the element, object, or method step listed after the term and its equivalents, without excluding other elements, objects, or method steps. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0042] As described in the background section, cracks are a typical type of pavement distress. However, in specific identification, the location and treatment methods of pavement joints often need to be considered. For example, joints in asphalt pavements generally require joint sealing tape, and joints in cement pavements significantly affect the determination of broken slabs and corner fractures. Currently, most products and technical solutions do not have the function of accurately locating and identifying joints in pavement images. Furthermore, most applications adopt a "one distress, one model" approach, which has low production efficiency and high complexity.

[0043] In light of the above practical situation, this application provides a road surface detection scheme. This application unifies the identification modes for joints and cracks, modeling their identification as a special instance segmentation problem. Based on the feature extraction backbone network, a multi-branch prediction structure is constructed. A center point detection branch predicts the center point of the joint and the center point of the crack region respectively; an offset detection branch predicts the offsets at both ends of the joint and the offset of the rectangular border covering the crack region; and a semantic segmentation branch achieves pixel-level initial identification of joints and cracks. In the inference stage, for joints, the predicted center point and the offsets of the two end points are fused to directly obtain the joint position and length; for cracks, the center point, the regressed border position, and the pixel-level identification results are fused to directly obtain the segmentation result of each crack region. Thus, accurate identification of both joints and cracks is achieved simultaneously in a single identification process, improving overall identification efficiency and user experience.

[0044] Figure 1 A flowchart illustrating an exemplary method provided in an embodiment of this application is shown.

[0045] like Figure 1 As shown in the embodiments of this application, the road surface detection method specifically includes the following steps.

[0046] Step 102: Obtain the road surface image.

[0047] In this step, the road surface image refers to image data containing all or part of the road surface, obtained through third-party software or directly input by the user. This image can be acquired in real-time or collected, processed, and then imported uniformly. In typical scenarios, the road surface itself covers a large area, making it difficult to represent the entire surface with a limited number of images. Therefore, the acquired road surface image is usually a partial image, which may contain joints, cracks, or simply a smooth, flat section of the road surface. In some embodiments, the acquired road surface image can be one or multiple images.

[0048] Step 104: Extract features from the road surface image to obtain feature data.

[0049] In this step, feature extraction needs to be performed on the acquired road surface image. Generally, a suitable feature extraction model can be used for feature extraction. In more specific scenarios, different training sets can be set to train the corresponding model, so that the trained model can obtain the features needed by the user and generate the corresponding feature map when used. The feature map here is the feature data obtained from feature extraction.

[0050] In some embodiments, feature extraction can be performed using a corresponding convolutional model. Since most seams or cracks are relatively simple line segments and are generally long and narrow aggregates, a more targeted model can be selected from among many convolutional models. Furthermore, since the objects being identified are relatively recognizable, to further reduce the overall model complexity, the EfficientNet convolutional neural network can be used as the backbone network for feature extraction. This network utilizes a compound scaling method to simultaneously optimize the network's depth, width, and input resolution, thereby providing excellent performance with limited computational resources. The core component of the EfficientNet convolutional neural network is the MBConv module, which integrates depthwise separable convolution and a Squeeze-and-Excitation (SE) mechanism to improve network efficiency. The specific network structure is as follows: Figure 2 As shown, the MBConv module includes a depthwise convolution and a corresponding SE attention mechanism module; the feature fusion module includes feature map concatenation and feature map fusion using convolution modules. All involved depthwise convolution and convolution modules consist of 3*3 convolutions, batch normalization, and ReLU activation functions.

[0051] Furthermore, to enhance the receptive field of individual pixels and thus improve the model's feature extraction and target recognition performance at different scales, Atrios Spatial Pyramid Pooling (ASPP) can be applied to the EfficientNet convolutional neural network. This allows for multi-scale feature fusion to improve the receptive field of individual pixels. ASPP uses convolutional operations with various dilation rates to capture contextual information at different scales, effectively improving the resolution of feature maps and the overall performance of the model. Specifically, in some embodiments, feature extraction from the road surface image includes: performing feature extraction on the road surface image using the adjusted EfficientNet convolutional neural network; wherein the adjusted EfficientNet convolutional neural network specifically refers to adjusting the EfficientNet convolutional neural network using Atrios Spatial Pyramid Pooling.

[0052] In more specific scenarios, such as Figure 2As shown, the pooling layer corresponding to the dilated spatial pyramid pooling can be concatenated after the EfficientNet convolutional neural network; that is, the output of the last MBConv module of the EfficientNet convolutional neural network is used as the input to the dilated spatial pyramid pooling layer. Simultaneously, to further improve the model's detection performance for features at different scales, the feature map output from the dilated spatial pyramid pooling layer can be further upsampled at least once. The feature map used for feature fusion during each upsampling can be the output of the last MBConv module in each size during convolution. For example... Figure 2 As shown, only six MBConv modules are illustrated here, referred to as MBConv module 1 to MBConv module 6. MBConv module 1 reduces the initial image size to 1 / 2, then MBConv module 2 reduces the image size to 1 / 4, MBConv module 4 reduces the image size to 1 / 8, MBConv module 6 reduces the image size to 1 / 16, and so on. Furthermore, during upsampling, the feature map to be fused is the output of the last MBConv module in each size, corresponding to MBConv module 1, MBConv module 3, and MBConv module 5 in the illustration. Although MBConv module 6 corresponds to the 1 / 16 size, it is not necessarily the last layer in this size, and therefore its output is not used for upsampling. Also, if only MBConv module 6 is used, the input and output of the holed spatial pyramid pooling layer are also 1 / 16 size; in a typical implementation, upsampling of the same size is not required. Subsequently, during upsampling, feature fusion can be performed sequentially in reverse order of size, such as... Figure 2 As shown, MBConv module 5 is upsampled first, then MBConv module 3 is upsampled, and finally MBConv module 1 is upsampled.

[0053] That is, in some embodiments, adjusting the EfficientNet convolutional neural network using dilated spatial pyramid pooling includes: adding a dilated spatial pyramid pooling layer after the EfficientNet convolutional neural network; determining the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and using the first feature map of the last convolutional layer in each image size as the feature map to be fused; performing feature fusion between the second feature map output by the dilated spatial pyramid pooling layer and the feature map to be fused, thereby performing the feature extraction; wherein the feature map to be fused is fused with the second feature map in reverse order of the image size. The first feature map corresponds to the output of each convolutional layer (MBConv module), and the second feature map corresponds to the output of the dilated spatial pyramid pooling layer.

[0054] Finally, the road surface image is input into the adjusted EfficientNet convolutional neural network for feature extraction, and the feature data is obtained.

[0055] Step 106: Perform center point detection, offset detection, and semantic segmentation on the feature data respectively; wherein, the center point detection determines the center point region of the cracks and / or joints in the road image, the offset detection determines the edge offset of the cracks and / or joints in the road image, and the semantic segmentation performs semantic recognition on the cracks and / or joints in the road image to obtain the recognition result.

[0056] In this step, to improve the accuracy of crack or joint identification, cracks and / or joints can be identified from multiple angles. In some embodiments, joints can be represented as straight lines, described using a center point and two endpoint offsets. Cracks, due to their variable shape, can be represented as a maximal rectangle (bounding rectangle), described using a center point, the width and height of a border, and offsets. Then, semantic segmentation can be performed on the joints and / or cracks, using a pixel value of 1 to describe them as foreground elements. By cross-referencing these multiple aspects, the final crack and / or joint identification result is obtained. It should be noted that since road surface images are usually local images, not every image will contain cracks and joints. Different images may only contain joints, only cracks, both cracks and joints, or nothing at all. Therefore, in this scenario, the identification generally focuses on cracks and / or joints.

[0057] In some embodiments, three branches can be set up on the backbone network for feature extraction to perform center point detection, offset detection, and semantic segmentation, respectively. For the first two detection branches, two channels can be set up, one corresponding to seams and the other to cracks. For the center point detection branch, the first channel detects the midpoint of the seam (straight line), and the second channel detects the center point of the largest rectangular bounding box forming the connected region of a crack, thus obtaining the center point region of the crack and / or seam. For the offset detection branch, the first channel predicts the offset of the seam endpoints from the center point, and the second channel predicts the length, width, and offset of the largest bounding box of the crack, thus obtaining the edge offset of the crack and / or seam. Then, semantic segmentation involves pixel-level recognition and segmentation of the seam and crack regions in the image to obtain the recognition result.

[0058] Specifically, in conjunction with the appendix Figure 3 As shown, for the seam detection task, firstly, on the first channel of the center point detection branch, the position of the pixel with the largest predicted value is obtained as the predicted seam center point position (x, y). Then, on the offset detection branch, the corresponding center point offset (x1, x2, y1, y2) is extracted, and then the two breakpoint positions of the predicted straight line are combined: x_left=x+x1, y_left=y+y1, x_right=x+x2, y_right=y+y2. Secondly, for the crack detection task, on the second channel of the center point detection branch, the position with the largest predicted value is obtained as the predicted crack region's maximum bounding box center point position (x, y). Then, on the offset detection branch, the crack target bounding box size (W, H) and offset (r1, r2) are extracted, and the corresponding crack region ranges are combined: x_left=x+r1-W / 2, y_left=x+r1-H / 2, x_right=x+r1+W / 2, y_right=x+r1+H / 2. Finally, the corresponding crack segmentation region can be extracted from the crack segmentation map and / or seam segmentation map of the semantic segmentation branch as the crack pixel-level segmentation result and / or the seam segmentation region as the seam pixel-level segmentation result, and finally combined with the crack region and / or seam region to form an instance.

[0059] In more specific scenarios, different branches can be trained together with the main body of the large model, and the model parameters can be adjusted using a more targeted loss function.

[0060] First, for center point detection, such as Figure 3 As shown, based on the features extracted from the basic backbone network, a cascade of 3x3 convolution-group normalization-GELU activation operations can be used to extract features from the center point detection branch. Figure 3 The cube block of branch 2), and then a combination of a 1x1 convolution and a sigmoid function ( Figure 3(See the right arrow of the cube in branch 2). This yields a prediction map where each pixel outputs a probability value of 0-1 to determine if it is the center of a certain type of object. During training of this model branch, different channels are used for detecting different object categories: the first channel of the final prediction map output by the detection branch focuses on predicting the midpoint of the seam, and the second channel focuses on predicting the center point of the maximum bounding box formed by a crack region. To highlight the contribution of cracks or seams to the center point detection task, the positive sample selection method for center point detection is adjusted. The region within a 7×7 pixel area around the center point of the crack bounding box and the region within a 3×3 pixel area around the center of the seam can be used as positive samples, with weighted attenuation based on pixel position and distance from the corresponding center point. This method treats all points within a region as center points during training, increasing the number of positive samples while reducing training difficulty. Therefore, in application, only the center point can be detected. For example, by calculating the confidence score of each pixel to determine if it is the center point, pixels with a confidence score higher than a certain threshold are considered the center points of a line segment. Alternatively, the detection can be expanded to include the central region, accelerating the fitting process and providing more efficient results. For each pixel within the window, its distance relative to the target center is calculated, and the weight is calculated using the following formula:

[0061]

[0062] in, and These are the x and y coordinates of the pixel. and These are the x and y coordinates of the center of the rectangle or the straight line of the seam. and These are the width and length of the rectangle, respectively. This weight varies with distance from the center point; pixels closer to the center point have a higher weight, and pixels farther away have a lower weight.

[0063] Subsequently, during the training of this branch, to improve the robustness of detection, a focal loss function can be used to optimize the loss function of the center point feature map. This effectively addresses the class imbalance problem. The specific formula for FocalLoss is as follows:

[0064]

[0065]

[0066] in, This refers to Focal Loss. This indicates the total number of joint and crack areas. Indicates the prediction confidence level. These are the weights after redistribution. It is an adjustment parameter used to adjust the deweighting rate of easily sampled data. In some examples, This reduces the contribution of easy samples to the loss function and increases the contribution of difficult samples to the loss function. It is also an adjustment parameter used to measure the contribution of negative samples; in some examples, This allows for the adjustment of the contribution of positive samples within the window range.

[0067] That is, in some embodiments, the center point detection of the feature data includes: using a trained first convolutional branch to predict the center point of the feature data, obtaining the predicted center point of any line segment in the road image, and the confidence level of whether the line segment is a straight line; according to the preset rules corresponding to cracks and / or joints, distinguishing between cracks and joints based on the confidence level, and expanding the region centered on the predicted center point to obtain a central region, thereby completing the center point detection. Here, the first convolutional branch is the model branch used for center point detection, the predicted center point is the center point obtained through the trained model, the confidence level is the probability of whether any line segment is a straight line, which can also be obtained through the trained model, and the central region is the crack region within a 7×7 pixel range or the joint region within a 3×3 pixel range of the aforementioned center point.

[0068] Secondly, for offset detection, such as Figure 3 As shown, based on the features extracted from the basic backbone network, a cascade of 3x3 convolution-group normalization-GELU activation operations can be used to extract features from the center point detection branch. Figure 3 The cube block of branch 3), then through a 1x1 convolution ( Figure 3 (See the right arrow of the cube in branch 3). This branch generates a predicted image and performs accurate regression of the target position based on the target center point and its category information determined by the detection branch. For the seam detection task, the regression branch predicts the relative position of the upper endpoint of the seam with respect to the center point. Based on the center point coordinates and the relative position of the endpoint with respect to the center point, the position of the endpoint can be calculated. For the crack detection task, the width and height (W, H) of the crack border and the offset (r1, r2) of the center point are regressed. Based on the width, height, and center point position, the target can be selected in the crack image. The offset is responsible for fine-tuning the specific position of the bounding box after generating it based on the center point.

[0069] Subsequently, when training this branch, the difference between the predicted and true values ​​can be calculated using L1 Loss (Mean Absolute Error, MAE) to ensure regression accuracy. The specific formula for L1 Loss is as follows:

[0070]

[0071] in, It is the number of positive samples. It is the first The first sample One predicted displacement, It is the first The first sample One actual displacement.

[0072] That is, in some embodiments, the offset detection of the feature data includes: obtaining the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and using the trained second convolutional branch to perform the offset detection on the feature data; wherein, the offset detection specifically involves: determining the offset from the two endpoints of any line segment to the predicted center point based on the predicted center point and according to the offset calculation method corresponding to the distinction result, or determining the offset from the circumscribed frame of any line segment to the predicted center point. The second convolutional branch is the model branch used for offset detection. Then, if a line segment is likely a seam based on confidence level and other data, the offset from the two endpoints of the line segment to the predicted center point is determined; correspondingly, if a line segment is likely a crack, the offset from the circumscribed frame of the line segment to the predicted center point is determined. Finally, the model can use these offsets to fine-tune the position of the seam and / or crack.

[0073] Third, for semantic segmentation, such as Figure 3 As shown, based on the basic features extracted from the backbone network, a cascade of 3x3 convolution-group normalization-GELU activation operations can be used to extract features from the center point detection branch. Figure 3 The cube block of branch 1, then combined with a 1x1 convolution-Sigmoid function ( Figure 3 Branch 1 (right arrow of the cube block) generates a prediction image. Each pixel in this image outputs a probability value of 0-1 to determine whether it belongs to a certain object category. The segmentation branch is responsible for generating preliminary segmentation results for seams or cracks, helping to improve the overall detection accuracy of the model.

[0074] Subsequently, when training this branch, its loss function can also adopt the Focal Loss loss function mentioned in the previous embodiment.

[0075] That is, in some embodiments, the semantic segmentation of the feature data includes: obtaining the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and performing the semantic segmentation on the feature data using the trained third convolutional branch; wherein, the semantic segmentation specifically involves: based on the predicted center point and the distinction result, determining whether each point in the road surface image belongs to a crack or a joint through the third convolutional branch, thereby completing the semantic segmentation. The third convolutional branch is used as the model branch for semantic segmentation.

[0076] Step 108: Combine the center point region, the edge offset, and the recognition result to generate the detection result of the road surface image.

[0077] In this step, after obtaining the center point region, edge offset, and recognition results, they can be cross-validated. Through training, the model can use these three results to obtain more accurate crack and / or joint detection results. Finally, the detection results are output.

[0078] When the confidence threshold is set close to the optimal value of the detected image, a relatively accurate recognition result is obtained. However, if the confidence threshold is set lower than the optimal value of the detected image, multiple redundant results may be generated for the same target. Therefore, in some embodiments, non-maximum suppression (NMS) can be further applied to eliminate these redundant line segments. Specifically, redundant line segments may appear in two forms: one is a line segment that highly overlaps with the real situation, and the other is a line segment that is shorter or longer than the real situation. To address both cases simultaneously, when obtaining the detection results, a non-maximum suppression scheme based on the minimum endpoint-to-line segment distance can be used to filter redundancy in the detection results.

[0079] Specifically, to reduce false positive predictions in seam detection—that is, redundant predictions that are very close to the actual seam location—a Non-Maximum Suppression (NMS) strategy is needed to filter the prediction results. Therefore, a point-to-line distance-based evaluation metric is proposed to implement the NMS method. This metric is defined as follows:

[0080] For the predicted line segment and , express From the endpoint to The vertical distance, and Indicates from endpoint to The vertical distance. The calculation method is as follows:

[0081]

[0082]

[0083] in and They represent the predicted line segments respectively. and The endpoints. Indicates from arrive The dangling foot, Indicates from arrive The dangling foot.

[0084] After that, if and The minimum value in the range is less than the specified threshold. If the line segment with higher confidence is selected, then the line segment with higher confidence is retained. This condition can be expressed as:

[0085]

[0086] in, It is a manually defined hyperparameter.

[0087] That is, in some embodiments, generating the detection result of the road surface image includes: performing non-maximum suppression detection on any two line segments in the detection result, and determining the vertical distance from one line segment to the other line segment respectively; in response to the minimum value of the vertical distance satisfying a preset condition, obtaining the confidence of the any two line segments determined in the center point detection, and deleting the line segment with the lower confidence among the any two line segments.

[0088] Finally, the test results can be displayed on the corresponding device to provide feedback to the operator. Of course, in other embodiments, the output method of the test results is not limited to display; it can also be used to store, display, use, or further process the test results. The specific output method of the test results can be flexibly selected according to different application scenarios and implementation needs.

[0089] For example, in application scenarios where the method of this embodiment is executed on a single device, the detection results can be directly output on the display component (monitor, projector, etc.) of the current device, so that the operator of the current device can directly see the content of the detection results on the display component.

[0090] For example, in application scenarios where the method of this embodiment is executed on a system composed of multiple devices, the detection results can be sent to other preset devices within the system, i.e., synchronization terminals, as receivers, via any data communication method (wired connection, NFC, Bluetooth, Wi-Fi, cellular network, etc.), so that the synchronization terminals can perform subsequent processing. Optionally, the synchronization terminal can be a preset server, which is generally located in the cloud and serves as a data processing and storage center, capable of storing and distributing the detection results; wherein, the receivers of the distribution are the terminal devices, and the owners or operators of these terminal devices can be road inspection operators, road repair operators, supervisors of relevant work, etc.

[0091] For example, in the application scenario where the method of this embodiment is executed on a system composed of multiple devices, the detection results can be directly sent to a preset terminal device through any data communication method. The terminal device can be one or more of the devices listed in the preceding paragraphs.

[0092] As can be seen from the above embodiments, the road surface detection method provided in this application includes: acquiring a road surface image; extracting features from the road surface image to obtain feature data; performing center point detection, offset detection, and semantic segmentation on the feature data respectively; wherein, the center point region of cracks and / or joints in the road surface image is determined by the center point detection, the edge offset of cracks and / or joints in the road surface image is determined by the offset detection, and the semantic segmentation is used to perform semantic recognition on cracks and / or joints in the road surface image to obtain a recognition result; combining the center point region, the edge offset, and the recognition result, a detection result of the road surface image is generated. This application unifies the recognition modes of joints and cracks, modeling their recognition as a special instance segmentation problem. Based on the feature extraction backbone network, a multi-branch prediction structure is constructed. The center point detection branch predicts the center point of the joint and the center point of the crack region respectively; the offset detection branch predicts the offsets at both ends of the joint and the offset of the rectangular border covering the crack region; and the semantic segmentation branch realizes the initial pixel-level recognition of joints and cracks. During the inference phase, for seams, the predicted center point and the offsets of the two endpoints are fused to directly obtain the seam position and length; for cracks, the center point, the regressed border position, and the pixel-level recognition results are fused to directly obtain the segmentation result of each crack region. In this way, accurate identification of both seams and cracks is achieved simultaneously in a single recognition process, improving overall recognition efficiency and user experience.

[0093] It should be noted that the method in this application embodiment can be executed by a single device, such as a computer or server. The method in this application embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the process. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this application embodiment, and the multiple devices will interact with each other to complete the method described.

[0094] It should be noted that the above description describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] Based on the same concept, corresponding to any of the above embodiments, this application also provides a road surface detection device.

[0096] refer to Figure 4 The road surface detection device includes:

[0097] The acquisition module 410 is used to acquire road surface images.

[0098] The extraction module 420 is used to extract features from the road surface image to obtain feature data.

[0099] The detection module 430 is used to perform center point detection, offset detection, and semantic segmentation on the feature data respectively; wherein, the center point detection determines the center point region of the cracks and / or joints in the road image, the offset detection determines the edge offset of the cracks and / or joints in the road image, and the semantic segmentation performs semantic recognition on the cracks and / or joints in the road image to obtain the recognition result.

[0100] The output module 440 is used to combine the center point region, the edge offset, and the recognition result to generate the detection result of the road surface image.

[0101] In some exemplary embodiments, the extraction module 420 is further configured to:

[0102] The road surface image is used to extract features using an adjusted EfficientNet convolutional neural network; specifically, the adjusted EfficientNet convolutional neural network is adjusted using dilated spatial pyramid pooling.

[0103] In some exemplary embodiments, the extraction module 420 is further configured to:

[0104] A dilated spatial pyramid pooling layer is added after the EfficientNet convolutional neural network.

[0105] Determine the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and use the first feature map of the last convolutional layer in each image size as the feature map to be fused.

[0106] The second feature map output from the hollow space pyramid pooling layer is fused with the feature map to be fused to perform feature extraction; wherein the feature map to be fused is fused with the second feature map in reverse order of the image size.

[0107] In some exemplary embodiments, the detection module 430 is further configured to:

[0108] The first convolutional branch, after training, is used to predict the center point of the feature data to obtain the predicted center point of any line segment in the road image, as well as the confidence level of whether the line segment is a straight line.

[0109] Based on the preset rules corresponding to cracks and / or joints, cracks or joints are distinguished according to the confidence level. The region is expanded with the predicted center point as the center to obtain the central region, thereby completing the center point detection.

[0110] In some exemplary embodiments, the detection module 430 is further configured to:

[0111] Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the completed training second convolution branch to perform the offset detection on the feature data; wherein, the offset detection specifically refers to: based on the predicted center point and according to the offset calculation method corresponding to the distinction result, determining the offset from the two endpoints of any line segment to the predicted center point, or determining the offset from the outer frame where any line segment is located to the predicted center point.

[0112] In some exemplary embodiments, the detection module 430 is further configured to:

[0113] Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the trained third convolutional branch to perform semantic segmentation on the feature data; wherein, the semantic segmentation specifically refers to: based on the predicted center point and the distinction result, using the third convolutional branch to determine whether each point in the road surface image belongs to a crack or a joint, thereby completing the semantic segmentation.

[0114] In some exemplary embodiments, the output module 440 is further configured to:

[0115] Non-maximum suppression detection is performed on any two line segments in the detection results to determine the perpendicular distance from one line segment to the other line segment.

[0116] In response to the minimum value of the vertical distance satisfying a preset condition, the confidence level of any two line segments determined in the center point detection is obtained, and the line segment with the lower confidence level among the two line segments is deleted.

[0117] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0118] The apparatus described above is used to implement the corresponding road surface detection method in the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0119] Based on the same concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the road surface detection method as described in any of the above embodiments.

[0120] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0121] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0122] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0123] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0124] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0125] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0126] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0127] The electronic devices described above are used to implement the corresponding road surface detection methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0128] Based on the same concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the road surface detection method as described in any of the above embodiments.

[0129] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, which can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0130] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the road surface detection method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0131] Based on the same concept, corresponding to any of the above-described embodiments, this application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processors to perform the road surface detection method. Corresponding to the execution entity for each step in each embodiment of the road surface detection method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0132] The computer program product of the above embodiments is used to cause the computer and / or the processor to execute the road surface detection method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0133] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0134] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0135] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0136] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A road surface detection method, characterized in that, include: Acquire road surface images; Feature extraction is performed on the road surface image to obtain feature data; The feature data are subjected to center point detection, offset detection, and semantic segmentation respectively; wherein, the center point detection determines the center point region of cracks and / or joints in the road surface image, the offset detection determines the edge offset of cracks and / or joints in the road surface image, and the semantic segmentation performs semantic recognition on cracks and / or joints in the road surface image to obtain the recognition result. The detection result of the road surface image is generated by combining the center point region, the edge offset, and the recognition result; wherein, for the joint, the predicted center point and the offsets of the two end points are fused to obtain the joint position and length; for the crack, the center point, the regressed border position, and the pixel-level recognition result of semantic segmentation are fused to obtain the segmentation result of the crack region. The process of center point detection for the feature data includes: The first convolutional branch, after training, is used to predict the center point of the feature data to obtain the predicted center point of any line segment in the road image, as well as the confidence level of whether the line segment is a straight line. Based on the preset rules corresponding to cracks and / or joints, cracks or joints are distinguished according to the confidence level. The region is expanded with the predicted center point as the center to obtain the center region, thereby completing the center point detection. The offset detection of the feature data includes: Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the completed training second convolution branch to perform the offset detection on the feature data; wherein, the offset detection specifically refers to: based on the predicted center point and according to the offset calculation method corresponding to the distinction result, determining the offset from the two endpoints of any line segment to the predicted center point, or determining the offset from the outer frame where any line segment is located to the predicted center point; The semantic segmentation of the feature data includes: Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the trained third convolutional branch to perform semantic segmentation on the feature data; wherein, the semantic segmentation specifically refers to: based on the predicted center point and the distinction result, using the third convolutional branch to determine whether each point in the road surface image belongs to a crack or a joint, thereby completing the semantic segmentation.

2. The method according to claim 1, characterized in that, The feature extraction of the road surface image includes: The road surface image is used to extract features using an adjusted EfficientNet convolutional neural network; specifically, the adjusted EfficientNet convolutional neural network is adjusted using dilated spatial pyramid pooling.

3. The method according to claim 2, characterized in that, The adjustment of the EfficientNet convolutional neural network using dilated spatial pyramid pooling includes: A dilated spatial pyramid pooling layer is added after the EfficientNet convolutional neural network. Determine the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and use the first feature map of the last convolutional layer in each image size as the feature map to be fused. The second feature map output from the hollow space pyramid pooling layer is fused with the feature map to be fused to perform feature extraction; wherein the feature map to be fused is fused with the second feature map in reverse order of the image size.

4. The method according to claim 1, characterized in that, The detection results used to generate the road surface image include: Non-maximum suppression detection is performed on any two line segments in the detection results to determine the perpendicular distance from one line segment to the other line segment. In response to the minimum value of the vertical distance satisfying a preset condition, the confidence level of any two line segments determined in the center point detection is obtained, and the line segment with the lower confidence level among the two line segments is deleted.

5. A road surface detection device, characterized in that, include: The acquisition module is used to acquire road surface images; The extraction module is used to extract features from the road surface image to obtain feature data; The detection module is used to perform center point detection, offset detection, and semantic segmentation on the feature data respectively; wherein, the center point detection determines the center point region of cracks and / or joints in the road surface image, the offset detection determines the edge offset of cracks and / or joints in the road surface image, and the semantic segmentation performs semantic recognition on cracks and / or joints in the road surface image to obtain the recognition result. The output module is used to combine the center point region, the edge offset, and the recognition result to generate the detection result of the road surface image; wherein, for the joint, the predicted center point and the offsets of the two end points are fused to obtain the joint position and length; for the crack, the center point, the regressed border position, and the pixel-level recognition result of semantic segmentation are fused to obtain the segmentation result of the crack region. The process of center point detection for the feature data includes: The first convolutional branch, after training, is used to predict the center point of the feature data to obtain the predicted center point of any line segment in the road image, as well as the confidence level of whether the line segment is a straight line. Based on the preset rules corresponding to cracks and / or joints, cracks or joints are distinguished according to the confidence level. The region is expanded with the predicted center point as the center to obtain the center region, thereby completing the center point detection. The offset detection of the feature data includes: Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the completed training second convolution branch to perform the offset detection on the feature data; wherein, the offset detection specifically refers to: based on the predicted center point and according to the offset calculation method corresponding to the distinction result, determining the offset from the two endpoints of any line segment to the predicted center point, or determining the offset from the outer frame where any line segment is located to the predicted center point; The semantic segmentation of the feature data includes: Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the trained third convolutional branch to perform semantic segmentation on the feature data; wherein, the semantic segmentation specifically refers to: based on the predicted center point and the distinction result, using the third convolutional branch to determine whether each point in the road surface image belongs to a crack or a joint, thereby completing the semantic segmentation.

6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method described in any one of claims 1 to 4.

Images