Road surface detection method and device, electronic equipment and storage medium

Abstract

The invention provides a road surface detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a road surface image; performing feature extraction on the pavement image to obtain feature data; respectively carrying out central point detection, offset detection and semantic segmentation on the feature data; wherein the central point area of a crack and / or a seam in the pavement image is determined through the central point detection, the edge offset of the crack and / or the seam in the pavement image is determined through the offset detection, and semantic recognition is performed on the crack and / or the seam in the pavement image through the semantic segmentation to obtain a recognition result; and generating a detection result of the road surface image in combination with the central point region, the edge offset and the recognition result.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a road surface detection method, device, electronic device and storage medium. Background Art

[0002] For highway pavements, cracks are a typical type of pavement disease. However, simply detecting pavement cracks cannot fully realize a scientific assessment of the pavement's technical condition. In many cases, the location and treatment methods of pavement joints need to be considered.

[0003] When inspecting the road surface, it is necessary to distinguish between joints and cracks and accurately locate them separately. Most current applications use the "one disease, one model" technical route, which has low production efficiency and there is no solution to accurately detect both at the same time. Summary of the invention

[0004] In view of this, the present application proposes a road surface detection method, device, electronic device and storage medium to solve or partially solve the above-mentioned problems.

[0005] Based on the above purpose, the present application provides a road surface detection method, including: Acquire road surface images; Extracting features from the road surface image to obtain feature data; The feature data are respectively subjected to center point detection, offset detection and semantic segmentation; wherein the center point area of ​​the cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of the cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically identified by the semantic segmentation to obtain an identification result; The detection result of the road surface image is generated by combining the central point area, the edge offset and the recognition result.

[0006] In some exemplary embodiments, extracting features from the road surface image includes: The feature extraction is performed on the road image using the adjusted EfficientNet convolutional neural network; wherein the adjusted EfficientNet convolutional neural network is specifically: the EfficientNet convolutional neural network is adjusted using atrous spatial pyramid pooling.

[0007] In some exemplary embodiments, adjusting the EfficientNet convolutional neural network using dilated spatial pyramid pooling includes: Add a dilated spatial pyramid pooling layer after the EfficientNet convolutional neural network; Determine the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and use the first feature map of the last convolutional layer in each image size as the feature map to be fused; The second feature map output by the atrous space pyramid pooling layer is subjected to feature fusion with the feature map to be fused, so as to perform the feature extraction; wherein the feature map to be fused is subjected to feature fusion with the second feature map in reverse order of the image size.

[0008] In some exemplary embodiments, the performing center point detection on the feature data includes: Using the trained first convolution branch to predict the center point of the feature data, obtain the predicted center point of any line segment in the road surface image, and the confidence level of whether the line segment is a straight line; According to preset rules corresponding to cracks and / or seams, cracks or seams are distinguished based on the confidence level, and the area is expanded with the predicted center point as the center to obtain the central area, thereby completing the center point detection.

[0009] In some exemplary embodiments, the performing offset detection on the characteristic data includes: Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the trained second convolution branch to perform the offset detection on the feature data; wherein the offset detection is specifically: based on the predicted center point and in accordance with the offset calculation method corresponding to the distinction result, determine the offset from the two end points of any line segment to the predicted center point, or determine the offset from the external frame in which any line segment is located to the predicted center point.

[0010] In some exemplary embodiments, performing semantic segmentation on the feature data includes: The predicted center point corresponding to any line segment and the distinction result determined according to the confidence level are obtained, and the feature data is semantically segmented using the trained third convolution branch; wherein the semantic segmentation is specifically as follows: based on the predicted center point and the distinction result, each point in the road surface image is judged as a crack or a seam through the third convolution branch, thereby completing the semantic segmentation.

[0011] In some exemplary embodiments, generating the detection result of the road surface image includes: Performing non-maximum suppression detection on any two line segments in the detection result, and determining the vertical distance from one line segment to the other line segment respectively; In response to the minimum value of the vertical distances satisfying a preset condition, the confidences of the arbitrary two line segments determined in the center point detection are obtained, and the line segment with a smaller confidence in the arbitrary two line segments is deleted.

[0012] Based on the same concept, the present application also provides a road surface detection device, including: An acquisition module, used for acquiring road surface images; An extraction module, used for extracting features from the road surface image to obtain feature data; A detection module, used to perform center point detection, offset detection and semantic segmentation on the feature data respectively; wherein the center point area of ​​the cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of the cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically recognized by the semantic segmentation to obtain a recognition result; An output module is used to generate a detection result of the road surface image by combining the central point area, the edge offset and the recognition result.

[0013] Based on the same concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.

[0014] Based on the same concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement any of the methods described above.

[0015] As can be seen from the above, the present application provides a road surface detection method, device, electronic device and storage medium, the method comprising: acquiring a road surface image; performing feature extraction on the road surface image to obtain feature data; performing center point detection, offset detection and semantic segmentation on the feature data; wherein the center point area of ​​cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically identified by the semantic segmentation to obtain the recognition result; combining the center point area, the edge offset and the recognition result, the detection result of the road surface image is generated. The present application unifies the seam and crack recognition modes, models the recognition of the two as a special instance segmentation problem, and constructs a multi-branch prediction structure based on the feature extraction backbone network, using the center point detection branch to predict the center point of the seam and the center point of the crack area respectively; the offset detection branch predicts the offsets at both ends of the seam and the offset of the rectangular frame covering the crack area; the semantic segmentation branch realizes the initial recognition of seams and cracks at the pixel level. In the inference stage, for seams, the predicted center point and the offset of the two end points are fused to directly obtain the seam position and length; for cracks, the center point, the regressed border position and the pixel-level recognition results are fused to directly obtain the segmentation result of each crack area. In this way, accurate recognition of seams and cracks is completed at the same time through one recognition, which improves the overall recognition efficiency and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of an exemplary method provided in an embodiment of the present application.

[0018] Figure 2 A schematic diagram of the structure of the feature extraction backbone network provided in an embodiment of the present application.

[0019] Figure 3 A schematic diagram of a model output process according to an exemplary method provided in an embodiment of the present application.

[0020] Figure 4 A schematic diagram of the structure of an exemplary device provided in an embodiment of the present application.

[0021] Figure 5A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of this specification more clear, this specification is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0023] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements, objects or method steps appearing before the word cover the elements, objects or method steps listed after the word and their equivalents, without excluding other elements, objects or method steps. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0024] As mentioned in the background technology section, cracks are a typical type of pavement disease, but in many cases, the location and treatment methods of pavement joints need to be considered when making specific identifications. For example, the joints of asphalt pavements generally need to be treated with seam tape, and the joints of cement pavements greatly affect the determination of broken plates and plate corner fractures. Currently, most products and technical solutions do not have the function of accurately locating and identifying joints in pavement images. Secondly, most applications use the "one disease one model" technical route, which has low production efficiency and high complexity.

[0025] In view of the above practical situation, the embodiment of the present application provides a road surface detection solution. The present application unifies the seam and crack recognition modes, models the recognition of the two as a special instance segmentation problem, and constructs a multi-branch prediction structure based on the feature extraction backbone network. The center point detection branch is used to predict the center point of the seam and the center point of the crack area respectively; the offset detection branch predicts the offsets at both ends of the seam and the offset of the rectangular border covering the crack area; the semantic segmentation branch realizes the initial recognition of seams and cracks at the pixel level. In the inference stage, for seams, the predicted center point is fused with the offsets of the two end points to directly obtain the seam position and length; for cracks, the center point, the regressed border position and the pixel-level recognition results are fused to directly obtain the segmentation results of each crack area. In this way, the accurate recognition of seams and cracks is completed at the same time through one recognition, which improves the overall recognition efficiency and user experience.

[0026] Figure 1 A flow chart of an exemplary method provided in an embodiment of the present application is shown.

[0027] like Figure 1 As shown, the road surface detection method exemplarily proposed in the embodiment of the present application includes the following steps.

[0028] Step 102: Acquire a road surface image.

[0029] In this step, the road surface image is the image data containing all or part of the road surface through third-party software or direct input by the user. The image can be acquired in real time, or it can be collected and sorted and uniformly imported. In general scenarios, the road surface itself is large in area, and it is difficult to present the whole picture through fewer images. The road surface image obtained is generally a partial image, which may contain seams, cracks, or of course, a flat piece of local road surface. In some embodiments, the road surface image obtained can be one or more.

[0030] Step 104: extract features from the road surface image to obtain feature data.

[0031] In this step, it is necessary to extract features from the acquired road surface image, and generally, the corresponding feature extraction model can be used for feature extraction. In more specific scenarios, different training sets can be set to train the corresponding models according to different scenarios, so that the trained models can obtain the features required by users when used and generate corresponding feature maps. The feature map here is the feature data obtained by feature extraction.

[0032] In some embodiments, features can be extracted with the help of corresponding convolution models. Since most general seams or cracks are in the form of relatively simple line segments and are generally in the form of relatively narrow and long sets, a more targeted model can be selected from many convolution models. At the same time, since the identified objects are more recognizable, in order to further reduce the complexity of the overall model, the EfficientNet convolutional neural network can be used as the backbone network for feature extraction in many convolutional models. The network uses a composite scaling method to simultaneously optimize the depth, width and input resolution of the network, thereby providing excellent performance with limited computing resources. Among them, the core component of the EfficientNet convolutional neural network is the MBConv module, which integrates depth-separable convolution and Squeeze-and-Excitation (SE) mechanisms to improve network efficiency. The specific network structure is as follows: Figure 2 As shown in the figure, the MBConv module includes deep convolution and the corresponding SE attention mechanism module; the feature fusion module includes feature map concatenation and fusion of feature maps using the convolution module. All the deep convolution and convolution modules involved are composed of 3*3 convolution, batch normalization and ReLU activation function.

[0033] Furthermore, in order to improve the receptive field of a single pixel, the model's feature extraction capability and target recognition performance at different scales are enhanced. Based on the EfficientNet convolutional neural network, it can be adjusted with the help of Atros Spatial Pyramid Pooling (ASPP), so as to improve the receptive field of a single pixel through multi-scale feature fusion. Atros Spatial Pyramid Pooling (ASPP) is used to capture contextual information of different scales through convolution operations with multiple expansion rates, which effectively improves the resolution of the feature map and the overall performance of the model. That is, in some embodiments, the feature extraction of the road image includes: using the adjusted EfficientNet convolutional neural network to extract the feature of the road image; wherein the adjusted EfficientNet convolutional neural network is specifically: adjusting the EfficientNet convolutional neural network using atrous spatial pyramid pooling.

[0034] In more specific scenarios, such as Figure 2As shown in the figure, the pooling layer corresponding to the dilated spatial pyramid pooling can be spliced ​​after the EfficientNet convolutional neural network, that is, the output of the last MBConv module of the EfficientNet convolutional neural network is used as the input of the dilated spatial pyramid pooling layer. At the same time, in order to further improve the detection performance of the model for features of different scales, the feature map output by the dilated spatial pyramid pooling layer can be further upsampled at least once, and the feature map of each upsampled feature fusion can be the output of the last MBConv module in each size when the MBConv module performs convolution. Figure 2 As shown, only 6 layers of MBConv modules are shown here, referred to as MBConv modules 1 to MBConv modules 6. Among them, MBConv module 1 reduces the size of the initial image to 1 / 2, then MBConv module 2 reduces the size of the image to 1 / 4, MBConv module 4 reduces the size of the image to 1 / 8, MBConv module 6 reduces the size of the image to 1 / 16, and so on. Furthermore, when upsampling, the feature map to be fused each time is the output of the last layer of the MBConv module in each size. In the 6 modules shown in the figure, it corresponds to MBConv module 1, MBConv module 3, and MBConv module 5. Although MBConv module 6 corresponds to 1 / 16 size, it is not necessarily the last layer of this size, and the output of this layer is not used for upsampling here. At the same time, if only MBConv module 6 is reached, the input and output of the hollow space pyramid pooling layer itself are also 1 / 16 size. In general embodiments, upsampling of the same size is not required. Afterwards, when upsampling, the features can be fused in reverse order of size, such as Figure 2 As shown, upsampling of MBConv module 5 is performed first, then upsampling of MBConv module 3 is performed, and finally upsampling of MBConv module 1 is performed.

[0035] That is, in some embodiments, the use of atrous spatial pyramid pooling to adjust the EfficientNet convolutional neural network includes: adding an atrous spatial pyramid pooling layer after the EfficientNet convolutional neural network; determining the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and using the first feature map of the last convolutional layer in each image size as the feature map to be fused; performing feature fusion on the second feature map output by the atrous spatial pyramid pooling layer and the feature map to be fused, thereby performing feature extraction; wherein the feature map to be fused is sequentially fused with the second feature map in the reverse order of the image size. The first feature map corresponds to the output of each convolutional layer (MBConv module), and the second feature map corresponds to the output of the atrous spatial pyramid pooling layer.

[0036] Finally, the road image is input into the above-adjusted EfficientNet convolutional neural network for feature extraction, and finally the feature data is obtained.

[0037] Step 106, performing center point detection, offset detection and semantic segmentation on the feature data respectively; wherein, the center point area of ​​the cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of the cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically recognized by the semantic segmentation to obtain a recognition result.

[0038] In this step, in order to improve the recognition accuracy of cracks or seams, the recognition of cracks and / or seams can be performed from multiple angles. In some embodiments, the seams can be characterized as straight lines, which can be described using a center point and two end point offsets. Since cracks have uncertain shapes, the independent crack area can be characterized as a maximum rectangle (circumscribed rectangle), which can be described using a center point and the width, height and offset of a border. Afterwards, the seams and / or cracks can be semantically segmented, and the seams and / or cracks can be described as foregrounds using a pixel value of 1. The above multiple aspects can then be mutually verified to finally obtain the recognition results of cracks and / or seams. It should be noted that since road images are usually local images, not every image will contain cracks and seams. Different images may contain only seams, only cracks, both cracks and seams, or nothing at all. Therefore, the recognition of cracks and / or seams is generally performed in the scene.

[0039] In some embodiments, three branches can be set on the main network of feature extraction to perform center point detection, offset detection and semantic segmentation respectively. For the first two detection branches, two channels can be set respectively, one corresponding to the seam and the other corresponding to the crack. For the center point detection branch, the first channel detects the midpoint of the seam (straight line), and the second channel detects the center point of the largest rectangular circumscribed frame constituting a certain crack connected area, thereby obtaining the center point area of ​​the crack and / or seam. For the offset detection branch, the first channel predicts the offset of the end point of the seam from the center point, and the second channel predicts the length, width and offset of the maximum circumscribed frame of the crack, thereby obtaining the edge offset of the crack and / or seam. Afterwards, semantic segmentation is to perform pixel-level recognition and segmentation of the seam area and crack area in the image to obtain the recognition result.

[0040] Specifically, in combination with Figure 3 As shown in the figure, for the seam detection task, first, on the first channel of the center point detection branch, the position of the pixel with the largest predicted value is obtained as the predicted seam center point position (x, y), and then the corresponding center point offset (x1, x2, y1, y2) is extracted on the offset detection branch, and then the two breakpoint positions of the predicted straight line are combined: x_left=x+x1, y_left=y+y1, x_right=x+x2, y_right=y+y2. Secondly, for the crack detection task, on the second channel of the center point detection branch, the position with the largest predicted value is obtained as the predicted center point position (x, y) of the maximum external bounding box of the crack area, and then the crack target bounding box size (W, H) and offset (r1, r2) are extracted on the offset detection branch, and the corresponding crack area range x_left=x+r1-W / 2, y_left=x+r1-H / 2, x_right=x+r1+W / 2, y_right=x+r1+H / 2 are combined. Finally, the corresponding crack segmentation region can be extracted from the crack segmentation map and / or the seam segmentation map of the semantic segmentation branch as a crack pixel-level segmentation result and / or the seam segmentation region as a seam pixel-level segmentation result, and finally form an instance with the crack region and / or the seam region.

[0041] In more specific scenarios, different branches can be trained together with the main model, and the model parameters can be adjusted with the help of more targeted loss functions.

[0042] First, for center point detection, Figure 3 As shown in Figure 1, based on the features extracted by the basic backbone network, a cascade of 3x3 convolution-group normalization-GELU activation operations can be used to extract the features of the center point detection branch ( Figure 3 The cube block of branch 2) is then passed through a 1x1 convolution-Sigmoid function combination ( Figure 3The prediction map is obtained by outputting a probability value of 0-1 for each pixel position on the map to determine whether the pixel is the center position of a certain type of object. When training this model branch, different channels are used to detect different types of objects: the first channel of the prediction map output by the detection branch focuses on the prediction of the midpoint of the seam, and the second channel focuses on the prediction of the center point of the maximum circumscribed border formed by a certain crack area. In order to highlight the contribution of the crack or seam area to the center point detection task, some adjustments have been made to the selection method of positive samples for center point detection. The area within the 7×7 pixel range around the center point of the crack border and the area within the 3×3 range around the center of the seam can be used as positive samples, and weighted attenuation is performed according to the pixel position and the distance from the corresponding center point. In this way, all points in an area are regarded as center points during training, which can increase positive samples and reduce the difficulty of training. In the application, only the center point can be detected. For example, the confidence of each pixel point is calculated to determine whether it is the center point, and the confidence point above a certain threshold is regarded as the center point of a line segment. The center point can also be expanded to the center area for detection to speed up the fitting and give the result more efficiently. For each pixel point within the window range, its distance relative to the target center is calculated, and the weight is calculated using the following formula:

[0043] in, and are the horizontal and vertical coordinates of the pixel points, and is the horizontal and vertical coordinates of the center of the rectangle or seam line, and The weights are the width and length of the rectangle respectively. The weight changes with the distance from the center point. The closer the distance is, the greater the weight of the pixel point, and the farther the distance is, the smaller the weight of the pixel point.

[0044] Later, when training this branch, in order to improve the robustness of detection, the focal loss function can be used to optimize the loss function of the center point feature map, which can effectively deal with the problem of category imbalance. The specific formula of FocalLoss is as follows:

[0045]

[0046] in, represents the focal loss, represents the total number of seams and crack areas, represents the prediction confidence, is the weight after redistribution, is a tuning parameter used to adjust the down-weighting rate of easy samples. In some examples, This can reduce the contribution of easy samples to the loss function and increase the contribution of difficult samples to the loss function. It is also a tuning parameter used to measure the contribution of negative samples. In some examples, In this way, the contribution of positive samples within the window range can be controlled.

[0047] That is, in some embodiments, the center point detection of the feature data includes: using the first convolution branch that has completed training to predict the center point of the feature data, and obtain the predicted center point of any line segment in the road surface image, and the confidence of whether any line segment is a straight line; according to the preset rules corresponding to the cracks and / or seams, distinguish cracks or seams based on the confidence, and expand the area with the predicted center point as the center to obtain the center area, so as to complete the center point detection. Among them, the first convolution branch is the model branch used for center point detection, and the predicted center point is the center point obtained by the trained model. After that, the confidence is the probability of whether any line segment is a straight line, which can also be obtained by the trained model. The center area is the crack area within the 7×7 pixel range of the aforementioned center point or the seam area within the 3×3 range.

[0048] Secondly, for offset detection, such as Figure 3 As shown in Figure 1, based on the features extracted by the basic backbone network, a cascade of 3x3 convolution-group normalization-GELU activation operations can be used to extract the features of the center point detection branch ( Figure 3 3 cube), and then through a 1x1 convolution ( Figure 3 Branch 3 (arrow on the right side of the cube block) obtains the prediction map. This branch performs accurate regression of the target position based on the target center point and its category information determined by the detection branch. For the seam detection task, the regression branch predicts the relative position of the endpoint on the seam relative to the center point. The position of the endpoint can be calculated based on the coordinates of the center point and the relative position of the endpoint relative to the center point; for the crack detection task, the width and height (W, H) of the crack border and the offset (r1, r2) of the center point are regressed. According to the width, height and center point position, the target can be selected from the image. The offset is responsible for fine-tuning the specific position of the bounding box after generating the bounding box based on the center point.

[0049] Later, when training this branch, the absolute error loss (L1 Loss, Mean Absolute Error, MAE) can be used to calculate the difference between the predicted value and the true value to ensure regression accuracy. The specific formula of L1 Loss is as follows:

[0050] in, is the number of positive samples, It is The sample The predicted displacement, It is The sample A real displacement.

[0051] That is, in some embodiments, the offset detection of the feature data includes: obtaining the predicted center point corresponding to any line segment and the distinction result determined according to the confidence, and using the second convolution branch that has completed the training to perform the offset detection on the feature data; wherein the offset detection is specifically: based on the predicted center point and according to the offset calculation method corresponding to the distinction result, determining the offset from the two end points of any line segment to the predicted center point, or determining the offset from the external frame where any line segment is located to the predicted center point. Wherein, the second convolution branch is the model branch for offset detection. Afterwards, if a line segment may be a seam according to the confidence and other data, the offset from the two end points of the line segment to the predicted center point is determined; correspondingly, if a line segment may be a crack, the offset from the external frame where the line segment is located to the predicted center point is determined. Finally, these offsets can be used by the model to fine-tune the position of the seam and / or crack.

[0052] Third, for semantic segmentation, such as Figure 3 As shown in Figure 1, based on the basic features extracted by the basic backbone network, a cascade of 3x3 convolution-group normalization-GELU activation operations can be used to extract the features of the center point detection branch ( Figure 3 The cube block of branch 1) is then passed through a 1x1 convolution-Sigmoid function combination ( Figure 3 The segmentation branch is responsible for generating preliminary segmentation results of seams or cracks, which helps improve the overall detection accuracy of the model.

[0053] Afterwards, when training the branch, its loss function can also adopt the Focal Loss loss function mentioned in the previous embodiment.

[0054] That is, in some embodiments, the semantic segmentation of the feature data includes: obtaining the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and performing the semantic segmentation on the feature data using the trained third convolution branch; wherein the semantic segmentation is specifically: based on the predicted center point and the distinction result, the third convolution branch is used to judge whether each point in the road surface image belongs to a crack or a seam, thereby completing the semantic segmentation. The third convolution branch is used for the model branch of the semantic segmentation.

[0055] Step 108 : generating a detection result of the road surface image by combining the center point area, the edge offset and the recognition result.

[0056] In this step, after obtaining the center point area, edge offset and recognition results, they can be verified with each other. Through training, the model can use these three results to obtain more accurate crack and / or seam detection results. Finally, the output of the detection results is completed.

[0057] When the previous confidence threshold is set close to the optimal value of the detected image, a more accurate recognition result will be obtained. However, if the confidence threshold is set lower than the optimal value of the detected image, multiple redundant results may be generated for the same target. Therefore, in some embodiments, the non-maximum suppression method (NMS) can be further applied to eliminate these redundant line segments. Specifically, redundant line segments may appear in two forms: one is a line segment that is highly consistent with the actual situation, and the other is a line segment that is shorter or longer than the actual situation. In order to solve these two situations at the same time, when the detection result is obtained, the non-maximum suppression scheme of the minimum endpoint to the line segment distance can be used to filter the redundant detection results.

[0058] Specifically, in order to reduce the false positive predictions of seam detection, that is, the redundant prediction results that are very close to the actual seam location, it is necessary to use the NMS (non-maximum suppression) strategy to filter the prediction results. Therefore, an evaluation index based on the point-to-line distance is proposed to implement the NMS method. The evaluation index is defined as follows: For the predicted line segment and , express From the endpoint to The vertical distance Indicates from The endpoint to The vertical distance is calculated as follows:

[0059]

[0060] in and Represent the predicted line segments and endpoint. Indicates from arrive The foot of Indicates from arrive The vertical foot.

[0061] Afterwards, if and The minimum value in is less than the specified threshold , then retain the line segments with higher confidence. This condition can be expressed as:

[0062] in, is a manually defined hyperparameter.

[0063] That is, in some embodiments, the generation of the detection result of the road surface image includes: performing non-maximum suppression detection on any two line segments in the detection result, and respectively determining the vertical distance from one line segment to the other line segment; in response to the minimum value of the vertical distances satisfying a preset condition, obtaining the confidence of the any two line segments determined in the center point detection, and deleting the line segment with the smaller confidence among the any two line segments.

[0064] Finally, the test results can be displayed and output on the corresponding device to give the operator corresponding feedback. Of course, in other embodiments, the output method of the test results may not be limited to output display, and it can also be used to store, display, use or reprocess the test results. According to different application scenarios and implementation needs, the specific output method of the test results can be flexibly selected.

[0065] For example, for an application scenario in which the method of this embodiment is executed on a single device, the detection results can be directly output in a displayed manner on a display component (display, projector, etc.) of the current device, so that the operator of the current device can directly see the content of the detection results from the display component.

[0066] For another example, for an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the detection results can be sent to other preset devices as receivers in the system, that is, synchronization terminals, through any data communication method (wired connection, NFC, Bluetooth, wifi, cellular mobile network, etc.), so that the synchronization terminal can perform subsequent processing on them. Optionally, the synchronization terminal can be a preset server, which is generally set up in the cloud as a data processing and storage center, which can store and distribute the detection results; wherein the recipient of the distribution is the terminal device, and the holder or operator of these terminal devices can be the operator of road surface detection, the operator of road surface repair, the supervisor of the corresponding work, etc.

[0067] For another example, in an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the detection results can be sent directly to a preset terminal device through any data communication method, and the terminal device can be one or more of the ones listed in the preceding paragraphs.

[0068] It can be seen from the above embodiments that the embodiment of the present application provides a road surface detection method, which includes: acquiring a road surface image; performing feature extraction on the road surface image to obtain feature data; performing center point detection, offset detection and semantic segmentation on the feature data; wherein the center point area of ​​cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically identified by the semantic segmentation to obtain a recognition result; combining the center point area, the edge offset and the recognition result, the detection result of the road surface image is generated. The present application unifies the seam and crack recognition modes, models the recognition of the two as a special instance segmentation problem, and constructs a multi-branch prediction structure based on the feature extraction backbone network, using the center point detection branch to predict the center point of the seam and the center point of the crack area respectively; the offset detection branch predicts the offsets at both ends of the seam and the offset of the rectangular frame covering the crack area; the semantic segmentation branch realizes the initial recognition of seams and cracks at the pixel level. In the inference stage, for seams, the predicted center point and the offset of the two end points are fused to directly obtain the seam position and length; for cracks, the center point, the regressed border position and the pixel-level recognition results are fused to directly obtain the segmentation result of each crack area. In this way, accurate recognition of seams and cracks is completed at the same time through one recognition, which improves the overall recognition efficiency and user experience.

[0069] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of the embodiment of the present application can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0070] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0071] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a road surface detection device.

[0072] refer to Figure 4 , the road surface detection device comprises: The acquisition module 410 is used to acquire a road surface image.

[0073] The extraction module 420 is used to extract features from the road surface image to obtain feature data.

[0074] The detection module 430 is used to perform center point detection, offset detection and semantic segmentation on the feature data respectively; wherein, the center point area of ​​the cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of the cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically recognized by the semantic segmentation to obtain a recognition result.

[0075] The output module 440 is used to generate a detection result of the road surface image by combining the central point area, the edge offset and the recognition result.

[0076] In some exemplary embodiments, the extraction module 420 is further configured to: The feature extraction is performed on the road image using the adjusted EfficientNet convolutional neural network; wherein the adjusted EfficientNet convolutional neural network is specifically: the EfficientNet convolutional neural network is adjusted using atrous spatial pyramid pooling.

[0077] In some exemplary embodiments, the extraction module 420 is further configured to: Add a dilated spatial pyramid pooling layer after the EfficientNet convolutional neural network; Determine the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and use the first feature map of the last convolutional layer in each image size as the feature map to be fused; The second feature map output by the atrous space pyramid pooling layer is subjected to feature fusion with the feature map to be fused, so as to perform the feature extraction; wherein the feature map to be fused is subjected to feature fusion with the second feature map in reverse order of the image size.

[0078] In some exemplary embodiments, the detection module 430 is further configured to: Using the trained first convolution branch to predict the center point of the feature data, obtain the predicted center point of any line segment in the road surface image, and the confidence level of whether the line segment is a straight line; According to preset rules corresponding to cracks and / or seams, cracks or seams are distinguished based on the confidence level, and the area is expanded with the predicted center point as the center to obtain the central area, thereby completing the center point detection.

[0079] In some exemplary embodiments, the detection module 430 is further configured to: Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the trained second convolution branch to perform the offset detection on the feature data; wherein the offset detection is specifically: based on the predicted center point and in accordance with the offset calculation method corresponding to the distinction result, determine the offset from the two end points of any line segment to the predicted center point, or determine the offset from the external frame in which any line segment is located to the predicted center point.

[0080] In some exemplary embodiments, the detection module 430 is further configured to: The predicted center point corresponding to any line segment and the distinction result determined according to the confidence level are obtained, and the feature data is semantically segmented using the trained third convolution branch; wherein the semantic segmentation is specifically as follows: based on the predicted center point and the distinction result, each point in the road surface image is judged as a crack or a seam through the third convolution branch, thereby completing the semantic segmentation.

[0081] In some exemplary embodiments, the output module 440 is further configured to: Performing non-maximum suppression detection on any two line segments in the detection result, and determining the vertical distance from one line segment to the other line segment respectively; In response to the minimum value of the vertical distances satisfying a preset condition, the confidences of the arbitrary two line segments determined in the center point detection are obtained, and the line segment with a smaller confidence in the arbitrary two line segments is deleted.

[0082] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0083] The device of the above embodiment is used to implement the corresponding road surface detection method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0084] Based on the same concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the road surface detection method as described in any of the above embodiments is implemented.

[0085] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0086] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0087] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0088] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0089] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0090] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0091] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0092] The electronic device of the above embodiment is used to implement the corresponding road surface detection method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0093] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the road surface detection method described in any of the above embodiments.

[0094] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0095] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the road surface detection method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0096] Based on the same concept, corresponding to any of the above-mentioned embodiments, the present application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the road surface detection method. Corresponding to the execution subject corresponding to each step in each embodiment of the road surface detection method, the processor that executes the corresponding step can belong to the corresponding execution subject.

[0097] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the road surface detection method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0098] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0099] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device may be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0100] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0101] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A road surface detection method, characterized in that: include: Acquire road surface images; Extracting features from the road surface image to obtain feature data; The feature data are respectively subjected to center point detection, offset detection and semantic segmentation; wherein the center point area of ​​the cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of the cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically identified by the semantic segmentation to obtain an identification result; The detection result of the road surface image is generated by combining the central point area, the edge offset and the recognition result.

2. The method according to claim 1, characterized in that The extracting features from the road surface image comprises: The feature extraction is performed on the road image using the adjusted EfficientNet convolutional neural network; wherein the adjusted EfficientNet convolutional neural network is specifically: the EfficientNet convolutional neural network is adjusted using atrous spatial pyramid pooling.

3. The method according to claim 2, characterized in that The use of atrous spatial pyramid pooling to adjust the EfficientNet convolutional neural network includes: Add a dilated spatial pyramid pooling layer after the EfficientNet convolutional neural network; Determine the image size of the first feature map output by each convolutional layer in the EfficientNet convolutional neural network, and use the first feature map of the last convolutional layer in each image size as the feature map to be fused; The second feature map output by the atrous space pyramid pooling layer is subjected to feature fusion with the feature map to be fused, so as to perform the feature extraction; wherein the feature map to be fused is subjected to feature fusion with the second feature map in reverse order of the image size.

4. The method according to claim 1, characterized in that: The performing center point detection on the feature data includes: Using the trained first convolution branch to predict the center point of the feature data, obtain the predicted center point of any line segment in the road surface image, and the confidence level of whether the line segment is a straight line; According to preset rules corresponding to cracks and / or seams, cracks or seams are distinguished based on the confidence level, and the area is expanded with the predicted center point as the center to obtain the central area, thereby completing the center point detection.

5. The method according to claim 4, characterized in that The performing offset detection on the characteristic data comprises: Obtain the predicted center point corresponding to any line segment and the distinction result determined according to the confidence level, and use the trained second convolution branch to perform the offset detection on the feature data; wherein the offset detection is specifically: based on the predicted center point and in accordance with the offset calculation method corresponding to the distinction result, determine the offset from the two end points of any line segment to the predicted center point, or determine the offset from the external frame in which any line segment is located to the predicted center point.

6. The method according to claim 4, characterized in that The performing semantic segmentation on the feature data includes: The predicted center point corresponding to any line segment and the distinction result determined according to the confidence level are obtained, and the feature data is semantically segmented using the trained third convolution branch; wherein the semantic segmentation is specifically as follows: based on the predicted center point and the distinction result, each point in the road surface image is judged as a crack or a seam through the third convolution branch, thereby completing the semantic segmentation.

7. The method according to claim 4, characterized in that The generating the detection result of the road surface image includes: Performing non-maximum suppression detection on any two line segments in the detection result, and determining the vertical distance from one line segment to the other line segment respectively; In response to the minimum value of the vertical distances satisfying a preset condition, the confidences of the arbitrary two line segments determined in the center point detection are obtained, and the line segment with a smaller confidence in the arbitrary two line segments is deleted.

8. A road surface detection device, characterized in that: include: An acquisition module, used for acquiring road surface images; An extraction module, used for extracting features from the road surface image to obtain feature data; A detection module, used to perform center point detection, offset detection and semantic segmentation on the feature data respectively; wherein the center point area of ​​the cracks and / or seams in the road surface image is determined by the center point detection, the edge offset of the cracks and / or seams in the road surface image is determined by the offset detection, and the cracks and / or seams in the road surface image are semantically recognized by the semantic segmentation to obtain a recognition result; An output module is used to generate a detection result of the road surface image by combining the central point area, the edge offset and the recognition result.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to implement the method according to any one of claims 1 to 7.

Images