Deep learning-based pavement crack image segmentation method and system

Abstract

The invention discloses a pavement crack image segmentation method and system based on deep learning, and the method can collect a pavement crack image through a camera, and introduces an image processing and semantic segmentation algorithm based on artificial intelligence and deep learning at a rear end to carry out the analysis of the pavement crack image. Shallow shape features and deep semantic features of pavement cracks in the pavement crack image are captured, and semantic segmentation of the pavement crack image is carried out based on feature interaction selection information between the shallow shape features and the deep semantic features, so that the crack length, the crack width and the crack area are determined. And carrying out grade evaluation on the severity of the crack based on the crack attribute information. Therefore, semantic segmentation and crack detection can be performed on the pavement crack image by using a deep learning semantic segmentation algorithm, so that the attribute information of the crack is determined in a more intelligent manner, the grade evaluation of the severity of the crack is facilitated, and a scientific basis is provided for urban road maintenance.

Description

Technical Field

[0001] The present application relates to the field of image segmentation, and more specifically, to a pavement crack image segmentation method and system based on deep learning. Background Art

[0002] As an important part of infrastructure, the health of roads directly affects the safety and efficiency of transportation. With the increase in traffic volume and the influence of the natural environment, various types of damage may occur on the road surface, among which cracks are one of the most common forms of damage. Pavement cracks not only affect traffic safety, but may also cause more serious structural damage and increase maintenance costs. Therefore, timely and accurate detection and evaluation of the severity of pavement cracks is of great significance for ensuring traffic safety and optimizing resource allocation.

[0003] However, traditional pavement crack detection methods mainly rely on manual inspection and simple image processing technology. This method is not only time-consuming and labor-intensive, but also its accuracy is affected by the experience and technical level of the inspectors, which easily leads to inconsistency and errors in the detection results. In addition, when dealing with crack images under complex backgrounds and different lighting conditions, traditional pavement crack detection methods often cannot effectively extract crack features. In other words, for low-contrast cracks or cracks with colors similar to the background, the detection difficulty increases. At the same time, it is also difficult to detect and assess the degree of damage for subtle or hidden cracks.

[0004] Therefore, an optimized pavement crack detection solution is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a road crack image segmentation method and system based on deep learning, which can collect road crack images through a camera, and introduce image processing and semantic segmentation algorithms based on artificial intelligence and deep learning in the back end to analyze the road crack images, so as to capture the shallow shape features and deep semantic features of the road cracks in the road crack images, and perform semantic segmentation of the road crack images based on the feature interaction selection information between the two features, so as to determine the crack length, crack width and crack area, and perform a grade assessment of the severity of the cracks based on these crack attribute information. In this way, the road crack images can be semantically segmented and crack detected by using a deep learning semantic segmentation algorithm, so as to determine the attribute information of the cracks in a more intelligent way, so as to perform a grade assessment of the severity of the cracks, and provide a scientific basis for urban road maintenance.

[0006] According to one aspect of the present application, a pavement crack image segmentation method based on deep learning is provided, which includes: Acquire a road crack image collected by a camera; Performing grayscale processing on the pavement crack image to obtain a pavement crack grayscale image; Performing multi-scale scanning and extraction of crack features on the pavement crack grayscale image to obtain shallow shape features of the pavement cracks and deep semantic features of the pavement cracks; Performing feature interactive selection on the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain shallow shape representation features of the pavement cracks guided by crack semantics; Among them, interactive feature selection is performed on the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain shallow shape characterization features of the pavement cracks under the guidance of crack semantics, including: linear transformation and masking processing are performed on the deep semantic features of the pavement cracks to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and a masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation; based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation, feature selection is performed on the shallow shape features of the pavement cracks to obtain shallow shape characterization features of the pavement cracks under the guidance of crack semantics; Based on the shallow shape characterization features of pavement cracks under the guidance of crack semantics, a pavement crack image semantic segmentation result is generated, and based on the pavement crack image semantic segmentation result, crack length, crack width and crack area are determined; Based on the crack length, crack width and crack area, a crack severity level type label is determined.

[0007] According to another aspect of the present application, a pavement crack image segmentation system based on deep learning is provided, which includes: An image acquisition module, used to acquire a road surface crack image collected by a camera; A grayscale processing module, used for performing grayscale processing on the pavement crack image to obtain a pavement crack grayscale image; A crack feature multi-scale scanning and extraction module is used to perform a crack feature multi-scale scanning and extraction on the pavement crack grayscale image to obtain the pavement crack shallow layer shape features and pavement crack deep layer semantic features; A feature interactive selection module, used for interactively selecting the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain the shallow shape representation features of the pavement cracks under the guidance of crack semantics; A crack attribute determination module, used to generate a pavement crack image semantic segmentation result based on the shallow shape characterization features of the pavement cracks under the guidance of the crack semantics, and determine the crack length, crack width and crack area based on the pavement crack image semantic segmentation result; The crack severity level type label generation module is used to determine the crack severity level type label based on the crack length, crack width and crack area.

[0008] Compared with the prior art, the present application provides a road crack image segmentation method and system based on deep learning, which can collect road crack images through a camera, and introduce an image processing and semantic segmentation algorithm based on artificial intelligence and deep learning in the back end to analyze the road crack image, so as to capture the shallow shape features and deep semantic features of the road cracks in the road crack image, and perform semantic segmentation of the road crack image based on the feature interaction selection information between the two features, so as to determine the crack length, crack width and crack area, and perform a grade assessment of the severity of the cracks based on these crack attribute information. In this way, the road crack image can be semantically segmented and crack detected by using a deep learning semantic segmentation algorithm, so as to determine the attribute information of the cracks in a more intelligent way, so as to perform a grade assessment of the severity of the cracks, and provide a scientific basis for urban road maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 is a flow chart of a pavement crack image segmentation method based on deep learning according to an embodiment of the present application; Figure 2 A schematic diagram of data flow of a pavement crack image segmentation method based on deep learning according to an embodiment of the present application; Figure 3 is a flowchart of sub-step S4 of the pavement crack image segmentation method based on deep learning according to an embodiment of the present application; Figure 4 is a structural schematic diagram of a Deeplabv3+ model of a pavement crack image segmentation method based on deep learning according to another embodiment of the present application; Figure 5 A schematic diagram comparing the segmentation results of the Deeplabv3+ model and other models of the pavement crack image segmentation method based on deep learning according to another embodiment of the present application; Figure 6 It is a block diagram of a pavement crack image segmentation system based on deep learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0012] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0013] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0014] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0015] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0016] Traditional pavement crack detection methods mainly rely on manual inspection and simple image processing technology. This method is not only time-consuming and labor-intensive, but also its accuracy is affected by the experience and technical level of the inspectors, which easily leads to inconsistency and errors in the detection results. In addition, when dealing with crack images under complex backgrounds and different lighting conditions, traditional pavement crack detection methods often cannot effectively extract crack features, that is, for low-contrast cracks or cracks with colors similar to the background, the detection difficulty increases. At the same time, it is also difficult to detect and assess the degree of damage for subtle or hidden cracks. Therefore, an optimized pavement crack detection solution is desired.

[0017] In recent years, deep learning technology has made remarkable achievements in the field of image recognition, especially in tasks such as object detection and semantic segmentation. Deep learning models can automatically learn features from a large amount of labeled data and have good recognition and evaluation capabilities for small targets (such as road cracks) in complex backgrounds. Semantic segmentation is a pixel-level classification technology that can accurately classify each pixel in an image into a specific category, which is crucial for the accurate detection of road cracks. Through semantic segmentation technology, not only can the location of the crack be located, but the length, width and area of ​​the crack can also be further calculated, thus providing a basis for the severity assessment of the crack.

[0018] In the technical solution of the present application, a pavement crack image segmentation method based on deep learning is proposed. Figure 1 This is a flowchart of a pavement crack image segmentation method based on deep learning according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the pavement crack image segmentation method based on deep learning according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the pavement crack image segmentation method based on deep learning includes the following steps: S1, acquiring a pavement crack image collected by a camera; S2, grayscale processing the pavement crack image to obtain a pavement crack grayscale image; S3, performing multi-scale scanning and extraction of crack features on the pavement crack grayscale image to obtain shallow shape features of pavement cracks and deep semantic features of pavement cracks; S4, interactively selecting the shallow shape features of pavement cracks and the deep semantic features of pavement cracks to obtain shallow shape characterization features of pavement cracks under the guidance of crack semantics; generating a pavement crack image semantic segmentation result based on the shallow shape characterization features of pavement cracks under the guidance of crack semantics, and determining the crack length, crack width and crack area based on the pavement crack image semantic segmentation result; S5, determining a crack severity level type label based on the crack length, crack width and crack area.

[0019] In particular, the S1 obtains a road crack image collected by a camera. It should be understood that the road crack image covers multiple aspects such as crack characteristics, location, time, environment, etc. This information is crucial for road health assessment, maintenance decisions and traffic management. Through effective image processing and data analysis, the road condition can be better understood and managed to ensure traffic safety.

[0020] In particular, the S2 performs grayscale processing on the pavement crack image to obtain a pavement crack grayscale image. Considering that color images contain a large amount of color information, crack detection usually focuses on features such as the shape, position and size of the cracks. Therefore, in order to reduce the complexity of data processing while retaining the basic characteristics of the cracks and providing a basis for subsequent crack severity detection and maintenance, in the technical solution of the present application, the pavement crack image is grayscale processed to obtain a pavement crack grayscale image. The grayscale processing method can remove the color information in the image and retain the brightness information, which can reduce the complexity of processing while retaining the basic characteristics of the cracks. In addition, in the grayscale image, the cracks usually present a different brightness level from other areas, which helps to enhance the contrast between the cracks and the background, making them easier to be recognized by the detection algorithm.

[0021] In particular, S3 performs a multi-scale scanning and extraction of crack features on the pavement crack grayscale image to obtain shallow shape features of pavement cracks and deep semantic features of pavement cracks. In a specific example of the present application, first, the pavement crack grayscale image is input into a crack multi-scale feature scanning network based on a hollow pyramid network to obtain a pavement crack deep semantic feature map and a pavement crack shallow shape feature map as the pavement crack shallow shape features; here, considering that the cracks on the pavement may have different sizes and shapes, and the cracks may be interfered by background factors in the image, therefore, in order to better perceive and capture the pavement crack-related features in the image, and extract different levels of crack features from multiple scales, in the technical solution of the present application, the pavement crack grayscale image is further input into a crack multi-scale feature scanning network based on a hollow pyramid network to obtain a pavement crack shallow shape feature map and a pavement crack deep semantic feature map. It is worth mentioning that the pavement crack shallow shape feature map mainly captures geometric information such as the edge, contour and shape of the crack, which is very important for identifying the existence and shape of the crack. The pavement crack deep semantic feature map contains the distribution pattern and contextual information of cracks in a larger range. This information helps to understand the position of cracks in the entire image and its relationship with the surrounding environment, and plays an important role in distinguishing cracks from the background environment and the subsequent crack severity assessment. Furthermore, the pavement crack deep semantic feature map is subjected to global mean pooling processing to obtain a pavement crack deep semantic description vector as the pavement crack deep semantic feature. In order to extract a compact and representative pavement crack deep semantic feature representation from the pavement crack deep semantic feature map, which is conducive to the subsequent feature interaction selection processing, in the technical solution of the present application, the pavement crack deep semantic feature map is further subjected to global mean pooling processing to obtain a pavement crack deep semantic description vector. The global mean pooling process can compress the features of a feature map into a vector. This process integrates all spatial information of each channel in the pavement crack deep semantic feature map into a single value, thereby greatly reducing the number of features and reducing the dimension. Furthermore, by calculating the average value of each feature matrix along the channel dimension of each feature map, the pavement crack deep semantic feature map retains the most important information of the channel, that is, those features that dominate the entire feature map. This can ignore some unimportant details and retain the most critical feature information related to the subsequent pavement crack detection and severity assessment tasks.

[0022] In particular, the S4 interactively selects the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain shallow shape characterization features of the pavement cracks under the guidance of crack semantics. It should be understood that since the shallow shape feature map of the pavement cracks and the deep semantic description vector of the pavement cracks respectively contain shallow shape features and deep semantic features related to the pavement cracks, specifically, the shallow shape feature map of the pavement cracks provides information about the appearance of the cracks, such as edges, textures and shape features, etc.; while the deep semantic description vector of the pavement cracks contains a more abstract and higher-level understanding of the cracks, such as crack type, location and severity, etc. Therefore, in order to be able to combine these two types of features at different levels and scales of pavement cracks, so as to interactively select features to obtain a richer and more representative pavement crack feature representation, and provide a basis for subsequent crack image semantic segmentation and crack severity level type judgment, in the technical solution of the present application, the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks are further interactively selected to obtain shallow shape characterization features of the pavement cracks under the guidance of crack semantics. In a specific example of the present application, such as Figure 3 As shown, the S4 includes: S41, performing linear transformation and masking processing on the deep semantic features of the pavement cracks to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and a masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation; S42, performing feature selection on the pavement crack shallow shape features based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation to obtain the pavement crack shallow shape characterization features guided by the crack semantics.

[0023] Specifically, the S41 performs linear transformation and masking on the pavement crack deep semantic features to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and a masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation. Specifically, firstly, the pavement crack deep semantic description vector is subjected to a linear transformation based on the first weight modifier matrix and the second weight modifier matrix to obtain a first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector and a second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector; the expression range of the pavement crack deep semantic features is expanded by linear transformation, so that the model can capture the relationship between the pavement crack deep semantic features and the pavement crack shallow shape feature map from different angles. Next, based on the gradient amplitude of each position in the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector, the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector is subjected to a masking process based on the gradient amplitude to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation; and based on the gradient amplitude of each position in the second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector, the second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector is subjected to a masking process based on the gradient amplitude to obtain a masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation. Among them, the gradient amplitude reflects the degree of change of each element in the pavement crack deep semantic coding auxiliary feature mapping pattern representation vector. Through masking, features with large changes can be highlighted, thereby enhancing attention to important pavement crack deep semantic features in the subsequent feature selection and expression process. Then, the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector is combined with the hyperbolic tangent function to perform preliminary screening on the pavement crack shallow shape feature map. The hyperbolic tangent function is a commonly used activation function with an output range between -1 and +1, which can effectively distinguish different areas in the feature map, thereby highlighting those feature representations that are highly related to the crack shallow shape features in the preliminary screening stage.

[0024] Among them, the process of performing a linear transformation on the deep semantic description vector of the pavement cracks based on the first weight modifier matrix and the second weight modifier matrix to obtain the first pavement cracks deep semantic coding auxiliary feature mapping mode representation vector and the second pavement cracks deep semantic coding auxiliary feature mapping mode representation vector includes: calculating the pavement cracks deep semantic description vector and multiplying it with the transpose of the first weight modifier matrix, and then adding it positionally with the first bias modifier vector to obtain the first pavement cracks deep semantic coding auxiliary feature mapping mode representation vector; calculating the pavement cracks deep semantic description vector and multiplying it with the transpose of the second weight modifier matrix, and then adding it positionally with the second bias modifier vector to obtain the second pavement cracks deep semantic coding auxiliary feature mapping mode representation vector.

[0025] More specifically, based on the gradient amplitude of each position in the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector, the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector is subjected to gradient amplitude-based masking processing to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation process, including: calculating the square of the difference between the previous position feature value and the next position feature value of each position feature value in the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector to obtain the first pavement crack deep semantic gradient amplitude representation vector. vector; calculate the absolute value of the square root of each first pavement crack deep semantic gradient amplitude in the first pavement crack deep semantic gradient amplitude representation vector, and input the obtained feature vector into the mask function for processing to obtain the masked first pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector; wherein, in the mask function, in response to the first pavement crack deep semantic gradient amplitude in the first pavement crack deep semantic gradient amplitude representation vector being greater than a predetermined threshold, the first pavement crack deep semantic gradient amplitude greater than the predetermined threshold is set to 1, otherwise it is set to 0 to obtain the masked first pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector.

[0026] Specifically, the S42 performs feature selection on the pavement crack shallow shape features based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation to obtain the pavement crack shallow shape characterization features under the crack semantics guidance. Specifically, firstly, based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector and the hyperbolic tangent function, the pavement crack shallow shape feature map is subjected to preliminary feature selection to obtain the pavement crack shallow shape feature map after preliminary screening; then, based on the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector and the Sigmoid function, the pavement crack shallow shape feature map after preliminary screening is subjected to feature screening again to obtain the pavement crack shallow shape enhancement feature map under the crack semantics guidance as the pavement crack shallow shape characterization features under the crack semantics guidance. That is, on the basis of preliminary screening, the masked second pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector combined with the Sigmoid function is used to further screen the pavement crack shallow shape feature map after preliminary screening to determine which pavement crack shallow shape features are truly important. In this way, the pavement crack deep semantic features can be used to guide and enhance the expression ability of the pavement crack shallow shape features, and generate the pavement crack shallow shape enhancement feature map under the guidance of crack semantics. Through feature interaction and screening, the relevant features in the shallow shape feature map can be enhanced under the guidance of crack semantics, so that it can better reflect the real state of the crack. In this way, the enhanced feature map not only contains the original shape information, but also incorporates deep semantic understanding, providing a basis for subsequent crack severity detection and evaluation tasks.

[0027] Among them, the process of performing preliminary feature screening on the pavement crack shallow shape feature map based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector and the hyperbolic tangent function to obtain the pavement crack shallow shape feature map after preliminary screening includes: using the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the channel weight vector, performing weighted enhancement on the pavement crack shallow shape feature map along the channel dimension, and then inputting the weighted enhanced feature map into the hyperbolic tangent function for activation to obtain the pavement crack shallow shape feature map after preliminary screening.

[0028] More specifically, the process of performing feature re-screening on the pavement crack shallow shape feature map after preliminary screening based on the masked second pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector and the Sigmoid function to obtain the pavement crack shallow shape enhancement feature map under the guidance of crack semantics as the pavement crack shallow shape characterization feature under the guidance of crack semantics includes: using the masked second pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector as the channel weight vector, performing weighted enhancement on the pavement crack shallow shape feature map after preliminary screening along the channel dimension, and then inputting the weighted enhanced feature map into the Sigmoid function for activation to obtain the pavement crack shallow shape enhancement feature map under the guidance of crack semantics.

[0029] In summary, the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks are interactively selected to obtain the shallow shape characterization features of the pavement cracks under the guidance of crack semantics, including: the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks are interactively selected to obtain the shallow shape enhancement feature map of the pavement cracks under the guidance of crack semantics as the shallow shape characterization features of the pavement cracks under the guidance of crack semantics by using the following feature interactive selection formula; wherein the feature interactive selection formula includes: in, is the deep semantic description vector of the pavement cracks, and are the first weight modifier matrix and the first bias modifier vector, respectively, and are the second weight modifier matrix and the second bias modifier vector, respectively, and are the first linear transformation and the second linear transformation respectively, and are respectively the first pavement crack deep semantic coding auxiliary feature mapping mode representation vector and the second pavement crack deep semantic coding auxiliary feature mapping mode representation vector, and are respectively the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector Position and The eigenvalues ​​at the positions, and are respectively the first and second pavement crack deep semantic coding auxiliary feature mapping pattern representation vectors. Position and The eigenvalues ​​at the positions, For mask operation, and The first masked pavement crack deep semantic coding auxiliary feature map pattern representation vector and the second masked pavement crack deep semantic coding auxiliary feature map pattern representation vector are respectively The eigenvalues ​​at the positions, is the preset threshold, is the shallow shape characteristic diagram of the pavement crack, A representation vector of a deep semantic encoding auxiliary feature map pattern for the masked first pavement crack, A representation vector of a deep semantic encoding auxiliary feature map pattern for the masked second pavement crack, is the position-wise weighted multiplication process along the channel dimension, is the hyperbolic tangent function, To preliminarily screen the shallow shape characteristics of pavement cracks, for function, It is a shallow shape enhancement feature map of pavement cracks guided by the crack semantics.

[0030] It is worth mentioning that in other examples of the present application, the following steps can be used to interactively select the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain the shallow shape characterization features of the pavement cracks under the guidance of crack semantics, for example: input the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks; splice the shallow shape features and the deep semantic features to form a comprehensive feature vector; use feature selection algorithms (such as recursive feature elimination (RFE), LASSO regression, random forest, etc.) to select the most important features for feature selection; based on the selected features, generate shallow shape characterization features of the cracks; combine semantic information to enhance the understanding and characterization of crack features.

[0031] In particular, the S5 generates a semantic segmentation result of a pavement crack image based on the shallow shape characterization features of the pavement cracks under the guidance of the crack semantics, and determines the crack length, crack width and crack area based on the pavement crack image semantic segmentation result. In a specific example of the present application, the pavement crack shallow shape enhancement feature map under the crack semantics guidance is subjected to image semantic segmentation to obtain the pavement crack image semantic segmentation result; that is, the pavement crack image semantic segmentation is performed using the crack shallow shape feature enhancement expression information under the guidance of the deep semantic features of the pavement cracks, thereby generating a pavement crack image semantic segmentation result. Furthermore, based on the pavement crack image semantic segmentation result, the crack length, crack width and crack area are determined. In this way, the pavement crack image can be semantically segmented and crack detected by using a deep learning semantic segmentation algorithm, thereby determining the attribute information of the crack in a more intelligent way, so as to perform a grade assessment of the severity of the cracks and provide a scientific basis for urban road maintenance.

[0032] In particular, considering that the pavement crack shallow shape feature map and the pavement crack deep semantic description vector represent the shallow image features and deep image semantic description features of the pavement crack image respectively, when the feature interaction screening encoder based on the metadata block is used, the different dimensional association structures based on the cross-modal feature distribution will also have cross-modal feature interaction screening encoding differences, resulting in insufficient long-distance joint perception representation of the pavement crack shallow shape enhancement feature map under the guidance of crack semantics, thereby reducing the expression effect of the pavement crack shallow shape enhancement feature map under the guidance of crack semantics, affecting the accuracy of the semantic segmentation results of the pavement crack image.

[0033] Preferably, in one example, performing image semantic segmentation on the pavement crack shallow shape enhancement feature map under the crack semantics guidance to obtain a pavement crack image semantic segmentation result includes: Arrange the characteristic values ​​in the pavement crack shallow layer shape enhancement feature map under the guidance of crack semantics in ascending order to obtain a pavement crack shallow layer shape enhancement encoding feature set under the guidance of crack semantics; In response to the first one of the pavement crack shallow shape enhancement coding feature set under the crack semantics guidance The eigenvalue and The absolute value of the difference between the eigenvalues ​​is less than or equal to the distance difference hyperparameter ,Right now ,in, and They represent the first The eigenvalue and Eigenvalue, calculate the The eigenvalues ​​are similar to the The weighted sum between the eigenvalues ​​is the optimized Eigenvalue; Calculate the square root of the sum of squares of all feature values ​​of the pavement crack shallow shape enhancement coding feature set under the guidance of crack semantics, multiply the square root by 2, and then divide it by the square of the scale of the pavement crack shallow shape enhancement feature map under the guidance of crack semantics to obtain the pavement crack shallow shape enhancement coding space primitive value under the guidance of crack semantics: Among them, the scale of the shallow shape enhancement feature map of pavement cracks guided by crack semantics is is equal to the width of the feature matrix of the pavement crack shallow shape enhancement feature map under the guidance of crack semantics multiplied by the height and then multiplied by the number of channels of the pavement crack shallow shape enhancement feature map under the guidance of crack semantics, represents the square root of the sum of squares of all feature values ​​of the shallow shape enhancement encoding feature set of pavement cracks guided by the crack semantics, Representing the shallow shape of pavement cracks guided by crack semantics to enhance the encoding of spatial primitive values; In response to the first one of the pavement crack shallow shape enhancement coding feature set under the crack semantics guidance The eigenvalue and The absolute value of the difference between the eigenvalues ​​is greater than the distance difference hyperparameter ,Right now , multiply the pavement crack shallow shape enhancement encoding space primitive value guided by the crack semantics by the first After the eigenvalue, calculate the product with the first The weighted reduction between the eigenvalues ​​is the optimized Eigenvalue; The optimization of the shallow shape enhancement coding feature set of pavement cracks guided by crack semantics is combined. The eigenvalue is used to obtain an optimized pavement crack shallow shape enhancement feature map under the guidance of crack semantics, wherein the minimum eigenvalue of the pavement crack shallow shape enhancement encoding feature set under the guidance of crack semantics remains unchanged; The optimized crack semantics-guided pavement crack shallow shape enhancement feature map is semantically segmented to obtain a pavement crack image semantic segmentation result.

[0034] Accordingly, in view of the problem that the feature set of the pavement crack shallow shape enhancement feature map under the crack semantics guidance has insufficient global joint perception representation capability due to the long distance exceeding the predetermined local distribution interval threshold under the predetermined feature value sequence distribution, the high-dimensional feature space primitive representation based on self-inner product fusion of the pavement crack shallow shape enhancement feature map under the crack semantics guidance is used to capture the complex structure of the global network interaction of its feature values, so as to reconstruct the joint perception relationship between the feature values ​​of the pavement crack shallow shape enhancement feature map under the crack semantics guidance by simulating the scale-based high-dimensional feature space potential primitive, so as to realize the encoding reconstruction of the real sequence distribution behavior of the pavement crack shallow shape enhancement feature map under the crack semantics guidance under the long distance, improve the joint perception expression effect of the pavement crack shallow shape enhancement feature map under the crack semantics guidance, and improve the semantic segmentation accuracy of the pavement crack image semantic segmentation result. In this way, the pavement crack image can be semantically segmented more accurately, so as to determine the crack length, crack width and crack area, and the crack severity level evaluation can be performed based on these crack attribute information, providing a scientific basis for urban road maintenance.

[0035] In particular, the S6 determines the crack severity level type label based on the crack length, crack width and crack area. It should be understood that the crack length, crack width and crack area belong to crack attribute information, so the crack severity level can be evaluated through the crack attribute information, and maintenance resources can be reasonably arranged, and long-term road maintenance and reconstruction plans can be formulated to extend the service life of the road. For example, in a specific embodiment of the present application, weight values ​​can be assigned to the crack length, crack width and crack area respectively, and the weighted sum of the crack length, crack width and crack area is calculated as the crack global evaluation value, and then the crack severity level type label is determined based on the crack global evaluation value. For example, when the crack global evaluation value is between the first threshold and the second threshold, the crack severity level type label is a moderate crack; when the crack global evaluation value is less than the first threshold, the crack severity level type label is a slight crack; and when the crack global evaluation value is greater than the second threshold, the crack severity level type label is a severe crack.

[0036] In summary, according to the deep learning-based pavement crack image segmentation method of the embodiment of the present application, it is explained that it can collect pavement crack images through a camera, and introduce an image processing and semantic segmentation algorithm based on artificial intelligence and deep learning at the back end to analyze the pavement crack image, so as to capture the shallow shape features and deep semantic features of the pavement cracks in the pavement crack image, and perform semantic segmentation of the pavement crack image based on the feature interaction selection information between the two features, so as to determine the crack length, crack width and crack area, and perform a grade assessment of the severity of the cracks based on these crack attribute information. In this way, it is possible to perform semantic segmentation and crack detection on pavement crack images by using a deep learning semantic segmentation algorithm, so as to determine the attribute information of the cracks in a more intelligent way, so as to perform a grade assessment of the severity of the cracks, and provide a scientific basis for urban road maintenance.

[0037] Furthermore, another embodiment of the present application also provides a pavement crack image segmentation method based on deep learning. The method uses the Deeplabv3+ model to extract features of the pavement crack image collected by the camera to obtain a segmentation result.

[0038] Specifically, the Deeplabv3+ model has been widely used in various image segmentation tasks. The model consists of two parts: an encoder and a decoder, and integrates deep convolutional neural networks (DCNN) and atrous spatial pyramid pooling (ASPP) modules. In the encoder part, DeeplabV3+ uses the Xception structure as the skeleton network and extracts high-order and low-order feature layers with the help of DCNN. The ASPP module increases the effective size of the convolution kernel and expands the receptive field through atrous convolution, combines 1×1 convolution, 3 layers of 3×3 convolution with different atrous rates, and a global average pooling layer to improve the efficiency and range of feature extraction. The decoder part uses 1×1 convolution to compress low-order features and effectively fuses them with high-order features. The feature information is further refined through 3×3 convolution, and 4 times upsampling is performed to restore the original size of the image, outputting high-precision segmentation results. The decoding part not only improves the model's ability to process edge details, but also ensures the comprehensive utilization of features, thereby fully understanding and processing image details and contextual information. The structure of the Deeplabv3+ model is shown in Figure 4.

[0039] Figure 5 Examples of pavement crack segmentation using four models. Figure 5It can be seen that the Deeplabv3+ model can extract the crack contour more completely and accurately without redundant information. Compared with the Deeplabv3+ model, the segmentation effect of the U-Net and PSPNet segmentation models needs to be improved. The segmentation effect of the SegNet model is poor and some redundant information will be generated.

[0040] Furthermore, a pavement crack image segmentation system based on deep learning is also provided.

[0041] Figure 6 FIG. 1 is a block diagram of a pavement crack image segmentation system based on deep learning according to an embodiment of the present application. Figure 5 As shown, according to an embodiment of the present application, a pavement crack image segmentation system 300 based on deep learning includes: an image acquisition module 310, used to acquire a pavement crack image collected by a camera; a grayscale processing module 320, used to perform grayscale processing on the pavement crack image to obtain a pavement crack grayscale image; a crack feature multi-scale scanning and extraction module 330, used to perform crack feature multi-scale scanning and extraction on the pavement crack grayscale image to obtain a pavement crack shallow shape feature and a pavement crack deep semantic feature; a feature interactive selection module 340, used to perform feature interactive selection on the pavement crack shallow shape feature and the pavement crack deep semantic feature to obtain a pavement crack shallow shape characterization feature under the guidance of crack semantics; a crack attribute determination module 350, used to generate a pavement crack image semantic segmentation result based on the pavement crack shallow shape characterization feature under the guidance of crack semantics, and determine the crack length, crack width and crack area based on the pavement crack image semantic segmentation result; a crack severity level type label generation module 360, used to determine the crack severity level type label based on the crack length, crack width and crack area.

[0042] As described above, the pavement crack image segmentation system 300 based on deep learning according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a pavement crack image segmentation algorithm based on deep learning. In a possible implementation, the pavement crack image segmentation system 300 based on deep learning according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the pavement crack image segmentation system 300 based on deep learning can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the pavement crack image segmentation system 300 based on deep learning can also be one of the many hardware modules of the wireless terminal.

[0043] Alternatively, in another example, the deep learning-based pavement crack image segmentation system 300 and the wireless terminal may also be separate devices, and the deep learning-based pavement crack image segmentation system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0044] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A pavement crack image segmentation method based on deep learning, characterized in that: include: Acquire a road crack image collected by a camera; Performing grayscale processing on the pavement crack image to obtain a pavement crack grayscale image; Performing multi-scale scanning and extraction of crack features on the pavement crack grayscale image to obtain shallow shape features of the pavement cracks and deep semantic features of the pavement cracks; Performing feature interactive selection on the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain shallow shape representation features of the pavement cracks guided by crack semantics; Among them, interactive feature selection is performed on the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain shallow shape characterization features of the pavement cracks under the guidance of crack semantics, including: linear transformation and masking processing are performed on the deep semantic features of the pavement cracks to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and a masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation; based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation and the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation, feature selection is performed on the shallow shape features of the pavement cracks to obtain shallow shape characterization features of the pavement cracks under the guidance of crack semantics; Based on the shallow shape characterization features of pavement cracks under the guidance of crack semantics, a pavement crack image semantic segmentation result is generated, and based on the pavement crack image semantic segmentation result, crack length, crack width and crack area are determined; Based on the crack length, crack width and crack area, a crack severity level type label is determined.

2. The pavement crack image segmentation method based on deep learning according to claim 1, characterized in that: The pavement crack grayscale image is subjected to multi-scale scanning and extraction of crack features to obtain shallow shape features of pavement cracks and deep semantic features of pavement cracks, including: Inputting the pavement crack grayscale image into a crack multi-scale feature scanning network based on a hollow pyramid network to obtain a pavement crack deep semantic feature map and a pavement crack shallow shape feature map as the pavement crack shallow shape feature; A global mean pooling process is performed on the pavement crack deep semantic feature map to obtain a pavement crack deep semantic description vector as the pavement crack deep semantic feature.

3. The pavement crack image segmentation method based on deep learning according to claim 2 is characterized in that: The pavement crack deep semantic features are linearly transformed and masked to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping mode representation and a masked second pavement crack deep semantic coding auxiliary feature mapping mode representation, including: Performing linear transformations based on a first weight modifier matrix and a second weight modifier matrix on the pavement crack deep semantic description vector to obtain a first pavement crack deep semantic coding auxiliary feature mapping mode representation vector and a second pavement crack deep semantic coding auxiliary feature mapping mode representation vector; Based on the gradient amplitude of each position in the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector, performing gradient amplitude-based masking processing on the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation; Based on the gradient amplitude of each position in the second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector, the second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector is subjected to gradient amplitude-based masking processing to obtain a masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation.

4. The pavement crack image segmentation method based on deep learning according to claim 3 is characterized in that: The pavement crack deep semantic description vector is subjected to a linear transformation based on a first weight modifier matrix and a second weight modifier matrix to obtain a first pavement crack deep semantic coding auxiliary feature mapping mode representation vector and a second pavement crack deep semantic coding auxiliary feature mapping mode representation vector, including: Calculating the pavement crack deep semantic description vector and the transpose of the first weight modifier matrix, multiplying the resultant vector by the first weight modifier matrix, and then adding the vector by position to the first bias modifier vector to obtain the first pavement crack deep semantic encoding auxiliary feature mapping mode representation vector; The pavement crack deep semantic description vector is calculated and multiplied by the transpose of the second weight modifier matrix, and then added to the second bias modifier vector by position to obtain the second pavement crack deep semantic encoding auxiliary feature mapping mode representation vector.

5. The pavement crack image segmentation method based on deep learning according to claim 4 is characterized in that: Based on the gradient amplitude of each position in the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector, the first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector is subjected to a masking process based on the gradient amplitude to obtain a masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation, including: Calculating the square of the difference between the previous position feature value and the next position feature value of each position feature value in the first pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector to obtain the first pavement crack deep semantic gradient amplitude representation vector; Calculating the absolute value of the square root of each first pavement crack deep semantic gradient amplitude in the first pavement crack deep semantic gradient amplitude representation vector, and inputting the obtained feature vector into a mask function for processing to obtain the masked first pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector; Among them, in the mask function, in response to the first pavement crack deep semantic gradient amplitude in the first pavement crack deep semantic gradient amplitude representation vector being greater than a predetermined threshold, the first pavement crack deep semantic gradient amplitude greater than the predetermined threshold is set to 1, otherwise it is set to 0 to obtain the masked first pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector.

6. The pavement crack image segmentation method based on deep learning according to claim 5, characterized in that: Based on the masked first pavement crack deep semantic coding auxiliary feature mapping mode representation and the masked second pavement crack deep semantic coding auxiliary feature mapping mode representation, feature selection is performed on the pavement crack shallow shape features to obtain the pavement crack shallow shape characterization features guided by the crack semantics, including: Performing preliminary feature selection on the pavement crack shallow shape feature map based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector and the hyperbolic tangent function to obtain a pavement crack shallow shape feature map after preliminary screening; Based on the masked second pavement crack deep semantic encoding auxiliary feature mapping pattern representation vector and Sigmoid function, the pavement crack shallow shape feature map after preliminary screening is subjected to feature screening again to obtain the pavement crack shallow shape enhancement feature map under the guidance of crack semantics as the pavement crack shallow shape characterization feature under the guidance of crack semantics.

7. The pavement crack image segmentation method based on deep learning according to claim 6, characterized in that: Based on the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector and the hyperbolic tangent function, the pavement crack shallow shape feature map is subjected to preliminary feature screening to obtain the pavement crack shallow shape feature map after preliminary screening, including: using the masked first pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the channel weight vector, performing weighted enhancement on the pavement crack shallow shape feature map along the channel dimension, and then inputting the weighted enhanced feature map into the hyperbolic tangent function for activation to obtain the pavement crack shallow shape feature map after preliminary screening.

8. The pavement crack image segmentation method based on deep learning according to claim 7, characterized in that: Based on the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector and the Sigmoid function, the pavement crack shallow shape feature map after the preliminary screening is subjected to feature screening again to obtain the pavement crack shallow shape enhancement feature map under the guidance of crack semantics as the pavement crack shallow shape characterization feature under the guidance of crack semantics, including: using the masked second pavement crack deep semantic coding auxiliary feature mapping pattern representation vector as the channel weight vector, performing weighted enhancement on the pavement crack shallow shape feature map after the preliminary screening along the channel dimension, and then inputting the weighted enhanced feature map into the Sigmoid function for activation to obtain the pavement crack shallow shape enhancement feature map under the guidance of crack semantics.

9. The pavement crack image segmentation method based on deep learning according to claim 8, characterized in that: Based on the shallow shape characterization features of pavement cracks under the guidance of crack semantics, a pavement crack image semantic segmentation result is generated, and crack length, crack width and crack area are determined based on the pavement crack image semantic segmentation result, including: Performing image semantic segmentation on the pavement crack shallow shape enhancement feature map guided by the crack semantics to obtain the pavement crack image semantic segmentation result; Based on the semantic segmentation result of the pavement crack image, the crack length, crack width and crack area are determined.

10. A road crack image segmentation system based on deep learning, characterized in that: include: An image acquisition module, used to acquire a road surface crack image collected by a camera; A grayscale processing module, used for performing grayscale processing on the pavement crack image to obtain a pavement crack grayscale image; A crack feature multi-scale scanning and extraction module is used to perform a crack feature multi-scale scanning and extraction on the pavement crack grayscale image to obtain the pavement crack shallow layer shape features and pavement crack deep layer semantic features; A feature interactive selection module, used for interactively selecting the shallow shape features of the pavement cracks and the deep semantic features of the pavement cracks to obtain the shallow shape representation features of the pavement cracks under the guidance of crack semantics; A crack attribute determination module, used to generate a pavement crack image semantic segmentation result based on the shallow shape characterization features of the pavement cracks under the guidance of the crack semantics, and determine the crack length, crack width and crack area based on the pavement crack image semantic segmentation result; The crack severity level type label generation module is used to determine the crack severity level type label based on the crack length, crack width and crack area.

Images